Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

August 11, 2022 22:39

Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: Cramming 'Papers, Please' onto Phones (August 06, 2022: 1289 points)

(1290) Cramming 'Papers, Please' onto Phones

1290 points 5 days ago by nycpig in 10000th position

dukope.com | Estimated reading time – 25 minutes | comments | anchor

Cramming 'Papers, Please' Onto Phones

Aug 06, 2022

I created Papers, Please in 2013 specifically for desktop computers with mouse control. Now, here, in 2022, desktop computers no longer exist and all computing is done via handheld mobile telephone. Time to update this dinosaur.

These thousands of words and megabytes of images will cover some bits of porting the game from big desktop to little phone. As a winking throwback to days past, this is a big dump of text and inline img tags instead of exciting splashes of quick-cut video. In 9 years, I'll port this post to VR.

Beyond the minimal amount necessary to make the interface work, there are no content changes or additions for this port. No wild story twists, no new characters, no voiced dialog, no stereoscopic ray-traced graphics, and most disappointingly, no cosmetic unlockables.

Same meal, different plate.

Step 1: To Unity

Papers, Please was originally written in Haxe/OpenFL, a combo of modern ECMA-ish language, Flash-alike API, and multi-platform build system. This was a great environment for getting the game together quickly in 2013 but over the years Haxe has moved away from its Flash-targeting roots, and keeping up with OpenFL changes to make game updates has required outsized effort.

When I finally committed to this port my first decision was to rewrite the game in C#/Unity. After Obra Dinn I'm a solid fan of Unity – the editor, the entity/component design, the build system, the ubiquity, just about everything.

I made it a few days into rewriting before finding that although I like C#, I like Haxe more. In making the game I'd leveraged a bunch of core Haxe features like super enums and implicit typing and it was all getting trickier and trickier to implement by hand in C#.

One option would've been to keep the Haxe part but switch from OpenFL to Heaps, another Haxe-based engine and build system. Dead Cells, Northgard, and a few other games are built with Haxe/Heaps so it's a perfectly capable system. Still it's hard to overstate how appropriate Unity is for someone in my position: a solo developer targeting multiple platforms and desperate for a popular, proven, and well-supported engine and build system.

Fortunately, Haxe is a transpiled language, meaning that you write in one language (Haxe) and it gets converted to another language (Javascript, PHP, Python, C++, C#, Lua, etc) before being compiled/interpreted for whatever target you've got. So it's possible to write code in Haxe and have it transpiled to C# that can be loaded and compiled in Unity. Which is how I decided to roll this.

Haxe code

The same code transpiled to C#

First, I stripped all usage of the Flash-like API that OpenFL was providing. The display tree, input events, bitmap manipulation, resource management, sound playback – basically everything that wasn't core game logic. Then I rebuilt it all with custom code, the goal being to create a sort of PAPERS-PLEASE black box that could take user input and spit out a list of quads to draw and audio commands to execute every frame.

This was probably the most fun part of the project. The full Flash API was overkill for Papers Please anyways, and building a fresh engine specifically for what I needed was cathartic.

I was slightly worried about performance here, since much of the OpenFL API is written in C++ and I was replacing it all with higher-level Haxe code. The requirements for the game are earth-core low though, and Haxe is pretty performant anyways so I only had to be a little bit careful to get good framerates.

The end result requires a minimal host shell to send input to the black box and render the quads and audio commands that it outputs every frame. This minimalism meant it was manageable for me to create two hosts: one in Heaps and one in Unity.

All perfectly clear

Haxe itself compiles almost instantly and is well-supported in Visual Studio Code so writing and debugging in Haxe/Heaps is quick and easy. For making releases, I transpile to C#, tab over to Unity, and build the project there.

Step 2: Phoneterface

Maintenance aside, the main thing holding me back from getting this game onto phones was its user interface. Papers, Please was always meant to be played on a large device. The actual resolution is laughably low (570x320) but these pixels need to be big.

The screen is divided into three always-visible regions:

Checkpoint, Booth, and Desk

Back in 2014, I separated out a chunk of interface layout code from the core logic in order to create an iPad version of the game. With a few adjustments I was able to preserve the "three regions" layout and drag-n-drop gameplay. The biggest changes are that the regions are stacked vertically with slight dynamic sizing ability, and the checkpoint area shows a horizontally scrollable window instead of the whole thing at once.

iPad layout. Checkpoint requires scrolling.

James Gray built on this to create the Vita version, which needed even more layout changes but kept all the same gameplay. The small screen required overlapping the checkpoint area with the booth and fullscreen vertical scrolling but the Vita's combination of physical and touch controls offset these compromises pretty well.

Vita layout. Overlapped booth/checkpoint and fullscreen vertical scroll.

For phones I wanted something that felt natural to the device, which meant a few subjective things:

  1. Played in portrait mode.
  2. All three regions visible at all times.
  3. No squinting, zooming, or precision required to read/manipulate documents.

Starting with the iPad layout and throwing together a quick mockup using the latest iPhone screen aspect:

Wrenched into an extreme 10:22 aspect ratio

I tried a few permutations of this, all alike. Converting 16:9 content to 10:22 involves a special kind of desperation. One variation that I didn't hate had the face a bit bigger:


Some conclusions after churning through a bunch of mockups:

  1. The border checkpoint view at the top will have to scroll left/right even more than the iPad version. Thankfully, the important stuff still fits without scrolling.
  2. The documents are too small and the desk area too crowded. There's a fundamental conflict between readability and having enough space for arranging things.
  3. Love that big face.

The jumbo face felt so nice that I concluded the documents should also be big, and that uncorked the brew that defines this port: no more desk, no more drag-n-drop. I swung all the way to readability and eliminated the arrangeability requirements completely:

Goodbye drag, so long drop

The desk where documents were manipulated has been replaced by two separate elements: a carousel for closeup work and a rack for quick navigation.

The Carousel and the Rack

The carousel extends offscreen on both sides

With the carousel, documents are displayed full-size in a long, horizontal, snap-scrolling list. The rack at the bottom shows a representation of all documents in smaller form. Navigation is done either by swiping the documents directly, or tapping/dragging anywhere in the rack.


My first worry when considering this interface was that it changed the gameplay too much. Instead of organizing documents on a 2D surface, you're just swiping around and viewing each document in near-isolation. The banner task from the desktop game is totally missing now.

That's a good clue to turn back but I knew this port would require some blood. After testing my first basic carousel implementation, the arterial spray I expected from losing document management turned out to be more of a minor abrasion.

In the end, I'm happy with the trade-offs. Swiping feels natural on the phone, and dealing with document inspection has its own rhythm – different from the desktop version but still fun. Instead of arranging the documents and darting your eyes around to correlate information, you manipulate the documents as a group and eye-dart a little less.

This carousel + rack represents the big sea-change adaptation for the phone port, and it had ripple effects throughout every single other element of the game's main interface. I'll go through the more interesting of these changes in exhausting animated-gif-supplemented detail below.

But first a word about the pixel grid.

The Pixel Grid

In working on a pixel-art game, one decision every developer has to make is how honest they'll be about their pixel grid. There's usually no actual restriction that all your pixels need to be the same size and even Nintendo's Mario Maker series aerial kicks pixel size consistency directly in the face.

Resolution-wise, the phone interface settled into 208x405 based on how the game's existing documents would fit legibly onto a modern non-Max iPhone. Not a lot of pixels quite frankly, and man that checkpoint region at the top is bulky.

Chunky checkpoint

I definitely prefer maintaining a single consistent pixel grid but we've all got our price. In this case, cheating to two pixel grids solved a lot of problems. With a base scale of 3x, the main booth and document regions can run at 3x while the checkpoint is a more manageable 2x.

Checkpoint at 2x, everything else at 3x

In code, this is handled with fractional 2/3 pixel scaling on the 2x stuff, which multiplies out to integer coordinates in the final 3x buffer.

With the pixel grid sorted there's still the problem of scaling the whole image onto a phone screen. Thankfully the effective resolution of the game is basically only 208x450 and phone screens – especially retina ones – are delirious with pixels. A combination of integer scaling and bilinear filtering doesn't hurt so bad here. The phone version handles aspect adjustments within a certain range and then scales the result to fill the device's screen.

Final image integer + bilinear scaled up to fill the screen

On to some specifics of how the core interface changes played out.

Too Narrow, Doesn't Fit

Several of the game's single visual elements don't fit in the base 208 pixel width. Instead of redrawing them, the extra 2/3 scale option made some fixes easy, as with the title screen:

Using selective 2/3 scale on that big eagle there

The night screen was a little more complex. It's got way too much business going on, coasting easily in the desktop's ocean of 570x320 but requiring a completely new layout and conditional 2/3 scaling to fit in 208x450.

Night screen's roomy desktop layout

Adjusted phone layout with 2/3 scaling to fit status icons, tokens, etc

The start-of-day newspaper was another big boy and needed both the 2/3 scale plus a narrower 3-column layout to replace the orignal 4-column layout:

Changed layout and scale to fit newspaper

This "needs new background image" requirement for the newspaper segues to a core engine feature implemented for the port: load-time image transmogrification.

Image Transmogrification

Papers, Please has a boatload of documents and images that contain text. A good share of these are generated outside the game and have to be localized for each supported language. There's tools to help with that but one of my goals for this port was to do as little localization work as possible.

Instead of generating new images just for the phone layout, the game has a "mogrification" step when loading image assets that makes edits procedurally based on which platform it's running. This is handled at a pretty low level so the higher level game logic doesn't need to worry about it.

One example is the newspaper background, which has the localized name baked into the image. The mogrification process loads both the desktop-formatted localized image and the phone-formatted unlocalized image, then copies a section from one to the other to get a phone-formatted, localized image.

Localizing the newspaper banner at load-time

Another case is the bulletin page with inspection correlation instructions. Here, the mogrifier combines the localized lower half of this desktop-specific bulletin page with the unlocalized phone-specific upper half to get a localized phone-specific bulletin.

This here plus that there

These load-time adjustments go beyond images and into some document layouts too. For example, the rulebook is the widest document in the game and it, horrifyingly, doesn't fit in 208 pixels. It's close but not quite, and there are important tabs and page-turn links on the edges that need to be comfortably onscreen.

Original rulebook, designed for landscape layout on desktops

I wanted to keep the entire booth/documents area in one pixel grid, so 2/3 scaling wasn't an option here.

Load-time mogrification works a number on this to crop each page, move all the active links, and rotate/re-position the tabs to fit within the 208 pixel width.

Mogrified rulebook for the phone portrait layout. RIP spiral.

Hand Over Your Documents

In the desktop build, each traveler drops their documents on the countertop after they enter the booth. Plain and simple.

Countertop docs

Due to the lack of space, the phone layout doesn't have a functional countertop. An obvious alternative would be to have the docs just drop all the way down to the rack at the bottom and appear in the carousel automatically.

That seemed a bit passive to me though, losing the initial "ok fine I'll take these" activity from the original desktop version when you drag them from the counter to the desk for reading. The solution I came up with is to float the documents first, requiring a tap to get them onto the rack and into the carousel. The order they're tapped determines their order in the list.

Floating docs

This design highlights something used throughout the phone version: overlapping elements. The main carousel is intended to feel like you're holding documents up to scrutinize them, enhanced a bit by overlapping the traveler and booth console. To stay practical, it's possible to pull the carousel down for a better look.

The original design of the booth was always pretty spacey but there are some distinct activity modes. For this port it was generally possible to optimize the use of space by masking/overlapping things that weren't needed in one activity with something else needed right now.

Custom UI API OK

A notable benefit of writing my own UI system for this was that I could do some very specific things with touch and pointer input. Papers, Please looks like a pretty simple 2D game at a glance but representing all the document and interface interactions through standard input event callbacks in the original Haxe/OpenFL required a lot of uncomfortable hacks. The new input system doesn't use callbacks and is cleaner now, which let me implement all the overlapping and passthrough input logic in a more manageable way.

Old event system, where input handling logic is spread around in various callback functions

New react() event system, more centralized and flexible


The desktop version has a stamp bar that slides out from the right side and hangs over the desk. To stamp a passport you just align it under the stamp and click.

Original desktop stamp bar

I had some trouble redesigning this for the phone layout because (A) you can't just drag the passport around to align it under the stamp, and (B) I didn't want to give up the satisfaction of stamping on press/touch. Without this second requirement it's simple to just make the stamp itself draggable, to be dropped on the passport where it automatically stamps. But then you're stamping on release/un-touch, which never felt right to me. Some concepts I worked through:

Ceiling attachment, Split bar, Laser sensor

These all felt weird, confusing or just flat out didn't work. There was something implicitly sensible with the desktop stamp bar that I wasn't getting.

Ater some more experiments I decided to re-introduce the original setup, behind a modal switch. From the carousel, passports can isolate into a temporary stamp desk to be dragged around and stamped willy nilly.

Opening a temporary desk just for stamping

This temporary stamp desk is opened with a pull chain, which allows me to save some screen space and helpfully indicate its availability only when a stampable document is focused. Also it's marginally fun to play with the chain.

Swinging chain

Key Desk Too, Sure Why Not

This same temporary-desk mechanic is used for the rifle and tranquilizer gun sequences. When a border attack is in progress, it quickly flips to the keys on an isolated desk. Dragging the keys and unlocking a gun works the same here as the desktop version.

Temporary key desk for unlocking guns


The need for a modal stamp/key desk was a gradual realization born from failed experimentation. In contrast, I knew from the start of this port that I'd need a way to let the player align certain documents on top of each other.


To replicate this functionality in the new carousel mobile design, I added an alligator-clip style pincher. Like with the stamp desk pullchain, this pincher appears only when certain documents are focused in the carousel.

Alligator-clip pincher appearing near pinchable document, for pinching

Tapping the pincher will grab the document and hold it above and out of the carousel, where it can be dragged around separately. The carousel remains scrollable underneath so things can be lined up.

Pincher action

None of this is explained in-game so let's assume the concept and mechanics are self-evident.

Wanted: Criminal Poster

One emergent play pattern in the desktop version is how to deal with the daily criminal bulletin.

Detain these people

Any traveler matching a photo here needs to be interrogated, and the best way to keep this in mind while working is to tuck it off to the side of the desk and leave it there throughout the day.

You look familiar

Since the freeform desk is gone in this new phone layout, it's easy to swipe away from the bulletin and completely forget about the mugshots. This is a problem I didn't catch until playing through the mostly-finished port, where I missed having a way to see those three faces at all times.

Space is so tight in the phone layout that there's no perfect place to put them but I did find a little spot on the back wall where they can be hung up.

Mugshots helpfully posted on the wall

To preserve some of the desktop version's intent, this isn't automatic. The player needs to tap a pin in the bulletin to post them on the wall at the start of each day.

Pinning from the bulletin to the wall

Action Buttons Don't Fit

This could've gone under Doesn't Fit. The desktop version's FINGERPRINT, SEARCH, and DETAIN buttons don't fit.

Too wide to arrange these horizontally in 208px

This was painful. These buttons drop down after an interrogation, and their appearance is a noisy eye-catching signal that the player has some new actions available. It's rare, but possible, that all three are available at the same time.

Plenty of space over here

With basically no place to put them and not wanting to scale them down I decided they couldn't go in the main booth interface. There's now an extra slide-out panel to hold them.

Panel of buttons. This is an older version -- I removed the icons and right-aligned everything at the last second.

The panel pops out automatically when a new action becomes available, closes when clicking anywhere outside, and shows a flashing lever when hidden.

Opening the panel with a flashing lever

These buttons are fully localized as images, so reusing the same buttons and just putting them in a pop-out saves a bunch of trouble with the localization process.

Enhanced Inspect Mode

The original game treats its discrepancy-highlighting inspect mode as a quick aside. First the player arranges documents to get a clear view, then finds a discrepancy, enables inspect mode, highlights the problem, and finally interrogates. While inspect mode is active, documents can't be moved and pages can't be turned – things have to be in order before inspecting.

Interrogating a discrepancy in the desktop version

This process is impossible if you can't see all your documents at once, as in the phone port. The fix was conceptually pretty simple but a fairly big technical pain in the ass: the carousel and page turn buttons are still active during inspect mode, and highlighted elements remain partially onscreen when their documents scroll away.

Inspect mode in the phone version

Polishing Tweaks

With all of the game's interactions there was plenty of room for small frustrations. I did my best to fix these whenever I found them. None are critical but half the battle is just recognizing there's some friction that could use smoothing, so I enjoy these kinds of tweaks.

Auto Sorting

The three most important documents are the all-day ones: the bulletin, audio transcript, and rulebook. Because everything's in a flat list now, it's important to keep these three close to where each entrant's new docs appear on the right. To do this, the game performs an instant re-sort when calling an entrant, without affecting the focused view.

Instant auto-sorting to keep important docs on right side

Post-Give Smart Shift

Once a passport is stamped, the only thing left to do is give the entrant all their documents back. Giving a document back removes it from the carousel list, shifting the next document into view. My first implementation was the naive one: if document n disappears, shift to n-1. For a sequence of giving, this means it may shift to a document that's not meant to be given back, requiring the player to manually swipe around. Not a huge deal but it's a lot less annoying when the carousel intelligently shifts to the next givable one.

Shifting to the next givable document


While looking for correlations and discrepancies there's a lot of flipping back and forth. With swiping this is straightforward but if you keep your finger held on the screen and just slide it left and right, the reach isn't enough to actually see whole documents on either side. Not immediately obvious but measurably frustrating while playing. My fix was to implement a "peek" feature that snaps the adjacent document fully into view when sliding just a little bit.

Peeking at adjacent documents

Auto Player Testing

The original desktop game has a debug "soak" test mode for stressing memory and gameplay logic. Basically, a process plays through the game by calling internal debug functions to load a day, stamp documents, shoot attackers, etc. This was useful for making sure there were no errant memory leaks or serious logic errors but because it called directly into high-level functions it wasn't much use for testing the whole game and engine environment.

Having full control over the entire software stack for this port enabled me to enhance this system considerably. Instead of a memory soak test, it's now a full automatic player that controls the game using the same input messages that a real player would.

Auto player in action, showing the route, day, traveler, time, memory, etc.

The auto player is scriptable and follows "routes" to:

  1. Play through to all 20 endings, starting from the title screen.
  2. Earn all tokens.
  3. Play a few minutes of each Endless style and course.
  4. Seed each playthrough with a different random seed.

Defining a route plan

And because the entire rewritten game+engine is modular, it's possible to run this auto player as fast as possible, with or without visuals. On a modern iPhone rendering every 1,000th frame, a full sweep through all ~30 routes takes about 15 minutes.

Creating this system was at least a solid month of initial work and a good amount of maintenance throughout. Quite a long time honestly but developing and using it has helped me find and fix countless bugs in the game. Since I develop on a Mac using Haxe/Heaps and deploy in Haxe/Unity, it's also an invaluable way to verify that Heaps and Unity give the same result, and that Mac/iOS/Android all behave correctly. Smaller unit testing can validate some of this but nothing beats full playthroughs.

Platforms, Plural

There are now three core interface modes for the game: desktop, tablet, and phone. This port deals with just the mobile interfaces and supports phone and tablet layouts in the same app/binary.

Both mobile layouts

Since everything is in one binary, the game needs to know where it's running in order to pick the right interface on startup. On iOS, this is easy as there's an explicit check you can make for iPhone or iPad. On Android, surprisingly, there's no clear delineation between phones and tablets and the game has to wing it based on resolution and DPI.

In the cases where it guesses wrong, there's an added setting to explicitly choose which mode you want.

Some devices get to choose

The final breakdown looks something like this:

  • iPhone: phone layout only.
  • iPad: default tablet layout, optional phone layout.
  • Android Device: Optional phone and tablet layouts, default best guess.


A proper mobile version of Papers, Please has been a long time coming. Since the desktop release 9 years ago I'd dismissed the idea that the game could even work on phones but I guess a switch must've flipped in my head last year.

From start to finish this port took around 8 months of work. That sounds like an eternity but I enjoyed most of it so won't complain too loudly. The sharpest cut is that it's kept me away from working on anything else.

After putting out whatever fires light up after release I'll take a short break and then get back to making new stuff. See you then.

All Comments: [-] | anchor

chairhairair(10000) 5 days ago [-]

The productivity of solo indie game devs is just really impressive.

While I'm aware of the "go fast alone, go far together" quote, I'm depressed by how inefficient app/ui development is at big tech companies compared to game studios, especially short staffed indies.

Really makes me wonder about the "engineering excellence" that tech leads and "architects" pride themselves in when games are developed much more quickly and are - to my eye - much more stable and performant.

oreally(10000) 5 days ago [-]

Yea the self wankery you see when these people place on insubstantial things like correctness against all possible situations in their PRs is amusing.

Elon once wanted to move his programmers to windows upon seeing how fast world of warcraft was developed. Goes to show how fast people can be if they care about the things that matter.

yieldcrv(10000) 5 days ago [-]

how far do you really need to go, alone

You need a win, but not that big of a win in the grand scheme of things

Six figures, a couple million

Whereas big studios need multiple hundred million+ hits

spaceman_2020(10000) 5 days ago [-]

I don't play a lot of games anymore, but the last one I spent a considerable amount of time with was Stardew Valley. And I couldn't believe it was made by just one guy.

999900000999(10000) 4 days ago [-]

Remember last Tuesday when you wanted to add something, but your manager said no.

Remember 2 weeks ago, and you really wanted to implement CICD, but the DevOps team told you not to.

When you're developing any solo projects, you don't have anyone else telling you what you can't do.

This is amazing, but you can also easily spend countless hours building something nobody really likes. I've made a small handful of games, a few of them have been released publicly. If I had to guess, at most maybe 20 people have played my games.

But I taught myself everything I know via learning game development, and my career is amazing.

Even now, I'm trying out different engines and having a blast. Odds are. I'll probably never produce anything that becomes all that popular. It'll just be another throwaway game on itch that nobody plays, but I can say without a doubt I had a hell of a time building it.

chairhairair(10000) 5 days ago [-]

Expanding on this in a reply to my own comment.

Grinding Gear Games, as an example, is a NZ based game studio that releases a new expansion to their famous RPG - Path of Exile - every three months.

The game is not a simple program by any stretch - which might be a fair criticism for Papers Please by comparison since it is purely client-side and all possible states of the game can be fairly easily enumerated.

Path of Exile is a persistent cross-platform 3D online multiplayer game that has daunting requirements for consistency, latency, and graphical performance.

Despite that, each three month release cycle introduces more new UI elements and features than I think the entirety of Google apps release in the same timeframe. And that's not because Google isn't trying to build new things - they are and I've written UI for several of those projects.

Does anyone else feel a real sense of deflation when it comes to app/web development velocity compared to games?

Jare(10000) 5 days ago [-]

A lot of it is about tradeoffs (long term vs one-off projects, business stakeholders, and a long etc), but there's also significant selection bias. Most 'indie' projects die before having much of a playable thing, and many successful solo indie projects can take many long and grueling years (e.g. Stardew Valley).

Engineering in most organisations will place stability and predictability very high among priorities, and pay a hefty price for it (sometimes, messing up along the way and getting neither benefit). An individually brilliant engineer can be a great asset if properly managed, but you need the majority of people to be less of an outlier.

ElevenLathe(10000) 5 days ago [-]

I've had my eyes opened recently onto the world of modern game editors (Unity, Unreal, Godot, etc.) by watching gamedev streams. The productivity of these environments, and their associated asset stores, is amazing. You can't help but think that other domains would also be well-served by a fully-integrated experience like this. Surely the universe of 'webapps' or 'unix-y network services' is at least as constrained as '3D games' is (possibly more?).

I would certainly be interested in a 'Unity for line-of-business apps.' I guess this was VB6 lol.

grapeskin(10000) 5 days ago [-]

Having a complete mental map of your work environment is the key. Nothing changes without your approval and you have some idea of when and where every change was made along with what it does.

Most of dev work is making sure other people can make sense of your work. With solo dev, that step is basically unnecessary.

I think it's also why some indie games (eg binding of Isaac) get completely remade instead of making updates a year later. Walk away from the project for a couple months and it's an untamable beast.

yieldcrv(10000) 5 days ago [-]

I thought this has been on ipad forever

MattRix(10000) 5 days ago [-]

Yes it has. He talks about the iPad version in the post.

keyle(10000) 5 days ago [-]

The blog post is really top tier. Consider how much time some engineers spend considering details when they're just being contracted to do a job (I've work with lots) and how this indie game project is being explained, nitpicked, loved and cherished.

The blog post alone makes you want to purchase whatever this person has been working on based on how much passion is oozing off the explanation of how it's made.

Side note... A lot of YouTubers also work like this: pushing the craftsmanship to beyond expectation to almost cause an emotional reaction.

In this case though, it's genuine engineering with high standards.

lemming(10000) 5 days ago [-]

IMO you should absolutely buy what he produces, partly as you say to support a genuine craftsman, but also because (unsurprisingly) the games he produces are really good. I also think of my purchases as a donation to keep his blog posts coming.

wly_cdgr(10000) 5 days ago [-]

This blog post is good, but is pretty average stuff for a real professional. You are just used to reading webdev tip&trick postlets from third-rate hacks

kergonath(10000) 5 days ago [-]

> The blog post alone makes you want to purchase whatever this person has been working on based on how much passion is oozing off the explanation of how it's made.

Papers Please is nice, but Return of the Obra Dinn is a masterpiece. Honestly. A very cool concept with some impressive attention to detail. Pixel-perfect 1-bit dithered graphics, amazing soundtrack, and a nice story. I don't have a link here but there was a forum thread somewhere where he discussed his progress as he was making the game. It's a bit long but we'll worth a read as well.

diebeforei485(10000) 5 days ago [-]

I own the iPad version. Anyone know if I need to pay again for the iPhone version?

snoopy_telex(10000) 5 days ago [-]


> He added that the game would be available as an update to the 2014 iPad app

routerl(10000) 5 days ago [-]

You don't. Go buy it and the price will be gone.

yojo(10000) 5 days ago [-]

I played through Papers, Please back when it came out on desktop, and at the time I remember wondering: 'why is this fun?' I enjoyed it tremendously, but on the surface that didn't make any sense. Who wants to play an immigration officer sim? Getting this peek behind the curtain helped me understand all the little decisions that add up to an unexpectedly fun experience.

The level of thought given to tiny UI interactions here is wonderful. Details like being able to swipe around to 'play' with the dangling pull chain. Any other dev would just make it a static image and call it a day. But these little bits of magic working together transform one of the most boring possible topics into a real gem of a game. This post should be required reading for interaction designers.

chii(10000) 5 days ago [-]

> I remember wondering: 'why is this fun?'

Other than the tactility of the UI (which is a major part of the game), the reason, i believe, it is fun is because the game's mechanics matches that of the actual role you play in the story. Many games don't really get this correct (https://en.wikipedia.org/wiki/Ludonarrative_dissonance).

In papers please, your decisions aren't clear cut, like in a regular RPG game, where you can 'choose' to be a good guy or a bad guy by selecting one of two options, and one is clearly meant to be the good choice with the good ending, and the other bad. Papers please actually make you think like someone surviving a authoritarian regime, and your actions reflect that role too (you would, for example, choose not to feed, if family isn't absolutely hungry, or that you would attempt to deny entry as fast as possible, since a denied entry doesn't make you any money - no room for sympathies).

It makes the game feel 'real'.

seba_dos1(10000) 5 days ago [-]

> Any other dev would just make it a static image and call it a day.

You think so? Games are often full of little things like that. That's a good thing to point out, but it certainly doesn't seem as unusual as you paint it. Letting the user be playful is generally what games are good at ;)

jordanmorgan10(10000) 5 days ago [-]

There's a certain energy that's either unmatched, or it's just new to me, specifically from indie game developers that invigorates me. His post is just kinda filled with a "love of the game" vibe. I don't feel like other tech related industries have that same art house feel like game developers seem to have, or at least write about. You don't really see it in the sass space, or in my world of iOS, etc.

Just something I've noticed about the indie games world. Seems like an inspiring, genuine space.

spaceman_2020(10000) 5 days ago [-]

Imo, indie game devs don't get nearly the amount of mainstream attention they get as artists. So many of them do everything on their own, from the art and music to, of course, the coding.

That kind of cross-discipline talent is so hard to find.

marvin(10000) 4 days ago [-]

I think indie game devs of Pope's talent, motivation and ability are exceedingly rare. We're seeing only those who survive a filter that removes 99%. But the results are absolutely formidable.

alexb_(10000) 5 days ago [-]

I actually didn't know it was on mobile, so I searched for 'Papers Please' on the Google Play store. Only complete garbage came up, and I had to actually go to the website to find a link to the game (which only has 5k downloads!)

I wonder if there's a solution to this.

djhworld(10000) 5 days ago [-]

Thanks, I searched the play store for both 'papers please' and 'papers, please' which didn't yield any meaningful results. If I hadn't have read this comment I would have (lazily) assumed it was iOS only.

Play store says there have been 4000 downloads so at least some people are getting to that page, but probably through direct links rather than search

mijoharas(10000) 5 days ago [-]

Weirdly I was lookingt to see if there was a papers please on Android last week and concluded it hadn't been ported to mobile because of the results.

Lucky this came up on HN so I found out. If that weren't the case I doubt I'd have thought about it again and wouldn't have ever bought/played it.

komadori(10000) 5 days ago [-]

Oh, wow, thanks! I did likewise and assumed the mobile port hadn't actually been released yet. Google's search is clearly awful :-/.

Hyperbolicum(10000) 5 days ago [-]

The issue is the missing comma, I ran into the same problem. Searching for 'Papers, please' will yield the expected result.

Maybe that's something that can be configured?

sintezcs(10000) 5 days ago [-]

Can't find it in the AppStore :(( looks like some regional restrictions are set

rolph(10000) 4 days ago [-]

'I created Papers, Please in 2013 specifically for desktop computers with mouse control. Now, here, in 2022, desktop computers no longer exist and all computing is done via handheld mobile telephone.'

No . Desktop and mobile are 2 different ecosystems. The users have desktops; those who are used, have mobile; those who are kings, have both.

mr_toad(10000) 4 days ago [-]

Satire: The use of humour, irony, exaggeration, or ridicule to expose and criticize people's stupidity or vices, particularly in the context of contemporary politics and other topical issues.

mastax(10000) 5 days ago [-]

Haxe is really unusual and interesting, and I don't think it gets talked about enough.

> Haxe can build cross-platform applications targeting JavaScript, C++, C#, Java, JVM, Python, Lua, PHP, Flash, and allows access to each platform's native capabilities. Haxe has its own VMs (HashLink and NekoVM) but can also run in interpreted mode.

Compiling from one language to another isnt particularly unusual, but compiling from one language to so many is very unusual. On first impression it sounds unserious—real compilers output machine code—it's tempting to denigrate it by calling it a transpiler. But there are a lot of advantages that come with this approach. You always have access to the full capabilities of your target platforms. From a single language you can write code that is massively portable while also targeting specific platforms with just an if statement.

The 'real' compiler authors spend months working on linkage, calling conventions, runtimes, symbol mangling, allocators, and debuginfo trying to get their native code to link properly to the objective-c frameworks on iOS—and it never feels quite right. If you instead compile to objective-c, a lot of things get easier. It's a very pragmatic approach.

bogwog(10000) 4 days ago [-]

Haxe's transpiling helped me in a project once. The client needed a small-ish web app written ASAP, but they didn't know anything about their backend/stack when asked since they were non-technical and using an external provider for everything. The provider was not responding to emails for some reason, and the deadline was approaching, so I decided to start writing the thing in Haxe.

I figured they were either running PHP/SQL or Node, so I wrote a simple backend in a way that would make it easy for to deploy to either one with minimal changes. By the time the provider finally replied, the project was nearly half way complete. It turned out that they were using a standard PHP/SQL stack, so had I gone with Javascript there would've been problems. Instead, all I had to do was change one flag in my build system.

I don't know if this is a big selling point for Haxe since it's such a highly specific situation... but it's probably at least worth mentioning :P

TazeTSchnitzel(10000) 4 days ago [-]

> The 'real' compiler authors spend months working on linkage, calling conventions, runtimes, symbol mangling, allocators, and debuginfo

Don't worry: targeting high-level languages gives you a different, equally-frustrating set of problems! Actually, many of them have equivalents to the low-level ones: you also have to worry about calling conventions, name mangling, runtimes... they just look quite different.

Especially you want to support any kind of dynamic functionality, there are many uncomfortable trade-offs involved.

vmladenov(10000) 5 days ago [-]

> On first impression it sounds unserious—real compilers output machine code—it's tempting to denigrate it by calling it a transpiler.

What? No, this is so wrong. "Real" compilers transform input from a source language to a target language. That's it. A program could compile a language to itself with functions inlined and it would still be a compiler. Transpiler is a dumb word made up to identify compilers that output valid source for a chosen language.

boondaburrah(10000) 5 days ago [-]

There's also that Haxe is older than a lot of the stuff we take for granted today. It's roots are in ActionScript, and it started as basically a successor to AS2 before Adobe came out with ActionScript 3. It did 'same codebase on server and client' by compiling to flash bytecode and PHP before node existed. It's ECMAScript/AS roots + static types + type inference make it feel like alternate-timeline TypeScript as it also compiles to JS.

So it's completely comfortable for me. AND I can hit C++ if the platform demands it!

lastdong(10000) 5 days ago [-]

Love this game, bought it for different platforms over the years. I'll prob end up doing the same for mobile :) Papers Please, FTL and Don't Starve (discovered them around the same time) are on my top list of games, it was a period of very fine releases.

macintux(10000) 5 days ago [-]

Thanks for reminding me I need to play Don't Starve again. Dangerously addictive though.

psyc(10000) 5 days ago [-]

I was ready to charge in here and pitch a fit because I thought the title was referring to storage / memory.

zbird(10000) 5 days ago [-]

Yeah, though on the other hand the title does not seem particularly taxing. 'Shove it down to Unity' and then the big bulk of the work is cramming the game into a small screen.

The 'Auto Player Testing' is smart and a token of good design, as he must have completely decoupled IO from the main game logic. That seemed the most interesting to me.

egypturnash(10000) 5 days ago [-]

Man it is gonna be so nice when the average young adult has a device in their pocket that folds out to the size of a paperback, or even a magazine, and we can start making popular culture that fits that size again instead of everything having to make sense through the tiny window of a phone.

soylentgraham(10000) 5 days ago [-]

Phones could be fine if 95% of the screen wasn't covered in popups, adverts, mailing list & cookie prompts.

blooalien(10000) 5 days ago [-]

> From the article: 'Now, here, in 2022, desktop computers no longer exist and all computing is done via handheld mobile telephone.'

They're kidding, right? Nobody in their right mind actually believes that, do they?

mr_toad(10000) 4 days ago [-]

> when the average young adult has a device in their pocket that folds out to the size of a paperback

The average young adult likes to use their phone one handed...

Animats(10000) 5 days ago [-]

Yeah. This is a big problem with the 'metaverse'. Peering into a high-resolution 3D world via a hand-sized screen is tunnel vision. We're still a long way from the 'swim goggles' form factor in VR headgear, which is what Carmack says is the minimum for reasonable consumer adoption levels.

Playing 'Papers Please' on a phone requires a good memory. Do you remember what the recognized visa issuing cities for Kolechia are? You need to know that. If you have to swipe to the rulebook to look it up, your productivity will drop and you won't make enough money for the day to keep your family fed.

aliqot(10000) 5 days ago [-]

I'm noticing a lot of the younger crowd don't seem as glued to the phone as their parents. Materialists will always be materialists, but as an adherent to Ordnung, I don't own/need a phone, so it sticks out and it's obvious to me that the normal garden variety youth these days are not as absorbed as the first generation to this little thing known to 'create fire' because it is now commonplace. A computer in your pocket has 0 novelty or wow-factor to this generation, as it should. Nobody fawns over a butane-lighter or debit card, they're commonplace despite being relatively new.

This isn't just in my community, it's noticeable enough in my travels that it seems to be a trend. I assume it is because of more short-format digestible content, along with the shift of social being one-to-one and one-to-many, to being many-to-many, in the sense that you're not necessarily seeking out those you had a direct relationship with, you're seeking out elements and segments of a topical zeitgeist, whether that be tech videos, memes, cat compilations etc.

I also have another hypothesis- when phones that provided a rich experience first debuted, it was the nerds and city folk who got it first. iPhone then brought this mobile-first-lifestyle to the stylemakers and artists and those whose inner monologue is narrated by Justin Long, folks who'd likely have bought anything apple anyway. From there, smartphones and rich experiences were disseminated into the lesser elements of the greater public who either are receptive to tastemaker's influences or have limited option to refute the convenience of popularity; popular hardware is cheap, ubiquitous and accessible, some might say in some regards modern smart phones are disposable.

What I'm getting at is this, this stuff is no longer a mystery to this generation. We are now 2 or 3 generations removed from this type of pocket-computer being anything wow-inducing. I think of it sometimes like when I was a youngster, the class of people who traveled via air vs everyone else at ground level. Air travel had a mystique and prestige, this person must be doing something to be enjoying a cigarette and being served a glass of wine however many feet in the air, direct to destination. The same way I might not be in admiration of my neighbors boots for having a good welt, because a good welt is a given, I assume the youngster of today are no longer enamored by the novelty of a mobile phone or pocket-computer. As such, it is no longer a status symbol for most. So what the new iPhone came out and you got one, that's only a valid status symbol for maybe a few weeks, for over 1,000 USD invested in some models.

Youth of today, I don't see them going for a pocket atlas or any such form factor, I see them going for augmented spectacles or lenses. Everything indicates that a new 'moores law' is taking effect around energy storage and thermodynamics - we are no longer optimizing per-core clock speed, we are optimizing core count and the amount of energy that can be stored to later be turned into CPU cycles rather than heat. As soon as the battery technology will allow it, you will see lenses, whether they be spectacles or contacts, that will take in and assimilate your surroundings, your focus, and the imperceptible changes to your heart rate, retinal dilation, and ocular pressure responses to commercial items. It's not far fetched, we already know of this research being done. Despite the cumbersome experience of VR, we are seeing a point where it is no longer 3D TV or bluray level tech, it's sub-standard as a whole but more and more people are buying it because it shows promise.

I see in the future that our interface devices, whether they be communicators like phones, or additive interfaces like AR spectacles that can dole out retail info in response to a brief biomarker-spike like pupil dilation when glancing at a new pair of shoes. These devices will be funded by corporations much the same way tech learning materials, operating systems, and software is today. It makes most sense that before wider adoption, they'd first be available to those with the most capacity for realizing an ad-prompt via converting to a purchase, so think of like snapchat goggles release, but at your local best buy.

pocket-held mobile phones are the least optimal form factor for every purpose or task it can accommodate other than 'fits in pocket'. Mark my words, as soon as it can be bonded to a wearable lens, it will be, and the corporations will subsidize it heavily. You think adtech is bad now, just wait.

funstuff007(10000) 5 days ago [-]

> here, in 2022, desktop computers no longer exist and all computing is done via handheld mobile telephone. Time to update this dinosaur.

What a witty fellow.

Underphil(10000) 5 days ago [-]

I took it as tongue-in-cheek but as a desktop loving dinosaur I couldn't help feel mildly personally attacked :)

simonbarker87(10000) 5 days ago [-]

I'm always amazed at how many programming languages there are. I'd never heard of Haxe, I don't think it would have crossed my mind to look for something like it but here it is powering a highly successful game with, from what I can tell, a vibrant eco system around the language as well.

Perhaps I'm not curious enough to go exploring for these languages. I've used a few smaller ones in my years (usually because of an external forcing factor - like Squirrel running on ElecticImp devices) but I tend to stick to the big names we all know

boondaburrah(10000) 5 days ago [-]

Haxe is wild to me since I recently started learning it and realised it's basically TypeScript but before TypeScript. Since it's statically typed and can hit C#/Java/C++/JS, I really want to try it in line of business applications as well.

MBCook(10000) 5 days ago [-]

I've only ever heard of it in one context. About seven years ago TiVo announced that they were going to start using it to program their devices when they made the new (terrible) interface.

I don't think I've heard about it since then.

dhosek(10000) 5 days ago [-]

I came of age in an era in which it was often important which version of a language you were programming in. Not all C code could compile with all C compilers (which was part of the motivation behind the C preprocessor), likewise with Pascal, BASIC, FORTRAN, etc. IBM had two different Pascal compilers for VM/CMS named, confusingly, VS/Pascal and Pascal/VS which were almost but not quite identical in functionality and features. On timesharing systems, you might discover all manner of legacy languages lurking on the (dishwasher-sized) hard drives. I checked out a book on SNOBOL from the library to understand what was happening in some SPITBOL code that I found on UIC's mainframe that was part of the source for a C compiler. Most personal computers came with some version of a Microsoft BASIC in ROM, but there were differences from one platform to the next so you couldn't necessarily just type in a program written in AppleSoft BASIC and run it in QBASIC under DOS. The fact that in 2022 JVM languages run identically anywhere and that Rust is (almost) platform-agnostic is, to be honest, kind of miraculous.

tokinonagare(10000) 5 days ago [-]

> I'd never heard of Haxe

Well it was born in France in a web game company (some of their games were pretty famous domestically) as an internal language, and the main selling point at first was the multiple compilation targets (PHP, JS) which included Flash. It wasn't something developed in English in the open at first like a lot of new languages are nowadays, so obviously it took some time to get some international exposure.

999900000999(10000) 5 days ago [-]

I really like Haxe.

I want smaller game engines to succeed, but the tooling is just painful to use.

From the time I spent with Haxe, it's very neat language. At the same time, half of why I make games is to learn.

If I have to use your custom language I'm not learning skills to use at work.

For example, with C# you can make games with several languages, you can also work on boring Fintech so you can pay your rent.

With .net core open source I'd love for more engines to use it. Godot 4 with Mono will be a very very strong contender.

MOARDONGZPLZ(10000) 4 days ago [-]

This is such a good blog post. I also didn't know this author was behind Obra Dinn, but that one is also great. I used to play Papers Please years ago and will certainly purchase it when it does come out for iPhone.

emllnd(10000) 4 days ago [-]

It's available on the iPhone App Store for me!

rossvor(10000) 5 days ago [-]

If you see a devlog post from Lucas Pope you know it's going to be a goldmine. No matter the topic. Dude has a real knack in writing these, clearly describing the problem and the thought process on possible solution. And making it all very interesting so you yourself start thinking how would you address it or what other cool thing could be built instead.

Here's some of his other huge devlogs on TIGSOURCE:

1. Papers, Please. https://forums.tigsource.com/index.php?topic=29750.0

2. Return of the Obra Dinn. https://forums.tigsource.com/index.php?topic=40832.0

franknine(10000) 5 days ago [-]

Another great devlog thread on TIGSOURCE is Leilani's Island by Craig Forrester, it's a treasure trove of 2D platformer development:


bradknowles(10000) 3 days ago [-]

TL;DR -- it's a game. That was designed for computers with a mouse and not for a touch-oriented environment, and so a lot of hard work had to be done to port it to phones.

BoredPuffin(10000) 4 days ago [-]

Thanks for recommending this! Haven't heard of him before but really interesting read.

collegeburner(10000) 5 days ago [-]

Arstechnica did a war stories with him on Return of the Obra Dinn, combining one of my fave video series with one of my fave gamedevs. https://www.youtube.com/watch?v=OMi6xgdSbMA

scyzoryk_xyz(10000) 5 days ago [-]

Yes - I remember even coming across those before Papers became a bigger thing. So thorough

dclowd9901(10000) 4 days ago [-]

From a practical standpoint, I can't even understand how he writes these! Presumably, this process took him months (a year?) to work through. What was he doing that whole time? Taking notes and captures for an eventual blog post? It just seems he has an incredible ability for organization and foresight, one that I am deeply jealous of.

aerovistae(10000) 5 days ago [-]

Yes, the guy is honestly some type of genius. To be this talented of a programmer, and across multiple platforms, languages, and toolsets just dazzles me. It's hard enough to just be competent with one platform. I can't imagine.

And on top of that he composes the music and makes the art and literally everything else.

I honestly just don't understand how a person can get that good at that many things.

frenchie14(10000) 5 days ago [-]

His youtube[0] also has some very interesting behind the scenes content. If you haven't seen the Obra Dinn ship building timelapse[1] you're in for a treat! (Major spoilers - don't watch unless you've already beaten the game!)

[0] https://www.youtube.com/user/dukope1 [1] https://www.youtube.com/watch?v=qZFoBvJf8Ug

Historical Discussions: Instagram can track anything you do on any website in their in-app browser (August 10, 2022: 1195 points)

(1199) Instagram can track anything you do on any website in their in-app browser

1199 points 1 day ago by the_mitsuhiko in 10000th position

krausefx.com | Estimated reading time – 18 minutes | comments | anchor

The iOS Instagram and Facebook app render all third party links and ads within their app using a custom in-app browser. This causes various risks for the user, with the host app being able to track every single interaction with external websites, from all form inputs like passwords and addresses, to every single tap.

Note: To keep this post simple, I'll use 'Instagram' instead of 'Meta' or 'Facebook'

What does Instagram do?

  • Links to external websites are rendered inside the Instagram app, instead of using the built-in Safari.
  • This allows Instagram to monitor everything happening on external websites, without the consent from the user, nor the website provider.
  • The Instagram app injects their tracking code into every website shown, including when clicking on ads, enabling them monitor all user interactions, like every button & link tapped, text selections, screenshots, as well as any form inputs, like passwords, addresses and credit card numbers.

Why is this a big deal?

Instagram is purposely working around the App Tracking Transparency permission system, which was designed to prevent this exact type of data collection. After its introduction, Meta announced:

Apple's simple iPhone alert is costing Facebook $10 billion a year

Facebook complained that Apple's App Tracking Transparency favors companies like Google because App Tracking Transparency "carves out browsers from the tracking prompts Apple requires for apps."

Websites you visit on iOS don't trigger tracking prompts because the anti-tracking features are built in.

Daring Fireball & MacWorld

With 1 Billion active Instagram users, the amount of data Instagram can collect by injecting the tracking code into every third party website opened from the Instagram & Facebook app is a staggering amount.

With web browsers and iOS adding more and more privacy controls into the user's hands, it becomes clear why Instagram is interested in monitoring all web traffic of external websites.

Facebook bombarded its users with messages begging them to turn tracking back on. It threatened an antitrust suit against Apple. It got small businesses to defend user-tracking, claiming that when a giant corporation spies on billions of people, that's a form of small business development.

EFF - Facebook Says Apple is Too Powerful. They're Right.

Note added on 2022-08-11: Meta is most likely following the ATT (App Tracking Transparency) rules (as added as a note at the bottom of the article). I explained the above to provide some context on why getting data from third party websites/apps is a big deal. The message of this article is about how the iOS Instagram app actively injects and executes JavaScript code on third party websites, using their in-app browser. This article does not talk about the legal aspect of things (as I'm not a lawyer), but the technical implementation of what is happening, and what is possible on a technical level.

FAQs for non-tech readers

  • Can Instagram/Facebook read everything I do online? No! Instagram is only able to read and watch your online activities when you open a link or ad from within their apps.
  • Does Facebook actually steal my passwords, address and credit card numbers? No! I didn't prove the exact data Instagram is tracking, but wanted to showcase the kind of data they could get without you knowing. As shown in the past, if it's possible for a company to get access to data for free, without asking the user for permission, they will track it.
  • How can I protect myself? For full details scroll down to the end of the article. Summary: Whenever you open a link from Instagram (or Facebook or Messenger), make sure to click the dots in the corner to open the page in Safari instead.
  • Is Instagram doing this on purpose? I can't say how the decisions were made internally. All I can say is that building your own in-app browser takes a non-trivial time to program and maintain, significantly more than just using the privacy and user-friendly alternative that's already been built into the iPhone for the past 7 years.

The external JavaScript file the Instagram app injects (connect.facebook.net/en_US/pcm.js) is the Meta Pixel, as well as some code to build a bridge to communicate with the host app. This is not just a pixel/image, but actual JavaScript code that gets executed:

The Meta Pixel is a snippet of JavaScript code that allows you to track visitor activity on your website. It works by loading a small library of functions which you can use whenever a site visitor takes an action that you want to track [...]

The Meta Pixel can collect the following data:

  • [...]
  • Button Click Data – Includes any buttons clicked by site visitors, the labels of those buttons and any pages visited as a result of the button clicks.
  • Form Field Names – Includes website field names like email, address, quantity, etc., for when you purchase a product or service. We don't capture field values unless you include them as part of Advanced Matching or optional values.

developers.facebook.com/docs/meta-pixel (June 2022)

'The Meta Pixel allows you to track visitor activity on your website' - This is the problem: It's perfectly okay for a website provider to decide to implement the Meta pixel to track visitor activity. However in this case, the website operator did not consent to having the Meta Pixel installed. On top of that, the website provider doesn't even have a way to opt-out.


I don't have a list of precise data Instagram sends back home. I do have proof that the Instagram and Facebook app actively run JavaScript commands to inject an additional JS SDK without the user's consent, as well as tracking the user's text selections. If Instagram is doing this already, they could also inject any other JS code. The Instagram app itself is well protected against human-in-the-middle attacks, and only by modifying the Android binary to remove certificate pinning and running it in a simulator.

Overall the goal of this project wasn't to get a precise list of data that is sent back, but to highlight the privacy & security issues that are caused by the use of in-app browsers, as well as to prove that apps like Instagram are already exploiting this loophole.

To summarize the risks and disadvantages of having in-app browsers:

  • Privacy & Analytics: The host app can track literally everything happening on the website, every tap, input, scrolling behavior, which content gets copy & pasted, as well as data shown like online purchases
  • Stealing of user credentials, physical addresses, API keys, etc.
  • Ads & Referrals: The host app can inject advertisements into the website, or replace the ads API key to steal revenue from the host app, or replace all URLs to include your referral code (this happened before)
  • Security: Browsers spent years optimizing the security UX of the web, like showing the HTTPs encryption status, warning the user about sketchy or unencrypted websites, and more
  • Injecting additional JavaScript code onto a third party website can cause issues and glitches, potentially breaking the website
  • The user's browser extensions & content blockers aren't available
  • Deep linking doesn't work well in most cases
  • Often no easy way to share a link via other platforms (e.g. via Email, AirDrop, etc.)

Instagram's in-app browser supports auto-fill of your address and payment information. However there is no legit reason for this to exist in the first place, with all of this already built into the operating system, or the web browser itself.

WhatsApp is opening iOS Safari by default, therefore no issues.

How it works

To my knowledge, there is no good way to monitor all JavaScript commands that get executed by the host iOS app (would love to hear if there is a better way).

I created a new, plain HTML file, with some JS code to override some of the document. methods:

document.getElementById = function(a, b) {
    appendCommand('document.getElementById('' + a + '')')
    return originalGetElementById.apply(this, arguments);

Opening that HTML file from the iOS Instagram app yielded the following:

Comparing this to what happens when using a normal browser, or in this case, Telegram, which uses the recommended SFSafariViewController:

As you can see, a regular browser, or SFSafariViewController doesn't run any JS code. SFSafariViewController is a great way for app developers to show third party web content to the user, without them leaving your app, while still preserving the privacy and comfort for the user.

Technical Details

  • Instagram adds a new event listener, to get details about every time the user selects any text on the website. This, in combination with listening to screenshots, gives Instagram full insight over what specific piece of information was selected & shared
  • The Instagram app checks if there is an element with the ID iab-pcm-sdk: surprisingly I found very little information about this online. Basically it seems to be a [cross-platform tracking SDK provided by IAB Tech Lab](https://iabtechlab.com/wp-content/uploads/2021/04/Authenticated-UID-APAC-v2.0-Deck.pdf), however I don't know enough about the relationship between Instagram and [IAB Tech Lab](https://iabtechlab.com/) (e.g. [this tweet](https://twitter.com/IABTechLab/status/1519414703239438336)) According to this tweet, the iab probably refers to "In App Browser" as is not related to the IAB Tech Lab.
  • If no element with the ID iab-pcm-sdk was found, Instagram creates a new script element, sets its source to https://connect.facebook.net/en_US/pcm.js, which is the source code for the Meta tracking pixel
  • It then finds the first script element on your website to insert the Meta Pixel right before, injecting the Meta Pixel onto your website
  • Instagram also queries for iframes on your website, however I couldn't find any indication of what they're doing with it

How to protect yourself as a user?

Escape the in-app-webview

Most in-app browsers have a way to open the currently rendered website in Safari. As soon as you land on that screen, just use that option to escape it. If that button isn't available, you will have to copy & paste the URL to open the link in the browser of your choice.

Use the web version

Most social networks, including Instagram and Facebook, offer a decent mobile-web version, offering a similar feature set. You can use https://instagram.com without issues in iOS Safari.

How to protect yourself as a website provider?

Until Instagram resolves this issue (if ever), you can quite easily trick the Instagram and Facebook app to believe the tracking code is already installed. Just add the following to your HTML code:

<span id='iab-pcm-sdk'></span>
<span id='iab-autofill-sdk'></span>

Additionally, to prevent Instagram from tracking the user's text selections on your website:

const originalEventListener = document.addEventListener
document.addEventListener = function(a, b) {
    if (b.toString().indexOf('messageHandlers.fb_getSelection') > -1) {
        return null;
    return originalEventListener.apply(this, arguments);

This will not solve the actual problem of Instagram running JavaScript code against your website, but at least no additional JS scripts will be injected, as well as less data being tracked.

It's also easy for an app to detect if the current browser is the Instagram/Facebook app by checking the user agent, however I couldn't find a good way to pop out of the in-app browser automatically to open Safari instead. If you know a solution, I'd love to know.


For Apple

Apple is doing a fantastic job building their platform with the user's privacy in mind. One of the 4 privacy principles:

User Transparency and Control: Making sure that users know what data is shared and how it is used, and that they can exercise control over it.

Apple Privacy PDF (April 2021)

At the moment of writing, there is no AppStore Review Rule that prohibits companies from building their own in-app browser to track the user, read their inputs, and inject additional ads to third party websites. However Apple is clearly recommending that to use SFSafariViewController:

Avoid using a web view to build a web browser. Using a web view to let people briefly access a website without leaving the context of your app is fine, but Safari is the primary way people browse the web. Attempting to replicate the functionality of Safari in your app is unnecessary and discouraged.

Apple Human Interface Guidelines (June 2022)

If your app lets users view websites from anywhere on the Internet, use the SFSafariViewController class. If your app customizes, interacts with, or controls the display of web content, use the WKWebView class.

Apple SFSafariViewController docs (June 2022)

Introducing App-Bound Domains

App-Bound Domains is an excellent new WebKit feature making it possible for developers to offer a safer in-app browsing experience when using WKWebView. As an app developer, you can define which domains your app can access, and all web requests will be restricted to them. To disable the protection, a user would have to explicitly disable it in the iOS settings app.

App-Bound Domains went live with iOS 14 (~1.5 years ago), however it's only an opt-in option for developers, meaning the vast majority of iOS apps don't make use of this feature.

If the developers of SocialApp want a better user privacy experience they have two paths forward:

  • Use SafariViewController instead of WKWebView for in-app browsing. SafariViewController protects user data from SocialApp by loading pages outside of SocialApp's process space. SocialApp can guarantee it is giving its users the best available user privacy experience while using SafariViewController.
  • Opt-in to App-Bound Domains. The additional WKWebView restrictions from App-Bound Domains ensure that SocialApp is not able to track users using the APIs outlined above.

I highlighted the 'want a better user privacy experience' part, as this is the missing piece: App-Bound Domains should be a requirement for all iOS apps, since the social media apps are the ones injecting the tracking code.

In July 2022 Apple introduced the Lockdown Mode to better protect people who are at high risk. Unfortunately the iOS Lockdown Mode doesn't change the way in-app web views work. I have filed a radar with Apple: rdar://10735684, for which Apple has responded with "This isn't what Lockdown Mode is for"

A few immediate steps for Apple to take:

Update the App Review Rules to require the use of SFSafariViewController or App-Bound Domains when displaying any third party websites.

  • There should be only a few exception (e.g. browser apps), that require two extra steps:
    • Request an extra entitlement to ensure it's a valid use-case
    • Have the user confirm the extra permission
  • First-party websites/content can still be displayed using the WKWebView class, as they are often used for UI elements, or the app actually modifying their first party content (e.g. auto-dismissing of their own cookie banners)

I've also submitted a radar (rdar://38109139) to Apple as part of my past blog post.

Do what Meta is already doing with WhatsApp: Stop modifying third party websites, and use Safari or SFSafariViewController for all third party websites. It's what's best for the user, and the right thing to do.

I've disclosed this issue with Meta through their Bug Bounty Program, where within a few hours they confirmed they were able to reproduce the "issue", however I haven't heard back anything else within the last 9 weeks, besides asking me to wait longer until they have a full report. Since there hasn't been any responses on my follow-up questions, nor did they stop injecting tracking code into external websites, I've decided to go public with this information (after giving them another 2 weeks heads-up)

Update 2022-08-11: After the blog post went live, Meta sent a reply explaining that the system they built honours the user's ATT choice. However I am still waiting for a follow-up reply on why injecting additional JavaScript code into third party websites is needed to check if a Meta Pixel is setup, considering websites with a Meta Pixel setup wouldn't need additional JavaScript code to be executed. I will update the post once I have heard back.

In the mean-time, everything published in this post is correct: the Instagram app is executing and injecting JavaScript code into third party websites, rendered inside their in-app browser.

Check out my other privacy and security related publications.

All Comments: [-] | anchor

joshstrange(10000) 1 day ago [-]

I was super confused by this since to the best of my knowledge SFSafariViewController blocks anything like this, you, as a developer, cannot inject anything or peak into the view it creates. Then I got to the bottom and realized I was correct, but FB/Meta/IG/etc aren't using SFSafariViewController and instead using the older ways to embed a web view.

Honestly I thought all other methods had been deprecated and had no idea apps could still make use of the less secure (for the user) options. Trust me, as a developer I've wanted to reach inside a SFSafariViewController many times to make my life easier but in the end I've just grumbled and assumed it's not possible and worked around it.

I wish there was a privacy-safe way to get the best of both worlds but due to bad actors I doubt that will be possible. I need to look more into App-bound domains but I don't think even that will give me what I really wish for (a way for the page loaded in SFSafariViewController to tell my app something). Something like postMessage support for SFSafariViewController would be amazing and be safe privacy-wise I think since the contained page would need to support sending/receiving messages instead of just having code injected against their will.

Roby65(10000) about 14 hours ago [-]

How come that my apps get rejected if i don't use SafariViewController but they can publish them?

secretsatan(10000) about 15 hours ago [-]

I wouldn't say WKWebView is 'the old way', it isn't marked as deprecated and just because there's a new class that's easier to use doesn't mean they replace older classes that let you do more stuff. As you've found out, you've run into problems, but I have no idea why you would think not to use WKWebView if you can't do what you need.

That said, in general, never, ever use in app browsers for surfing the web. We use them for specific pages on our website that should interact back to the app, otherwise we tell the system to open the URL in the user's browser

interpol_p(10000) about 18 hours ago [-]

I have started moving from an SFSafariViewController to a WKWebView in my app. In my case, we have a discussion forum that I wanted to display in a sidebar alongside the app, so people could get help while they work. I also wanted to intercept clicks on certain files to open them directly in the app — making it easier for users to check out other work on the forums.

However after reading this article what I will be doing is intercepting any links from my WKWebView to domains outside of my forums and opening them in an SFSafariViewController. I have no intention of tracking anyone, but neither do I want the responsibility or reputation for domains outside of my control

YourGrace(10000) 1 day ago [-]

Yes, developers are able to leverage WKWebview on iOS and a Webview on Android.

One thing about both webviews is that there are callbacks with these implementations that developers can choose to open a link in the embed webview or not. It might be useful for privacy/security for Apple/Android to force developers to allow-list a domain (like iOS's Associated Domains) or such that an embedded webview can load (besides local html and files). It might be something in addition to the developer's callback.

iOS WKWebview: https://developer.apple.com/documentation/webkit/wkwebview Android Webview: https://developer.android.com/guide/webapps/webview Associated Domains: https://developer.apple.com/documentation/Xcode/supporting-a...

koenvdb(10000) about 14 hours ago [-]

At our company we use Cordova for all our apps, which means we have to use WKWebView in order to render our application. As does ever Cordova app.

joshu(10000) 1 day ago [-]

i don't understand why apple allows in-app browsers.

nkozyra(10000) 1 day ago [-]

You mean custom ones, right? WebViews are incredibly useful, but it definitely seems like implementing your own browser gives people a false sense of security, like they've been sandboxed when they haven't.

What would be nice here is a permission requirement if you're injecting code into a browser view.

navanchauhan(10000) 1 day ago [-]

Do you want to cripple the entire app industry? Apps built using React Nativ / Flutter e.t.c use the WebView to render themselves. So they're basically already running "in-app browsers"

But then how do you differentiate when the app is rendering its own view rather than another website? You could apply some restrictions like <iFrame> has nowadays where you need extra security privileges (I think) to render pages / execute scripts not on the same domain

Otherwise you can always open safari from all of these in-app browser views and they could implement a toggle which forces all of them to be opened in Safari automatically

atwood22(10000) 1 day ago [-]

At the very least webviews should treat contents as a subframe and respect the frame option headers.

oconnor663(10000) 1 day ago [-]

Last I heard (years ago), iOS forced everyone to use Safari for webviews, which lots of people also complained about. Did that change? Or is the Safari webview the subject of this story?

hnburnsy(10000) 1 day ago [-]

I'm confused I thought Apple only allowed web browsing via Safari...

'2.5.6 Apps that browse the web must use the appropriate WebKit framework and WebKit Javascript.'

If apps can use their own in app browser, why can't say Brave for example, create an app that does very little, except it browses the web with its own in app browser?

superjan(10000) 1 day ago [-]

This just forbids developers to write their own browser engine. It is OK to use the iOs provided webbrowser as a control in your app.

mbrochh(10000) about 20 hours ago [-]

There is nothing of real value on Instagram. Just delete your account and never look back.

neop1x(10000) about 6 hours ago [-]

My words. People are just wasting their time in these useless ad-selling services.

zsims(10000) about 20 hours ago [-]

Um excuse me, how will people know I had an açai bowl for lunch?

EGreg(10000) 1 day ago [-]

I once wrote an email to Steve Jobs, saying that operating systems like MacOS and iOS should have a secret phrase or icon that they show to you whenever they show a system-level security dialog. (And of course implement the same restrictions on screenshots of that dialog as they do for movies.)

Because otherwise, an app can totally fake the interface of a security dialog. The only way you know, these days, is that password managers and cookie jars work with the 'approved' sites, but they can simply show you a site that doesn't require those, and then fool you into entering your passwords!

Steve never replied to me. And Apple never implemented it.

Mordisquitos(10000) 1 day ago [-]

Fun fact, the Spanish word for password, 'contraseña', originally implied this behaviour as 'seña' —> 'contraseña' ('sign' —> 'countersign').

cloudyporpoise(10000) 1 day ago [-]

The battle for control continues. I started noticing this personally when using social media and took note of the fact that the browsing was still being done within the app when clicking on an external link.

The war on control of data continues on.

thallium205(10000) about 22 hours ago [-]

Facebook used to have a feature in their settings where you could opt out of their in app browser. No longer.

asadlionpk(10000) 1 day ago [-]

I hope Apple doesn't disable JS injection in WKWebViews in response to this. JS injection is the (only?) way to call native Swift methods from JS ie. bridging.

I am not sure what the solution here is. Maybe only allow injection to sites you control (via apple association file).

tehlike(10000) about 15 hours ago [-]

Check app bound domains.

nofunsir(10000) 1 day ago [-]

I hope they do.

AtNightWeCode(10000) 1 day ago [-]

Should only be allowed on domains one owns. Could be solved by DNS records or certificates.

throwaway290(10000) 1 day ago [-]

Wait, websites can call native Swift methods from JS?

jedberg(10000) 1 day ago [-]

Apple can just disallow in app browsers in the store policy. Require apps to call out to the default external browser.

Aulig(10000) about 14 hours ago [-]

Same here. So many apps legitimately depend on these features, which makes me worried about an overreaction from Apple here.

A domain verification would be a huge hassle for me, since I provide an app builder that allows my non-technical customers to build an app (which includes a webview). Asking them to do domain verification would be tricky.

andix(10000) 1 day ago [-]

It would be interesting if this violates rights of the website owner the user is visiting. I known that embedding content of other websites into your own via an iframe can be a copyright violation. And what Meta does here is more or less like an iframe.

kevin_thibedeau(10000) about 18 hours ago [-]

Copyright violations stem from distributing an unauthorized copy of a protected work. Modifying an authorized copy shown to one recipient can't be a violation.

User agents are expected to be empowered to transform the data they receive to suit the rendering requirements of the end user. Having a third party perform part of the transformation by supplying supplementary code executed by the user agent doesn't change anything.

AtNightWeCode(10000) 1 day ago [-]

I believe so. Copyright and TOS of the sites. Copyright also in the sense that content have been changed. This should be on pair with banner swap techs.

upupandup(10000) 1 day ago [-]

It isn't. By that logic any browser is liable for violating ToS of websites, which btw isn't the law and you are not obligated to follow anyhow.

This of course is a different case for corporations with a dedicated legal team.

bacan(10000) 1 day ago [-]

In-App browsers have always been a security nightmare. Similar issues exist with Electron apps as well.

But developers continue to use them as HTML + CSS + JS is the easiest way to develop a graphical dynamic UI, for a newbie. Many schools & colleges even teach basic HTML, CSS & JS, so the barrier to entry is very low.

I am not sure what a good solution here would be, but maybe we could start by limiting access. Or another way could be to have some way to convert the rendered UI to compiled binary code

xfitm3(10000) about 24 hours ago [-]

Good call out on Electron apps, I try to avoid Electron as much as possible. I use Slack's web interface for example.

I never made the connection until you brought it up, but yes, Electron apps are just like using Webkit on iOS. Abstracting UI/UX to a browser engine which has identical security pitfalls to a browser but with far less control and inspection capability.

chadlavi(10000) 1 day ago [-]

It's really concerning that everyone treats their children like prisoners. Your kids are gonna find a way to look at what they want anyway, why make it MORE appealing to them by making it verboten? Are you protecting them or are you controlling them?

twodave(10000) 1 day ago [-]

Some kids (and adults) literally aren't capable of impulse control. It's actually nice to be able to hand that control over to somebody else in some cases. And, as a parent if I know my child struggles with this it would be negligent of me to let them harm themselves knowing they can't stop themselves. I have four children and if there is any generalization about raising kids that I have learned it is that each child has different needs.

notatoad(10000) 1 day ago [-]

>I've disclosed this issue with Meta through their Bug Bounty Program

lol. and this is why companies can be hesitant to run bug bounty programs. it's not a place to complain about things you don't like. Meta/instagram has made a design decision here. just because you don't like it, doesn't mean it's a vulnerability.

lrvick(10000) 1 day ago [-]

Remember this is the same company that just gave police DMs that aided in an abortion investigation. If those had been end to end encrypted that risk would not have existed, but they made a business decision to leave the application vulnerable to spying for profit reasons. That is a vulnerability, in the same way we call it a vulnerability when an entity man-in-the-middles a browser to spy on people.

Personal user browsing or communications leaking in plain text to private companies without explicit and obvious user consent puts users at risk, and is a vulnerability. It just so happens to be one arising from malicious profit seeking behavior that happens to be the status quo.

Not having https was once the status quo, and a boon for corporate spying, but we call that a vulnerability now because the abuses became too big too ignore.

samstave(10000) 1 day ago [-]

>just because you don't like it, doesn't mean it's a vulnerability.

Technical Vuln or Business Vuln?

vade(10000) 1 day ago [-]

It is a vulnerability. You the user are just vulnerable to them...

sleepyhead(10000) 1 day ago [-]

It should be reported as a vulnerability. To Apple. Yes they made a decision for this as well but a decision can still be reported as a vulnerability.

riazrizvi(10000) 1 day ago [-]

Only Instagram? Every app maker who makes an in app browser can see what you do, that's the point of embedding it. And why not? You arrive there because of a link in the container app you clicked on. They want to see what you do with the link they gave you. Otherwise only Google/Apple can see what you do with it. Someone can see what you do with the link no matter what.

mwexler(10000) about 9 hours ago [-]

There is a UX benefit I've found, and maybe the app folks felt it as well.

Android has a nice universal 'back' button which can close an opened tab and return you the app that spawned it. iOS tried a similar option with a 'go back to App' link at the top left, but it doesn't fully close out browser tabs spawned by the app. Safari has gotten smarter about this, with browser 'back' buttons closing spawned tabs in many cases, but not all.

So, in all those emails where I click to launch a browser to 'read more' but see no reason to keep it around, the embedded browser actually plays a maintenance role: no need to get rid of those tabs that were just a funny comment I wanted to see but never need to keep.

Yes, the tracking is disappointing, but the UX of not having a zillion tabs open (and in old iOS version, hitting tab limits) makes my life easier.

ikurei(10000) about 12 hours ago [-]

> They want to see what you do with the link they gave you.

Except they didn't give it to me, my partner/sibling/parent/friend did, in a message sent through their app. We should expect that kind of interaction, to not be tracked; should expect, but obviously can't expect it today, these news are not surprising.

This is just one more aspect of their tracking that may be non-obvious, and that it's good to know about.

altairprime(10000) 1 day ago [-]

Not necessarily. If they're using a WKWebView, they can't see what you're doing, which is why Safari Autofill remains enabled in some but not all app-embedded web views.

Tepix(10000) about 14 hours ago [-]

The problem is that users cannot tell if they are being spied upon or not as long as they use in-app browsers. Avoid them! Go web apps instead of native apps for better privacy in general.

Regardless, i don't consider it to be acceptable behaviour by Instagram to inject their tracking into all web pages i access through their app just because they can.

senttoschool(10000) 1 day ago [-]

No wonder. I recently opened a link on Instagram and the website's responsive elements were completely broken. Then I opened the link in Safari and it worked fine.

Does this script injection break Apple's ToS?

I thought Apple required Safari/Webkit for all in-app browsers?

Zuckerberg has no shame.

PS. I hate in-app browsers. They don't sync with my main browser states such as authenticated sessions.

yieldcrv(10000) 1 day ago [-]

> Does this break Apple's ToS? I thought Apple required Safari/Webkit for all in-app browsers?

Doesn't apply to special companies.

ffpip(10000) 1 day ago [-]

> Does this break Apple's ToS? I thought Apple required Safari/Webkit for all in-app browsers?

They are still using Safari/Webkit, but just injecting a script into every page.

navanchauhan(10000) 1 day ago [-]

It probably is still running Webkit underneath with some additional JavaScript to track everything

kube-system(10000) 1 day ago [-]

> PS. I hate in-app browsers. They don't sync with my main browser states such as authenticated sessions.

Seems like that's probably a good thing :)

mh-(10000) 1 day ago [-]

> They don't sync with my main browser states such as authenticated sessions.

And this is exactly why Apple gives them their own cookie jar. The alternative would be [more of] a security nightmare.

samstave(10000) 1 day ago [-]

>They don't sync with my main browser states such as authenticated sessions.

Under what circums do you want this?

dzikimarian(10000) 1 day ago [-]

I'm surprised that so many people write 'yeah, any in-app browser can do that - nothing to see here'.

Anyone can potentially steal your wallet, so we shouldn't point out when someone actually does? Especially when there's hard evidence in article?

MomoXenosaga(10000) about 23 hours ago [-]

Every app and or website steals your wallet and when you point this out to people in tech who code this shit they respond with 'just following orders'.

Installed system wide ad blockers ages ago and got on with my life.

benbristow(10000) 1 day ago [-]

One thing I've noticed is that content-blockers/adblock don't seem to work within the Facebook/Instagram etc. in-app browsers so I usually end up jumping out of them anyway.

vuln(10000) 1 day ago [-]

Yup same. I jump out as soon as it attempts to load and I have the ability.

saagarjha(10000) 1 day ago [-]

Content blockers only work in Safari and SFSafariViewController.

thallium205(10000) about 22 hours ago [-]

The worst is when the link is YouTube and I have to watch an ad even though I pay for YouTube Premium but because it launched in an app browser I'm not logged in.

graham1776(10000) 1 day ago [-]

I've meant to write a blog post about this, but here goes: In-app browsers allow users to view inappropriate content, often against the wishes of sensitive individuals. People especially at risk for this include addicts and children.

Nearly every app, even 'safe apps' including children-rated apps, allow access to an in app browser. Even when iOs has locked down all access to Safari, a parent has removed access to all the 'apparent' unsafe sites, there are still ways to access the unfiltered internet inside of these safe apps.

How? Usually buried in App Settings. Almost all apps use some instance of an in-app browser to (lazily) reference thier privacy policies, EULAs, or TOCs. A buried link leads to a homepage, leads to an instagram link, leads to an unfiltered internet. Yes they are long, inefficient paths to reach the internet, but curious (or motivated) individuals or children will use almost any app to reach the internet. Even boring apps like MS Teams or adding a Gmail account to iOS mail uses a secret in-app browser.

This obviously presents a problem: should developers restrict any and all app access to in-app browsers, or leave policing to individuals/parents? An easy approach is to disable the in-app browser functionality in iOs, but obviously with grave cost to developers. At the same time, at what cost is in-app browser functionality being implemented.

davet91(10000) 1 day ago [-]

The in-app browsers could use a domain whitelist if parental controls are turned on.

polote(10000) 1 day ago [-]

A feature doesn't become a problem because 1% have an issue with it (people who use parental control).

The internet is the internet if you want to restrict what people can see on the internet the only solution is to not have access to it at all

nodamage(10000) about 22 hours ago [-]

> Nearly every app, even 'safe apps' including children-rated apps, allow access to an in app browser. Even when iOs has locked down all access to Safari, a parent has removed access to all the 'apparent' unsafe sites, there are still ways to access the unfiltered internet inside of these safe apps.

Last time I checked, WKWebView will follow the parental control settings set on the device.

rahkiin(10000) 1 day ago [-]

It is interesting how this would apply for custom browser engines in the future of iOS.

xfitm3(10000) about 24 hours ago [-]

Doesn't the harm of surveillance outweigh the harm of viewing 'inappropriate content'?

Think of the addict is a new one, but I am automatically suspicious any time someone cites child protection.

j2bax(10000) 1 day ago [-]

Why don't you just make sure there are no unsavory links on whatever page you are using the in-app browser for and disable/hide the address bar so they can't just jump onto the open web? Seems like you can have your cake and eat it!

registeredcorn(10000) about 22 hours ago [-]

Interesting! This reminds me of the classic Windows 95 bypass. You abuse the help screen to gain access to the desktop without having to login.[1]

I'm currently going through HTB Academy and once you mentioned unsecured in-app browsers, the first thing I thought of was either a Web Shell[2], or better yet, directing the in-app browser to a malicious website to download additional software to better exploit the phone. If the in-app browsers aren't filtering explicit content, I have to assume they aren't filter malicious content either.

If this isn't already a well-known route of exploitation, I'm interested to see how that might change in the near future. It sounds surprisingly easy to exploit, provided you can get momentary physical (remote?) access to the phone for a short time.

[1] https://www.youtube.com/watch?v=1UfNlRe-goY [2] https://en.wikipedia.org/wiki/Web_shell

CodeSgt(10000) 1 day ago [-]

I'm glad to see someone mention addicts. I feel as if internet addiction, and especially subsets of it such as porn addiction, aren't given enough weight by either the addiction treatment community or the technical community.

Before someone accuses me of being a conservative religious zealot as tends to happen when anyone denounces porn, I'll say that I'm far from a puritan and am extremely liberal in my social views. That said, I firmly believe that easy access porn is one of the worst things happening to the young men and women today. I (23) know many men around my age who suffer from chronic porn addictions to the point that it severely impacts their ability for form real relationships and median age of first exposure is getting lower and lower.

It's an absolutely crucial issue that no one seems to be talking about or taking seriously.

michannne(10000) 1 day ago [-]

We used to exploit these types of paths when school IT admins didn't know how to filter traffic properly but knew to block proxies.

t8ty2evj(10000) 1 day ago [-]

This seems like a non-issue. Where's the damage? I'm tired of people using children and a miniscule population of users w/ severe content sensitivities as excuses to justify features that are really just tools for asserting norms. The children are fine. We've been talking about how bad the internet is for children so long that those children grew up, led fufilling lives, had their own children, and now those children are apparently being ruined by the internet. What children need isn't more protection it's an escape hatch from all the forces trying to manipulate them during their most vulnerable years.

qwertox(10000) 1 day ago [-]

I think on Android they could use Chrome Custom Tabs [0] instead of WebViews. IIRC this also protects the browser content from being accessed by the hosting app, but there is still a limited communication which is possible between the app and the tab.

[0] https://developer.chrome.com/docs/android/custom-tabs/

BolexNOLA(10000) about 22 hours ago [-]

> Yes they are long, inefficient paths to reach the internet, but curious (or motivated) individuals or children will use almost any app to reach the internet.

I don't think this can be overstated. How many people tell you stories of watching signal-scrambled porn on TV when their parents are asleep? How many of us waited until our parents are asleep to play video game late at night? How many millions covertly downloaded Napster/Kazaa/etc. and downloaded 30 versions of a song before they finally got the one they wanted?

Being "motivated" as a kid or a teen is a low bar.

smoldesu(10000) 1 day ago [-]

Or maybe... just don't give your kids an iPhone?

Seriously, using the internet/computers should be treated with the same level of caution as grown-up scissors or fillet knives; powerful tools, but they need training to avoid hurting yourself with them. If this is what you're worried about, why are you even giving them a small computer in the first place? Your kids will always be more cunning than your security policy (a hard pill to swallow for HN users), so control their access to technology unless you're ready to have a serious sit-down discussion about the internet, personal privacy, and all that jazz. Put yourself in their shoes; if you're given a small black brick with an indeterminate number of capabilities, wouldn't your response be pushing it as far as it can go? I know that was my reaction when I was a kid, after buying a Pentium desktop at a garage sale.

Fogest(10000) 1 day ago [-]

I have a browser based game I play that makes use of many userscripts and browser extensions to further improve/enhance the game. However mobile users suffer from a problem of not having such extras. There is a very nice app someone made on Android and iOS that uses in-app browsers in order to be able to add a lot of custom things.

There are many useful instances for the in-app browsers and I don't think they should be removed because of some bad actors. It's similar to how Android has had password managers making use of autofill tools via accessibility tools. Android was butchering that access, but luckily started adding some official autofill support.

I don't think removing capabilities in the favour of 'safety' is usually the right approach in my opinion.

franga2000(10000) 1 day ago [-]

If someone is knowledgeable and committed enough to dig through all their apps, find any in-app browsers and try to break out onto the web, they will also realize that simply using another device will bypass all your silly blocks.

wepple(10000) 1 day ago [-]

Tangential, but these same links have always been a great way to break out of poorly designed kiosk systems.

I recall noodling with a huge interactive display on the side of a bus stop that had an embedded map, and surely enough the TOS link launched a browser, and from there you could use the Save As dialog to get to anything to execute

O__________O(10000) about 24 hours ago [-]

Reminds me of stories I have heard about users of computer systems with "strong" access controls figuring out ways to make it to unfiltered internet; examples include: student/prisoner computer labs, public libraries, flight entertainment systems, public kiosks, operating system logins, etc.

amenghra(10000) 1 day ago [-]

In the early 1990s, we used to break out of Macintosh's AtEase at our middle school by writing a two line MacBasic program which launched Finder. We would then bring games on floppies. Everything old is new again!

kart23(10000) 1 day ago [-]

surprised this is at the top of HN. isn't it obvious that every app does this? tiktok, snapchat, even linkedin all open links in their built-in browser and can track what you're doing. click open in safari if you're doing anything more than visiting a single page.

SnowHill9902(10000) 1 day ago [-]

It's not obvious but it is reasonable.

joshstrange(10000) 1 day ago [-]

I was/am a little surprised since I thought everyone had to use SFSafariViewController for stuff like this (which doesn't allow the developer to reach in). I 'eject' out to Safari almost always when I get in in-app-browser (if only for cookies/logged in status) so this doesn't affect me much but it did come as a surprise.

InCityDreams(10000) 1 day ago [-]

>isn't it obvious that every app does this?

Not if you never have/ don't use them.

M4v3R(10000) 1 day ago [-]

Not every app does this. Twitter for example doesn't, because it uses SFSafariViewController which doesn't allow for script injecting.

stevage(10000) about 24 hours ago [-]

As a non mobile developer, no, this was completely surprising to me.

webercoder(10000) 1 day ago [-]

I naively assumed that they were using a WebView object and that Apple had tight controls over source code injection. Silly me!

madeofpalk(10000) 1 day ago [-]

It's not surprising, but it's not obvious.

davidmurdoch(10000) about 21 hours ago [-]

It's so obvious to me I'm flabbergasted others here on HN don't think so.

In app browsers that display content unrelated to the app itself (like links from creators) serve zero purpose to the user and offer a horrible user experience. So why is the browser in-app then? I thought the answer was obvious: to track you.

Maybe some people prefer in-app browsers?

plif(10000) 1 day ago [-]

Yep, this is a feature, not just for tracking but also containment when navigating to external links. Big reason why all of those apps and others aggressively push users from web to mobile.

somerando7(10000) 1 day ago [-]

To me it's not obvious. I wouldn't think that an app can inject JS into a website because I'm using a web-browser from their app.

sixothree(10000) 1 day ago [-]

Also why is the headline 'Instagram _can_ track anything you do on any website in their in-app browser'?

spoonjim(10000) 1 day ago [-]

Every app that uses an in-app browser (which is most of them) can do this. This is a clickbait headline that relies on 'Blue Company Bad' sentiments.

bhelkey(10000) about 23 hours ago [-]

> The [iOS] Instagram app injects their tracking code into every website shown, including when clicking on ads, enabling them monitor all user interactions

The allegation isn't that Instagram can do this but that they are.

mirkodrummer(10000) 1 day ago [-]

I'm not surprised and it's really annoying apps still use in-app browsers. I remember even Telegram had that at one point, with link opening only on in-app browser(at least on iOS). But what really annoys me is that most of the users, e.g. my girlfriend, have NO IDEA about the difference, it's just a browsing window, no matter in-app, which engine, with which privacy feature. Perhaps os vendors should show more obvious UI, and UX wise, tell you you're leaving a safe browsing experience?

makoto12(10000) about 13 hours ago [-]

in-app browsers are a much better UX imo, solution is to make them safe, not get rid of them, as they solve a real ux use case

pphysch(10000) 1 day ago [-]

Isn't this the main reason why social media pushes their apps over their (once) perfectly functional websites?

Better analytics = better product*.

* for the true customers, i.e. marketing & communication firms, governments, etc.

l33t2328(10000) 1 day ago [-]

Why do you say "true customers"?

Is anyone under the impression that they are a customer of a service they don't pay for?

People would readily identify as a "Twitter user" instead of a "Twitter customer"

scraplab(10000) 1 day ago [-]

As a provider is it possible to defend against this with a Content Security Policy or does this mechanism override the site's CSP?

the_mitsuhiko(10000) 1 day ago [-]

External sources yes, preventing an app to inject inline HTML and JavaScript is tricky.

RKearney(10000) about 23 hours ago [-]

Not only would CSP block it, but this type of behavior only strengthens Apple's decision to not allow third party rendering engines. Could you imagine the privacy nightmare that would ensue if Facebook could release a browser that bypassed any and all safeguards implemented by site operators?

xfitm3(10000) about 24 hours ago [-]

Strides have been made in web security, check out the permissions policy[0] along with COOP and COEP[1].

[0] https://www.w3.org/TR/permissions-policy-1/ [1] https://scotthelme.co.uk/enabling-coop-and-coep-reports-on-r...

robocat(10000) 1 day ago [-]

MDN docs for Content Security Policy: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP (for anyone unfamiliar with that browser feature that should in theory disallow injection for websites you control).

eis(10000) 1 day ago [-]

They not only track very invasively what you are doing but they create real problems for websites because certain features wont work anymore. Be it due to them disabling them or third party services having to block their usage because of the huge privacy and data safety issue.

Simple example: try to use 'Login with Google' from within one of those in-app browsers and you will notice Google had to actively detect them and block the attempt because otherwise the app could spy on the login credentials without anyone noticing.

Instagram, Twitter, Facebook Messenger, TikTok.... the list goes on and on.

I am very confident that these companies are breaking GDPR laws left and right on an absolutely massive scale. They are spyware at this point.

klabb3(10000) 1 day ago [-]

I'm with you. If you're navigating to a website, you should open the web browser. The app shouldn't monitor or inject crap when I'm going to an unaffiliated site. The worst offenders are the ones that force you to enter credentials for a 3p account with an in-app browser.

That said, it's a huge UX failure that navigating between the web and an app is so broken. That doesn't mean that it's motivated to break the fundamental models of the web. Long term it does much more harm than good. How do you teach non-technical users good practices if developers circumvent these barriers anyway? 'Trust us, we won't steal your Google account' is not exactly reassuring, but ok say that you trust a reputable app to do that. What happens when the user normalizes this behavior and a less reputable app does the same thing? Obviously many users will have no idea of the risk.

AtNightWeCode(10000) 1 day ago [-]

I believe this is not legal. It is a grey area for users to do things like this but for a browser to change the actual contents is illegal on most sites. Or at least, there is no general way for a browser to validate if it is legal or not.

AtNightWeCode(10000) 1 day ago [-]

HN should really get rid of the down votes... Please explain why you think it is legal for a proxy to inject custom scripts. I am sure our TOS states that this is not allowed. Also, I think it basically is a copyright infringement.

nemothekid(10000) 1 day ago [-]

I also noticed TikTok does this as well; at the very least they are snooping inside their in-app browser to prevent you from visiting adult sites.

Flimm(10000) 1 day ago [-]

The article isn't complaining about in-app browsers per se, but that Instagram implements a special version of an in-app browser that injects Javascript code to track user behaviour. If you have noticed TikTok doing the same thing, please publish a blog post about it, and I expect it would get attention here on Hacker News, at least.

zahma(10000) 1 day ago [-]

Any reason why Google Maps wouldn't* use the same in-app tracking?

Edit: meant why Google wouldn't do this. I guess what I really mean, is what are the chances they don't do this?

smitty1110(10000) 1 day ago [-]

Yeah, tracking your behavior. If you searched for a bar, did you look at other bars? Parking? What other things did you look at? All of this could potentially be used for segmentation.

rawling(10000) about 24 hours ago [-]

Yes, Google Maps probably tracks your usage of Google Maps. But when you click through to a location's website, it doesn't open that in a local webview and track how you use their website.

Whether Chrome tracks how you use it...

wonderbore(10000) 1 day ago [-]

Please tell every newspaper to publish this so Apple puts a stop to this. I have no idea why they allow this. All apps should use Safari unless they're a browser and this rated "18+"

TaylorAlexander(10000) 1 day ago [-]

Well I like when browsing reddit that when I open links they are sandboxed. The in-app browser in that case has an easy button to open to get to my normal safari if I want to.

tiku(10000) 1 day ago [-]

I still remember the LinkedIn app ripping all my contacts, so no apps for me. I just use the sites.

AtNightWeCode(10000) 1 day ago [-]

I remember when the Twitter app asked if I wanted to sync the mobile contacts every time I opened the app. Thankfully Android has become better when it comes to this even if there are still flaws.

testfoobar(10000) 1 day ago [-]

Quite a few apps from the early mobile days did this.

jimbob45(10000) 1 day ago [-]

I just keep an old phone around for when I need to use apps (banking, especially). Can't steal the information off my device if there's nothing on there taps forehead

PenguinCoder(10000) 1 day ago [-]

> use the sites.

Which are increasingly user hostile, if not down right impossible to view on mobile. Go try using Reddit or Twitter on your Mobile browser.

onlyrealcuzzo(10000) 1 day ago [-]

This theoretically can't happen anymore, right?

You have to give apps permission to get your contacts, right?

cloudking(10000) 1 day ago [-]

They can also track anything you do outside their browser, on a website with their tracking pixel.

croes(10000) 1 day ago [-]

You can install blockers in your browser but not in in-app browsers

hashishen(10000) 1 day ago [-]

Firefox has built in tracking protection to prevent this iirc

ledauphin(10000) 1 day ago [-]

I can't imagine why anyone would expect otherwise. If you're still 'inside' an application, why wouldn't that app be able to track everything you do?

To completely hijack the discussion here, I believe that Apple is actually one of the strongest forces for anti-privacy in the world, because of their long-term, successful push for the convention of app > website (not fully supporting PWAs, disallowing web push, etc). A website may spy on you, but it can only do so in ways constrained by the browser, which has to serve many 'masters'. Mobile apps are completely unconstrained in their spying, and in-app browsers are just the logical extension of that pattern.

Thanks largely to Apple, we've conditioned ourselves to expecting that you can't have good mobile UX without a mobile-native application, and it's hard to imagine ever escaping back into the relatively open web now that we're this far down this path. Most people will never question the privacy implications of installing the Facebook app, and most of Apple's privacy-directed efforts on iOS are basically playing walled-garden whack-a-mole on problems that are better solved at a societal level with web browser standards.

Yes, it's quite likely that I'm scapegoating here, but it's the way I see it.

ezfe(10000) 1 day ago [-]

Apps that use Safari View Controller cannot view the page - of course Facebook doesn't use SVC for this reason.

While you're right that the Facebook/Instagram app can spy on links opened within the app, it can't plant cookies in your web browser - so those go both ways.

iamjk(10000) 1 day ago [-]

Isn't this... what everyone (that uses in-app browsers) does? I just assumed that's a big reason why one would use in-app over sending a person to their native environment, which is decidedly a better browsing experience.

yreg(10000) 1 day ago [-]

I think that in times when user just quickly checks some website the better UX is to stay in the app, so there would be legitimate use cases.

e.g. Apollo by iamthatis here on hn does this and I very much doubt he is doing it for tracking reasons.

solarkraft(10000) 1 day ago [-]

No shit! Instagram tracks what I do in the Instagram app!

eis(10000) 1 day ago [-]

You get a link inside Instagram to some website that does not belong to Instagram. It is none of Instagrams business what you do on that website. People do not even realise they are still inside Instagram while logging into their bank account and Instagram keeping a log of some of their activity inside that bank website. It's insane.

dazbradbury(10000) 1 day ago [-]

Websites need cookie notices, but apps can track your full web usage (albeit within the in-app browser) without any such notice or opt in? Doesn't seem like this would be legal. Anyone know how this could be compliant in the EU?

It's also frustrating that on an android device you can't simply disable in-app browsers globally.

flipbrad(10000) 1 day ago [-]

The EU+UK e-privacy 'cookie' rule applies to apps in the same way as anything else that's sending/receiving data over a public network (e.g. the Internet): all storage of information to, or reading of information from, the end-user device requires their free, informed and specific consent, unless it's a technical necessity for the service they requested, or certain limited (technical) purposes like load balancing. How strictly this is enforced by regulators has waxed and waned over time and from one country to another. Civil litigants, however, have had pretty good results in the courts (or just threatening litigation) - e.g. the Lloyd and Vidal-Hall cases against Google in the UK

fleddr(10000) 1 day ago [-]

I'm not familiar with the Instagram signup flow but it may very well be that the user did opt-in at one point. The opt-in would of course only be valid if there's also a clear 'reject' option.

Nextgrid(10000) about 12 hours ago [-]

GDPR enforcement is significantly lacking. There's basically zero ways this is compliant unless it's opt-in (but who in their right mind would opt-in?) but the regulators aren't interested in standing up to these nasty companies.

nodejsthrowaway(10000) 1 day ago [-]

Is this different from my android experience where I open a link from an app and it opens my default browser, Firefox, but kind-of within the app, but allows me to instantly switch over to the Firefox app instead using a drop-down menu option?

Flimm(10000) 1 day ago [-]

iOS provides a way of showing a browser that looks like it's within the app from which it is launched. This is not what Instagram is doing. Instagram is doing something different from what other apps like Telegram do, according to the article:

> Comparing this to what happens when using a normal browser, or in this case, Telegram, which uses the recommended SFSafariViewController:

> As you can see, a regular browser, or SFSafariViewController doesn't run any JS code. SFSafariViewController is a great way for app developers to show third party web content to the user, without them leaving your app, while still preserving the privacy and comfort for the user.

izacus(10000) 1 day ago [-]

Android has two ways of doing that - Chrome Custom Tabs which are secured against this (iirc) and WebView which isnt.

Custom Tabs always have a title bar and a small writing 'Powered by <browser>' at the end of the menu.

dilDDoS(10000) 1 day ago [-]

I generally don't see any appeal to in-app browsers in the first place. They often have extremely broken navigation controls (i.e. attempting to swipe back to a previous page usually just returns back to the app), block the ability to navigate to a specific URL, content blockers don't work, don't allow opening 'smart links' that would typically open in another app if opened from a normal browser, etc. From what I'm gathering from this article, it sounds like in-app browsing allows apps to give you all of the 'benefits' of being tracked (for their benefit only), with none of the (actual) benefits of using a real browser.

inlined(10000) 1 day ago [-]

The appeal of in-app browsers is that apps like Facebook can boost their "time in app" metrics while you read linked articles.

the_gipsy(10000) 1 day ago [-]

They lock users into the app. Every app and website tries hard to not let the user follow a link. Engagement.

darth_avocado(10000) 1 day ago [-]

I frankly am surprised why anyone would think otherwise? The "In-app" in the name should kind of give it away that it is, after all, in the app. Anything you do will be available for the app to track.

zippergz(10000) 1 day ago [-]

I'm sure this has gotten better as people have become more used to smartphones, but I worked on a popular app for a big company a number of years ago, and we would send people out to Safari to open links. The number of customer service calls we got from people who couldn't figure out how to get back to the app after that was ASTOUNDING. We eventually gave in and did an in-app browser. Not only did it get rid of that category of call, but it also noticeably helped our key metrics because fewer people were leaving the app to never come back again.

I realize that doesn't address the appeal FOR USERS, but it is why we did it as developers.

samtheprogram(10000) 1 day ago [-]

Ironically the whole point of it originally was sandboxing, and it's true at least on iOS. Thus, you won't be logged into the same sites within an in-app browser, and clicking a link from within an app (whether it appears to be an link or not) can't automatically connect you to cookies and any other tracking from your actual browser.

mrtksn(10000) 1 day ago [-]

On iOS this is traditionally done with UIWebView or WKWebView(like the former but better performance, runs as separate process) and you are right about the problems it creates.

However, the developers do have options to incorporate SFSafariViewController since iOS9.0 and that gives the user full Safari experience with Autofill and everything and without giving access to its contents to the app developer.

It actually makes a lot of sense from users perspective when the context is that the app temporary needs to take you to a webpage for something with the intention of you going back to the app. With SFSafariViewController this is done securely and with good user experience but unfortunately most apps business model revolves around tracking everything you do and as a result, most developers would use UIWebView/WKWebView instead of SFSafariViewController just to be able to track you.

The UIWebView/WKWebView has legitimate uses like letting you sign in from a web interface and transfer the session into the app but I kind of feel like we would be better off to depreciate it in favour of using alternative methods to do the web/app connection and improve privacy significantly.

Personally, I would never do anything sensitive from within a browser that is in an app. It looks like very obvious attack vector to me.

zionic(10000) 1 day ago [-]

> i.e. attempting to swipe back to a previous page usually just returns back to the app

Is there any way to turn that damn functionality off? I can't tell you how many times I've been navigating some newfangled web UI and had a swipe go "back".

That and disabling pinch to zoom backing out to the tabs UI. I wanna zoom out dammit. Is hitting a back or tab button really so hard that you have to break basic pan/zoom mechanics?!

I know I'm putting off "old man yells at cloud" vibes here, but come on

tolmasky(10000) 1 day ago [-]

It's even worse than that:

1. Nothing you visit gets saved in your history. So many times I'm looking through my history thinking 'I could have sworn I read an article about this...' only to eventually discover (if I'm lucky) that it was in Twitter's stupid in-app browser. But oh well, never going to find that article again! The irony of the APP knowing everything you visit but you never getting to remember what you visited.

2. All your logins are gone! I actually pay a bunch of stupid newspapers just to click on links in Twitter and STILL be told I can't read the article because of course I'm not logged-in in the in-app browser. UGH.

You could imagine a world where iOS tried to balance the desire of an app to not bounce you out with a more 'integrated experience' by providing an 'in-app' browser that was completely controlled by the OS, modifying your history, keeping you logged in, running out of process, and being able to be 'adopted' as a tab in Safari, but instead they just made 'SFSafariViewController' which does none of these things and instead just makes it really really easy for all apps to incorporate these infuriating in-app browsers.

sayrer(10000) 1 day ago [-]

Well, I'm sure there are 'growth hacker' types out there abusing the ability to observe browsing. But I think the real reason they don't bounce you to Safari, Chrome, etc is because users don't stay in the app if they do that.

I think all of the various bad things people talk about here must happen sometimes, but it's mostly just retention I'd guess.

stingrae(10000) 1 day ago [-]

My assumption is that it is a Product managers play to get people to stay in the app for longer. If you give people a link out of the app, then they are less likely to come back after.

You get a bump in engagement and time spent in the app at the cost of UX.

nerdponx(10000) 1 day ago [-]

There is no appeal for users and there never has been.

systemvoltage(10000) 1 day ago [-]

Instagram isn't doing it for the benefit of the user.

rconti(10000) about 24 hours ago [-]

The very first thing I do, every time, is click 'open in browser', just because, if nothing else, the framing of the site always feels 'off' to me when using one of those in-app browsers.

Historical Discussions: Man who built ISP instead of paying Comcast $50K expands to hundreds of homes (August 10, 2022: 1090 points)

(1102) Man who built ISP instead of paying Comcast $50K expands to hundreds of homes

1102 points 1 day ago by carride in 10000th position

arstechnica.com | Estimated reading time – 5 minutes | comments | anchor

Enlarge / A truck delivery of fiber conduit and other materials for Jared Mauch's broadband network.

Jared Mauch

Jared Mauch, the Michigan man who built a fiber-to-the-home Internet provider because he couldn't get good broadband service from AT&T or Comcast, is expanding with the help of $2.6 million in government money.

When we wrote about Mauch in January 2021, he was providing service to about 30 rural homes including his own with his ISP, Washtenaw Fiber Properties LLC. Mauch now has about 70 customers and will extend his network to nearly 600 more properties with money from the American Rescue Plan's Coronavirus State and Local Fiscal Recovery Funds, he told Ars in a phone interview in mid-July.

Enlarge / Fiber installed at one of the homes on Mauch's network.

The US government allocated Washtenaw County $71 million for a variety of infrastructure projects, and the county devoted a portion to broadband. The county conducted a broadband study before the pandemic to identify unserved locations, Mauch said. When the federal government money became available, the county issued a request for proposals (RFP) seeking contractors to wire up addresses 'that were known to be unserved or underserved based on the existing survey,' he said.

'They had this gap-filling RFP, and in my own wild stupidity or brilliance, I'm not sure which yet, I bid on the whole project [in my area] and managed to win through that competitive bidding process,' he said. Mauch's ISP is one of four selected by Washtenaw County to wire up different areas.

Mauch's network currently has about 14 miles of fiber, and he'll build another 38 miles to complete the government-funded project, he said. In this sparsely populated rural area, 'I have at least two homes where I have to build a half-mile to get to one house,' Mauch said, noting that it will cost 'over $30,000 for each of those homes to get served.'

$55 a month for 100Mbps with unlimited data

The contract between Mauch and the county was signed in May 2022 and requires him to extend his network to an estimated 417 addresses in Freedom, Lima, Lodi, and Scio townships. Mauch lives in Scio, which is next to Ann Arbor.

Although the contract just requires service to those 417 locations, Mauch explained that his new fiber routes would pass 596 potential customers. 'I'm building past some addresses that are covered by other [grant] programs, but I'll very likely be the first mover in building in those areas,' he said.


Under the contract terms, Mauch will provide 100Mbps symmetrical Internet with unlimited data for $55 a month and 1Gbps with unlimited data for $79 a month. Mauch said his installation fees are typically $199. Unlike many larger ISPs, Mauch provides simple bills that contain a single line item for Internet service and no extra fees.

Mauch also committed to participate in the Federal Communications Commission's Affordable Connectivity Program, which provides subsidies of $30 a month for households that meet income eligibility requirements.

The contract requires all project expenses to be incurred by the end of 2024, and for the project to be completed by the end of 2026. But Mauch aims for a much quicker timeline, telling Ars that his 'goal is to build about half of it by the end of this year and the other half by the end of 2023.' The exact funding amount is $2,618,958.03.

Comcast wanted $50K, AT&T offers just 1.5Mbps

Operating an ISP isn't Mauch's primary job, as he is still a network architect at Akamai. He started planning to build his own network about five years ago after being unable to get modern service from any of the major ISPs.

As we wrote last year, AT&T only offers DSL with download speeds up to 1.5Mbps at his home. He said Comcast once told him it would charge $50,000 to extend its cable network to his house—and that he would have gone with Comcast if they only wanted $10,000. Comcast demands those up-front fees for line extensions when customers are outside its network area, even if the rest of the neighborhood already has Comcast service.

Mauch was using a 50Mbps fixed wireless service before switching over to his own fiber network. In addition to his home Internet customers, Mauch told us he provides free 250Mbps service to a church that was previously having trouble with its Comcast service. Mauch said he also provides fiber backhaul to a couple of cell towers for a major mobile carrier.

All Comments: [-] | anchor

qwe----3(10000) 1 day ago [-]

> over $30,000 for each of those homes to get served

This doesn't seem very efficient to me.

rvnx(10000) 1 day ago [-]

To say the least, it's more about siphoning public taxes

sgerenser(10000) 1 day ago [-]

Yeah seems like some sort of mix of fiber and wireless for the 'last mile' would make more sense for installations like this.

toast0(10000) 1 day ago [-]

If utilities are underground, it can be pretty expensive to install anything. I have an estimate for municipal fiber that's about that much to get fiber a mile or two down the street overhead, and then about that much to go down my driveway underground 400 feet.

It's hard to justify when the local phone company is probably going to roll out fiber in the next few years without a direct charge, at least for the portion on the street. Of course, that'll probably be PPPoE, maybe asymetrical, likely limited to 1G, etc. Comcast won't even quote me to come down my driveway, even though they serve my neighbor across the street from the pole at the corner of my driveway.

rasz(10000) about 21 hours ago [-]

Its more than he personally was willing to pay ;-)

>Comcast once told him it would charge $50,000 to extend its cable network to his house—and that he would have gone with Comcast if they only wanted $10,000.

Im guessing being a nerd working at akamai he wont be the one spending ~1-2 days on a Ditch Witch/trencher to make those. He probably wont even hire anyone to work a rental from United Rentals. He will subcontract to same company that does trenches for Comcast.

inopinatus(10000) 1 day ago [-]

It isn't, but that's the norm for all internet infrastructure, both last-mile and backbone.

Since time immemorial, the gap between the amortized cost of building it, and anyone's willingness to pay for transport or transit, has been a) huge (that is, commercially insurmountable), and b) traditionally covered by one of two means:

1. Government subsidy, or

2. Attempting to offer services at the high prices necessary to recoup the investment, consequently going bust due to low volumes, selling the infrastructure for a pittance in a fire sale, and the next owner gets to offer services for prices the market is willing to tolerate. With this approach, it merely remains to find some VCs to sucker for the build phase.

It was also possible, back in the day, to run tunnels across your peers since they would announce the IXP networks at each end into their IGP, but folks got wise to that scam.

There is a variation on (2) involving anti-trust laws during M&A but it amounts to the same thing.

Vaslo(10000) 1 day ago [-]

Agreed - that much money could put in a computer lab in a local library for everyone to use. I'm very supportive of rural people and the life they choose to live, but you are right - they should understand the drawbacks.

burntsushi(10000) 1 day ago [-]

It's funny because he said one of the houses needed 0.5 miles of cable. My jaw dropped when he said it would only be $30K for that.

I'm speaking as someone who has had a few hundred foot trenches dug in my yard for running cable. Extrapolating it to 0.5 miles would come out to a lot more than $30K.

throwaway787544(10000) 1 day ago [-]

.....have you ever dug fiber in Michigan?

omvtam(10000) 1 day ago [-]

You can hang your fiber on existing infrastructure like electric distribution poles. edit: If you're the electric company.

adrr(10000) 1 day ago [-]

Friend of mine needed to run fiber across the street. They had to dig up the road. Cost was $50k. This was in a city where there aren't large pools of money from the government to get people decent Internet address.

fourthark(10000) 1 day ago [-]

At $55/mo, he'll start making a profit in 45 years.

bodfinch(10000) 1 day ago [-]

Same sentiment here. Maybe he could look into some WAN to CPE connections from the fibre terminations

jtap(10000) 1 day ago [-]

He presented at an online nanog event. You can watch it here https://www.youtube.com/watch?v=Twe6uTwOyJo I did enjoy listening.

kloch(10000) 1 day ago [-]

Jared has been participating in Nanog since forever. I have always looked up to him as a top-tier engineer.

slim(10000) about 24 hours ago [-]

is he visually impaired? I'm asking because he's presenting using slides including pictures

brentm(10000) 1 day ago [-]

This seems like a fun project to work on but what is the financial game here? Does he invest in building the network, operate at a loss and then sell to someone like Comcast? I assume building a remote fiber network that can reach 600 houses has to incur huge CapEx (way more than $2.6M right?) and at $50/mo a very long payback period.

However it works, pretty awesome project, kudos.

alexb_(10000) 1 day ago [-]

If I were really rich, I would spend a gigantic amount of money for the sole purpose of fucking over Comcast.

dominotw(10000) 1 day ago [-]

Its a misleading title. Govt 'built ISP', this guy led the effort.

wollsmoth(10000) 1 day ago [-]

looks like he's been able to find some deals on equipment and stuff since he operates on such a small scale. I guess he can just continue as a small business indefinitely if he gets enough cash flow.

failrate(10000) 1 day ago [-]

It is government subsidized. He just wanted good internet in the area.

pwinnski(10000) 1 day ago [-]

The initial investment is paid by subscribers or financed by the government grant, both mentioned in the linked article.

The monthly income of $55 or $79 times 70 people is $3850-5530/month gross right now, which is likely not a full-time income, but with potentially 600 more customers soon, it's possible he could achieve a full-time income for himself, which many people would consider a worthy goal.

In 1994 or 1995, I used an ISP in Sioux Falls, South Dakota, that was just one guy providing decent service. If there were issues, I'd call David and he'd fix them. His goal was to have good internet service--which was difficult to come by then and there--and to underwrite it by sharing it with others. I know he made a go of it for a number of years, although I'm not sure how it ended.

jbverschoor(10000) 1 day ago [-]

People in tech tend to forget that proper tech doesn't actually need 100s of engineers to keep it operational. That's the whole point of a computer. It does what you programmed it to do, and it does so automatically.

tryptophan(10000) 1 day ago [-]

Crazy idea, but why can't we just buy some armored cable and let it lie on the ground? People can bury it themselves if it really bothers them.

A lot of these people dont seem rich enough to justify caring about it being pretty...

throwaway0a5e(10000) 1 day ago [-]

Traditionally the solution is to have a tiny outbuilding with your electric meter, water valve (if you're on town water) and landline connection and then let the homeowner deal with the bulk of the length of the line run.

Getting electrical and water in those situations is always a town by town crap shoot because the trades are constantly lobbying to disallow it because they want more work. I assume ISPs are the same way.

cptcobalt(10000) 1 day ago [-]

I think it's acceptable to expect better. If we didn't, we'd probably have surface level sewer, water, fiber, cable, etc; all laying about, probably causing trip hazards. And these industries would probably lobby and set archaic and asinine rules for how the burial happens, and make you pay 10x the cost of what it really takes to use one of their approved contractors, because you're indulging in the luxury of having hidden basic-needs infrastructure.

dboreham(10000) 1 day ago [-]

There are many reasons why this isn't done and isn't a good idea. One of them is: animals will eat the cable. Another is: people will trip over the cable. Another is: eventually someone will dig the cable up with an excavator, even if the operator of the excavator is the same person who carefully laid the cable a few years earlier. I don't explain how I know that...

jleahy(10000) 1 day ago [-]

My wife did this, about 6 months of digging up roads in central London. Would recommend. AMA.

tinfever(10000) 1 day ago [-]

How did you meet your wife with an ASN? Asking for a friend...

KingFelix(10000) 1 day ago [-]

How long did it take to complete? Central London seems like high density, how many users do you have? Can you share website, I can forward to some central London folks!

bitcoinmoney(10000) 1 day ago [-]

What kind of research did you do to achieve this? Any workshops or did you talk to other ISPs to gain knowledge?

moritonal(10000) 1 day ago [-]

As a fellow Londerer please please expand on this. Like, why, what were the returns, what'd you peer into?

vlunkr(10000) 1 day ago [-]

Anyone else amused by the title? To me it reads as "Man [...] expands to hundreds of homes."

d23(10000) 1 day ago [-]

I suppose that's impressive too!

tikiman163(10000) 1 day ago [-]

I find it a little weird and off putting that thierprivate business is having its expansion funded by state funds for coronavirus recovery. I get that this is generally a good thing, and many ISPs, especially the smaller ones, receive government funds for developing and maintaining infrastructure. However, why is the Coronavirus recovery fund paying for this?

WorldMaker(10000) 1 day ago [-]

In this specific case there is an easy answer (mentioned in the article): Access to reasonably priced broadband internet was seen as one of the biggest, most easily addressable (with targeted government infrastructure funding) dividing lines between people that were able to easily work from home and those that experienced larger hardships during the height of the pandemic.

atentaten(10000) 1 day ago [-]

Is he connecting to a backbone or to another ISP?

woah(10000) 1 day ago [-]

The 'backbone' is made up of other ISPs

guywithahat(10000) 1 day ago [-]

Isn't this just called starting a business? Don't get me wrong it's very cool but this just seems like the thing people should do when there isn't enough competition in the market

treesknees(10000) 1 day ago [-]

He did start an LLC but it's not a business in the sense that he's hiring a corporate structure around it or kicking up VC funding, or even trying to make a profit. It's admirable because how many other ISPs can you point to with this model? I can't think of any.

shakezula(10000) 1 day ago [-]

Sure, at face value you're right about that, but I think the main difference is a lot of people don't get annoyed at , for example, Ford's customer service and turn around and start an auto manufacturer, and for most non-technical people I think they'd consider the two nearly equal in terms of feasibility and effort.

bluedino(10000) 1 day ago [-]

> Comcast once told him it would charge $50,000 to extend its cable network to his house—and that he would have gone with Comcast if they only wanted $10,000.

Starts his own company and finds out it costs $30,000 to do it.

You need big trucks, drills, excavating equipment, skilled union workers making good wages, safety concerns around water, gas, sewer, electrical and other communication lines, you can't mess up peoples lawns, you have to go out and maintain these systems after storms.

And people want this all for $55/month!

mschuster91(10000) 1 day ago [-]

As someone who actually was working in excavation for internet... well, some points to unpack here:

- You don't hire your own workers to dig trenches as an ISP, you sub-contract that stuff out to contractors - they can spread out the cost of, say, a backhoe not over the one year or two you need to build out a district's fiber, but over twenty years.

- Other underground stuff isn't much of an issue in rural areas - you have the central map register of the district which shows exactly where active lines are, and there aren't many. Usually it's the 10 kV/220V electricity line, water mains and the huge POTS cable. Sewers in most cases aren't much of a concern as they tend to be built very deep (here in Germany, minimum 100cm below ground level, and usually it's more like 2-3 meters). In rural areas you can usually get away with shooting a mole through the ground or a plough for a trench that a following tractor immediately closes after the pipe is laid in.

- That pipe or whatever you're building out underground can last literally for decades. POTS cable in many cases is over fifty years old, personally I have seen stuff that was covered in clay protection plates with swastikas meaning it was well over 70 years old. At 50 years, the life time earning of a connection is 33.000$.

- Governments usually subsidize the cost because broadband is an extremely net-positive investment. Assume a small village of 100 people gets broadband Internet uplink - now a small company moves into some farmer's shed because the rent is cheap and now pays tens of thousands a year in corporate and employment taxes.

dkhenry(10000) 1 day ago [-]

Its so expensive that Comcast only made a profit of 42 Billion in 2021, while providing a lower quality of service than what a small ISP in Michigan can give you for a one time 2M in government grants.

LatteLazy(10000) 1 day ago [-]

The correct price in cities is $10 a month. The correct price in rural areas is $500 a month plus. But we have to average them because we insist on taxing cities to subsidise rural lifestyles...

criddell(10000) 1 day ago [-]

> Starts his own company and finds out it costs $30,000 to do it.

There are two homes that are a half mile away from the others. The $30k number relates to those two properties.

the_optimist(10000) 1 day ago [-]

Average cost of ~ $40k per connection to the government. How is this better than Starlink?

bell-cot(10000) 1 day ago [-]

The area served is close to Ann Arbor, MI - so remember Starlink's 'satellites are in random-ish orbits around the Earth, not magically hovering over areas with more potential customers' issue.

It's possible that the county is trying to get tough with Comcast here - 'stop gouging our residents so badly, or we'll help a local competitor (to you) grow into a real thorn in your bottom line'. Starlink isn't credible for that.

And the money is from a 'State and Local Fiscal Recovery' fund that the county has access to - so spending it on Starlink would probably be a legal non-starter regardless.

supernova87a(10000) 1 day ago [-]

I greatly respect the initiative and scrappy-ness of someone doing this. And the legacy providers are clearly sitting on their monopoly position in a way that makes their pathetic alternative so starkly unattractive.

But isn't it also true that once his network grows above a certain customer base (and gets into the maintenance phase), he will start to see all the effects that eat into being able to do this cheaply?


-- customers who don't behave as well or kindly as before

-- customers who need 24 hour customer service

-- maintenance that can't be done himself, and he has to employ people

-- customers and vendors who sue you for breach of contract, or other simply nuisance lawsuits

-- upgrading the network to the next technology requirement, or when he's unable to get 2nd-hand parts so cheaply, etc.

-- or a natural disaster that unexpectedly forces replacement of (and charging for) equipment that wasn't anticipated in the original subscriber price

Maybe none of this rises to the level of making it fundamentally different or unsustainable? But it seems to me the honeymoon phase doesn't last long, and it's got to hit some unavoidable realities soon. At least, if you think you can replicate this, it requires finding people and neighbors who are willing to do actual work and investment/concern to make something like this possible, and not simply pay a vendor a premium to phone it in. It must be treated like a neighbor-to-neighbor community project, not a faceless commercial transaction with its attendant obligations.

kalleboo(10000) 1 day ago [-]

There are lots of ISPs that don't suck

dimitrios1(10000) 1 day ago [-]

> -- customers who don't behave as well or kindly as before

Easy. Refuse service. You aren't legally obligated to offer your service to assholes. Any business has the right to do or not do business with whoever they want, provided they're not refusing service for a reason that violates local, state, or federal law.

> -- customers who need 24 hour customer service

Also easy. You are under no obligation to meet peoples unrealistic demands or needs.

> -- maintenance that can't be done himself, and he has to employ people

He already is familiar with third party contracting.

> -- customers and vendors who sue you for breach of contract, or other simply nuisance lawsuits

Frivolous lawsuits are a risk in any business in America.

> -- upgrading the network to the next technology requirement, or when he's unable to get 2nd-hand parts so cheaply, etc.

What is this 'next technology requirement'? My area cable company still runs most their network on 30 year old lines.

> -- or a natural disaster that unexpectedly forces replacement of (and charging for) equipment that wasn't anticipated in the original subscriber price

Cost of doing business, doesn't matter the size.

I think people don't understand just how profitable municipal broadband can be. It's why big players spend so much lobbying and bribing so they can keep their established position running and keep the gravy train running, but really the economics of it are fantastic once you've done the initial digging and running the lines, which sounds like he has here.

At $55 /mo for 400 households he's bringing in $22,000 a month plus whatever federal and local government subsidies and grants. The odds of a disaster, or one of the other scenarios you mentioned happening anytime soon is low, so he will have runway to build a decent sized war-chest to be able to easily afford handling any of these scenarios with third party contractors. The more houses he brings on line, the better it gets.

pessimizer(10000) 1 day ago [-]

I'm going to skate past the fact that difficult customers and maintenance aren't why monopolies are expensive, in fact they're the things that are most amenable to economies of scale, so bigger gets cheaper.

The real question is: why does he have to get larger than the 600 homes in his nearby rural area, ever? Why does his goal have to be to defeat and replace Comcast rather than to supply internet service to his neighbors?

chriscappuccio(10000) 1 day ago [-]

With a fiber based service he would be getting very few calls

Rackedup(10000) about 20 hours ago [-]

Are you saying that Comcast provides decent customer service? because I think it is probably the first or second reason everyone hates them... another one could be the doubling cost yearly unless you call them and are serious about cancelling.

Where I'm at Comcast is very reliable but I've had different experiences.

wmf(10000) 1 day ago [-]

Hopefully with the government funding he can turn it into a real business.

connorlads(10000) 1 day ago [-]

Not sure how Canada compares but these concerns haven't stopped the biggest telecoms in Canada from providing subpar service under very restrictive terms and conditions with no accountability. Namely, a 12 hour complete outage by Rogers to which the reply was basically a big shrug. If they can get away with that I am sure a small independant provider can get away with that as well.

margarina72(10000) about 21 hours ago [-]

Not everything need to scale. A good way to handle this kind of project is keep it at a certain community size, and if people want in, beyond a certain threshold, they need to build their own. This is how federated internet providers work usually.

kevin_nisbet(10000) 1 day ago [-]

I'm not convinced this is the case. The big thing that makes telco's such profit making machines is that wires in the ground are generally a large capital expense that doesn't really provide a great marketplace for competition. But once you've got that infrastructure, it's hard to duplicate. The rest of the equipment and employees relatively aren't that expensive.

So the power is on the provider here, there isn't really another choice for customers if the article is to be believed, no matter how good or bad the company is. Sure there might be disputes with vendors, but that's just part of any business.

The biggest threat IMO is probably some sort of competition. Maybe a big telco decides to wire up the area, although then they would be the second player in the market trying to steal customers who may not be interested in switching. Or if this really is a rural area, things like wireless last mile (basically LTE), Starlink, OneWeb, etc may start to be more compelling options if they get the capacity, latency, and price point to the right spot to be competitive.

colechristensen(10000) 1 day ago [-]

In Minneapolis there is a local fiber provider which charges about the same for the same level of fiber connectivity. I think it's pretty sustainable.

It looks like his revenue is going to be $50k/mo in not so long and that's more than enough to have a couple of people willing to work on an as-needed hourly rate and to cover whatever issues come up.

chiefalchemist(10000) about 23 hours ago [-]

I have to presume his marketing costs will be close to zero. On tge other hand, in my area (central NJ) both Comcast and Verizon spend a ton on marketing.

He'll also have zero churn. So that's got to help the bottomline.

Finally, I'm willing to bet it helps raise local home prices as those who had to have proper broadband were effectively excluded from that market. The point being, some homes will be able and will to pay more.

Certainly the future will be different, the comparison to traditional ISPs might not be reliable either.

bentobean(10000) 1 day ago [-]

I, too, greatly respect the scrappy-ness of this individual. Kudos to him for sticking it to Comcast. That said, I'm not wild about the notion of dropping $30K of our collective money on running fiber to a single home out in the country.

Octoth0rpe(10000) 1 day ago [-]

A couple of fun facts about this guy:

His little ISP is AS267, which is a SHOCKINGLY low number. That's like.. the ISP equiv of a 4 digit slashdot id, or owning something like sodapop.com.

He's also one of the authors of RFC 5575, which is a pretty big deal in the DDoS world.

notyourday(10000) 1 day ago [-]

Jared is not a rando who built an ISP. He is someone who forgot more about networking and running NSPs than most people know.

kloch(10000) 1 day ago [-]

I don't know (or care) about how he got that ASN but ARIN does occasionally recycle returned 3 or 4 digit ASN's, including very recently:

The first time ARIN did this with a lot of 4-digit ASN's was 2009 and was how Netflix was able to get AS2906.

There is also a market for reselling ASN's that aren't needed anymore: https://auctions.ipv4.global (filter by ASN)

upupandup(10000) 1 day ago [-]

can somebody ELI5? what is this code mean? what is RFC 5575?

hammock(10000) 1 day ago [-]

What is an ASN and what advantage is there to have a low number?

bad416f1f5a2(10000) 1 day ago [-]

I recognized his name from providing hosting for the outages.org list[0] – if you haven't subscribed, and you do anything operations at all, go hit the button now.

[0]: https://puck.nether.net/mailman/listinfo/outages

ajdude(10000) 1 day ago [-]

My university's is number 2; is there any significance to that?

samwhiteUK(10000) 1 day ago [-]

I'm going to put my hand up and say I have absolutely no idea how an ISP works. He runs cables to each house in the area... now where does the other end go?

andix(10000) 1 day ago [-]

I think you more or less just buy connections from bigger ISPs, so for example you get a 100 Gbps connection to one location and distribute it to your end users from there.

Most of the equipment you can buy, you can even get a lot of the needed things as a service. You just need to organize all those hardware and software things, and get the economic and legal part right too. And in the end it needs to tie together in a way, that your earnings are bigger then your expenses.

I think it's not so different to opening a car repair shop for example. Just more nerdy.

beezlebroxxxxxx(10000) 1 day ago [-]

There is a very good Ars Technica article on how an ISP works. It traces the whole network, from submarine cable through to last mile into a house. It was written in 2016, but I imagine it's still relevant:


Bloating(10000) 1 day ago [-]

There are wholesalers that provide 'dark fiber', then you buy data services from another 'wholesaler'. When I looked into it, dark fiber was available through some utilities and through a government funded non-profit. Data to light-up the fiber was available through several different data centers that connected to that dark fiber.

You still had to build-out the last mile though, and thats what will get you. You either need private easements, or be a registered telecom utility to use public utility easements. That last mile is $20k +/-, depending on your circumstances. If your semi-rural or less, there's ROI sucks. Hence, many smaller ISPs are wireless.

At least in area, there are already a number of wISPs, 5G is rolling out, Starlink eventually. and lots of gov't funding going to the big players to expand their networks (and drive the start-ups out of business.)

There some other business models out there too that look interesting. Underline in Co Springs, for example. They provide a basic tier of service, in order to qualify as a telecom, install the fiber and then allow multiple competing ISPs to use their network.

IMHO, any utility that has the benefit of government privilege should be required to allow competors to use the infrastructure that the taxpayers funded.

I'm waiting on one of you brilliant folks to defy the laws of physics to create a decentralized, wireless mesh internet.

southerntofu(10000) 1 day ago [-]

As the other commenters have pointed out, a possibility is simply to 'resell' transit from other providers. However, on the Internet all peering networks are somewhat equal and it's entirely possible to extend the 'other end' over time to establish dedicated peering with other networks, so that for example traffic from your network to Youtube doesn't have to go through (paid-for) 3rd parties.

There's good chances there are Internet eXchange Points around where you live where for a small maintenance fee anyone can come and place their router and cables to interconnect with others.

So the likely steps are:

1) Find a transit provider, that will serve your trafic to any other network, and where to connect with this provider 2) (Optional) If you don't have the necessary infrastructure, find another provider to get from your last-mile network to your transit provider 3) (Optional) Find other networks to peer with so that you can significantly reduce your transit bill and provide better routes (therefore better service)

Some non-profit ISPs take the problem from the other side, and build a core network without necessarily owning any last-mile infrastructure, which is leased from other operators (opérateurs de collecte) with whom they interconnect at some datacenter/IXP. The most famous example of that in France is FDN.fr which has been operating since early 90s. That approach is more cost-effective in high-density area where the local infrastructure is already quite good, and construction jobs to lay new cables is very costly, but will still set you back 10-30€/month/line.

the_only_law(10000) 1 day ago [-]

Not sure if it's what the person in question did, but there's a whole guide that pops up on here occasionally regarding building a wireless ISP.


wil421(10000) 1 day ago [-]

Depending on how close they are he could run cables (ethernet) or fiber. Single mode fiber can go 10km according to some Ubiquiti spec sheets I found on google. Ubiquiti also sells AirMax products that can do PTP or PTMP over the air, although some will be affected by rain. They could even rent space from a radio/cell tower. There are probably a decent amount of other products out there I am only familiar with Ubiquiti.

dbelson(10000) about 22 hours ago [-]

Also very much worth reading on this topic: Tubes by Andrew Blum (https://www.andrewblum.net/tubes-2)

boplicity(10000) 1 day ago [-]

He's getting $2.6 million to set up access to 417 homes. That works out to $6,235 per home. At $55 per month, it would take 113 months, or over 9 years just to get $2.6 million in revenue.

Horrible economics! What a crazy business to be in. No wonder grants like this are necessary.

judge2020(10000) 1 day ago [-]

This is how the ISPs work as well, typically 10 years is common ROI for any neighborhood and 5-10 years for multi-family housing (apartment) runs. This is also the reason AT&T/Comcast won't run new installations to small (less than 40 residents in my experience) or rural neighborhoods since the ROI time gets longer the fewer potential customers they have.

pphysch(10000) 1 day ago [-]

Has a road or water line ever paid for itself?

FredPret(10000) 1 day ago [-]

It's a utility. Utilities have very stable revenues and very long payback periods. Nine years is pretty short in this context

cool_dude85(10000) 1 day ago [-]

Not that bad. A lot of utility-type businesses expect to have much longer return on investment times, the electric business is usually wanting to get 50 years of life out of a new baseload generating unit, and it might be 30 or 40 to get your investment back.

jrajav(10000) 1 day ago [-]

So taxpayer dollars are necessary to make this business viable, and the product of that business is something that, realistically, everyone absolutely needs access to - certainly seems like this should not be a private business at all but a public utility. Have we ever asked this kind of question for interstate highways?

capableweb(10000) 1 day ago [-]

The actual price they are offering seems to be $55 or $79/month + ~$200 installation fee. Also missing in your calculation, is a $30/month subsidy from FCCs 'Affordable Connectivity Program'.

I didn't make the calculation myself, but a sub-10 year horizon for a project someone seems to do from the goodness of their heart, doesn't seem so bad.

andrewallbright(10000) 1 day ago [-]

...And they say 10x engineers are a myth.

intelVISA(10000) 1 day ago [-]

It's a coping mechanism like lying on the couch watching the Olympics and getting angry that some people are able to push themselves to incredible feats instead of being happy for them.

Never understood that mindset, when I see 100x engineering feats like TempleOS or αcτμαlly pδrταblε εxεcμταblεs it inspires me to learn more and think outside the box.

banannaise(10000) 1 day ago [-]

10x engineers are a myth when it comes to productivity working within a team. There are absolutely 10x engineers when they're working on a project more or less completely solo.

NaturalPhallacy(10000) about 23 hours ago [-]

I've long felt that there's a relatively simple formula for productivity:

Productivity = (Time * Effort)^Talent

People like Buckminster Fuller come to mind. Especially because of this quote of his:

>"We should do away with the absolutely specious notion that everybody has to earn a living. It is a fact today that one in ten thousand of us can make a technological breakthrough capable of supporting all the rest. The youth of today are absolutely right in recognizing this nonsense of earning a living. We keep inventing jobs because of this false idea that everybody has to be employed at some kind of drudgery because, according to Malthusian Darwinian theory he must justify his right to exist. So we have inspectors of inspectors and people making instruments for inspectors to inspect inspectors. The true business of people should be to go back to school and think about whatever it was they were thinking about before somebody came along and told them they had to earn a living."

vaidhy(10000) 1 day ago [-]

There are extremely competent programmers (10x) like there are outstanding players in sports and music. They do have an outsized impact on the projects they work on. However, they are also extremely rare. The problem, IMHO, comes from cult-startups where they think they can (a) identify these people in an interview (b) build a team of only 10x programmers.

This results in (c) calling a whole lot of average programmers they hired as 10x programmers because of (a). After all, they are smart and their interview process is infallible.

So, if you meet one of those rare folks, enjoy the intellectual banter :).

mi_lk(10000) 1 day ago [-]

Whoever says that never met one and isn't one of them. It's so obvious once you see it

jononomo(10000) 1 day ago [-]

A+ comment. I've been hearing this idea that 'there is no such thing as a 10x engineer' for almost a decade now and from the very first moment I heard it I considered it one of the most definitively untrue ideas circulating in the tech industry. In fact, there are 100x engineers.

thrwyoilarticle(10000) 1 day ago [-]

If we get to expand the definition from a software engineer on a team to a business founder, do we also get to call the fiber optics 10X engineers? Is a truck driver delivering laptops a 10X engineer?

thankful69(10000) 1 day ago [-]

That also depends on the X, from my experience working at FAANGs, startups, etc... I have never seen a 10x engineer in good teams, I have only seen '10x engineers' on teams without great engineers. The comparison with sports and music is pretty silly, as those are environment where the winner(s) take all (there can only be one Billie Eillish (lol) even tho there are many singers who are better), engineering is often a team effort. In the other hand, the best engineers I have seen, just spend more time than anybody else working on a problem, and often are the ones who like to show off more, and very often lack the skills in other areas of life.

zzzeek(10000) 1 day ago [-]

> Jared Mauch is expanding with the help of $2.6 million in government money.

> Mauch told us he provides free 250Mbps service to a church that was previously having trouble with its Comcast service.

That's interesting, he's taking money from the government and giving free internet to a religious organization? Do all 'churches' get free internet or just the ones he prefers? Taxpayers are OK subsidizing a specific church based on one person's personal whim???

xupybd(10000) 1 day ago [-]

He picked a charity to help and this is your response?

He won a government contract to with specific deliverables. I'm not sure how he would have any responsibilities beyond those deliverables.

aisengard(10000) 1 day ago [-]

I know this is a troll, but I'll respond anyway. If someone can prove he's discriminating against institutions on the basis of religion, he can be sued. Whether he takes money from the government or not doesn't matter in the slightest.

bell-cot(10000) 1 day ago [-]

> user: zzzeek

> about: ...I am a strong proponent of sarcasm.

So - difficult to interpret this comment.

An atheist might reasonably do the same for churches in his service area, for P.R. and Marketing reasons.

How is this different from Bob - who (say) the township pays to mow the lawn & plow the parking lot at the township hall - deciding that he'll mow the lawn & plow the parking lot for free at some local church?

jquery(10000) 1 day ago [-]

Meanwhile I live in San Francisco and I still can't get affordable symmetric gigabit fiber internet to my home.

fragmede(10000) 1 day ago [-]

More frustrating is to chart how close you actually are to gigabyte symmetric. AT&T and Sonic has wired up large parts of the city but if they don't serve you, it's often by just a block or two, depending on where you are in the city. Rumor has it that local ISP MonkeyBrains is also getting in on the fiber game.

CliffStoll(10000) 1 day ago [-]

2 years ago, Sonic pulled fiber in my neighborhood in the East Bay. Gigabit is $65/month (including taxes/fees + 1 unused phone line). Very happy with Sonic!

Ancapistani(10000) about 11 hours ago [-]

That's nuts to me.

I live in a town of ~15k people in a Southern state, and I have symmetric gigabit. Granted, I pay ~$110/mo, but I have the option of symmetric 300Mbps for ~$50/mo. Neither plan has a data cap.

Then again, I chose my home based on the local ISPs' physical network topology. I didn't rely on their service maps, either - I physically went to their installation folks and got a copy of their maps.

notatoad(10000) 1 day ago [-]

>'I have at least two homes where I have to build a half-mile to get to one house,' Mauch said, noting that it will cost 'over $30,000 for each of those homes to get served.'

is this really a valuable use of taxpayer money? sending a wireless link over a half-mile isn't that difficult, surely there's a better way to spend $60k of public money than delivering internet service to two families. especially now that starlink exists.

i'm all in favour of scrappy upstart ISPs, but this just seems wasteful.

lsllc(10000) 1 day ago [-]

You can do that with 2 Ubiquiti Nanobeams 5AC gen2's for $130 each and get a ~650Mbps link (source, I've done this a number of times!).

a2tech(10000) 1 day ago [-]

Especially since he's burying the lede about the people he's servicing--its true 'in general' that the area is lower income, but most of the homes he's serving will be millionaires.

mrb(10000) 1 day ago [-]

'I have at least two homes where I have to build a half-mile to get to one house,' Mauch said, noting that it will cost 'over $30,000 for each of those homes to get served.'

That's over $11 per feet. That sounds about right. I paid $18 per feet to have a private fiber optic line of 1000 feet installed at one of my houses (in the US), going down a very long driveway, with 3 patch panels, 2 at each end and one in the middle at a gate. That was just for my LAN, not internet access. I needed the link to hook up intercoms and security cameras. I absolutely wanted 100% reliability of the network link, so wireless solutions wouldn't have been adequate. The previous homeowner had buried a cat5e line in the first 500 feet, with a cat5e repeater (underground), but its electronics failed after a couple years and its exact location couldn't be found. And he had not even put the cable in conduit.

zy0n911(10000) about 23 hours ago [-]

You mean 'foot' instead of 'feet' here surely?

system2(10000) about 22 hours ago [-]

> I paid $18 per feet to have a private fiber optic line of 1000 feet installed

Are you saying you paid $18,000 for fiber optic installation at your house?

H1Supreme(10000) 1 day ago [-]

> 1Gbps with unlimited data for $79 a month

Wow, sign me up. Comcast, which has a monopoly on my market, charges me a few bucks more per month, for 150mbps.

nodunutshere992(10000) 1 day ago [-]

Comcast charges $100/mo for 1Gbps where I'm at in a suburb of Salt Lake City. Our city announced a partnership with Google Fiber that will begin rolling out in 6-8 months. After that happened, I've started getting Comcast adverts to sign a 2 year contract...I also expect to see their prices start dropping soon.

capableweb(10000) 1 day ago [-]

The costs for internet in the US still surprises me, how on earth can it be so expensive?! I understand some countries, but in the US, it seems high costs are because 'because we can', not because it has to be like that.

In comparison, you get 1 Gbps symmetric fiber connection in most countries in Europe for under ~$30/month. In some, you even get it for under $10/month (like Romania, which has surprisingly awesome internet infrastructure).

jer0me(10000) 1 day ago [-]

1Gbps is $40/mo from Sonic in the Bay Area

Tsukiortu(10000) 1 day ago [-]

I only can use Windstream as the other providers are right on the edge of my area and refuse to move in. I only get "50Mbps" (It's never gone above 45) for $90+ a month, and they have been forever increasing it because well, what choice do we have.

colechristensen(10000) 1 day ago [-]


Come to Minneapolis. 1 Gbps for $70.

IE6(10000) 1 day ago [-]

> charges me a few bucks more per month, for 150mbps

And, in my experience, they will slowly ratchet up the cost until you call in and complain or change your plan, so a negotiated 80 dollars slowly can become 160+

mtnGoat(10000) 1 day ago [-]

I use a smaller ISP in Washington state and my 1G symmetrical line just went from $79 to $59 a month and they increased my upload, it used to not be symmetrical.

woah(10000) 1 day ago [-]

This is kind of an interesting illustration of how little people know about how the internet works, and how news is ultimately entertainment.

Full respect to the man in the article for the hard work and initiative he took in starting a small independent ISP, but this story is the story of thousands of small ISPs in the US and many more around the world.

In a basic sense, this story is not 'newsworthy' since there is nothing new about it. It's more of a human interest piece, like if the reporter wrote a story about the lady who started a coffee shop after being overcharged for a Frappuccino.

I'm guessing this ISP has gotten more attention here and on Ars Technica than others because the founder is fluent in the software engineering world, as well as having started an ISP. Ironically there is a pretty big gulf between the world of techies who know how to write the code on the internet and the people who actually build the internet who are more blue collar.

Spivak(10000) 1 day ago [-]

One of my coworkers also did this but went the cell tower route. Had no idea you could just install a cell tower without mountains of red tape and huge expense but hey. Then all his 'customers' (i.e neighbors) have antennas on their house pointed right at it and boom, internet. He only had to front the cost of getting the lines run to one location.

Historical Discussions: NSA, NIST, and post-quantum crypto: my second lawsuit against the US government (August 05, 2022: 971 points)

(971) NSA, NIST, and post-quantum crypto: my second lawsuit against the US government

971 points 6 days ago by trulyrandom in 10000th position

blog.cr.yp.to | Estimated reading time – 38 minutes | comments | anchor

The cr.yp.to blog

Table of contents (Access-I for index page)
2022.08.05: NSA, NIST, and post-quantum cryptography: Announcing my second lawsuit against the U.S. government. #nsa #nist #des #dsa #dualec #sigintenablingproject #nistpqc #foia
2022.01.29: Plagiarism as a patent amplifier: Understanding the delayed rollout of post-quantum cryptography. #pqcrypto #patents #ntru #lpr #ding #peikert #newhope
2020.12.06: Optimizing for the wrong metric, part 1: Microsoft Word: Review of 'An Efficiency Comparison of Document Preparation Systems Used in Academic Research and Development' by Knauff and Nejasmic. #latex #word #efficiency #metrics
2019.10.24: Why EdDSA held up better than ECDSA against Minerva: Cryptosystem designers successfully predicting, and protecting against, implementation failures. #ecdsa #eddsa #hnp #lwe #bleichenbacher #bkw
2019.04.30: An introduction to vectorization: Understanding one of the most important changes in the high-speed-software ecosystem. #vectorization #sse #avx #avx512 #antivectors
2017.11.05: Reconstructing ROCA: A case study of how quickly an attack can be developed from a limited disclosure. #infineon #roca #rsa
2017.10.17: Quantum algorithms to find collisions: Analysis of several algorithms for the collision problem, and for the related multi-target preimage problem. #collision #preimage #pqcrypto
2017.07.23: Fast-key-erasure random-number generators: An effort to clean up several messes simultaneously. #rng #forwardsecrecy #urandom #cascade #hmac #rekeying #proofs
2017.07.19: Benchmarking post-quantum cryptography: News regarding the SUPERCOP benchmarking system, and more recommendations to NIST. #benchmarking #supercop #nist #pqcrypto
2016.10.30: Some challenges in post-quantum standardization: My comments to NIST on the first draft of their call for submissions. #standardization #nist #pqcrypto
2016.06.07: The death of due process: A few notes on technology-fueled normalization of lynch mobs targeting both the accuser and the accused. #ethics #crime #punishment
2016.05.16: Security fraud in Europe's 'Quantum Manifesto': How quantum cryptographers are stealing a quarter of a billion Euros from the European Commission. #qkd #quantumcrypto #quantummanifesto
2016.03.15: Thomas Jefferson and Apple versus the FBI: Can the government censor how-to books? What if some of the readers are criminals? What if the books can be understood by a computer? An introduction to freedom of speech for software publishers. #censorship #firstamendment #instructions #software #encryption
2015.11.20: Break a dozen secret keys, get a million more for free: Batch attacks are often much more cost-effective than single-target attacks. #batching #economics #keysizes #aes #ecc #rsa #dh #logjam
2015.03.14: The death of optimizing compilers: Abstract of my tutorial at ETAPS 2015. #etaps #compilers #cpuevolution #hotspots #optimization #domainspecific #returnofthejedi
2015.02.18: Follow-You Printing: How Equitrac's marketing department misrepresents and interferes with your work. #equitrac #followyouprinting #dilbert #officespaceprinter
2014.06.02: The Saber cluster: How we built a cluster capable of computing 3000000000000000000000 multiplications per year for just 50000 EUR. #nvidia #linux #howto
2014.05.17: Some small suggestions for the Intel instruction set: Low-cost changes to CPU architecture would make cryptography much safer and much faster. #constanttimecommitment #vmul53 #vcarry #pipelinedocumentation
2014.04.11: NIST's cryptographic standardization process: The first step towards improvement is to admit previous failures. #standardization #nist #des #dsa #dualec #nsa
2014.03.23: How to design an elliptic-curve signature system: There are many choices of elliptic-curve signature systems. The standard choice, ECDSA, is reasonable if you don't care about simplicity, speed, and security. #signatures #ecc #elgamal #schnorr #ecdsa #eddsa #ed25519
2014.02.13: A subfield-logarithm attack against ideal lattices: Computational algebraic number theory tackles lattice-based cryptography.
2014.02.05: Entropy Attacks! The conventional wisdom says that hash outputs can't be controlled; the conventional wisdom is simply wrong.

2022.08.05: NSA, NIST, and post-quantum cryptography: Announcing my second lawsuit against the U.S. government. #nsa #nist #des #dsa #dualec #sigintenablingproject #nistpqc #foia

The Black Chamber was founded by the U.S. Army and the U.S. State Department in 1919. The Secretary of State terminated funding in 1929, famously writing that 'Gentlemen do not read each other's mail.'

The Black Chamber was succeeded by the Signal Intelligence Service in 1930, the Armed Forces Security Agency in 1949, and the National Security Agency (NSA) in 1952. NSA's Project Minaret began spying on anti-war protesters in 1967. NSA's targets under this project included Martin Luther King, New York Times journalist Tom Wicker, U.S. senator Frank Church, and many more.

NSA's policy decision to sabotage public cryptographic standards. In 1968, the National Bureau of Standards (NBS) 'went to NSA for help', in the words of an internal NSA history book. Work by journalists over several years forced NSA to release the relevant portions of the book in 2013, and before that smaller portions in 2008 and 2009.

NBS was an agency inside the U.S. Department of Commerce, another part of the U.S. government. Later NBS was renamed the National Institute of Standards and Technology (NIST). The reason NBS went to NSA is that NBS had decided to develop a U.S. government encryption standard.

According to the same history book, this triggered an internal debate within NSA, culminating in NSA deciding to manipulate public standards to make sure they were 'weak enough' for NSA to break them:

Narrowing the encryption problem to a single, influential algorithm might drive out competitors, and that would reduce the field that NSA had to be concerned about. Could a public encryption standard be made secure enough to protect against everything but a massive brute force attack, but weak enough to still permit an attack of some nature using very sophisticated (and expensive) techniques?

NSA then worked with NBS and IBM's Walter Tuchman on the design of what later became the Data Encryption Standard (DES):

NSA gave Tuchman a clearance and brought him in to work jointly with the Agency on his Lucifer modification ... The relationship between NSA and NBS was very close. NSA scientists working the problem crossed back and forth between the two agencies, and NSA unquestionably exercised an influential role in the algorithm.

Back in the 1970s, Tuchman and NSA told a completely different story to the public. For example, regarding accusations that IBM and NSA had 'conspired', Tuchman told an interviewer 'We developed the DES algorithm entirely within IBM using IBMers. The NSA did not dictate a single wire!'

As another example, here's a 1979 statement from NSA director Bobby Inman:

NSA has been accused of intervening in the development of the DES and of tampering with the standard so as to weaken it cryptographically. This allegation is totally false.

See Section 3.6 of my paper Cryptographic competitions for further quotes and references.

The breakability of DES. The cryptographic core of NSA's sabotage of DES was remarkably blunt: NSA simply convinced Tuchman to limit the key size to 56 bits, a glaring weakness.

Whit Diffie and Marty Hellman wrote a paper explaining in considerable detail how to build a machine for $20 million that would break each DES key with an amortized cost of just $5000/key using mid-1970s technology. They predicted that the cost of such a brute-force attack would drop 'in about 10 years time' to about $50/key, simply from chip technology improving.

Diffie and Hellman already distributed drafts of their paper before DES was standardized. Did NSA say, oh, oops, you caught us, this isn't secure?

Of course not. NSA claimed that, according to their own estimates, the attack was 30000 times more expensive: 'instead of one day he gets something like 91 years'.

Meanwhile NSA claimed that 'for the next n years, up to 10, we stand by the statement that this is more than adequate', as if DES were going to be replaced soon. In fact, DES remained an official U.S. government standard until 2005.

(Remember that, internally, NSA had observed that 'narrowing the encryption problem to a single, influential algorithm might drive out competitors, and that would reduce the field that NSA had to be concerned about'. Externally, NSA was playing dumb.)

Diffie and Hellman proposed a low-cost modification to DES to use longer keys. Questionable performance arguments were raised in response.

For example, various government contractors claimed at a 1976 NBS workshop that DES was 'close to the maximum that could be implemented on a chip with present technology' and that a manufacturing delay 'of one to two years might be encountered if a longer key were required'. Even if this was true, how could it possibly justify establishing a breakable standard for the next 10 years, never mind the next three decades?

In 1980, Hellman published 'A cryptanalytic time-memory tradeoff'. This was an algorithmic improvement showing that, compared to a brute-force machine, a more sophisticated machine could be constructed with an even smaller cost of breaking each key.

In 1993, Mike Wiener wrote a paper 'Efficient DES key search' giving an even more detailed description of a brute-force DES attack machine, not including Hellman's time-memory tradeoff. In a 1997 update, Wiener estimated that it would cost about $1 million to build a machine that would break each key in 35 minutes.

Wiener's estimate corresponded to an amortized cost of $13/key, assuming a 5-year hardware lifetime, or perhaps twice as much if one includes the costs of electricity. Extrapolating from the 1970s brute-force estimates by Diffie and Hellman, one might instead guess $1/key. Extrapolating from the 1970s brute-force estimates by NSA, one might instead guess $30000/key, an indefensible and dangerous overestimate.

DES was so cheap to break in 1997 that one could throw away orders of magnitude of efficiency, using off-the-shelf computers instead of optimized attack circuits, and still break DES. This was demonstrated by DESCHALL in 1997.

In response to DESCHALL, Federal Bureau of Investigation Director Louis Freeh testified as follows in a 1997 hearing:

If we hooked together thousands of computers and worked together over 4 months we might, as was recently demonstrated decrypt one message bit. That is not going to make a difference in a kidnapping case, it is not going to make a difference in a national security case. We don't have the technology or the brute force capability to get to this information.

NSA Deputy Director William Crowell testified at the same hearing that 'There is no brute force solution for law enforcement,' again highlighting that 'It took 78,000 computers 96 days to break one message, and the headline was, DES has weak encryption.'

Did NSA admit that optimized hardware was much more efficient? Of course not. It was taking the inefficiency of off-the-shelf computers and misrepresenting this as security of DES.

In 1998, the Electronic Frontier Foundation assembled a $250000 machine, the DES Cracker, to publicly break DES in a few days. This was, obviously, much cheaper and faster than 96 days on 78000 computers.

Did NSA admit that its public estimates of the cost of breaking DES (1) were wild overestimates even for a brute-force attack machine and (2) were ignoring algorithmic improvements such as Hellman's time-memory tradeoff? Of course not.

The Digital Signature Algorithm. NIST was busy in the meantime issuing more cryptographic standards. One of these was an influential standard for a Digital Signature Algorithm (DSA).

NSA had proposed DSA in 1991. The DSA proposal had an obvious, glaring flaw: a 512-bit key size.

This sounds much larger than the 56-bit DES key size. But DSA is a different type of cryptographic system, using a type of mathematical structure that was already publicly known in 1991 to require much larger key sizes for security.

With attack algorithms publicly known in 1991, 512 bits for DSA seemed somewhat stronger than 56 bits for DES. But chips were faster in 1991 than they were in the 1970s. The DSA attack algorithms known in 1991 were also much more complicated than the DES attack algorithms. Many aspects of the DSA attacks hadn't been thoroughly studied. The attacks were publicly superseded by even faster, even more complicated attacks.

Beyond this glaring flaw, DSA had further flaws that weren't as obvious. For example, DSA had interesting possibilities for back doors. It also had pitfalls that would trap implementors. Ron Rivest wrote in 1992 that 'the poor user is given enough rope with which to hang himself'.

At the beginning, NIST presented DSA as a NIST proposal, making no mention of NSA. A FOIA lawsuit by Computer Professionals for Social Responsibility (CPSR) revealed, however, that DSA had been secretly designed by NSA.

(FOIA is the Freedom of Information Act, a law generally requiring the U.S. government to promptly provide records to the public upon request. Sometimes agencies disobey the law, and one has to go to court to have the law enforced. That's what CPSR did.)

Public backlash forced NIST to allow larger keys in the DSA standard in 1994. But the keys were still limited to 1024 bits; 1024 bits for DSA are much less secure than, e.g., 128 bits for AES. The DSA standard also did nothing to address DSA's further flaws.

The specific 'hang himself' problem that Rivest had highlighted was publicly exploited in a 2010 break of the Sony PlayStation security system, which used ECDSA, a variant of DSA. Presumably the same problem was secretly exploited by large-scale attackers against higher-value targets.

The scale of attacks. What do I mean by 'large-scale attackers?' Let's take the Chinese government as an example. Here's a 2012 quote from an 'Investigative report on the U.S. national security issues posed by Chinese telecommunications companies Huawei and ZTE':

Chinese intelligence collection efforts against the U.S. government are growing in 'scale, intensity and sophistication.' Chinese actors are also the world's most active and persistent perpetrators of economic espionage. U.S. private sector firms and cyber-security specialists report an ongoing onslaught of sophisticated computer network intrusions that originate in China, and are almost certainly the work of, or have the backing of, the Chinese government. Further, Chinese intelligence services, as well as private companies and other entities, often recruit those with direct access to corporate networks to steal trade secrets and other sensitive proprietary data.

The large-scale attacker whose behavior seems most comprehensively documented is the U.S. government. The European Parliament already issued a 194-page 'Report on the existence of a global system for the interception of private and commercial communications (ECHELON interception system)' in 2001:

The existence of a global system for intercepting communications, operating by means of cooperation proportionate to their capabilities among the USA, the UK, Canada, Australia and New Zealand under the UKUSA Agreement, is no longer in doubt ... The US authorities have repeatedly tried to justify the interception of telecommunications by accusing the European authorities of corruption and taking bribes. It should be pointed out to the Americans that all EU Member States have properly functioning criminal justice systems. If there is evidence that crimes have been committed, the USA must leave the task of law enforcement to the host countries. If there is no such evidence, surveillance must be regarded as unproportional, a violation of human rights and thus inadmissible.

Some Americans trust their government and happily swallow whatever the government's latest excuse is for spying on billions of people around the world. 'U.S. kills al Qaeda leader Zawahiri in Kabul drone missile strike', of course with the help of the espionage system? Sounds great! Last year the same system murdered 10 innocent civilians without a trial? Isolated mistake! Also, doesn't the Constitution say that the only people entitled to a trial are rich white male normal-looking Americans?

The same people tend to have trouble grasping that most of the vulnerabilities exploited and encouraged by NSA are also exploitable by the Chinese government. These people start with the assumption that Americans are the best at everything; ergo, we're also the best at espionage. If the Chinese government stole millions of personnel records from the U.S. government, records easily usable as a springboard for further attacks, this can't possibly be because the U.S. government made a policy decision to keep our computer systems 'weak enough to still permit an attack of some nature using very sophisticated (and expensive) techniques'.

New directions in cryptographic sabotage. Cryptographic weaknesses aren't always exploitable by everybody. Sometimes it's possible to design a cryptographic system with a back door that can be opened only by someone who has a secret key. A spectacular example is the Dual Elliptic Curve Deterministic Random Bit Generator (Dual EC).

The Dual EC standard includes two random-looking constants P and Q, points on an 'elliptic curve'. Dual EC uses P, Q, and some initial randomness provided by the user to generate a long sequence of random-looking numbers. Cryptography often needs many random numbers.

The secret key to the Dual EC back door is the 'discrete logarithm' of Q base P. It's easy to generate this secret key while generating P and Q. An attacker who knows this secret key can exploit secret patterns in the Dual EC output, even without being told the initial randomness from the user. This is a severe weakness in Dual EC, but it's exploitable only if you know the secret key.

A 2013 New York Times report said that internal NSA documents provided by Ed Snowden 'appear to confirm that the fatal weakness ... was engineered by the agency'. The same report gave an idea of the magnitude of NSA's sabotage efforts:

According to an intelligence budget document leaked by Mr. Snowden, the N.S.A. spends more than $250 million a year on its Sigint Enabling Project, which 'actively engages the U.S. and foreign IT industries to covertly influence and/or overtly leverage their commercial products' designs' to make them 'exploitable.' ... One goal in the agency's 2013 budget request was to "influence policies, standards and specifications for commercial public key technologies," the most common encryption method.

Did NSA admit that, okay, you caught us, we designed Dual EC to be exploitable? Of course not. NSA's Dickie George gave a talk in 2014 making the following claims (minutes 32-33 and 57-61):

  • NSA couldn't use cryptography that wasn't standardized by NIST. (In fact, NSA has its own classified suite of algorithms, Suite A.)
  • 'So I had to go down to my friends at NIST, and I know 'em well, cause I work with them on other things ...' (This part is true: NSA does work with NIST.)
  • 'We're gonna use the Dual Elliptic Curve randomizer. And I said, if you can put this in your standard, nobody else is gonna use it, because it looks ugly, it's really slow, it makes no sense for anybody to go there, but I'll be able to use it. And so they stuck it in.' (In fact, NSA paid the RSA company $10 million to make Dual EC 'the preferred, or default, method for number generation in the BSafe software'.)
  • 'And I said, by the way, these parameters that we have here, as long as they're in there so we can use them, you can let anybody else put any parameters in that they want.' (In fact, the Dual EC standard specifically discouraged implementors from switching away from NSA's P and Q. NIST also set up validation procedures specifically requiring NSA's P and Q.)
  • 'Sticking a bunch of digits of pi in the middle or something like that, so you can show it's not some kind of hoked-up thing, we just don't do that.' (In fact, NSA did something like that to generate the NIST P-256 elliptic curve a few years earlier, first publishing supposedly random numbers but then feeding the random numbers through a hash function to generate the curve parameters, so the public could check the hash. If NSA had similarly used a hash function for both P and Q in Dual EC then it would have been throwing away the key for the Dual EC back door.)
  • 'We don't care what the parameters are for anybody else as long as the government ones are there for government use.' (In fact, FOIA requests to NIST showed Don Johnson telling NIST that P and Q 'could also be generated like a(nother) canonical G, but NSA kyboshed this idea, and I was not allowed to publicly discuss it, just in case you may think of going there.')

George also challenged researchers 'to actually generate their own parameters and show me that in real life they can recover that.' He offered dinner. Did he pay up after a paper 'On the practical exploitability of Dual EC in TLS implementations' appeared at the USENIX Security Symposium? Of course not.

Typical cryptographic weaknesses are algorithms that, if discovered by the public, can be demonstrated to work. The weakness in Dual EC is different. Shumow and Ferguson announced in 2007 that there mathematically exists a back door with a secret key, and that anyone generating P and Q can easily generate a secret key at the same time; but this still doesn't demonstrate that NSA did generate a secret key along with P and Q. We have no way to prove that NSA's P and Q are weak.

For the same reason, if someone changes P and/or Q, a code reviewer can't tell whether this is being done safely or is stealthily opening up a back door. In the Juniper Dual EC incident, Juniper chose its own Q for its NetScreen VPN routers, but then an attacker managed to modify the code to substitute a new choice of Q. So much for the idea that this weakness could be exploited only by NSA.

Technical interlude: bamboozling people with fake mathematics. Internally, Dual EC generates more elliptic-curve points. Each point has an x-coordinate and a y-coordinate. The numbers that Dual EC releases as output are truncated versions of the x-coordinate of each point. The truncation removes 16 bits.

An attacker exploiting the back door has to try 216 possibilities for the missing bits. This has low cost, since 16 is so small. If, however, 16 were replaced by a much larger number, such as 128, then the back door would become so expensive as to be worthless.

Why did NSA include truncation in the first place? Answer: Releasing the full x-coordinates, with no truncation, would have shown glaring statistical biases, and this would have been too embarrassing for NIST to standardize.

Attack papers in 2006 by Kristian Gjøsteen and then Berry Schoenmakers and Andrey Sidorenko showed that Dual EC was still detectably biased: it wasn't removing enough bits to meet the standard definition of security for random-number generators.

Did NSA admit that, oops, Dual EC was broken? Of course not. Let's look at what NSA wrote instead, specifically pages 88–91 of NIST's Dual EC standard (2012 version), SP 800-90A.

NSA didn't even acknowledge the standard security goal of being indistinguishable from truly random numbers. NSA spent two pages on calculations related to a weaker security goal, having nearly full 'entropy'. NSA concluded that the entropy loss with 16 bits of truncation 'has been demonstrated to be minimal (see the above chart)'.

This still wouldn't stop people from removing more bits (and thus making the back door harder to exploit). So NSA also played the performance card: 'One might wonder if it would be desirable to truncate more than this amount. The obvious drawback to such an approach is that increasing the truncation amount hinders the performance.'

But, hmmm, random-number generation usually isn't the most important bottleneck in cryptography. That's why something as slow as Dual EC could be deployed in the first place. So NSA pulled out another argument:

However, there is an additional reason that argues against increasing the truncation. Consider the case where the low s bits of each x-coordinate are kept. Given some subinterval I of length 2s contained in [0, p), and letting N(I) denote the number of x-coordinates in I, recent results on the distribution of x-coordinates in [0, p) provide the following bound: |N(I)/(p/2)-2s/p| < (k*log2 p)/sqrt(p) where k is some constant derived from the asymptotic estimates given in [Shparlinski]. For the case of P-521, this is roughly equivalent to: |N(I)-2(s-1)| < k*2277, where the constant k is independent of the value of s. For s < 2277, this inequality is weak and provides very little support for the notion that these truncated x-coordinates are uniformly distributed. On the other hand, the larger the value of s, the sharper this inequality becomes, providing stronger evidence that the associated truncated x-coordinates are uniformly distributed. Therefore, by keeping truncation to an acceptable minimum, the performance is increased, and certain guarantees can be made about the uniform distribution of the resulting truncated quantities. Further discussion of the uniformity of the truncated x-coordinates is found in [Gurel], where the form of the prime defining the field is also taken into account.

Any mathematician who checks this sees how ludicrously inaccurate it is. The central error is already visible in the third sentence, which counts the number of integers in an interval of length 2s. That's the number of integers that produce a given output when one throws away the bottom s bits, not when one keeps the bottom s bits. Correcting this error and following through the rest of the argument leads to the conclusion that, for guarantees regarding the distribution, one should truncate more than half of the bits.

But NIST never checked. NIST's friends at NSA were providing text and references arguing against increasing the truncation; so NIST accepted this, redistributed it, and didn't increase the truncation.

Dual EC fallout. After Shumow and Ferguson announced the vulnerability in Dual EC, Bruce Schneier wrote an article 'Did NSA put a secret backdoor in new encryption standard?' ending as follows:

My recommendation, if you're in need of a random-number generator, is not to use Dual_EC_DRBG under any circumstances. If you have to use something in SP 800-90, use CTR_DRBG or Hash_DRBG.

In the meantime, both NIST and the NSA have some explaining to do.

But most cryptographers didn't take the threat seriously. Consider, for example, Shumow's 2008 talk with the title 'Shumow and Schneier escape from Guantanamo Bay'. Shumow mocked 'conspiracy theories', accused Schneier of omitting facts 'in a way to make the NSA (as well as Microsoft) look far worse than they otherwise would', and argued that Dual EC exploits were 'possible but improbable':

I found this, and I am neither a talented mathematician nor a talented cryptographer. I was just the first person to commercially implement the algorithm.

The probability of getting caught trying to sneak this in is too high.

Neither NIST nor the NSA told anyone to use this (it is not the Clipper Chip.) ...

The NSA is not the cryptographic research power house it once was.

In 2013, the Snowden documents finally forced NIST to do some soul-searching. NIST's Dual EC post-mortem drew the following conclusion:

It is of paramount importance that NIST's process for developing cryptographic standards is open and transparent and has the trust and support of the cryptographic community.

The same post-mortem shows NIST's invited reviewers recommending clear transparency rules, such as 'full documentation of all decisions, and clear processes for the disposition of each and every comment received', along with being open about 'what authorities were consulted'.

Note that it's not always obvious who's providing input. For example, NIST received the draft Dual EC proposal from another standardization organization, ANSI. Dual EC was proposed to ANSI not by NSA, but by Johnson, who was working for Cygnacom, a defense contractor. But NIST did end up working directly with NSA on the Dual EC standard. Prompt reporting of NIST's communications and evaluations would have given the public many more opportunities to promptly catch what was happening with Dual EC.

Post-quantum cryptography. A 2014 Washington Post article 'NSA seeks to build quantum computer that could crack most types of encryption' said that NSA had an $80 million/year research program called 'Penetrating hard targets', including research aimed at building a 'cryptologically useful quantum computer'.

This is only part of the U.S. government budget for breaking pre-quantum cryptography. My 2012 talk 'Cryptography for the paranoid' pointed to a $2.2 million grant for defense contractor Raytheon as 'one of many publicly announced quantum-computing grants from government agencies'.

Put yourself in NSA's shoes for a moment. You have a budget to build a quantum computer to break pre-quantum cryptography. You are, of course, aware of public efforts to design and deploy post-quantum cryptography. You also have a quarter-billion-dollar-a-year budget to 'covertly influence and/or overtly leverage' deployed cryptography to make it 'exploitable'. What do you do?

Some people seem to be unable to rationally consider the possibility that NSA is sabotaging post-quantum cryptography. I've heard people saying, for example, that submissions to the NIST Post-Quantum Cryptography Standardization Project (NISTPQC) were publicly designed and evaluated by top experts, and that NSA can't have bribed the submission teams.

Let's look at the facts.

Almost all of the submissions to NISTPQC have less security against the best attacks publicly known today than they had against the best attacks publicly known when they were submitted in 2017. I'm not talking about chips getting faster: I'm talking about new attack algorithms.

For many of the submissions, the attack improvements have been so dramatic that the submissions have been publicly demonstrated to be rapidly breakable on a laptop. Last month a new attack broke SIKE, one of just eight submissions that was still under consideration by NIST, one of just two submissions that had been selected for a high-profile Cloudflare–Google TLS experiment in 2019.

As for 'top experts', here's a quote from a document 'Risks of lattice KEMs' by the NTRU Prime Risk-Management Team:

Consider the fact that the Institute for Defense Analyses, an NSA consulting company, many years ago hired Buhler, one of the original developers of the number-field sieve [94] for integer factorization; Gordon, the first developer of a discrete-logarithm version [166] of the number-field sieve; Miller, who as part of introducing ECC [261] was one of the first authors to probe the limits of discrete-logarithm algorithms; and Coppersmith. Much less data is available regarding the cryptanalytic capabilities of, e.g., the Chinese government. Surely large-scale attackers know many more attacks than this public does.

At the risk of belaboring the obvious: An attacker won't have to say 'Oops, researcher X is working in public and has just found an attack; can we suppress this somehow?' if the attacker had the common sense to hire X years earlier, meaning that X isn't working in public. People arguing that there can't be sabotage because submission teams can't be bribed are completely missing the point.

I coined the phrase 'post-quantum cryptography' in 2003. It's not hard to imagine that the NSA/IDA post-quantum attack team was already hard at work before that, that they're years ahead of the public in finding attacks, and that NSA has been pushing NISTPQC to select algorithms that NSA secretly knows how to break.

Could such a weakness also be exploited by other large-scale attackers? Best bet is that the answer is yes. Would this possibility stop NSA from pushing for the weakness? Of course not.

Hybrids. When Google rolled out its first post-quantum experiment in 2016, it didn't switch from encrypting with a well-established pre-quantum system to encrypting with a post-quantum system. Instead it encrypted with a well-established pre-quantum system and encrypted with a post-quantum system, so that at least it wouldn't be losing security if the post-quantum system turned out to be breakable.

The 2019 Cloudflare–Google experiment worked the same way. The general view today is that of course post-quantum cryptography should be an extra layer on top of well-established pre-quantum cryptography. As the French government cybersecurity agency (Agence nationale de la sécurité des systèmes d'information, ANSSI) put it at the end of 2021:

Acknowledging the immaturity of PQC is important: ANSSI will not endorse any direct drop-in replacement of currently used algorithms in the short/medium term. However, this immaturity should not serve as an argument for postponing the first deployments. ANSSI encourages all industries to progress towards an initiation of a gradual overlap transition in order to progressively increase trust on the post-quantum algorithms and their implementations while ensuring no security regression as far as classical (pre-quantum) security is concerned. ...

Given that most post-quantum algorithms involve message sizes much larger than the current pre-quantum schemes, the extra performance cost of an hybrid scheme remains low in comparison with the cost of the underlying post-quantum scheme. ANSSI believes that this is a reasonable price to pay for guaranteeing an additional pre-quantum security at least equivalent to the one provided by current pre-quantum standardized algorithms.

But NSA has a different position: it says that it 'does not expect to approve' hybrids. Publicly, NSA justifies this by

Does that mean the original NISTPQC process, or the current NISTPQC process in which NIST, evidently surprised by attacks, announced plans to call for new submissions?

Of course, if NSA/IDA have secretly developed an attack that works for a particular type of post-quantum cryptosystem, then it makes sense that they'd want people to start using that type of cryptosystem and turn off the existing pre-quantum cryptosystem.

Transparency for NISTPQC. NIST issued final 'Submission requirements and evaluation criteria for the post-quantum cryptography standardization process' in December 2016, including a promise that NIST would 'perform a thorough analysis of the submitted algorithms in a manner that is open and transparent to the public'.

It became clear to me in 2020 that, despite this promise, most of NIST's evaluation process was happening behind closed doors. I tweeted the following at 13:01 GMT on 22 July 2020:

After NIST's Dual EC standard was revealed in 2013 to be an actual (rather than just potential) NSA back door, NIST promised more transparency. Why does NIST keep soliciting private #NISTPQC input? (The submissions I'm involved in seem well positioned; that's not the point.)

Coincidentally, at 13:02 GMT on 22 July 2020, NSA suddenly made its first public appearance in NISTPQC. Slides from NIST in September 2020 admitted that before this there was already 'feedback' from NSA to NIST (slide 20). The public still hasn't seen the contents of NIST's communications with NSA, defense contractors, etc., let alone the records of how NIST processed the input it received.

The same September 2020 NIST slides tried to downplay NSA's influence: 'NIST alone makes the PQC standardization decisions, based on publicly available information, and stands by those decisions.' One is reminded of Tuchman saying 'We developed the DES algorithm entirely within IBM using IBMers. The NSA did not dictate a single wire!'

In 2021, NIST claimed that 'We operate transparently. We've shown all our work'. In fact, most of the information on NIST's web site for this project is simply copies of submissions. NIST has posted some extra information, but the total volume of information in NIST's reports, web pages, and mailing-list messages obviously falls far short of 'all our work'. Anyone trying to obtain more than a superficial understanding of what has happened in NISTPQC rapidly discovers that critical information is missing. See my paper 'A discretization attack', specifically Section 5, for various concrete examples of mysteries regarding the NIST process.

I've filed seven FOIA requests with NIST since mid-2020. NIST has released a few dribbles of information, but in general NIST's responses have been very slow and obviously not complete.

For example, I filed a FOIA request in June 2021 asking for 'copies of all NIST records of communication between NSA and NIST regarding the NIST Post-Quantum Cryptography Standardization Project'. This request has, so far, produced zero records. NIST has stonewalled, ignoring the FOIA deadlines.

My seventh FOIA request, in March 2022, said the following:

Analyzing NSA's impact on this project will require not just seeing NSA's communication with NIST, but also tracing how NIST's decisions were made and analyzing the influence of the information that NIST received from NSA. If each step of this analysis requires dealing with another round of stonewalling from NIST then the analysis will obviously not be done in time to help the public make safe decisions regarding post-quantum cryptography.

NSA's documented history of sabotage, along with its evident sway over NIST, makes NSA's influence on NIST a high priority to review, but it also seems likely that other entities have also been trying to sabotage NIST's process. As far as I can tell, NIST has no procedures in place to prevent attackers from influencing the project through pseudonyms, proxies, etc. Anything short of a full review of project records could easily miss evidence of attacks.

Even without sabotage, getting cryptography right is challenging. Public review has identified security flaws in dozens of submissions and has identified many errors in the limited additional information released by NIST. Having NIST keep most of its analysis secret is a recipe for disaster. Given that NIST promised to be 'open and transparent', and recently claimed to have 'shown all our work', it's hard to understand why the full project records aren't already available to the public.

I asked for the full NISTPQC records, and for 'all records of NIST/NSA meetings mentioning the word 'quantum', whether or not NIST views those meetings as part of this project'.

NIST has produced zero records in response to this FOIA request. Civil-rights firm Loevy & Loevy has now filed suit on my behalf in federal court, the United States District Court for the District of Columbia, to force NIST to comply with the law.

[2022.08.06 edits: Fixed 'requests since' -> 'requests with'. Added the 'At the risk of belaboring the obvious' paragraph. Added various further links under existing text.]

Version: This is version 2022.08.06 of the 20220805-nsa.html web page.

All Comments: [-] | anchor

er4hn(10000) 6 days ago [-]

> The same people tend to have trouble grasping that most of the vulnerabilities exploited and encouraged by NSA are also exploitable by the Chinese government. These people start with the assumption that Americans are the best at everything; ergo, we're also the best at espionage. If the Chinese government stole millions of personnel records from the U.S. government, records easily usable as a springboard for further attacks, this can't possibly be because the U.S. government made a policy decision to keep our computer systems 'weak enough to still permit an attack of some nature using very sophisticated (and expensive) techniques'.

I'm not sure if I understand this part. I was under the impression that the OPM hack was a result of poor authn and authz controls, unrelated to cryptography. Was there a cryptography component sourced somewhere?

danielheath(10000) 6 days ago [-]

If, rather than hoarding offensive tools & spying, the NSA had interpreted its mission as being to harden the security of government infrastructure (surely even more firmly within the remit of national security) and spent its considerable budget in that direction, would authn and authz controls have been used at the OPM?

woodruffw(10000) 6 days ago [-]

This is my understanding as well. I asked this very same question less than a week ago[1], and now it's the first Google result when you search 'OPM Dual_EC_DRBG.'

The response to my comment covers some circumstantial evidence. But I'm not personally convinced; human factors are a much more parsimonious explanation.

[1]: https://news.ycombinator.com/item?id=32286528

throwaway654329(10000) 6 days ago [-]

The history in this blog post is excellently researched on the topic of NSA and NIST cryptographic sabotage. It presents some hard won truths that many are uncomfortable to discuss, let alone to actively resist.

The author of the blog post is also well known for designing and releasing many cryptographic systems as free software. There is a good chance that your TLS connections are secured by some of these designs.

One of his previous lawsuits was critical to practically protecting free speech during the First Crypto War: https://en.m.wikipedia.org/wiki/Bernstein_v._United_States

I hope he wins.

nimbius(10000) 6 days ago [-]

the author was also part of the Linux kernel SPECK cipher talks that broke down in 2013 due to the nsa's stonewalling and hand waving for technical data and explanations.

nsa speck was never adopted.


aliqot(10000) 6 days ago [-]

Given his track record, and the actual meat of this suit, I think he has a good chance.

- He is an expert in the domain

- He made a lawful request

- He believes he's experiencing an obstruction of his rights

I don't see anything egregious here. Being critical of your government is a protected right for USA. Everyone gets a moment to state their case if they'd like to make an accusation.

Suing sounds offensive, but that is the official process for submitting an issue that a government can understand and address. I'm seeing some comments here that seem aghast at the audacity to accuse the government at your own peril, and it shows an ignorance of history.

fossuser(10000) 6 days ago [-]

I remember reading about this in Steven Levy's crypto and elsewhere, there was a lot of internal arguing about lots of this stuff at the time and people had different opinions. I remember that some of the suggested changes from NSA shared with IBM were actually stronger against a cryptanalysis attack on DES that was not yet publicly known (though at the the time people suspected they were suggesting this because it was weaker, the attack only became publicly known later). I tried to find the specific info about this, but can't remember the details well enough. Edit: I think it was this: https://en.wikipedia.org/wiki/Differential_cryptanalysis

They also did intentionally weaken a standard separately from that and all the arguing about 'munitions export' intentionally requiring weak keys etc. - all the 90s cryptowar stuff that mostly ended after the clipper chip failure. They also worked with IBM on DES, but some people internally at NSA were upset that they shared this after the fact. The history is a lot more mixed with a lot of people arguing about what the right thing to do is and no general consensus on a lot of this stuff.

matthewmcg(10000) 6 days ago [-]

Right came here to make the same point. The first lawsuit alluded to in the blog post title resulted in an important holding that source code can be protected free expression.

sigil(10000) 6 days ago [-]

Near the end of the post – after 50 years of axe grinding – djb does eventually get to the point wrt pqcrypto. I find the below excerpt particularly damning. Why not wrap nascent pqcrypto in classical crypto? Suspect!


The general view today is that of course post-quantum cryptography should be an extra layer on top of well-established pre-quantum cryptography. As the French government cybersecurity agency (Agence nationale de la sécurité des systèmes d'information, ANSSI) put it at the end of 2021:

Acknowledging the immaturity of PQC is important: ANSSI will not endorse any direct drop-in replacement of currently used algorithms in the short/medium term. However, this immaturity should not serve as an argument for postponing the first deployments. ANSSI encourages all industries to progress towards an initiation of a gradual overlap transition in order to progressively increase trust on the post-quantum algorithms and their implementations while ensuring no security regression as far as classical (pre-quantum) security is concerned. ...

Given that most post-quantum algorithms involve message sizes much larger than the current pre-quantum schemes, the extra performance cost of an hybrid scheme remains low in comparison with the cost of the underlying post-quantum scheme. ANSSI believes that this is a reasonable price to pay for guaranteeing an additional pre-quantum security at least equivalent to the one provided by current pre-quantum standardized algorithms.

But NSA has a different position: it says that it 'does not expect to approve' hybrids. Publicly, NSA justifies this by

- pointing to a fringe case where a careless effort to add an extra security layer damaged security, and

- expressing 'confidence in the NIST PQC process'.

Does that mean the original NISTPQC process, or the current NISTPQC process in which NIST, evidently surprised by attacks, announced plans to call for new submissions?

Of course, if NSA/IDA have secretly developed an attack that works for a particular type of post-quantum cryptosystem, then it makes sense that they'd want people to start using that type of cryptosystem and turn off the existing pre-quantum cryptosystem.

tptacek(10000) 5 days ago [-]

This is the least compelling argument Bernstein makes in the whole post, because it's simply not the job of the NIST PQC program to design or recommend hybrid classical/PQC schemes. Is it fucky and weird if NSA later decides to recommend against people using hybrid key establishment? Yes. Nobody should listen to NSA about that, or anything else. But NIST ran a PQC KEM and signature contest, not a secure transport standardization. Sir, this is a Wendy's.

jcranmer(10000) 6 days ago [-]

If anyone is curious, the courtlistener link for the lawsuit is here: https://www.courtlistener.com/docket/64872195/bernstein-v-na...

(And somebody has already kindly uploaded the documents to RECAP, so it costs you nothing to access.)

Aside: I really wish people would link to court documents whenever they talk about an ongoing lawsuit.

Natsu(10000) 6 days ago [-]

> Aside: I really wish people would link to court documents whenever they talk about an ongoing lawsuit.

I just want to second that and thank you for the link. Most reporting is just horribly bad at covering legal stuff because all the stuff that makes headlines that people click on is mostly nonsense.

tptacek(10000) 6 days ago [-]

It's just a vanilla FOIA lawsuit, of the kind hundreds of people file every month when public bodies fuck up FOIA.

If NIST puts up any kind of fight (I don't know why they would), it'll be fun to watch Matt and Wayne, you know, win a FOIA case. There's a lot of nerd utility in knowing more about how FOIA works!

But you're not going to get the secrets of the Kennedy assassination by reading this thing.

xenophonf(10000) 6 days ago [-]

Good god, this guy is a bad communicator. Bottom line up front:

> NIST has produced zero records in response to this [March 2022] FOIA request [to determine whether/how NSA may have influenced NIST's Post-Quantum Cryptography Standardization Project]. Civil-rights firm Loevy & Loevy has now filed suit on my behalf in federal court, the United States District Court for the District of Columbia, to force NIST to comply with the law.

Edit: Yes, I know who DJB is.

jcranmer(10000) 6 days ago [-]

That is truly burying the lede...

I spent most of the post asking myself 'okay, I'm guessing this is something about post-quantum crypto, but what are you actually suing about?'

kube-system(10000) 6 days ago [-]

Well, he is an expert in cryptic communication

lizardactivist(10000) 6 days ago [-]

An expert, prominent, and someone who the whole cryptography community listens to, and he calls out the lies, crimes, and blatant hypocrisy of his own government.

I genuinely fear that he will be suicided one of these days.

ok_dad(10000) 6 days ago [-]

I think the United States is more about charging people with crimes and ruining their lives that way rather than disappearing people. Russia might kill you with Polonium and make sure everyone knows it, but America will straight up "legally" torture you in prison via several means and then argue successfully that those methods were legal and convince the world you weren't tortured. Anyone who's a target for that treatment, though, knows that's a lie.

bumper_crop(10000) 6 days ago [-]

This definitely has the sting of bitterness in it, I doubt djb would have filed this suit if NTRU Prime would have won the PQC NIST contest. It's hard to evaluate this objectively when there are strong emotions involved.

cosmiccatnap(10000) 6 days ago [-]

It's funny how often the bitterness of a post is used as an excuse to dismiss the long and well documented case being made.

pixl97(10000) 6 days ago [-]

When it comes to the number of times DJB is right versus the number of times that DBJ is wrong, I'll fully back DJB. Simply put the NSA/NIST cannot and should not be trusted in this case.

lawrenceyan(10000) 6 days ago [-]

Here's an interesting question. Even if post-quantum cryptography is securely implemented, doesn't the advent of neurotechnology (BCIs, etc.) make that method of security obsolete?

With read and write capability to the brain, assuming this comes to fruition at some point, encryption as we know it won't work anymore. But I don't know, maybe this isn't something we have to worry about just quite yet.

Banana699(10000) 6 days ago [-]

The thing you're missing is that BCIs and friends are, themselves, computers, and thus securable with post-quantum cryptography, or any cryptography for that matter, or any means of securing a computer. And thus, for somebody to read-write to your computers, they need to read-write to your brain(s), but to read-write to your brain(s), they need to read-write to the computers implanted in your brain(s). It's a security cycle whose overall power is determined by the least-secure element in the chain.

Any sane person will also not touch BCIs and similar technology with a 100 lightyear pole unless the designing company reveals every single fucking silicon atom in the hardware design and every single fucking bit in the software stack at every level of abstraction, and ships the device with several redundant watchdogs and deadmen timers around it that can safely kill or faraday-cage the implant on user-defined events or manually.

Alas, humans are very rarely sane, and I come to the era of bio hacking (in all senses of the word) with low expectations.

yjftsjthsd-h(10000) 6 days ago [-]

The encryption is fine, that's just a way to avoid it. Much like how tire-iron attacks don't break passwords so much as bypass them.

lysergia(10000) 6 days ago [-]

Yeah I've even had very personal dreams where my Linux root password was spoken in the dream. I'm glad I don't talk in my sleep. There's also truth serums that can be weaponized in war scenarios to extract secrets from the enemy without resorting to torture.

xenophonf(10000) 6 days ago [-]

Cryptographic secrets stored in human brains are already vulnerable to an attack mechanism that requires $5 worth of interface hardware that can be procured and operated with very little training. Physical security controls do a decent job of preventing malicious actors from connecting said hardware to vulnerable brains. I assume the same would be true with the invention of BCIs more sophisticated than a crescent wrench.

politelemon(10000) 6 days ago [-]

So, question then, isn't one of the differences between this time's selection, compared to previous selections, that some of the algorithms are open source with their code available.

For example, Kyber, one of the finalists, is here: https://github.com/pq-crystals/kyber

And where it's not open source, I believe in the first round submissions, everyone included reference implementations.

Does the code being available make it easy to verify whether there are some shady/shenanigans going on, even without NIST's cooperation?

aaaaaaaaaaab(10000) 6 days ago [-]

What? :D

Who cares about a particular piece of source code? Cryptanalysis is about the mathematical structure of the ciphers. When we say the NSA backdoored an algorithm, we don't mean that they included hidden printf statements in 'the source code'. It means that mathematicians at the NSA have knowledge of weaknesses in the construction, that are not known publicly.

gnabgib(10000) 6 days ago [-]

Worth noting DJB (the article author) was on two competing (losing) teams to Kyber[0] in Round 3. And has an open submission in round 4 (still in progress). That's going to slightly complicate any FOIA until after the fact, or it should. Not that there's no merit in the request.

[0]: https://csrc.nist.gov/Projects/post-quantum-cryptography/pos...

lostcolony(10000) 6 days ago [-]

Not really. For the same reason that 'here's your github login' doesn't equate to you suddenly being able to be effective in a new company. You might be able to look things up in the code and understand how things are being done, but you don't know -why- things are being done that way.

A lot of the instances in the post even show the NSA giving a why. It's not a particular convincing why, but it was enough to sow doubt. The reason to make all discussions public is so that there isn't an after the fact 'wait, why is that obviously odd choice being done?' but instead a before the fact 'I think we should make a change'. The burden of evidence is different for that. A 'I think we should reduce the key length for performance' is a much harder sell when the spec already prescribes a longer key length, than an after the fact 'the spec's key length seems too short' 'Nah, it's good enough, and we need it that way for performance'. The status quo always has inertia.

ehzy(10000) 6 days ago [-]

Ironically, when I visit the site Chrome says my connection is not secured by TLS.

encryptluks2(10000) 6 days ago [-]

Are you logging into the site?

kzrdude(10000) 6 days ago [-]

I was hoping for chacha20+Poly1305

bsaul(10000) 6 days ago [-]

side question :

I've only recently started to digg a bit deeper into crypto algorithms ( looking into various types of curves etc), and it gave me the uneasing feeling that the whole industry is relying on the expertise of only a handful of guys to actually ensure that crypto schemes used today are really working.

Am i wrong ? are there actually thousands and thousands of people with the expertise to actually proove that the algorithms used today are really safe ?

chasil(10000) 6 days ago [-]

This 'monoculture' post raised this point several years ago.


aumerle(10000) 6 days ago [-]

Proof! the entire field of cryptography can prove absolutely nothing other than that a single use of One time pad is secure. the rest is all hand waving, that boils down to no-one I know knows how to do this, and I cant do it myself, so I believe it's secure.

So the best we have in cryptography is trusting 'human instincts/judgements' about various algorithms. Which then further reduces to trusting humans.

NavinF(10000) 6 days ago [-]

Most programmers don't need to prove crypto algorithms. There are many situations where you can just use TLS 1.3 and let it choose the ciphers. If you really need to build a custom protocol or file format, you can still use libsodium's secretbox, crypto_box, and crypto_kx functions which use the right algorithms.

kibibyte(10000) 6 days ago [-]

I don't know if that's easily quantifiable, but I had a cryptography professor (fairly well-known nowadays) several years ago tell us that she only trusted 7 people (or some other absurdly low number), one of them being djb, to be able to evaluate the security of cryptographic schemes.

Perhaps thousands of people in the world can show you proofs of security, but very few of them may be able to take into account all practical considerations like side channels and the like.

benlivengood(10000) 6 days ago [-]

There may be thousands of people in the entire world who understand cryptanalysis well enough to accurately judge the security of modern ciphers. Most aren't living or working in the U.S.

It's very difficult to do better. The mathematics is complex and computer science hasn't achieved proofs of the hypotheses underlying cryptography. The best we can achieve is heuristic judgements about what the best possible attacks are, and P?=NP is an open question.

jacooper(10000) 6 days ago [-]

Flippo valrosida and Matthey green aren't too happy.


ghoward(10000) 6 days ago [-]

Thanks for letting me know. I think I'll consider both of them compromised.

throwaway654329(10000) 6 days ago [-]

Dismissing this lawsuit as a conspiracy theory is embarrassing for both of them.

There is ample evidence to document malfeasance by the involved parties, and it's reasonable to ask NIST to follow public law.

jeffparsons(10000) 6 days ago [-]

I think this is a sloppy take. If you read the full back-and-forth on the FOI request between D.J. Bernstein and NIST, it becomes readily apparent that there is _something_ rotten in the state of NIST.

Now of course that doesn't necessarily mean that NIST's work is completely compromised by the NSA (even though it has been in the past), but there are other problems that are similarly serious. For example, if NIST is unable to explain how certain key decisions were made along the way to standardisation, and those decisions appear to go against what would be considered by prominent experts in the field as 'good practice', then NIST has a serious process problem. This is important work. It affects everyone in the world. And certain key parts of NIST's decision making process seem to be explained with not much more than a shrug. That's a problem.

silisili(10000) 6 days ago [-]

What's with the infighting here? Nothing about the post comes across as conspiracy theory level or reputation ruining. It makes me question the motives of those implying he's crazy, to be honest.

jacooper(10000) 6 days ago [-]

Man, mobile typos suck.

svnpenn(10000) 6 days ago [-]

Filippo Valsorda seems to be happy to ignore the fact that NIST already let an NSA backdoor in, as recently as 2014:


is he really just going to ignore something from 8 years ago?

dt3ft(10000) 6 days ago [-]

Perhaps the old advice ("never roll your own crypto") should be reevaluated? If you're creative enough, you could combine and apply existing algorithms in such ways that it would be very difficult to decrypt? Think 500 programmatic combinations (steps) of encryption applying different algorithms. Content encrypted in this way would require knowledge of the encryption sequence in order to execute the required steps in reverse. No amount of brute force could help here...

TobTobXX(10000) 6 days ago [-]

> Would require knowledge of the encryption sequence...

This is security by obscurity. Reputable encryptions work under the assumption that you have full knowledge about the encryption/decryption process.

You could however argue that the sequence then becomes part of the key. However, this key [ie. the sequence of encryptions] would then be at most as strong as the strongest encryption in this sequence, which kindof defeats the purpose.

Tainnor(10000) 6 days ago [-]

No, an important property of a secure cryptographic cipher is that it should be as close to a random permutation of the input as possible.

A 'randomly assembled' cipher that just chains together different primitives without much thought is very unlikely to have that, which will mean that it will probably have 'interesting' statistical properties that can be observed given enough plaintext/ciphertext pairs, and those can then be exploited in order to break it.

anfilt(10000) 6 days ago [-]

No not at all, that advice is still good. Even more important if your are talking about modifying algorithms. Your gonna want proofs of resistance or immunity to certain classes of attacks. A subtle change can easily make a strong primitive useless.

thrway3344444(10000) 6 days ago [-]

Why is the link in the URL http: not https: ? Irony?

cosmiccatnap(10000) 6 days ago [-]

If you spend all day making bagels do you go home and make bagels for dinner?

It's a static text blog, not a bank

sam0x17(10000) 6 days ago [-]

Well https uses the NIST standards so.... ;)

creatonez(10000) 5 days ago [-]

This is just due to the way that the OP posted it, not how it was originally published. This website forces HTTPS using ChaCha20-Poly1305 standard.

efitz(10000) 6 days ago [-]

Why don't we invert FOIA?

Why don't we require that all internal communications and records be public, available within 24 hours on the web, and provide a very painful mechanism involving significant personal effort of high level employees for every single communication or document that is to be redacted in some way? The key is requiring manual, personal (non-delegatable) effort on the part of senior bureaucrats, and to allow a private cause of action for citizens and waiver of immunity for bureaucrats.

We could carve out (or maybe not) specific things like allowing automatic redaction of employee PII and PII of citizens receiving government benefits.

After many decades, it's clear that the current approach to FOIA and sunshine laws just isn't working.

[ed] fixed autocorrect error

gorgoiler(10000) 6 days ago [-]

The old Abe rhetoric was powerful but it always felt like it was only hitting home on two of the three points. Obviously government, by definition really, is of the people. The much better parts were for the people and by the people.

chaps(10000) 5 days ago [-]

The carve-out you mention is a decent idea on paper, but in practice is a difficult process. There's really no way to do it in any significant degree without basically putting all gov to a complete halt. Consider that government is not staffed with technical people, nor necessarily critically minded people to implement these systems.

There are ways to push for FOIA improvements that don't require this sort of drastic approach. Problem is, it takes a lot of effort on the parts of FOIA requesters, through litigation and change in the laws. Things get surprisingly nuanced when you really get down into what a 'record' is, specifically for digital information. I definitely wouldn't want to have 'data' open by default in this manner, because it would lead to privacy hell.

Another component of this all is to consider contractors and subcontractors. Would they fall under this? If so, to what degree? If not, how do we prevent laundering of information through contractors/subcontractors?

To a large degree, a lot of 'positive' transparency movements like the one you suggest can ironically lead to reduced transparency in some of the more critical sides of transparency. A good example of that is 'open data', which gives an appearance of providing complete data, but without the legal requirements to enforce it. Makes gov look good but it de-incentivizes transparency pushback and there's little way to identify whether all relevant information is truly exposed. I would imagine similar would happen here.

elif(10000) 6 days ago [-]

Perhaps the best way to build trust in a cryptographic algorithm is to have it devised by certifiably neutral general purpose mathematic neural net.

It could even generate an algorithm so complicated it would be close to impossible for a human mind to comprehend the depth of it.

creatonez(10000) 5 days ago [-]

> It could even generate an algorithm so complicated it would be close to impossible for a human mind to comprehend the depth of it.

Okay... then some nefarious actor's above-human-intelligence neural network instantly decodes the algorithm deemed too complicated for human understanding?

I don't see how opaque neural nets are suddenly going to make security-through-obscurity work.

tooltower(10000) 6 days ago [-]

'Certifiably neutral'

So, by a process that hasn't been designed yet. Especially when one considers how opaque most neutral nets are to human scrutiny.

pyuser583(10000) 6 days ago [-]

Please include links with https://

oittaa(10000) 6 days ago [-]

NSA employees downvoted this?

tptacek(10000) 6 days ago [-]

I may believe almost all of this is overblown and silly, as like a matter of cryptographic research, but I'll say that Matt Topic and Merrick Wayne are the real deal, legit the lawyers you want working on something like this, and if they're involved, presumably some good will come out of the whole thing.

Matt Topic is probably best known as the FOIA attorney who got the Laquan McDonald videos released in Chicago; I've been peripherally involved in some work he and Merrick Wayne did for a friend, in a pretty technical case that got fierce resistance from CPD, and those two were on point. Whatever else you'd say about Bernstein here, he knows how to pick a FOIA lawyer.

A maybe more useful way to say the same thing is: if Matt Topic and Merrick Wayne are filing this complaint, you should probably put your money on them having NIST dead-to-rights with the FOIA process stuff.

daneel_w(10000) 6 days ago [-]

> 'I may believe almost all of this is overblown and silly, as like a matter of cryptographic research ...'

Am I misunderstanding you, or are you saying that you believe almost all of DJB's statements claiming that NIST/NSA is doctoring cryptography is overblown and silly? If that's the case, would you mind elaborating?

api(10000) 6 days ago [-]

I don't think it's a bad thing to push back and demand transparency. At the very least the pressure helps keep NIST honest. Keep reminding them over and over and over again about dual-EC and they're less likely to try stupid stuff like that again.

encryptluks2(10000) 6 days ago [-]

I have no doubt that they are great at their job, but when it comes to lawsuits the judge(s) are equally as important. You could get everything right but a judge has extreme power to interpret the law or even ignore it in select cases.

taliesinb(10000) 6 days ago [-]

Why is the submission URL using http instead of https? That just seems... bizarre.

sdwr(10000) 6 days ago [-]

Cryptography experts know when to care about security. Cryptography enthusiasts try to slap encryption on everything.

effie(10000) 6 days ago [-]

Why? Http is simpler, less fragile, not dependent on good will of third parties, the content is public, and proving authenticity of text on Internet is always hard, even when served via the https scheme. I bet Bernstein thinks there is little point in forcing people to use https to read his page.

msk20(10000) 6 days ago [-]

Just FYI, On my Firefox its saying 'Connection Secure (upgraded to https)', its actually using ECDHE CHACHA20 SHA256.

Note: I have 'Enable HTTPS-Only Mode in all windows' on by default.

ForHackernews(10000) 6 days ago [-]

Maybe this is too much tinfoil hattery, but are we sure DJB isn't a government asset? He'd be the perfect deep-cover agent.

rethinkpad(10000) 6 days ago [-]

Though 99% of the time I would agree with you, the public has to have faith in people who claim to be fighting (with previously noted successes in Bernstein v. US) in our best interests.

throwaway654329(10000) 6 days ago [-]

Please don't do the JTRIG thing. Dan is a national treasure and we would be lucky to have more people like him fighting for all of us.

Between the two, material evidence shows that NIST is the deep-cover agent sabotaging our cryptography.

crabbygrabby(10000) 6 days ago [-]

Seems like a baaad idea lol.

yieldcrv(10000) 6 days ago [-]

seems like they just need a judge to force the NSA to comply with a Freedom of Information Act request, its just part of the process

I'm stonewalled on an equivalent Public Record Act request w/ a state, and am kind of annoyed that I have to use the state's court system

Doesn't feel super partial and a couple law journals have written about how its not partial at all in this state and should be improved by the legislature

gruturo(10000) 6 days ago [-]

Yeah, terrible idea, except this is Daniel Bernstein, who already had an equally terrible idea years ago, and won. That victory was hugely important, it pretty much enabled much of what we use today (to be developed, exported, used without restrictions, etc etc etc)

zitterbewegung(10000) 6 days ago [-]

He won a case against the government representing himself so I think he would be on good footing. He is a professor where I graduated and even the faculty told me he was interesting to deal with. Post QC is his main focus right now and also he published curve25519.

josh2600(10000) 6 days ago [-]

I just want to say, the problem here is worldwide standards bodies for encryption need to be trustworthy. It is incredibly hard to know what encryption is actually real without a deep mathematics background and even then, a choir of peers must be able to present algorithms, and audits of those algorithms with a straight face.

Presenting broken-by-design encryption undermines public confidence in what should be one of our most sacrosanct institutions: the National Institute of Standards and Technology (NIST). Many enterprises do not possess the capability to audit these standards and will simply use whatever NIST recommends. The danger is that we could be engineering embedded systems which will be in use for decades which are not only viewable by the NSA (which you might be ok with depending on your political allegiance) but also likely viewable by any capable organization on earth (which you are probably not ok with irrespective of your political allegiance).

In short, we must have trustworthy cryptography standards. If we do not, bedlam will follow.

Please recall, the last lawsuit that DJB filed was the one that resulted in essentially 'Code is speech' in our world (https://en.wikipedia.org/wiki/Bernstein_v._United_States).

bananapub(10000) 6 days ago [-]

how could NIST possibly be 'one of our most sacrosanct institutions' after the NSA already fucked them with Dual_EC_DRBG?

whoever wants to recommend standards at any point since 2015 needs to be someone else

https://en.wikipedia.org/wiki/NIST_SP_800-90A for this who have forgotten.

tptacek(10000) 6 days ago [-]

There's an easier problem here, which is that our reliance on formal standards bodies for the selection of cryptography constructions is bad, and, not hardly just at NIST, has been over the last 20 years mostly a force for evil. One of the most important 'standards' in cryptography, the Noise Protocol Framework, will probably never be a formal standard. But on the flip side, no formal standards body is going to crud it up with nonsense.

So, no, I'd say that bedlam will not follow from a lack of trustworthy cryptography standards. We've trusted standards too much as it is.

mort96(10000) 6 days ago [-]

Weirdly, any time I've suggested that maaaybe being too trusting of a known bad actor which has repeatedly published intentionally weak cryptography is a bad idea, I've received a whole lot of push-back and downvotes here on this site.

throwaway654329(10000) 6 days ago [-]

Indeed. Have my upvote stranger.

The related "just ignore NIST" crowd is intentionally or unintentionally dismissing serious issues of governance. Anyone who deploys this argument is questionable in my mind, essentially bad faith actors, especially when the topic is about the problems brought to the table by NIST and NSA.

It is a good sign that those people are actively ignoring the areas where you have no choice and you must have your data processed by a party required to deploy FIPS certified software or hardware.

morpheuskafka(10000) 6 days ago [-]

I'm working on a project that involves a customized version of some unclassified, non-intelligence software for a defense customer at my job (not my ideal choice of market, but it wasn't weapons so okay with it). Some of the people on the project come from the deeper end of that industry, with several TS/SCI contract and IC jobs on their resumes.

We were looking over some errors on the sshd log and it was saying it couldn't find the id_ed25519 server cert. I remarked that that line must have stayed even though the system was put in FIPS mode which probably only allowed the NIST-approved ECC curve and related this story, how everyone else has moved over to ed25519 and the government is the only one left using their broken algorithm.

One of the IC background guys (who is a very nice person, nothing against them) basically said, yeah the NSA used to do all sorts of stuff that was a bad idea, mentioning the Clipper chip, etc. What blew my mind is that they seemed to totally have reasonable beliefs about government surveillance and powers, but then when it comes to someone like Snowden, thinks their are a traitor and should have used the internal channels instead of leaking. I just don't understand how they think those same people who run NSA would have cared one bit, or didn't know about it already. I always assumed the people that worked in the IC would just think all this stuff was OK to begin with I guess.

I don't know what the takeaway is from that, it just seems like a huge cognitive dissonance.

616c(10000) 6 days ago [-]

Another upvote from someone with many friends and colleagues in NIST. I hope transparency prevails and NISTers side with that urge as well (I suspect many do).

glitchc(10000) 6 days ago [-]

Many government or government affiliated organizations are required to comply with NIST approved algorithms by regulation or for interoperability. If NIST cannot be trusted as a reputable source it leaves those organizations in limbo. They are not equipped to roll their own crypto and even if they did, it would be a disaster.

dataflow(10000) 6 days ago [-]

Tangential question: while some FOIA requests do get stonewalled, I continue to be fascinated that they're honored in other cases. What exactly prevents the government from stonewalling practically every request that it doesn't like, until and unless it's ordered by a court to comply? Is there any sort of penalty for their noncompliance?

Tangential to the tangent: is there any reason to believe FOIA won't be on the chopping block in a future Congress? Do the majority of voters even know (let alone care enough) about it to hold their representatives accountable if they try to repeal it?

linuxandrew(10000) 6 days ago [-]

I know someone who works in gov (Australia, not US) who told me all about a FOI request that he was stonewalling. From memory, the request was open ended and would have revealed more than it possibly intended it to, and would have revealed some proprietary trade secrets from a third party contractor. That said, it was probably a case that would attract some public interest.

The biggest factors preventing governments from stonewalling every FOI case are generally time and money. Fighting FOI cases is time consuming and expensive and it's simply easier to hand over the information.

Panzer04(10000) 6 days ago [-]

Presumably most government employees are acting in good faith - why wouldn't they fulfil a reasonable FOIA request?

This is likely the result of some actors not acting in good faith, and so have no choice but to stonewall lest their intransigence be revealed.

xiphias2(10000) 6 days ago [-]

An interesting thing that is happening on Bitcoin mailing list is that although it would be quite easy to add Lamport signatures as an extra safety feature for high value transactions, as they would be quite expensive and easy to misuse (they can be used only once, which is a problem if money is sent to the same address twice), the current concensus between developers is to ,,just wait for NSA/NIST to be ready with the algorithm''. I haven't seen any discussion on the possibility of never being ready on purpose because of a sabotage.

potatototoo99(10000) 6 days ago [-]

Why not start that discussion yourself?

jack_pp(10000) 6 days ago [-]

Indeed as potato said, link this article in the ML for them to see that NIST can not be fully trusted

Historical Discussions: To uncover a deepfake video call, ask the caller to turn sideways (August 08, 2022: 854 points)

(855) To uncover a deepfake video call, ask the caller to turn sideways

855 points 3 days ago by Hard_Space in 10000th position

metaphysic.ai | Estimated reading time – 4 minutes | comments | anchor

However, none of the accompanying videos show the subject in acute profile. We asked Sensity's CEO and Chief Scientist Giorgio Patrini if the experiments and tests included the subject making 90° turns from camera as part of the deception technique, and he confirmed that they did not.

We also asked him if he considers that there is any possible merit in soliciting profile views as an anti-deepfake measure during videoconferencing calls. Patrini responded:

'Lateral views of people faces, when used as a form of identity verification, may indeed provide some additional protection against deepfakes. As pointed out, the lack of widely available profile view data make the training of deepfake detector very challenging.

'Additionally, I'd argue that most state of the art deepfake software simply fails if applied for faceswapping or re-enacting faces fully rotated on their side. This is because deepfake software needs to accurately detect faces and their landmarks on the target video; profile views make this more difficult as the detector has to work with only half of the facial key points.

'Indeed, one tip for performing deepfake detection "by eye" today is to check whether one can spot face artefacts or flickering while a person is turning completely to their side — where it's more likely that a face landmarks detector would have failed.'

Further, we asked deepfake expert Dr. Siwei Lyu, Professor of Computer Science and Engineering at the University at Buffalo School of Engineering and Applied Sciences, if this 'lateral' approach potentially has any value in deepfake detection in a live video scenario. He agreed that it does:

'The profile is a big problem for current deepfake technologies. The FAN network works extremely well for frontal faces, but not very well for side-on faces.

'The lack of available data is certainly an obstacle. Another aspect is that these algorithms do have a fundamental limitation: the alignment mechanism works well if you cover only part of your face, and is quite robust in those circumstances – but when you turn around, more than half the landmarks are missing.

'Probably the best an algorithm can do is to roughly estimate the profile, particularly if the person is enacting various expressions, or taking requests from the other correspondent in a video call, or from an automated liveness detection system.

'In a case like that, profile estimation is going to be kind of a 'guess', even if you were to have some depth information from some of the more recent sensors in smartphones.'

However, Dr. Lyu believes that the emerging new generation of 3D landmark location systems could improve on FAN's performance (though FAN itself can also enact 3D landmarks, and is used by DeepFaceLab in this way for 'full head' pose capture), but notes that this would not solve the problem of the lack of profile data for 'non-famous' people who deepfake attackers might want to train into models for deceptive purposes in a videoconference scenario.

In the absence of high-quality profile images as source training input, Dr. Lyu does not feel that novel-view synthesis systems such as NeRF, Generative Adversarial Networks (GANs) and Signed Distance Fields (SDF) are likely to be able to provide the necessary level of inference and detail in order to accurately imitate a person's profile views – at least, to a level that could stand up to the reasonably high-resolution capabilities of modern smartphone cameras and laptop webcams, which represent the likeliest environments for a deepfake attack.

'The problem is that you would have to make up information on the basis of inadequate, estimated data. Perhaps you could use a GAN model and get something similar, but the data would not be likely to stand up to cross-checking on a user who has already been legitimately enrolled into a liveness detection system, for instance.'

All Comments: [-] | anchor

yarg(10000) 3 days ago [-]

The existence of GANs as a mechanism for learning is reason enough to err away from static detection methods.

yarg(10000) 3 days ago [-]

We can already resolve the interframe wobbles that happen due to the inability to preserve spatial invariants on static meshes allowing for dynamic camera motion.

Of course, the spatial invariants of meat-suits in motion require an understanding of volumetric structure, and not just restricted depth surface meshes.

But it's not some unencodable computational enigma.

3jckd(10000) 3 days ago [-]

Source: I work in the field.

This is a current limitation, and an artifact of the data+method but not something that should be relied upon.

If we do some adversary modelling, we can find two ways to work around this:

1) actively generate and search for such data; perhaps expensive for small actors but not well equipped malicious ones.

2) wait for deep learning to catch up, e.g. by extending NERFs (neural radiance fields) to faces; matter of time.

Now, if your company/government is on the bleeding edge of ML-based deception, they can have such policy, and they will update it 12-18-24 months (or whenever (1) or (2) materialises). However, I don't know one organisation that doesn't have some outdated security guideline that they cling to, e.g. old school password rules and rotations.

Will 'turning sideways to spot a deepfake' be a valid test in 5 years? Prolly no, so don't base your secops around this.

dylan604(10000) 3 days ago [-]

>Will 'turning sideways to spot a deepfake' be a valid test in 5 years? Prolly no, so don't base your secops around this.

We'll just ask them to do 'the Linda Blair'. If they can turn their head 360 degrees, prolly a deepfake ;P

stcredzero(10000) 3 days ago [-]

1) actively generate and search for such data

What about doing a bunch of video calls, and asking for callers to show their profile, 'to guard against deepfakes?'

cpach(10000) 3 days ago [-]

As far as I can see, secops is an eternal cat-and-mouse game.

dheera(10000) 3 days ago [-]

The other thing is, why is this even important, when you shouldn't be basing decisions off the other person's race or face in general?

Base everything off the work they do, not how they look. Embracing deepfakes is accepting that you don't discriminate on appearances.

Hell, everyone should systematically deepfake themselves into white males for interviews so that there is assured to be zero racial/gender bias in the interview process.

simonswords82(10000) 3 days ago [-]

Interesting to bump in to somebody that works in this field.

What do you do in this field?

What's the direction of travel on it?

What makes it worth pursuing at a commercial level? In other words - how is this tech going to be abused/monetized?

bushbaba(10000) 3 days ago [-]

Asking for entropy that's easy for a real human to comply with and difficult for a prebuilt AI is at least a short term measure. Such-as show me the back of your head sideways then go from head to feet without cutting the feed.

Easy for a human, difficult for ML/AI

elondaits(10000) 3 days ago [-]

"The Impossible Mission Force has the technical capabilities to copy anyone's face and imitate their voice, so don't base your secops around someone's appearance."

... yes, because that worked well.

wildmanx(10000) 3 days ago [-]

It saddens me how many smart people are working in such an unethical field.

SoftTalker(10000) 3 days ago [-]

Last time I applied for a credit card online, they asked me to take a video of myself and turn my head from side to side.

kazinator(10000) 3 days ago [-]

OK, you passed the yokogao test. Now take a crayon and draw an X on your cheek.

Strom(10000) 3 days ago [-]

> This is a current limitation

The thing with any AI/ML tech is that current limitations are always underplayed by proponents. Self-driving cars will come out next year, every year.

I'd say that until the tech actually exists, this is a great way to detect live deepfakes. Not using the technique just because maybe sometime in the future it won't work isn't very sound.

For an extreme opponent you may need additional steps. So this sideways trick probably isn't enough for CIA or whatnot, but that's about as fringe as you can get and very little generic advice applies anyway.

kortex(10000) 3 days ago [-]

What about reflections? When I worked on media forensics, the reflection discrepancy detector worked extremely well, but was very situational, as pictures were not guaranteed to have enough of a reflection to analyze.

Asking the subject to hold up a mirror and move it around pushes the matte and inpainting problems to a whole nother level (though it may require automated analysis to detect the discrepancies).

I think that too might be spoofable given enough time and data. Maybe we could have complex optical trains (reflection, distortion, chromatic aberration), possibly even one that modulates in real time...this kind of just devolves into a Byzantine generals problem. Data coming from an untrusted pipe just fundamentally isn't trustable.

mrandish(10000) 3 days ago [-]

> so don't base your secops around this.

If it's a high-threat context I don't think live video should be relied on regardless of deep fakes. Bribing or coercing the person is always an alternative when the stakes are high.

hugobitola(10000) 3 days ago [-]

What if the real person draws something on his face? Does the deepfake algorithm removes it from the resulting image? Can you ask the caller to draw a line on his face with a pen as a test?

peoplefromibiza(10000) 3 days ago [-]

> Will 'turning sideways to spot a deepfake' be a valid test in 5 years? Prolly no, so don't base your secops around this.

couldn't the same thing be said about passwords, 2FA with SMS or asymmetric cryptography?

meanwhile real IDs have been easy to replicate for decades, but are still good enough for the job.

tmm84(10000) 3 days ago [-]

I was thinking of those two cases. Stuff like this is always a cat and mouse game.

neximo64(10000) 3 days ago [-]

But currently, it's pretty much a guarantee that you can pick out a deepfake with this method as there is no way for current methods to account for it that are in use.

As with any interaction with more than one adversary, there is an infinite escalation and evolution with time. And similarly then something will come up then that is unaccounted for and so on, and so on.

WalterBright(10000) 3 days ago [-]

I wonder how good the deepfake would be for things it didn't have training data on. For example, making an extreme grimace. Or have the caller insert a ping pong ball in his cheek to continue, or pull his face with his fingers.

One thing I notice with colorized movies is the color of the actor's teeth tends to flicker between grey and ivory. I wonder if there are similar artifacts with deep fakes.

roessland(10000) 3 days ago [-]

Might be a great article but I had to stop reading since I couldn't bear the scroll hijacking.

budafish(10000) 3 days ago [-]

100% agree. Made me feel a bit nauseous.

mdp2021(10000) 3 days ago [-]

No issue here. It appears your system allows it.

nominusllc(10000) 3 days ago [-]

I did not experience this, my system doesn't allow it

vrecan(10000) 3 days ago [-]

Agreed, as soon as I scrolled once and I noticed it I was gone.

jwilk(10000) 3 days ago [-]

https://archive.today/6Dis6 may work better.

JohnJamesRambo(10000) 3 days ago [-]

Is audio harder to fake than video? I was watching the Keanu one and wondered if it is harder to real time fake Keanu's voice than his face?

0xedd(10000) 3 days ago [-]

No. Both face the same challenge - quality of data. The rest has already been solved.

12ian34(10000) 3 days ago [-]

Is this to be an empathy test? Capillary dilation of the so-called blush response, fluctuation of the pupil, involuntary dilation of the iris?

JorgeGT(10000) 3 days ago [-]

We call it Voight-Kampff for short.

bobkazamakis(10000) 3 days ago [-]

shoe on head

eesmith(10000) 3 days ago [-]

Vermin Supreme, the leader in the fight against deepfakes. https://en.wikipedia.org/wiki/Vermin_Supreme

isusmelj(10000) 3 days ago [-]

Deepfake models are trained on very similar data. They don't generalize well, usually. E.g. we take lots of data from YouTube videos of a single person under a specific condition (same time, same day, same haircut etc.) I know that as I spent quite some time researching these models and worked on a deepfake detection startup. Purely looking at it from a technological side, it's a cat mouse game. Similar to an antivirus software. A new method appears to create deepfakes. A new detection method is required.

However, we can also make use of the models to not properly generalize and their limitations of the training process. Anything that is out of distribution (very rare occurrence in training data) will be hard for the model: - blinking (if the model has ever only seen single frames it will create rather random unusual blinking behavior - turn around (as mentioned by the author, side views are rarer in the web) - take off your glasses - slap your cheek - draw something on your cheek - take scissors and cut a piece of your hair

The last two would be especially difficult and funny (:

cypress66(10000) 3 days ago [-]

Looking at how fast dall-e is improving, and how it 'understands' concepts even if you mix them in crazy ways, all of your later examples seem solvable in less than a decade.

But I don't know much about ML so I might be wrong.

eckza(10000) 3 days ago [-]

> Put shoe on head

diydsp(10000) 3 days ago [-]

8/9/2022: To prevent against uncovering, train your models to generate sideviews.

tommoor(10000) 3 days ago [-]

Yes, but one of the points of the article is a distinct lack of source material to train models on profile views

david_draco(10000) 3 days ago [-]

'profile view challenge' coming in 3, 2, 1 ...

basilgohar(10000) 3 days ago [-]

It's probably not obvious to many that there's nearly a limitless source of training data on social media at this point. Your comment is eerily prescient and now all trends can become suspect as being a plant for additional training to circumvent, well, known circumventions!

florbo(10000) 3 days ago [-]

multiple pan angle 360 arc shot challenge

schroeding(10000) 3 days ago [-]

'Hey! To make sure you stay secure, we require a short video. Please look straight into the camera and tap the screen.'

'You look great! We just need you to blink 5 times, and you're almost done!'

'Almost done! Just show us your best side and turn your head to the left like shown above.'

'Of course, you only have best sides. Just turn your head to the right like displayed above, and we can continue.'

'You've almost got it! Please open your mouth and show us your teeth.'

'Wow, look at you go! Just one step remaining: Tilt your head to the right like shown above.'

'Now, to complete your verification, hold your national ID beside your face. Make sure it does not obstruct your head! We need to be able to see your pretty face!'

(Tongue in cheek, of course. But my banking app actually uses this kind of language, even for verification stuff, and I don't like it :D)

Traubenfuchs(10000) 3 days ago [-]

Show your left side like this and your right side like this and let others comment which side looks prettier OwO.

DenisM(10000) 3 days ago [-]

Patiently waiting for the government(s) to step in and start providing a modern ID service - a driver license with a built in private key, a fingerprint unlock, and a PIN.

The combination of the three can still be defeated by someone following you, stealing the card, lifting fingerprint from a glass, and spying the PIN, but that's a lot of trouble to go through and online identity fraud will become extinct.

xmprt(10000) 3 days ago [-]

> a driver license with a built in private key

IDs should never be used as secrets. That's like mixing up your username and password.

orthoxerox(10000) 3 days ago [-]

Or by someone kidnapping you and applying a rubber hose to your kidneys until you tell them the PIN.

londons_explore(10000) 3 days ago [-]

Slightly more robust method...

Ask the caller to move out of the frame and then back in again.

You will see a noticable 'step' as the face that is partially in the frame suddenly gets detected as a face and the deepfake is applied.

The only way around this is to crop the input video quite heavily - by at least one face diameter, which is a lot if the user is near the camera.

robocat(10000) 3 days ago [-]

Or pass a piece of paper or splayed fingered hand slowly in front of your face?

mkl(10000) 3 days ago [-]

I don't think that's very robust. The entire image could easily be fake, as the face is fake, and people already have fake backgrounds. If the entire image is fake, there's no reason for the actual camera's view to match the fake image, so a wider angle camera would keep you in view as you move, and the system could generate a tighter fake view.

IshKebab(10000) 3 days ago [-]

> The only way around this is to crop the input video quite heavily

I mean that sounds a lot easier than making deep fakes work well with profile data surely?

ape4(10000) 3 days ago [-]

Also if Nicolas Cage is calling me its probably fake.

rexreed(10000) 3 days ago [-]

What if it's your CEO? Or someone from the bank? Or a college professor? Or a political prisoner?

wakahiu(10000) 3 days ago [-]

I was recently looking for designers for my company when I came across an interesting profile on Dribbble. I reached out and quickly scheduled a time when we could talk over zoom. At the meeting time, in comes this person who seems to have a strange-looking, silicone-like face. I was using my Zoom account (I rarely use other peoples zooms unless I trust them), to avoid situations like this. One thing I noticed is that when the candidate touched their face, their fingers would appear to sink into their skin - almost as if it were made of liquid. Secondly, their face appeared larger, lighter and smoother than their neck. I got spooked an immediately let the candidate know that I was not comfortable moving forward.

More interestingly, what exactly are them mechanics of getting a deep fake into video call? How is it possible that a what seems like a deepfake could make its way into my Zoom? Is Zoom enabling external plugins that alter video details?


simoncion(10000) 3 days ago [-]

I do very much hope that you told the candidate what spooked you. Ideally, you would have done this early in the interview, giving them a chance to disable any video filtering / face-beautifying software that they may have been running.

If you didn't do either one of those, perhaps you now know enough so that next time you will be able to give the interviewee a chance to demonstrate whether or not they're using a 'Smooth over my facial blemishes because I'm uncomfortable with how my face looks and want it to look 'prettier'.' filter.

Best of luck with your interviews!

valarauko(10000) 3 days ago [-]

For what it's worth, it looks more like an aggressive filter rather than a deepfake.

EliotBee(10000) 3 days ago [-]

Things like OBS (streaming software) can create a virtual camera. I am guessing its something like that where Zoom does not even know the camera is not actually real hardware.

PullJosh(10000) 3 days ago [-]

The live-streaming software OBS has a "virtual webcam" feature that can make a generated video feed behave like a hardware webcam. Perhaps something similar is being used to feed generated video into zoom?

0xedd(10000) 3 days ago [-]

Input for software can be anything. Camera feed can be a generated one and the software consuming it doesn't have to be aware it isn't a real physical camera.

Zoom isn't aware.

thrashh(10000) 3 days ago [-]

You can just make Zoom use any webcam on your system

And you can write your own webcam drivers to use in any program

Or use existing software with virtual webcam output like OBS or ManyCam and write a plug-in for that

Our emit a network video stream and just play your video in that kind of software instead of writing a plugin

Benjammer(10000) 3 days ago [-]

It's fairly trivial to have a virtual camera source and point Zoom to that as it's input. It has nothing to do with integrating deeply with Zoom or getting "into" your Zoom. Check out Snap Camera[0] for an example.

[0] https://snapcamera.snapchat.com/

mathverse(10000) 3 days ago [-]

Out of curiosity was that person asian?

Maybe they were just using some of those beautifying filters like chinese streamers do.

jjk166(10000) 3 days ago [-]

Probably a more robust test would be asking the caller to run their hand through their hair a few times. Maybe you could pre-render a few samples, but it would be trivial to request the person pass their hands through their hair in a specific way, or simply do it again after their hair is already messed up a bit from the first time. It could still be defeated by the caller having the same hair style (or wearing a good wig) as the person they are imitating, but then making someone look like someone else with practical effects has been a thing forever and it has not been a huge problem.

robocat(10000) 3 days ago [-]

> run their hand through their hair

That would have trouble passing anti-discrimination requirements: disability (no hands), medical (bandanna covering cancer treatment hair loss), religious (burka, rasta, yarmulke, sheitel), racist (cornrow).

And trouble with: dreadlocks (can't run fingers through), bald headed guys (as mentioned by sibling comment), and people with hairdo's (coiffure, hairspray, topknots, plaits etcetera).

0xJRS(10000) 3 days ago [-]

in my career i've personally worked with no less than half a dozen bald coworkers. i do think this is a good idea but won't work for everyone

shiftpgdn(10000) 3 days ago [-]

Tangentially related but a simple way to bust a chatbot is to ask "What is larger, the Eiffel Tower or a shoe box?"

BitwiseFool(10000) 3 days ago [-]

I think you're on to something. The modern day chat-bot/answer engines seem very susceptible towards trying to answer fact-based, yet obviously incorrect questions. They seem unable to parse the entire question and instead focus on the most generic terms. For instance, the 'What year did Neil Armstrong land on Mars?' example that shows up on HN from time to time.

elicash(10000) 3 days ago [-]

Here's what Meta's blenderbot replied with: 'The Tokyo tower is taller than the eiffel tower. Interesting facts like that interest me. Do you know about it?'

hotpotamus(10000) 3 days ago [-]

I remember someone posting a chat thread from one of the more advanced AIs within the last few years wherein they asked it who the president of the US is, and it was not able to answer.

Interestingly, this is a question my father would ask patients as a paramedic who was trying to assess people's consciousness. Another would be, 'what day of the week is it?'.

I'd say that these technologies are just like magic - they can seem to do things that defy your expectations, but oftentimes they fall apart when looked at from a different angle.

vivegi(10000) 3 days ago [-]

Hybrid Video/audio/semantic Captcha, perhaps?

An audio prompt like 'Using your <right | left> hand, repeat the numbers that I am signaling. Use <a different | the same> set of fingers from what I am using'.

Scoundreller(10000) 3 days ago [-]

This is why I got really upset when my employer said the swimsuit competition segment of the interview was past its time. Its time is now!

mike_hearn(10000) 3 days ago [-]

Long term, the only robust way to solve this is going to involve a remote attestation chain i.e. video that's being signed by the web cam as it's produced, and then transformed/recompressed inside e.g. SGX enclaves or an SEV protected virtual machine that's sending an RA to the other side. Although hard to set up (you need a lot of people to cooperate and CPU vendors have to bring these features back to consumer hardware), it has a lot of advantages over what you might call trick-based approaches:

1. Robust to AI improvements.

2. Blocks all kinds of faking and tampering, not just deepfakes.

3. With a bit of work can securely timestamp the video such that it can become evidence useful for dispute resolution.

4. Also applies to audio.

5. Works in the static/offline scenario where you just get a video file and have to check it.

There are probably other advantages too. The way to do such things has been known about for a long time. The issue is not any missing pieces of tech but simply building a consensus amongst hardware vendors that there's actual market demand for [deep]fake-proof IO.

In reality, deepfakes have been around for some years now but have there been any reports of actual real world attacks using them? Not sure, I didn't hear of any but maybe there's been one or two. Problem is, that's not enough to sustain a market. Attacks have to become pretty common before it's worth throwing anything more than cheap heuristics at it.

feanaro(10000) 3 days ago [-]

The solution you propose sounds vastly overengineered. Why would we need remote attestation, tampering resistance and enclaves when this is simply a problem of your peers being unauthenticated?

If you care about the identity of who you are speaking to remotely, the only solution is to cryptographically verify the other end, which just requires plain old key distribution and verification. It's just not widespread enough today for videocalls because up to now, there wasn't much need for this.

IMSAI8080(10000) 3 days ago [-]

I think it would be useful if news outlets signed their video content using watermarking techniques. Then social media sites where news is shared could automatically check for recognised signatures for major outlets and give it a checkmark or something. The signature could be easily removed but video without the checkmark would then be suspicious. It would also be useful if they added signed timecodes to frames so it could be checked if the video has been edited.

jedberg(10000) 3 days ago [-]

The only solution will be in person meetings, as it has always been. Faking audio has been around a really long time. If you needed to be absolutely sure the person you're talking to is legit, you met them in person (mission impossible style disguises not withstanding).

Nothing has really changed with deepfake, other than the fact that for a brief period we could be sure the person we were having a video chat with was legit because the tech didn't exist to fake it.

ballenf(10000) 3 days ago [-]

Then you just point the webcam at a screen or microphone at a speaker?

I really don't think moving our trust to unknown, unnamed manufacturers of hardware in far away places is a solution.

The solution is not going to be high tech, imho. Just like we have learned a skepticism resulting from Photoshop, we'll learn a skepticism of live video or audio.

xen2xen1(10000) 3 days ago [-]

So your answer is .. more DRM?

Starlevel001(10000) 3 days ago [-]

Applying technological solutions to social problems hasn't worked a single time before, but SURELY it'll work this time

xyzal(10000) 3 days ago [-]

I wonder how much work would it entail to swap one actor's face for another's in a movie. Just finished watching Fury Road, and Tom Hardy just feels a bit off to me.

bsenftner(10000) 3 days ago [-]

That's 'bread and butter' work in VFX. I used to be a stunt double actor replacement specialist. These daze, ML enhanced tools make the work for a face replacement shot exponentially faster and easier - as is needed for the huge number of superhero stunts insurance companies will not let the stars perform.

Arkadin(10000) 3 days ago [-]

Why not just use the standard Voight-Kampff test?

aqw137(10000) 3 days ago [-]

it would be good if we could just ask to look up and to the left

neogodless(10000) 3 days ago [-]

The pitfalls have been thoroughly documented.

night-rider(10000) 3 days ago [-]

Signed up for one of those 'neobanks' (that don't have physical branches) and part of the signup required me to turn my head sideways. I wondered why they wanted me to do that. Now I know.

InCityDreams(10000) 3 days ago [-]

Thanks for contributing to the dataset.


anonu(10000) 3 days ago [-]

Is there a 'client side' way to detect this? Similar to how we can detect photoshopped still images: checking edges, shadows, pixels, etc...

The benefit is you would not have to rely on issuing commands to the remote party.

kortex(10000) 3 days ago [-]

Media forensics algorithms do work on various forms of rebroadcast, transmission, and compression, so yes this should be possible (for now). look up darpa medifor project. Siwei Lyu (in the article) did a bunch of work in this space. Also see Hany Farid and Shruti Agarwal. They've worked specifically with deep fake detection.


bearjaws(10000) 3 days ago [-]

Serendipitously over the weekend I was thinking about a future where for key sensitive data access (e.g. production main) you may need to have a quick 5 minute call (4th factor, '3D verification') where you would be asked to turn on your camera and be asked to answer some simple question and in different positions...

Main thinking was how out of control it would get, it would probably end up looking like anti-cheat systems where its a constant cat and mouse game due to growing sophistication of deep fake models.

function_seven(10000) 3 days ago [-]

> Main thinking was how out of control it would get...

From a job listing, circa 2024:

- Job may require occasional lifting. (No more than 20kg)

- Expected to travel up to 25% of the year.

- Proprietary access control requires users be able to do handstands and/or simple juggling. (Feats subject to change)

- EEOC employer.

BHSPitMonkey(10000) 3 days ago [-]

This is already a thing multiple vendors provide, e.g. https://www.idnow.io/products/idnow-videoident/ (used for use cases like banking and electronic contract signing)

phpnode(10000) 3 days ago [-]

in the UK this is already relatively common for online banking. I asked my bank to raise my daily transfer limit the other day for a property purchase and part of the process was recording a video of myself in their app.

comboy(10000) 3 days ago [-]

We have PK cryptography you know, yubikeys and such.

t_mann(10000) 3 days ago [-]

Omg, looking forward to Yoga-based Captchas in the future: That ain't looking like a proper downward dog to me, pal, no access for you

amelius(10000) 3 days ago [-]

Or this:


> On receipt of the form, we will require a photograph of you, or a trusted representative as proof of identity. You will have to get a NEW photograph taken, holding two symbol of ours. The two symbols we need you to hold are a loaf of BREAD and a FISH (the name of our church). This proves that the person in the photograph is genuine. Passport or other photographs will NOT be accepted.

> (...)

> As dumb as he looks, I'm not happy. I asked for the fish to be on his head AND a loaf of bread. I got neither!

elygre(10000) 3 days ago [-]

No no... this does look like a proper downward dog, but no way you could do that!

TheAceOfHearts(10000) 3 days ago [-]

Please drink verification can.


Historical Discussions: An incident impacting 5M accounts and private information on Twitter (August 09, 2022: 792 points)
An incident impacting some accounts and private information on Twitter (August 06, 2022: 2 points)
An incident impacting some accounts and private information on Twitter (August 05, 2022: 1 points)

(792) An incident impacting 5M accounts and private information on Twitter

792 points 2 days ago by WaitWaitWha in 10000th position

privacy.twitter.com | Estimated reading time – 3 minutes | comments | anchor

We want to let you know about a vulnerability that allowed someone to enter a phone number or email address into the log-in flow in the attempt to learn if that information was tied to an existing Twitter account, and if so, which specific account. We take our responsibility to protect your privacy very seriously and it is unfortunate that this happened. While there's no action for you to take specific to this issue, we want to share more about what happened, the steps we've taken, and some best practices for keeping your account secure.

What happened

In January 2022, we received a report through our bug bounty program of a vulnerability in Twitter's systems. As a result of the vulnerability, if someone submitted an email address or phone number to Twitter's systems, Twitter's systems would tell the person what Twitter account the submitted email addresses or phone number was associated with, if any. This bug resulted from an update to our code in June 2021. When we learned about this, we immediately investigated and fixed it. At that time, we had no evidence to suggest someone had taken advantage of the vulnerability.

In July 2022, we learned through a press report that someone had potentially leveraged this and was offering to sell the information they had compiled. After reviewing a sample of the available data for sale, we confirmed that a bad actor had taken advantage of the issue before it was addressed.

We will be directly notifying the account owners we can confirm were affected by this issue. We are publishing this update because we aren't able to confirm every account that was potentially impacted, and are particularly mindful of people with pseudonymous accounts who can be targeted by state or other actors.

How to Protect Your Account

If you operate a pseudonymous Twitter account, we understand the risks an incident like this can introduce and deeply regret that this happened. To keep your identity as veiled as possible, we recommend not adding a publicly known phone number or email address to your Twitter account.

While no passwords were exposed, we encourage everyone who uses Twitter to enable 2-factor authentication using authentication apps or hardware security keys to protect your account from unauthorized logins. If you're concerned about the safety of your account, or have any questions about how we protect your personal information, you can reach out to our Office of Data protection through this form.

To learn more about reporting a security vulnerability, visit our Help Center, and to learn more about our efforts to protect Twitter from platform manipulation and state-backed activity, visit the Twitter Transparency Report.

All Comments: [-] | anchor

londons_explore(10000) 2 days ago [-]

Remember that phone numbers are only 10 digits long, so brute forcing all phone numbers is totally doable.

Considering that, if you implement any flow that involves checking if a phone number is already in use, then you are effectively leaking to an attacker a list of every phone number that uses your product.

WaitWaitWha(10000) 2 days ago [-]

In the USA.

They range from 4 (St. Helena) to 13 (Austria), I believe.

quickthrower2(10000) 2 days ago [-]

Maybe they should store salted hashes of phone numbers.

The purposes of phone numbers:

1. Verify you are a not a bot: no need to store anything except TRUE once verified.

2. 2FA - well use something better than SMS, but if you must, store the hash, and make me enter my number for the 2FA each time. Compare with hash and then send SMS.

sbf501(10000) 2 days ago [-]

CPU throughput =/= endpoint throughput

drusepth(10000) 2 days ago [-]

It's interesting to wonder why only 5M accounts were affected by this exploit, especially if it's brute forceable. IIRC this vulnerability was widely known about for at least months before it was fixed, so I can't imagine nobody in the know had access to the resources/botnets necessary to enumerate through every account.

Have only 5M accounts linked their phone numbers on Twitter? That's less than 2% of their total accounts (~290M). I don't know what the industry average is for linking phone numbers, but this seems like an exceptionally low ratio.

dncornholio(10000) 1 day ago [-]

already doable with e-mail addresses. doing this with just a phone number is not really a problem. It is a problem when you can link the phone and email. But discovering a phonenumber in itself is nothing more then pressing random numbers and see who answers?

hunter2_(10000) 2 days ago [-]

Rate limiting should be used to mitigate this, although I suppose a botnet could overcome that to some extent proportional to the size of the botnet.

And for anyone who didn't read TFA, this incident goes well beyond leaking what phone numbers use the product, it leaked the usernames associated with each as well.

unethical_ban(10000) 2 days ago [-]

It's a solved problems that you never confirm or deny the registration of an identity (like email or phone) for your service.

Bad login? 'Not a valid user/pass combo'

Password recovery? No matter what email or phone provided, simply say 'If the email matches our records, we will send a recovery link'.

moretestaccoun(10000) 2 days ago [-]

Another reminder not to use Twitter. It's not worth it. Mastodon is better.

keepquestioning(10000) 2 days ago [-]

Could not have picked a worse name for a social network.

whywhywhywhy(10000) 2 days ago [-]

Pretty disgusting they don't have a thing to check if they leaked my personal information, which lets not forget they screamed and stamped their feet to force me to hand over in the first place.

I never wanted to give you my phone number, Twitter. You demanded it.

mellosouls(10000) 2 days ago [-]

Pretty disgusting they don't have a thing to check if they leaked my personal information

From the linked notice, fwiw: 'We will be directly notifying the account owners we can confirm were affected by this issue.'

NaturalPhallacy(10000) 2 days ago [-]

>To keep your identity as veiled as possible, we recommend not adding a publicly known phone number or email address to your Twitter account.

And yet they actually demanded I give them mine, and have repeatedly, recently demanded a confirmation.

Phone numbers are one of the worst 2fas.

klabb3(10000) 2 days ago [-]

> To keep your identity as veiled as possible, we recommend not adding a publicly known phone number or email address to your Twitter account.

I had to look up whether this was actually official communication, since it sounds like a kafkaesque fever dream, but yes it's real.

Tech CO's have been doing everything in their power to get your number and email, used it for advertising, and deliberately disabled non-regular phone numbers. And now suddenly you're being gaslit that it's your fault for complying with their demands.

The cream of the cake is the vague 'if your phone number is publicly known' stuff. Well yeah, every single phone number is publicly known because it's enumerable. Even if it weren't, almost everyone's number is harvested and resold by gray-market data brokers. Sounds like they want to muddy the waters and make it sound like a targeted vulnerability when in reality it is indiscriminate.

hot_gril(10000) 2 days ago [-]

Also, you can't really use Twitter with just an email. Sooner or later, anti-bot misidentifies you, locks you out, and asks you to 'verify' by entering a phone number.

tehbeard(10000) 2 days ago [-]

It's gonna be fun when this happens to Microsoft.

They recently (early this year) onboarded a few million kids with the Minecraft account migration, and a lot of those new accounts will have flagged as 'suspicious activity' and demanded a mobile number to verify who they are..

ArrayBoundCheck(10000) 2 days ago [-]

Some days I wish I could go without a phone number and have all my communications through an open protocol

bergenty(10000) 2 days ago [-]

I just have a physical burner phone for all this nonsense. Costs like $15 bucks and totally worth the tiny investment.

phreack(10000) 2 days ago [-]

Amen. Google is asking me to add 2FA to an account for work, and there's no way to do so except from phone numbers or Google Authenticator which I'd rather not use. It's the only service that doesn't let me use something like Authy for OTP.

jmprspret(10000) 2 days ago [-]

I have wanted to create a twitter a few times now, but they refuse to let me use my account/complete my signup until I give them my phone number.

joegahona(10000) 2 days ago [-]

I was just able to remove my phone number from my account settings and wandered into a Fred Sanford-level of junk data -- Twitter had me identified as a female (I'm male), had 'interests' tied to me for both 'Alexandria Ocasio-Cortez' and 'Ben Shapiro' (they're most certainly not), and had my languages as 'French' and 'Indonesian' (I know only English). Bad digital hygiene.

politelemon(10000) 2 days ago [-]

How did you arrive at the 5M figure, I didn't see the number of affected people in their post?

nishs(10000) 2 days ago [-]

Same, I don't see the number in the article. Was it removed from the official Twitter post?

An older external article[1] about the hack mentions 5.4M accounts.

[1]: https://www.cshub.com/attacks/news/54-million-twitter-accoun...

notpushkin(10000) 2 days ago [-]

Interesting how just throwing the 5M figure in the title changed everything for this post: https://u.ale.sh/some-accounts.png

wmeredith(10000) 2 days ago [-]

Well yeah. Some accounts could be two. If I see language like that in a headline, I pretty much ignore it. It's like when I see the word 'may' in a headline. 'New wonder drug may cure cancer.' That isn't even news.

matsemann(10000) 2 days ago [-]

> How to Protect Your Account

> (...) To keep your identity as veiled as possible, we recommend not adding a publicly known phone number or email address to your Twitter account.

Well, you're the ones constantly temporarily banning my account for not providing a phone number...

mandeepj(10000) 2 days ago [-]

> How to Protect Your Account

I'm thinking out loud for various other options that can be utilized: a private 256 char length key? You can also store it in a (Azure) key vault, so that it's easily accessible to you from other devices as well. I hope social media companies get open to more secure alternates, but security seems to be their after-thought.

lysergia(10000) 2 days ago [-]

Strange you say that. I'm six months into my pseudonymous account and they haven't tried to extort my phone number. It's like they know from my behavior that I don't want to be doxxed by Twitter Inc. I signed up using a VPN and a weird email address, and used an AD blocker.

SkyMarshal(10000) 2 days ago [-]

They said don't add a publicly known phone number to your account, so you have to create a Google Voice account that you'll never use except for account credentials like this. But Twitter will probably ban you for not using a real phone number. Or, you'll reuse that phone number across other accounts until one of them gets hacked and that phone number sold on the dark net, and now it's a public phone number again.

jjulius(10000) 2 days ago [-]

> How to Protect Your Account

> Don't sign up.


hnburnsy(10000) 2 days ago [-]

I believe this is the vulnerability reported to Twitter which awarded $5000 from its bug bounty program.


thenoblesunfish(10000) 1 day ago [-]

Anyone have any idea how many of these bounties are collected by people who actively look (seems like a hard way to make a living) vs. say people with some knowledge who stumble across the issue and wouldn't take the time to properly report, otherwise (might convince me to take a couple of hours)?

Nexxxeh(10000) 2 days ago [-]

$5k seems embarrassingly low so something with such horrendous impact. Potentially allowing for doxing, and because phone numbers are the lynchpin for many 2FA and consumer-facing telco security is generally lax, total user hijacking across multiple platforms. What an absolute disaster.

oars(10000) 2 days ago [-]

Thanks for sharing this link. Twitter should've shared it in their post...

deadonarrival(10000) 2 days ago [-]
pvg(10000) 2 days ago [-]

These make sense if there's discussion there otherwise they are just links to nowhere.

tdehnel(10000) 2 days ago [-]

Anyone else annoyed by the growing use of the word 'impact' to speak increasingly passively?

People are so afraid to make a claim nowadays, even if it's obviously true. They speak of 'impacts' or that something will be 'impacted'. But they seem to want to avoid saying who or what will be impacted.

'I was impacted by today's layoffs.' 'We expect there to be impacts to website traffic.'

These meaningless words do nothing except to say 'something has happened' which puts the reader in the mindset of having to unravel a mystery.

Anytime you write it's your job to make yourself understood. I don't want to have to be Encyclopedia Brown to figure out what you're trying to tell me.


Edit: A better headline for OP would have been 'Private phone numbers + email addresses leaked for 5M Twitter accounts'

ben174(10000) 2 days ago [-]

Exactly. They're giving the least information possible to formulate a coherent headline which is technically accurate. If they told the truth in the headline, it would get WAY more clicks. These are clicks they don't want.

whydid(10000) 2 days ago [-]

Head wobble

aquaduck(10000) 2 days ago [-]

Orwell's 'Politics and the English Language' really should be required reading for all high school students. Personally, I re-read it every few years - chronic exposure to terrible English makes the bad habits grow back, so you need to pull the weeds regularly.

bink(10000) 2 days ago [-]

> When we learned about this, we immediately investigated and fixed it. At that time, we had no evidence to suggest someone had taken advantage of the vulnerability.

> In July 2022, we learned through a press report that someone had potentially leveraged this and was offering to sell the information they had compiled. After reviewing a sample of the available data for sale, we confirmed that a bad actor had taken advantage of the issue before it was addressed.

Yikes. Sounds like they either didn't dig deep enough to see if it was exploited or they don't keep records long enough to be sure.

nolok(10000) 2 days ago [-]

You seem to believe 'we had no evidence to suggest someone had taken advantage of the vulnerability' implies 'we looked for any evidence of it', it doesn't, not in that case nor in any similar situation.

hertzrat(10000) 2 days ago [-]

Out of curiosity, why is it only 5M and not 500M? You would think the same vulnerability applied to every server, not just one or one cluster, if they are using automated deployments

addingnumbers(10000) 2 days ago [-]

'We have no evidence that this was exploited' is a standard psychological trick they pull in vulnerability announcements to give an unfounded impression that it hasn't been exploited.

londons_explore(10000) 2 days ago [-]

Probably the latter - all companies operating in the EU have had short (ie. 30 days) retention policies on anything user-identifiable (ie. http logs) for a while now.

But if they didn't keep sufficient logs, they should have alerted the users back then, not now.

grogenaut(10000) 2 days ago [-]

Or they don't monitor that system for that type of access at all and so literally don't know.

i_have_an_idea(10000) 2 days ago [-]

Facebook had a very similar information leak just a couple of years ago. It is amazing these companies seem to learn very little from each other when it comes to protecting personal information.

jlarocco(10000) 1 day ago [-]

And look how bad it turned out for them...

jmount(10000) 2 days ago [-]

This evidence Twitter has 5.4mm actual accounts?

aeyes(10000) 2 days ago [-]

Could be that they have 5.4M accounts with a phone number.

pmlnr(10000) 2 days ago [-]

What unit is mm? Millimetres?

ewhanley(10000) 2 days ago [-]

million (m = 1000, m * m = 1000000)

Thrymr(10000) 2 days ago [-]

This abbreviation is not in the article (nor is the number). And the HN headline now says '5M' which is maybe a more common abbreviation for 'million'.

eternalban(10000) 2 days ago [-]

MM is 1 million. mm = 1 millimeter.

RonMarken(10000) 2 days ago [-]

Perhaps Twitter needs to make it easier to create accounts anonymously and stop virtue signaling (i.e suspend accounts created over Tor onion-service)

With pseudonymous usage of public services information minimisation to maintain operational-security against private user-data being disclosed by external hackers or rogue insiders is a mantra that needs to be followed religiously.

refulgentis(10000) 2 days ago [-]

Virtue signaling? Preventing completely anonymously accounts doesn't seem to fit that colloquial definition of that, I always assumed it meant taking an action simply for social signalling, that has no benefit to you otherwise.

lysergia(10000) 2 days ago [-]

I'm six months in and they haven't asked for a phone number yet. I dread the day when they do. This is where proficiency in the Twilio API comes in handy.

slater(10000) 2 days ago [-]

is this the same security issue that was used to unmask who was behind that 'libs of tiktok' account?

yieldgap(10000) 2 days ago [-]

no, I think the 'libs of tiktok' person included their personal info when registering a domain

londons_explore(10000) 2 days ago [-]

So after forcing users to enter a phone number to continue using twitter, despite twitter having no need to know the users phone number, they then leak the phone numbers and associated accounts. Great.

But it gets worse... After being told of the leak in January, rather than disclosing the fact millions of users data had been open for anyone who looked, they quietly fixed it and hoped nobody else had found it.

It was only when the press started to notice they finally disclosed the leak.

That isn't just one bug causing a security leak - it's a chain of bad decisions and bad security culture, and if anything should attract government fines for lax data security, this is it.

myself248(10000) 2 days ago [-]

The whole announcement reeks of 'Stop hitting yourself!'

What scum. They had lots of chances to fix this, the first one being not collecting phone numbers in the first place. They chose to do that, and then they didn't adequately protect it, and now they're oh so very surprised that someone might be doxing their most vulnerable users.

If anyone is harmed by this, Twitter should be held liable.

junon(10000) 2 days ago [-]

Discord is also like this and it drives me nuts.

eli(10000) 2 days ago [-]

Requiring a phone number is part of fraud & spam prevention. Maybe you'd make a different tradeoff but that's not 'no reason.'

spicybright(10000) 2 days ago [-]

I know the answer is money in politics, SV culture, etc. But it's near certainty twitter will continue as they do in and 2 weeks everyone will move on.

Maybe they get a small boo-boo in the form of a symbolic fine, mangers scramble for a bit, and then the whole thing happens again and again.

Why is this?

smm11(10000) 2 days ago [-]

I already fixed it, by not using Twitter.

cowtools(10000) 2 days ago [-]

Truly. It is infuriating dealing with the phone number rigamarole.

Why does X company require me to use a certain phone number/IPv4 address/2FA? It doesn't improve security, it does not protect against sybil attacks. The reason is vendor lock-in and data collection.

It's not worth dealing with this crap to access another time-wasting/brainwashing app.

At the same time, there is no shortage of users here willing to give lip service to these backwards practices.

NelsonMinar(10000) 2 days ago [-]

Isn't this the second or third time for Twitter to have this exact same flaw? From 2020: https://www.socialmediatoday.com/news/twitter-uncovers-secur...

I might be confused; this is a very old feature of Twitter that does have an opt out. Maybe this new disclosure is the opt out didn't work? https://help.twitter.com/en/safety-and-security/email-and-ph...

It's a different problem, but this year Twitter also got a $150M fine for illegally using the phone numbers they demand from users for marketing purposes. https://www.theverge.com/2022/5/25/23141968/ftc-doj-twitter-...

hackernewds(10000) 2 days ago [-]

We consistently have to go through Data protection practices, and limit the purpose of what the data collected can be used for. This seems like either a blatant miss in process, or willful ignore where $150m is under the EXPECTED value of the rewards through marketing

lumost(10000) 2 days ago [-]

You know... in the last major tech bust, downsized teams working on oversized software didn't have thousands of productions services to maintain. What's a company with 10k services, and 10 languages going to do when when it comes time to patch security vulnerabilities. Or merely keep them from emerging?

1-6(10000) 2 days ago [-]

Put together a 'scrappy team' no longer sounds professional and sustainable.

LeoPanthera(10000) 2 days ago [-]

'we recommend not adding a publicly known phone number or email address to your Twitter account.'

This is literally impossible. You can't create a Twitter account without a phone number. It sometimes allows you to do so, but then is blocked within 24 hours until you add one.

It's insulting that Twitter should lie about that.

alx__(10000) 2 days ago [-]

> publicly known

Note the PR words they used. Which amounts to, 'If you want privacy, it's not our problem. Go create a virtual number somewhere.'

HL33tibCe7(10000) 2 days ago [-]

So what you're saying is that you discovered a vulnerability that leaked the private information of your users, said absolutely nothing for 6 months, then finally came clean, but only because you were forced to because people were selling data on the deep web.

Please take your "sorry" and shove it where the sun doesn't shine. You don't "take our privacy seriously". This is utterly ridiculous and unacceptable, and in a fair world you would be punished heavily for it.

Edit: an earlier version of this comment criticised Twitter for not doing an investigation earlier to uncover the fact that a leak occurred. This accusation was based on me misreading the press report - see one of the child comments for details. I've removed that part of the comment.

winternett(10000) 2 days ago [-]

The methods to scrape numbers from social media have been published on YouTube for ages now. They share those numbers publicly because they themselves run services that share user data with other companies openly... Twitter (for example) is used as an authentication service with Disqus and a few other online apps too, an online comment service which could easily save/track sensitive ID data across comments on multiple sites unwittingly to the user, so it's a really shady overreach if that is indeed the case. These numbers are gathered under the guise of security, but they are used for entirely different purposes.

I think the real fault is in them forcing users to enter this type of data to begin with, because that makes the only options to surrender your data to them or to not use the app at all.

It would be interesting to see if numbers from verified accounts were included in the leak, that would be very telling.

gameshot911(10000) 2 days ago [-]

>didn't bother to do an investigation into whether it leaked data (which clearly is possible, because you've done it now)

It sounds like they confirmed the exploit by looking at the hacked data, not by a renewed search of previously available logs.

vinay_ys(10000) 1 day ago [-]

I have seen too many services that ask phone number for account recovery purposes and then end up using it for other purposes for which the user didn't consent. Given how insecure SMS OTP is, I try not to enable that if I can avoid it. Then, on top of it, bugs like this make the service behave like a globally accessible open reverse-directory of mobile numbers to names.

How is twitter notifying users? Has anyone posted screenshots of this notification? I want to know where this notice will appear.

spurgu(10000) 1 day ago [-]

Not defending them but I think a major reason why Twitter (and for example Gmail nowadays) is asking for phone numbers is to decrease spam accounts (which is of course a good thing in itself).

londons_explore(10000) 2 days ago [-]

I think you will see more of this class of attack.

Lots of companies have various 'forgot my username'/'forgot my password'/'trying to sign up for a new account with a new email address but existing phone number'/'add a friend by email or phone' flows. It's very easy to accidentally leak some info that shouldn't be leaked while implementing such a flow, since you are peering into the users database querying by email/phone/other identifier while the user hasn't properly authenticated yet.

jdmichal(10000) 2 days ago [-]

Yes. The proper way to implement this flow is to ask for the information, and then present the exact same result screen regardless of the actions taken. Any additional information or action should be done exclusively through the contact information you have on record.

some_random(10000) 2 days ago [-]

>If you operate a pseudonymous Twitter account, we understand the risks an incident like this can introduce and deeply regret that this happened. To keep your identity as veiled as possible, we recommend not adding a publicly known phone number or email address to your Twitter account.

I'm so sick of this kind of victim blaming, you're forced to add a phone number to use twitter.

CommanderData(10000) 2 days ago [-]

Twitter would actively block one time numbers. This seems like a lie.

I tried to use onoff numbers with Twitter on multiple occasions but failed to receive anything. They are being very misleading here.

JacobThreeThree(10000) 2 days ago [-]

Yeah, it's shocking they would say this when they do require phone numbers in many instances.

londons_explore(10000) 2 days ago [-]

No mention of that fact that 'use another phone number' is quite an expensive thing to do in countries where a phone number has an annual fee of hundreds of dollars.

Suddenly 'use twitter securely' has gone from 'free' to 'hundreds of dollars a year'. Perhaps they should announce this as a price change instead?

rex_lupi(10000) 2 days ago [-]

Tip: If you email(anonymously ofc) twitter support that you do not have a phone number to receive the OTP for verification during account creation, they generally approve your request.

iszomer(10000) 2 days ago [-]

I have yet to add my phone number to my account. My guess is that it isn't applicable for legacy accounts circa 2008.

mhh__(10000) 2 days ago [-]

My twitter account does not have a phone number attached

skybrian(10000) 2 days ago [-]

They seem to be recommending that you use an unlisted phone number. That seems like a good idea.

devmunchies(10000) 2 days ago [-]

> you're forced to add a phone number to use twitter.

is that true for the desktop web client?

strulovich(10000) 2 days ago [-]

It might have some PR speak sprinkled, but it's genuinely good advice, put more bluntly:

"We can screw up, if it's important enough for you to stay anonymous you should get a separate phone number and email"

That is a good tip with every company. If you want better security, have less trust in the services you're using.

This goes to what victim blaming is. Yes. It would be great if the victim lived in a better world. But sometimes extra caution could help them now without waiting for the entire world to change.

tnzk(10000) 2 days ago [-]

> a publicly known phone number or email address

I don't get the definition of 'publicly' here. Does it mean something on Internet, or include numbers I tell people in-person? If the former, not so many people put their number online I suppose...

spullara(10000) 2 days ago [-]

You can remove your phone number after creating the account.

traceroute66(10000) 2 days ago [-]

> I'm so sick of this kind of victim blaming, you're forced to add a phone number to use twitter.

I had some old accounts that did not require a phone number.

At least until I wanted to enable TOTP 2FA.

At which point the numnuts at Twitter would not just let me 'just' enable TOTP, I was forced to provide a phone number (which, to add insult to injury, for at long time they refused to accept because they would only send messages to a limited number of carriers).

bena(10000) 2 days ago [-]

I mean, originally twitter was an SMS based service. It was made for phones.

yodsanklai(10000) 2 days ago [-]

> If you operate a pseudonymous Twitter account

If you operate a pseudonymous account anywhere, you should always assume there's a slight possibility that one day your identity is known.

I think it's not far stretched to think that in the future, malevolent governments will have access to whatever things we may have posted and use it against us.

echelon(10000) 2 days ago [-]

Fucking vile victim blaming.

Twitter should not be requiring phone numbers, especially when they don't care enough to protect them.

This is why we should get back to protocols for communication instead of platforms.

627467(10000) 2 days ago [-]

Agreed but, in what jurisdiction does Twitter require phone numbers?

franklampard(10000) 2 days ago [-]

> add a phone number

They do this to combat spammers, don't they?

kordlessagain(10000) 2 days ago [-]

The company entity requires blaming others. It can't blame itself, otherwise stakeholder value is affected. If you want to blame anyone, blame the environment that allows these types of actions by companies, or simply stop using them.

BTW, no Twitter account is 'ours'. If it was, we could download everything (friends and all) and move it somewhere else. Twitter needs to take ownership of all data on their platform - user accounts included. Trying to separate them into different entities is ridiculous.

remram(10000) 2 days ago [-]

> we recommend not adding a publicly known phone number or email address to your Twitter account.

> While no passwords were exposed, we encourage everyone who uses Twitter to enable 2-factor authentication

I think those things are incompatible, or at least Twitter really gives that impression. Great recommendation /s

tomjakubowski(10000) 2 days ago [-]

How are those things incompatible?

stygiansonic(10000) 2 days ago [-]

The full sentence states:

While no passwords were exposed, we encourage everyone who uses Twitter to enable 2-factor authentication using authentication apps or hardware security keys to protect your account from unauthorized logins.

So it actually does not imply adding a phone number, which is seemingly what you have tried to imply with the cut-off quote provided.

nomilk(10000) 2 days ago [-]

> If you operate a pseudonymous Twitter account, we understand the risks an incident like this can introduce and deeply regret that this happened. To keep your identity as veiled as possible, we recommend not adding a publicly known phone number or email address to your Twitter account.

First time I've heard a company actually say this. It's obvious to people who understand a bit about tech and security, but not obvious to the layperson. Twitter actually deserve a tiny amount of credit for giving practical advice that reduces adversity for users in the event of a breach.

civilized(10000) 2 days ago [-]

Except for a long time they shut down accounts without a phone number under the pretense of 'suspicious activity'. For some reason, these suspicions could be immediately allayed only by providing your phone number.

Being forced to do something and later being advised not to do that thing out of deep concern for my well-being? Yeah, that's the Twitter UX vibe: the most self-regarding, passive-aggressive person you know, in software form.

rbut(10000) 2 days ago [-]

No, that's just shifting the blame onto the user. If they are asking for something as sensitive as a mobile number, then they need to protect it properly.

They ask for a mobile number to verify you're a real human, then they say 'Ha it's your fault you gave us a sensitive mobile number'. 99.9% of users only have one mobile, and have no idea how to get an alternate number, so they just give the number they have.

vageli(10000) 1 day ago [-]

If they actually cared they would make that statement in bold at the time they ask for your phone number and email address.

winternett(10000) 1 day ago [-]

Twitter often FORCED users to enter a valid phone number by locking accounts, and then verified if it was active in comparison to accounts. To this day there is no way to remove the phone number or disassociate it with an account. Please do not oversimplify the offense, it does not do justice to the cited issues involved.

jtbayly(10000) 2 days ago [-]

No. That's not practical advice. Twitter is gaslighting us. You can't use Twitter without a phone number. They require it.

misterS(10000) 1 day ago [-]

Two days ago, I've tried to create an account tied only to an email. During account creation, the wizard suddenly inserted an additional step and required my to enter a phone number.

I realise though that this is possibly an anti-spam measure (which I'm in favour of), since I've connected through Tor when creating the account. But this procedure stands in stark contrast to the advise given in the article.

skrtskrt(10000) 2 days ago [-]

Is this going to be the thing that gets Elon Musk off the hook for his billion dollar fine for backing out of the deal?

They had a breach and actively actively hid it for an extended period of time. Obviously both sides have good lawyers, but it's hard to see how this doesn't hurt Twitter in regards to the legal battle over the Musk deal unwinding

collegeburner(10000) 2 days ago [-]

This starts getting toward 'everything everywhere is securities fraud'. This probably would have come up in tech diligence but he waived that.

mromanuk(10000) 2 days ago [-]

Is there a way to know if your own account is compromised?

bluetidepro(10000) 2 days ago [-]

> We will be directly notifying the account owners we can confirm were affected by this issue. We are publishing this update because we aren't able to confirm every account that was potentially impacted, and are particularly mindful of people with pseudonymous accounts who can be targeted by state or other actors.

So they may contact you, or may not. It would be nice if this gets added to something like haveibeenpwned

cecilpl2(10000) 2 days ago [-]

> In January 2022, we received a report through our bug bounty program

> This bug resulted from an update to our code in June 2021

Does this mean the problem existed for 7 months and nobody at Twitter noticed until they received a bug report?

ameliaquining(10000) 2 days ago [-]

That's not unusual for a security bug; it's not like this stopped people from using the app in a way that they'd loudly complain about or that would show up in metrics.

mcintyre1994(10000) 2 days ago [-]

Given they didn't think it was exploited they must have pretty poor logging and analytics around that part of their infrastructure. Someone managed to abuse it millions of times and they didn't know about it even after they'd fixed it and knew exactly where to look for abuse.

bpodgursky(10000) 2 days ago [-]

Cleaning house before due diligence.

HL33tibCe7(10000) 2 days ago [-]

Can we have a proper postmortem about this please, with information about the exact process that was required to obtain this information?

> We take our responsibility to protect your privacy very seriously and it is unfortunate that this happened.

Patently not seriously enough.

ryandrake(10000) 2 days ago [-]

It's always hilarious: Whenever any company is caught not taking X seriously, the first thing they do is issue a press release that starts with 'Here at COMPANY, we take X very seriously!'

hnburnsy(10000) 2 days ago [-]

here is the bug report...


taesu(10000) 2 days ago [-]

5.4mm == some

kurupt213(10000) 2 days ago [-]

All the non-bot active user accounts

nlitened(10000) 2 days ago [-]

I wonder how it affects the Musk's case of declining to buy Twitter? Surely they concealed from him the fact of this breach?

tns-hn(10000) 2 days ago [-]

Yes I wonder about this as well. Say Musk had good reasons to suspect some private information was at risk and Twitter kept denying anything was going on. No matter how minor the actual impact would be in the end, this would not paint Twitter in a favourable light especially in a legal battle where Musk claims Twitter held back vital information.

cheshire137(10000) 2 days ago [-]

The page isn't loading for me and I notice Twitter itself is either slow or not loading at all right now. I also see a spike in reported problems for Twitter on DownDetector.

zrobotics(10000) 2 days ago [-]

Archive link in case anyone is still having issues: https://archive.ph/HujUg

vidarh(10000) 2 days ago [-]

It loads for me in the browser, but the app is failing to load data.

EDIT: A few refreshes shows it's slow and occasionally failing in the browser for me...

Historical Discussions: "It's time for Apple to fix texting" (August 09, 2022: 742 points)

(768) "It's time for Apple to fix texting"

768 points 2 days ago by Fabricio20 in 10000th position

www.android.com | | comments | anchor

Texting between iPhones uses iMessage.

Texting between Android phones uses RCS, the modern industry standard for messaging.

Texting between iPhones and Android phones use SMS and MMS, outdated systems from the 90's and early 00's. The result is a poor experience—they do not support modern texting features like end-to-end encryption, high quality media sharing, read receipts, typing indicators and more.

If Apple upgraded SMS/MMS to RCS, texting issues between iPhones and Android phones would be fixed.

All Comments: [-] | anchor

simonjgreen(10000) 2 days ago [-]

SMS to me is solely the channel for machines to force a bad MFA implementation on me and couriers to tell me something is on the way/nearly there/delivered. All person to person comms, without exception, iPhone or Android users, is via WhatsApp. Anecdata from UK.

benknight87(10000) 1 day ago [-]

If the current dumb phone comeback continues, SMS won't be going anywhere any time soon

pineconewarrior(10000) 2 days ago [-]

I agree that SMS is obsolete, but I will not so willingly jump to another zuckerberg platform

baby(10000) 2 days ago [-]

Same, from the US

Andrex(10000) 2 days ago [-]

I far, far prefer a protocol over Zuckerware, but to each their own.

raxxorraxor(10000) 1 day ago [-]

I think the argument about SMS being insecure is very real but exaggerated. We send password reset with capability URLs through non encrypted mail. Sure, the channel is most likely encrypted but anyone at mail service providers could easily take over a massive amount of accounts. Although the user would notice at least.

Sure, there are tools to intercept SMS without the user noticing, but as a second factor an attacker still doesn't have access to other factors.

The successor RCS has the problem that users cannot reset the password. If you are compromised you need to urgently contact your ISP.

MBCook(10000) 2 days ago [-]

RCS itself does not support E2E encryption. That's an extension Google developed. It only works if BOTH people are using Google's Messages app.

Group chats? Totally plain text in the clear. No encryption at all.

RCS is not good enough. Fix the issues, develop something better, I don't care.

Only E2E is good enough.

jvolkman(10000) 2 days ago [-]

SMS/MMS, which is the only iMessage fallback today, is not E2E.

nneonneo(10000) 2 days ago [-]

Right now Apple is facing quite a lot of regulation in various places to open up their platform: open up app loading, open up repairs, etc. It's hard not to imagine this being yet another salvo in forcing Apple to open up their messaging platform (and it aligns with recent regulatory efforts).

Except, unlike app stores and repairs, the standard being pushed here, RCS, is not a good solution by comparison. It's locked to carriers, who have different and inconsistent implementations, rather than being tied to an identity like iMessage.

It'll be a shame if Apple is forced to adopt an inferior standard here...

jeroenhd(10000) 2 days ago [-]

Apple will probably be forced to open up their iMessage implementation once the Digital Markets Act will be adopted, forcing large messenger companies to make their messaging services interoperable.

If they're smart, they work together with Google and other large messenger providers to form some sort of secure standard. If they keep being stubborn, they'll be forced to either stop selling iMessage in Europe or accept consequences to their technology much worse than cooperation. I'm no fan of breaking E2EE for interoperation, but since none of the big market players seem interested in working together, I think this will be unavoidable. It's a shame, really, that it had to come to regulation to get the market to work in the users' favour.

This probably won't matter to users outside the EU but big changes are coming over here.

lern_too_spel(10000) 2 days ago [-]

This is about RCS vs. SMS. Apple and Google support SMS as a cross-platform standard. They should support RCS, which is superior.

rhacker(10000) 2 days ago [-]

Apple connect to our messaging platform voice, i mean messages i mean allo i mean duo i mean hangouts i mean...

tick_tock_tick(10000) 2 days ago [-]

Funny thing is the EU doesn't give a fuck and is going to force them.

hbn(10000) 2 days ago [-]

Google acting like RCS is the hot new standard is pretty disingenuous. In theory it's a standard, but in reality most carriers haven't been interested and haven't implemented it so the vast majority of RCS messages are routed through Google.

I can't really blame Apple for not being interested in adopting a 'standard' that's mostly Google pretending to be a standard.

pwpw(10000) 2 days ago [-]

This is such a weirdly US specific issue. It's hard to understand why people in this country refuse to adopt a data-based messaging service such as Signal or WhatsApp like the rest of the world has. Why are US citizens so set on having a terrible experience when messaging half of the population? How did other countries decide that using platform agnostic messaging services are better? I believe the UK has a similar split in Android/iOS users, yet they largely use WhatsApp.

In a way, it feels perfectly inline with America. We use Imperial when everyone else uses Metric. We use Fahrenheit when everyone else uses Celsius. But in this case, it's not as if our government led us down this path. The problem was entirely created by our market of users.

Ultimately, poor communication stifles society and innovation. It's in all of our best interests to improve the current situation. Sure, better alternatives such as Signal exist, but we will have to move mountains to convert everyone onto a new service. For now, I think it's best if we all apply pressure to Apple to adopt RCS. It's significantly better than where we are now, and that's a good thing.

kingrazor(10000) 2 days ago [-]

I can expect anyone who has a cell phone to be able to receive SMS. When I first started using a cell phone in 2007, that's all that was available. Or at least, it's all I knew that was available. Every phone I've owned since then has had the capability to use SMS, and I've never had a need to use anything else. So I don't see why I'd bother with something like Signal or WhatsApp, when there's no guarantee that the person I want to talk to will have it, and they don't offer anything I need that SMS/MMS doesn't already provide.

joenot443(10000) 2 days ago [-]

This might be news to you, but huge amounts of land in Canada and the US is still not well served by fast cell service. In those places, you often have an unreliable single bar 3G or 2G connection, and using a data heavy messenger like WhatsApp is entirely unrealistic.

asdff(10000) 2 days ago [-]

Because of historical reasons. Americans had free texting for decades. Entire segments of phones were released dedicated to the texting experience with full keyboards. When the iphone came out it had unlimited data, but it was the exception not the norm, and it was slow as hell. People don't want to use data for something they already got for free, plus there's the whole network effect issue. Everyone has a phone number, not everyone is going to be on whatever chat platform you insisted on using. Imo I think its better this way, using this do it all technology that works on a modern phone or the flip phone my dad has, versus being reliant on whatsapp and therefore Meta.

stetrain(10000) 2 days ago [-]

The problem is which one?

I don't use iMessage much, but there isn't One Alternative that everyone has conglomerated on.

Some groups use Whatsapp. Some use Facebook Messenger. Some use Discord. I jump around between all of them.

But if you have someone's phone number you can at least text them.

hot_gril(10000) 2 days ago [-]

Cause it's not terrible. It still works ok the way it is, and people tend to use FB Messenger or something if they want a more complex group chat with Android users in it. Considering that they do this even if it's an all-Android chat, I don't think it's Apple's fault, as scummy as they act with SMS.

Also, people don't care for metric cause it's not better. Basing your system around the physical properties of water doesn't help for daily usage, just for some sciences and yes for nerds to feel better. For example, many digital thermostats have to add increments of 0.5 when you switch to Celsius since each degree is twice as coarse, and you can feel the difference. Put ̊F and ̊C weather forecasts side by side, and ask people which they prefer. It's a nice spread of 0-100 ̊F outdoors usually, and you basically don't go outside if it's not. Freeway speeds, 0-85MPH legally and 100 means trouble. Freeway miles, about 1 per minute. Feet/mile is weird, but that's cause nobody converts miles to feet; you don't care to compare say a trip distance to the height of a human.

dt2m(10000) 2 days ago [-]

A good argument I've heard is that in Europe, when people got their first mobile phone contracts, you were charged per SMS message, whereas in the US, unlimited texting was included in your plan.

The consequence of this was that in Europe, people preferred free online messaging over costly SMS.

20+ years down the line and old habits die hard.

baby(10000) 2 days ago [-]

Most people I know in the US use whatsapp (except for like one person)

crazygringo(10000) 2 days ago [-]

> This is such a weirdly US specific issue. It's hard to understand why people in this country refuse to adopt a data-based messaging service

It makes sense if you look at the history.

For various reasons, the US was late to the party on text messaging at all. I remember back in ~2002, traveling the world and discovering people used SMS, when back home in the US it didn't really exist, at least not between different carriers.

BUT internationally, text messages were also often expensive, a few might add up to the price of a coffee.

So, as soon as data plans became a thing, it was a no-brainer for everyone internationally to switch to data-based messaging (like WhatsApp) -- it could save you like a hundred dollars a month.

On the other hand, when carriers in the US sorted out compatibility issues and SMS's became a thing later on, it was generally unlimited on most monthly plans. So people in the US never switched from SMS to data messaging because there was no financial incentive to do so, like there had been around the world.

Seen in this light, it shouldn't be hard to understand at all. It's all about whether SMS's were expensive or free when data plans became popular.

And so for most people in the US most of the time, SMS does everything you need, so why on earth would you go download a separate app? Nobody's 'refusing', they're just following the path of least friction and getting on with their day.

TheRealPomax(10000) 2 days ago [-]

Ah yes, whatsapp, that bastion of privacy, and not at all a messaging service that exists primarily for Meta to mine.

If everyone was on Signal, sure, but if everyone's on Whatsapp, maybe not the kind of thing to go 'why don't you just do this too, why are North Americans so backward?' for something owned by what is basically still just Facebook.

1270018080(10000) 2 days ago [-]

It's just too late for anyone to switch. The network effect went in Apple's favor. It could've happened to any country.

xnx(10000) 2 days ago [-]

I'm all for standards, but this is mainly sour grapes by Google. If they hadn't shot themselves in the foot dozens of times with messaging they could've dominated using the head start they had with Google Talk. Google should put all messages from iPhone users in comic sans.

hbn(10000) 2 days ago [-]

They had a surefire strategy starting in 2013 when they added SMS integration to Hangouts and made it a default-installed app on all Androids. It was tied to your Google account so most people (and basically all Android users) already had an account. It was pre-installed, meaning you didn't need to pitch people to install another app, which is usually a big ask. Instead you say 'hey open this app you already have installed, we can chat here and it's better, and you can text all your other contacts who don't have it too.' It had video calling too, basically all you needed.

But then Allo and Duo came along. Remember Allo? Me neither! It was Hangouts' death sentence anyway! And now Duo is being rebranded/merged into Meet for some reason.

Get out of the Google ecosystem wherever you can. They're only getting worse.

worthless-trash(10000) 2 days ago [-]

This is genius.

ElijahLynn(10000) 2 days ago [-]

This is sour grapes for users. Google doesn't own the RCS standard, fwiw. I still use SMS/MMS and it is really, really nice when another user is using RCS because modern messaging features actually work. I can send long voice memos/song ideas to others, high resolution photos, see if a message was read etc. RCS is a huge upgrade, and really has nothing to do with Google.

sudden_dystopia(10000) 2 days ago [-]

I have never experienced blurry photos or videos as this alleges. I hate read receipts and typing bubbles anyway. I do agree that Apples group chats are highly annoying. Personally, I think it is Androids text platform that is bloated and inferior.

bagacrap(10000) 2 days ago [-]

you've definitely experienced blurry media if someone texted you from the other kind of phone (note that your own media will still show up crisp in the conversion window even though the other end gets a mega compressed version)

aaaaaaaaaaab(10000) 2 days ago [-]

Ah, another 'grassroots' initiative to adopt a 'standard' (RCS) from Google! No thanks, I don't want this RCS crap on my phone.

iMessage works seamlessly on my multiple phones, iPads, and Macs. Fuck off with that carrier lock-in trash.

radiojasper(10000) 2 days ago [-]

I still don't get why people use SMS/MMS anyway? I've been using WhatsApp for ages now and so does everybody else in my country - and every country I've been in, apart from China and Japan. My friend who's from the US once said 'I've paid for those text messages, so I'm going to use them!' But if I send him a text from Europe to the US, I pay 1 damn euro per delivered text. WhatsApp is free! Is there any viable reason why Apple users use SMS so much?

kelnos(10000) 2 days ago [-]

I use WhatsApp as well, but not many of the people I communicate with have it. SMS/MMS is a common denominator that everyone with a phone number has, and can always be relied upon to work without foreknowledge that the other party has a particular app installed.

> But if I send him a text from Europe to the US, I pay 1 damn euro per delivered text

How the tables have turned! It's no secret that the US has more expensive cellular plans than the rest of the world, but with my carrier, international texting is free.

macintux(10000) 2 days ago [-]

WhatsApp was appealing before it got sucked into the FB vortex. Thanks, but no thanks.

If there were a single 3rd-party messaging platform that I trusted, and my friends started using it, sure. But since 75% use iPhones/iMessage, and the rest SMS, why in the world would I use WhatsApp?

cgrealy(10000) 2 days ago [-]

> Is there any viable reason why Apple users use SMS so much?

They don't. At least not in my experience. I have an iPhone, but there's about a 50/50 split ios/android in my friends and family.

Group chats are almost entirely WhatsApp, and single messages are a blend of WhatsApp, iMessage and SMS.

I probably use SMS/MMS once a week

abawany(10000) 2 days ago [-]

I've seen this position a lot throughout this thread and I have a question: all of these apps (whatsapp, signal, etc.) appear to be 'free' - how do you suppose they will make money? In the US, the users of sms/mms/imessage paid in some way for this service and can have some reasonable expectation for delivery and availability.

lotsofpulp(10000) 2 days ago [-]

There are many people in the US who have no international contacts, and so they grew up using only the default messaging app. And they are not sufficiently incentivized to install another app like WhatsApp.

Between NYC/SF, I do not know a single person that does not use both iMessage and WhatsApp. But typically it is people who are not children of immigrants and whose social circles have no one outside the country that tend to not have WhatsApp.

themagician(10000) 2 days ago [-]

Apple users don't use SMS—they use iMessage. It's seamless and automatic. All your contacts are automatically there as long as you have a phone number or email address which is an AppleID. It's so seamless most people don't even realize they are using it.

parkingrift(10000) 1 day ago [-]

I'd rather skip messaging friends and family altogether than use a service owned and operated by sociopath anti-human Mark Zuckerberg.

stonemetal12(10000) 2 days ago [-]

It comes preinstalled, works, is free. Why would I look for a different messaging app? What does WhatsApp do that the preinstalled, free, messaging app doesn't?

kevin_thibedeau(10000) 2 days ago [-]

SMS is the only federated messaging system guaranteed available on all cell phones. That makes it more useful than any walled garden.

asdff(10000) 2 days ago [-]

Because most Americans are texting for free

rootusrootus(10000) 2 days ago [-]

Almost nobody I know uses WhatsApp. On the other hand, a significant number of people I meet do have iMessage. There's no incentive for me to install WhatsApp. Even my friends internationally all have iPhones. I don't install third-party apps unless there is a very good reason. SMS is an inferior but acceptable fallback for edge cases.

nr2x(10000) 2 days ago [-]

The USA is very much iMessage driven relative to rest of the world.

lern_too_spel(10000) 2 days ago [-]

It's also time for Google to fix texting. Google Voice still doesn't support RCS despite people asking for it for many years. It would be great if someone just copied this web page and filled in Google and Google Voice everywhere it talks about Apple and iMessage, but I get the feeling that Google doesn't even care how embarrassing it is.

ocdtrekkie(10000) 2 days ago [-]

It's more embarrassing than that: For most of the time I used Google Voice, it couldn't even forward MMS. I used Google Voice as my primary number for years, and I had to tell people that I couldn't receive group texts or pictures, which always got me weird puzzled looks.

And of course, now they've removed SMS forwarding entirely, and basically completely made the service useless/redundant. I'm glad I ported my main number out years ago.

tristor(10000) 2 days ago [-]

This is a complete non-issue. I have no idea why people are complaining about this, except as another way in which Android users are trying to force their ecosystem choices on everyone else. The majority of people globally don't even use built-in messengers, they use WhatsApp or a similar application, most of which uses similar stylistic design choices as Apple uses for SMS, so it's hardly an issue.

I'm well aware that high schoolers get bullied for being poor and not being able to afford an iPhone. High schoolers were getting bullied for being poor and not being able to afford fancy clothes before cell phones were even a thing, and prior to smartphones were bullied for being poor and not being able to afford pagers or a cell phone (or a car, or ... or ...).

High schoolers are assholes and will find some excuse to torment each other regardless of what aesthetic and design choices somebody in Apple's UX team makes for their built-in messaging app. This is a massive nothingburger and I honestly have no idea why the media gives this any credence other than the shift of the media generally towards being anti-tech.

collsni(10000) 2 days ago [-]


refulgentis(10000) 2 days ago [-]

I really appreciated this comment. It subverted my expectations by inverting the situation: its not Apple's phone being locked to either Apple's messaging service or the texting standard from 1997, its people forcing Apple to implement a more modern standard.

pipeline_peak(10000) 2 days ago [-]

The second the guy in the video complained about Android users getting blamed as if it were an actual problem I questioned the age demographic they were targeting.

thomasahle(10000) 2 days ago [-]

> regardless of what aesthetic and design choices somebody in Apple's UX team makes for their built-in messaging app

It's not about aesthetic design, and it's not just about high schoolers. Lots of grown up Apple users don't like adding Android users to their group chats, because falling back to MMS degrades the messaging experience for everyone. Things like images sending in low enough resolution to be useless.

As a European moving to the US, I can attest how hard it is to get Apple users to switch their group chats to WhatsApp/Facebook/Signal. I don't think I'm in a single such group that isn't majority non-Americans.

You can say Apple doesn't need to care about Android users being socially isolated. But some Apple users might like an easier way to include their non-Apple friends.

maxsilver(10000) 2 days ago [-]

> This is a complete non-issue.

No it's not. iPhones intentionally don't support modern cellular messaging. There's nothing wrong with insisting they support LTE spec (RCS), like every modern smartphone does.

> The majority of people globally don't even use built-in messengers, they use WhatsApp or a similar application,

Completely unrelated. This discussion has nothing to do with iMessage / WhatsApp / etc. The problem is that Apple forces your SMS texts into effectively-2G-mode, Apple lies about the quality of the messaging you're getting from your native cellular service.

Imagine if they did the same thing to, say, your telephone calls from your phone number by running it through a ton of fake compression, and then said, 'well, you could iMessage or Skype or Zoom instead if you don't want us fucking up your phone call audio'.

zeusk(10000) 2 days ago [-]

As someone who worked around Windows Phone and had to deal with Google's unwillingness to support other eco systems; Pound sand android.

broodbucket(10000) 2 days ago [-]

Companies will always act in their own best interest, that doesn't mean you should let spite get in the way of what would be a positive change for consumers. Google sucks. Apple sucks. Microsoft sucks. Unifying a broken ecosystem is a net positive regardless of which entity happens to be championing it

altairprime(10000) 2 days ago [-]

Most businesses, consumers, and developers universally continue to ignore the primary reason that iMessage is a closed platform, rather than an app on every platform as iTunes is:

Apple is using device serial numbers for anti-spam, supported by a fully-authenticated hardware and software stack that does not allow user modification. This permits Apple to simply "console ban" any Apple device that spams on iMessage. This makes it prohibitively expensive to send spam over iMessage. They have been doing so since iMessage was launched.

Android offers no such attestation that I'm aware of. Windows, on Pluton, could offer this attestation securely — and that is a key deliverable of Pluton.

It's easy, then, to predict what Apple's first non-Apple platform will be: Microsoft Windows 12, only if secure-booted, with Pluton-signed attestation that the kernel is unmodified. And it's easy to predict how Apple will implement anti-spam: by applying "console" bans to specific Pluton chips by their serial number.

If Android wants to join the party, then Android phone builders need to implement secure boot with hardware-signed attestation of non-rooted-ness, in the style of Apple T2 + macOS or Microsoft Pluton + Secure Boot. Until then, Apple iMessage will remain single platform.

(I recognize that this is extremely unpalatable to device hackers, but the same freedom to modify an OS kernel that hackers desire is also the freedom to spam all users, as we have seen repeatedly with all messaging software platforms operated without hardware-backed attestation for the past thirty years — including email, Jabber, and HN itself.)

(No, I do not work at Apple.)

pca006132(10000) 1 day ago [-]

Or this is because more iTunes users means more potential customers to their iTunes store, i.e. more revenue when it is cross platform. While opening up iMessage will not incease their revenue but makes people easier to switch to other platform such as Android.

It is not like other platforms cannot deal with spam...

saulrh(10000) 2 days ago [-]

How does unmodified software relate in any way to the ability to console-ban bad actors? It's apple's servers, apple's accounts, and apple's devices. They are perfectly capable of burning a private key into the fuses of every device they sell, keeping a revocation list, and requiring a valid signature from an unrevoked key to log in and send messages. You can't get around that with any quantity of homebrew or custom software. Same reason that you don't see spam on Nintendo Switch games - if Nintendo bans your hardware you're not getting back online unless you buy a new Switch, and that's enough of a cost to make spam uneconomical. You can't do that with Android because maintaining a single revocation list across many manufacturers would be impossible - or because Google would have to host it and they'd get mobbed by angry HNers frothing at the mouth about their privacy - but Apple is totally capable of it and already gets a free pass on whatever walled garden shenanigans they can imagine.

harles(10000) 2 days ago [-]

iMessage spam has been through the roof for me the last couple of months. 1 or 2 messages a day with no obvious reporting mechanism. Whatever Apple is doing, it's not working and it's disingenuous to claim this is the reason iMessage isn't on Android.

upbeat_general(10000) 2 days ago [-]

This is just wrong because as others have pointed out, you can have a fully virtualized macOS environment with no secure boot or any kind and iMessage will run just fine.

Also, since basically every device that receives message also receives sms, isn't this irrelevant?

kelnos(10000) 2 days ago [-]

I'd rather live in a world with spam than a world where corporations get to decide what I run on my devices, and cripple a bunch of critical applications if I decide I want to, y'know, actually do whatever I want with the hardware I own.

I'm not unsympathetic to Apple's difficulties and goals here (assuming this spam problem is actually the reason, though I'm skeptical that there aren't also self-serving reasons that would be sufficient for Apple), but I'm so tired of society's slide toward 'security at any cost, and to hell with freedom' since the 9/11 attacks over 20 years ago.

(It's possible and likely that slide has been going on much longer, but I was a teenager in the 90s and not really aware of such things. But I think it's undeniable that the aftermath of 9/11 was a big turning point for the surveillance state and for average citizens being so scared of everything that they'd be willing to give up essential freedoms just to quell that fright.)

stetrain(10000) 2 days ago [-]

That has nothing to do with allowing RCS alongside SMS and iMessage.

My iPhone gets plenty of spam SMS messages, alongside my iMessage chats. The sanctity of iMessage communications doesn't stop that.

Swapping SMS for RCS support messages doesn't increase the spam surface.

tadfisher(10000) 2 days ago [-]

> If Android wants to join the party, then Android phone builders need to implement secure boot with hardware-signed attestation of non-rooted-ness, in the style of Apple T2 + macOS or Microsoft Pluton + Secure Boot. Until then, Apple iMessage will remain single platform.

This exists and has existed for years, via the SafetyNet Attestation API [1].

[1]: https://developer.android.com/training/safetynet/attestation

MetroWind(10000) 1 day ago [-]

Are we really at the point where letting a corporation decide what we can/cannot do on our own hardware is a good thing now?

Though the more I think about it the more I realize that we are indeed already at that point, and people really think that's a good thing. That's really sad to me.

a2128(10000) 2 days ago [-]

I'm having trouble understanding how this is a good solution... If a customer purchases a used iPhone from another person, and that person had used it for spam, is the customer now screwed and unable to message their friends without buying a new phone?

robbomacrae(10000) 2 days ago [-]

And yet I get plenty of spam via text on my iPhone. What is more, I cannot block numbers from texting me (unless there's an option I haven't found). What is more, a clearly spam text will stay as an alert number grabbing at my attention until I open up and see whatever spam image text was sent my way to dismiss it which is surely a security risk.

I used to work at Apple but this messaging stuff is really damning.

bergenty(10000) 1 day ago [-]

What's the point though. People still receive spam that doesn't originate in the iMessage ecosystem. The end result is the same.

AnthonyMouse(10000) 2 days ago [-]

This is a lawyer excuse. I've had Signal for years and the number of spam messages I've received over it is none. It's not a real problem.

SMS on the other hand... but iPhones receive SMS too, don't they?

tomxor(10000) 1 day ago [-]

> This permits Apple to simply "console ban" any Apple device that spams on iMessage.

This does nothing to protect users from non-Apple devices.

donatj(10000) 1 day ago [-]

I don't see what any of this has to do with Apple not supporting RCS.

JustSomeNobody(10000) 2 days ago [-]

This is about adopting RCS, NOT about pushing iMessage to other platforms.

gandalfff(10000) 2 days ago [-]

Would it be possible to have Android devices that have attestation, but with a one-way switch to disable attestation and allow users to root?

kalleboo(10000) 1 day ago [-]

There are literally multiple internal Apple emails released through court testimony where Apple executives clearly explain how important iMessage is to lock-in to iPhone and how if parents can just buy an Android and install an iMessage app it would mean disaster.

In none of these emails is spam or privacy or security even mentioned.

The primary reason Apple is doing it for platform lock-in, plain and simple. They literally said so themselves internally. Any other explanation is fanboyism.

stusmall(10000) 2 days ago [-]

It isn't about opening up iMessage. The article is about using RCS instead of SMS/MMS as the fallback. It's a pretty reasonable ask that will raise the quality of service when texting with the majority of the market. They can continue to lock down iMessage however they want.

raxxorraxor(10000) 1 day ago [-]

> and that is a key deliverable of Pluton

But there are very important key disadvantages that come with that. And I don't believe fighting spam is Microsofts MO. Just open Edge and look at the ads. This is very close to selling penis enlargement pills.

'device hackers' - seriously? You mean people that like to have control about what their devices do. Installing software you want should never be hacking.

That aside I am very skeptical of forcing Apple to open their messaging. The responsibility to choose a different medium is on the user.

dzikimarian(10000) 1 day ago [-]

There are literally emails leaked, that say iMessage is closed, because Apple wants monopoly in this area.

Yet in every thread recently someone spreads FUD how without uncle's Apple protection, bad world will hurt you, when reality shows that's nonexistent problem on other platforms.

alickz(10000) 1 day ago [-]

From the tech emails it seems they're more concerned about lockin than security.


>short version - don't make mail, calendar, iMessage work on Android and it's impossible to switch

yalogin(10000) 1 day ago [-]

Thank you. I have tried explaining this to people but the "freedom" people overwhelmingly flood the discussion and prevent any meaningful debate about it. Of course for them that point is not debatable but still for majority of people no spam is a huge deal.

cbsmith(10000) 2 days ago [-]

I mean... the spam texting I get is annoying, but it doesn't seem much different between iOS vs. Android devices...

thayne(10000) 2 days ago [-]

If that is the main reason, then why not use RCS when communicating with Android devices, and their own proprietary system when communicating with other iPhones. And or push to add an optional attestation to RCS that apple can use.

trissylegs(10000) 1 day ago [-]

You can connect to iMessage from a hackintosh though?

mjevans(10000) 2 days ago [-]

A good enough and low hanging fruit solution to spam is an allowed list. Generally allow contacts (initially at least). Track spam feedback by age against contacts.

If someone does end up in a spam list (and they don't rack up a high score across multiple targets), let them know they're in such a list and where to start looking to resolve that issue. A good enough solution for this is to have number carriers attest to have verified the government issued ID of the individual in question; and if spam happens shortly after that to yield the government ID number of that individual.

An alternate form I've considered, for email, is to pay a postage (transfer + storage) micro-transaction fee, and possibly an attention fee for prompt review. The custom might be to refund these in cases of legitimate messages.

calsy(10000) 2 days ago [-]

Any limitations/restrictions that Apple imposes on their devices that usually provides them some competitive advantage is ALWAYS explained away as 'protecting' the user. It's a joke how often this corporate spin is used as an excuse.

bearmode(10000) 1 day ago [-]

You're talking as if spam via text messages is a common occurence? I've had maybe one spammy text message on Android in the last 10 years, even though most websites I use have my phone number. Spammy phone calls are constant, but Apple doesn't do anything to prevent that.

soperj(10000) 2 days ago [-]

"I am concerned that the iMessage on Android would simply serve to remove and obstacle to iPhone families giving their kids Android phones," Craig Federighi, Apple software senior vice president, wrote in 2013.'

They'll find another reason not to implement it on other platforms.

GekkePrutser(10000) 2 days ago [-]

I never get spam on any other networks either. Be it WhatsApp, telegram, signal or even matrix.

It's not that big a problem apparently, and doesn't require giving up that much control.

On the other hand I social never use iMessage. It's not very popular here in Spain at all because of the Apple-only thing. Android is far bigger in marketshare here.

matheusmoreira(10000) 2 days ago [-]

> the same freedom to modify an OS kernel that hackers desire is also the freedom to spam all users

Yes, and that is absolutely fine. Computer freedom is more important than the ability to prevent spam. It should be illegal to prevent the rooting of devices or even put up any roadblocks for the user. It doesn't really matter how much this freedom impacts their networks. The freedom to run whatever software we want and interoperate with everything without being discriminated against should be our inviolable right.

runjake(10000) 2 days ago [-]

You can build a hackintosh, generate a serial number, and get on iMessage without any fully-authenticated hardware or even a legitimate secondary Apple device. Spammers use these setups to iMessage spam to great effect.

I think the onus is on Apple to open the platform.

jerryzh(10000) 1 day ago [-]

I have two cell phone, android and iOS. On Android I install my own spam filter message app and see no spam at all. On contrary, I still get plenty of spam from iMessage.

So I don't think it works.

sangnoir(10000) 2 days ago [-]

> Until then, Apple iMessage will remain single platform.

This seems to be a strawman - no one is asking for cross-platfrom iMessage, just for Apple to upgrade it's officially-supported cross-platform messaging stack (SMS) from the 90s.

s3p(10000) about 9 hours ago [-]

Since iMessage is restricted to those with an Apple ID, what's stopping them from releasing cross-platform apps that function only if the user has a valid Apple device? I think it's a business choice, not a spam one.

calsy(10000) 2 days ago [-]

The device still receives SMS messages, which makes all that wonderful iMessage security completely useless when receiving spam SMS messages with fake headers.

throwaway290(10000) 1 day ago [-]

I get imessage spam every day. I report junk every time, but it seems like they have infinite accounts.

pxeboot(10000) 2 days ago [-]

> Apple is using device serial numbers for anti-spam, supported by a fully-authenticated hardware and software stack that does not allow user modification.

This can't be true. It is trivial to get iMessage working in a macOS VM with randomly generated hardware IDs.

BiteCode_dev(10000) 1 day ago [-]

> Apple is using device serial numbers for anti-spam, supported by a fully-authenticated hardware and software stack that does not allow user modificatio

Ah, perfect tracking. Let's add that to Pluton list of promises.

dt2m(10000) 2 days ago [-]

This is a great point which I haven't heard before in this age-old debate.

But until Apple's dominance starts to wane, there's no chance in hell they will provide iMessage for other platforms unless forced by regulation.

If push comes to shove, they can implement heuristics which run texts from non-Apple devices through a harder spam filter. Spam isn't non-existent on the iMessage network, and there already seems to be a rudimentary spam filter in place.

mort96(10000) 1 day ago [-]

But... this is ridiculous on its face. SMS and iMessage both live in the Messages app. The only thing you achieve by locking down iMessage is that spam messages appear as green bubbles in Messages instead of as blue bubbles in Messages. It does literally nothing to prevent spam.

wilde(10000) 2 days ago [-]

This doesn't work though. I receive enough iMessage spam specifically through Apple ids that I wish I could disable the ability to message me unless you use a phone number.

rowanG077(10000) 1 day ago [-]

I literally never receive spam on telegram. And I have been using it for years. And by never I really mean never. I'm very doubtful spam is the crux of the issue here.

comex(10000) 2 days ago [-]

You can disable Secure Boot on a Mac and still use iMessage.

In this state, it would still be theoretically possible to attest to some kind of unique hardware ID, as the Secure Enclave is still locked down. But even if it weren't, it would be good enough to just distribute a unique key with each device. Sure you could take it off the device, but who cares? If it got banned, you'd still need to buy a new device for a new key.

...But given the sibling comment (by pxeboot) about using iMessage in a VM, I'm not sure whether any of this is actually done.

lern_too_spel(10000) 2 days ago [-]

Attestation is service that can only be provided by the builder of the phone. Most commercially available Android phones provide this, and banks and DRM rely on it. https://developer.android.com/training/safetynet/attestation and https://developer.android.com/google/play/integrity/overview

dcow(10000) 1 day ago [-]

What do you mean Android doesn't have an analog? It has both secure boot and device attestation. It has multiple APIs that can be used to design applications requiring varying levels of trusted computing context.

There's the high-level SafeyNet API which essentially lets you assert that you're running on a non-modified device running non-modified software in the context of a verified boot:

* https://developer.android.com/training/safetynet/attestation

It also has the lower-level Keymaster 3 API (since 2017) which provides HSM-signed certificates with the device attestiation extension, including the system trust level and verified device identifiers:

* https://source.android.com/security/keystore/attestation

* https://datatracker.ietf.org/doc/draft-bweeks-acme-device-at...

Microsoft is the one that's late to party... And your hypothesis seems pretty dependent on an argument that Apple can't build iMessaging on other platforms because they're the only platform with device attestation. That's simply not true.

If Apple wanted iMessage on other platforms, they've had at least 5 years to build it in the way you theorize must be required.

Rackedup(10000) 2 days ago [-]

That is just ridiculous.

I don't get random spam on Matrix/Element... it even handles video calls and more...

Do you get spam calls on your iphone?

Calamitous(10000) 2 days ago [-]

Then why do I still get texting spam on my iPhone?

yunohn(10000) 1 day ago [-]

Apple apologists always find some wacky reason to justify things, but this is the first I'm hearing of "iMessage exists to prevent spam"...

Have you used WhatsApp or Telegram? Neither need hardware attestation. No spam, in the decade I've used them. I have an iPhone and a Pixel, neither have spam.

Unless the gov allows free-for-all SMS, which is not the case in the continents I've lived in. Sounds like a uniquely US problem, which iMessage can't stop? You still get them, and Apple just hides them in a folder.

What are you on about really?

martius(10000) 1 day ago [-]

SMS spam isn't a thing on iOS?

shireboy(10000) 2 days ago [-]

The gist of the article has been a soapbox of mine for years. We wouldn't stand for 'you can only send Gmail email to other Gmail users' (Fidonet people know), and shouldn't settle for similar with messaging. That said, this spam angle is an aspect I hadn't fully considered.

tonymet(10000) 2 days ago [-]

RCS rollout has been a mess with a poor UX , unclear consent etc. Apple definitely doesn't want to inconvenience users with that garbage.

There are tons of great cross-device messaging apps e.g. Whatsapp , Signal, Telegram. The market has provided solutions.

Victerius(10000) 2 days ago [-]

How does have RCS have poor UX?

zaps(10000) 2 days ago [-]

"RCS is the modern standard adopted by most of the mobile world..."

(logos of Motorola, Samsung, OnePlus, Google Pixel, Snapdragon)

So.... Android then.

kelnos(10000) 2 days ago [-]

This may not be true in the US, but in the 'mobile world', Android is 'most of'.

collsni(10000) 2 days ago [-]

These comments are a firestorm. It would be nice for apple to adobt the rcs standard, would benefit native communications.

Gigachad(10000) 2 days ago [-]

Why are 'native' communications worth saving? Personally I prefer the current state where you can use whatever internet based platform you want and all these platforms are free to implement new features without spending decades trying to negotiate a new version between vendors and carriers. RCS right now is already and outdated and substandard platform.

tony-allan(10000) 2 days ago [-]

Two systems are suboptimal for developers. It would be great if Apple and Google could get their heads together and find a way to interoperate.

I don't mind the bubble colour... I'm OK with an Apple colour, Google colour, and an interoperable colour. It would be useful to signal what features might be available with each technology.

What I would really like is for the interoperable solution to have a common API so I can programmatically sent a message to anyone (at least the two main players with an option for other phone makers — and also apps like Telegram — to join).

fbanon(10000) 1 day ago [-]

>What I would really like is for the interoperable solution to have a common API so I can programmatically sent a message to anyone

No thanks, we don't want your spam.

Daunk(10000) 2 days ago [-]

I haven't called a 'normal' phone call or used SMS/MMS in many many years. Everyone I know (or care about rather) uses Telegram, and it's been great for us all.

Tagbert(10000) 2 days ago [-]

Telegram? Never heard of it.

Messaging currently requires you and the people you are communicating with to agree on a platform. If all you use is Telegraph, then you are not communicating with those who don't.

bagacrap(10000) 2 days ago [-]

that's terrific, but if telegram were the universal standard used for 95%+ of messaging then Google wouldn't bother with this effort.

balls187(10000) 2 days ago [-]

Does Twilio support RCS?

drcongo(10000) 2 days ago [-]

This is one of the funniest things I've seen in a long time. The company that has had 47 different messaging apps and changes them weekly trying to lecture the company that nailed it first time. Grow up Google.

ypeterholmes(10000) 2 days ago [-]

But isn't the request for a cross platform standard? Sure Apple got their own internal standard right, but the experience across platforms still matters.

seydor(10000) 2 days ago [-]

How did they nail it if it doesnt work well with 80% of phones?

etchalon(10000) 2 days ago [-]


Google can't act indignant that Apple isn't following their lead after they tried, and failed, repeatedly, to follow Apple's.

Google wanted a proprietary messaging service like iMessage for Android. They failed. They failed so many times they gave up and became champions of RCS, a standard the carriers were limping towards supporting.

Google pretending they're now champions of open standards and Apple is the big-bad meanie is ridiculous.

j1elo(10000) 2 days ago [-]

EDIT: ok I read other comments and learned that it's mostly a US-specific, cultural thing. I'm impressed that people in the US are still sending SMS, to be honest. Even with cheapest or free SMS, more modern chat apps became mainstream long ago in Europe, (probably) because they allowed to send photos and videos seamlessly, and messages didn't have length limitations.

Original message follows:

Could someone explain what this is about?

The page complains that pictures or videos are poor quality on SMS/MMS, and they lack encryption, like if those were relevant amd not a thing from the 90's.

Also I am reading things like 'Since I'm on Android, I'm stuck using SMS a lot, since most people I know have iPhones', and that adds to my confusion.

I guess the issue is that I simply cannot start to understand who would in today's context use SMS to chat. I too use Android and have a lot of friends with iPhones... we just send encrypted, good-enough quality pictures with whatsapp, telegram, or what have you. SMS or other cellular-based services are a thing of the past that nobody would use. So what's all the fuss about?

ch33zer(10000) 2 days ago [-]

If you friends an family use text you will too. It's simple network effects.

saxonww(10000) 1 day ago [-]

This is so funny because they don't even do a good job themselves with their own service.

I'm a Google phone user on Google Fi, and make heavy use of the web app (message.google.com/web) in addition to texting via the phone. Once you get a long conversation history, the phone experience becomes poor - conversations won't load, messages won't send - and the web will often lose sync and need to be re-paired (which may or may not itself work). On top of that, some days messages just will not send quickly, not over wifi or 5g, to the point that it's too hard to converse via text and we just give up.

Google has an alternative configuration for Fi users, btw: you can do all your calls and texts through Fi, have Fi store your messages and voicemails, etc. Except you have to turn off RCS for this, because Fi doesn't support it. So you get the SMS/MMS experience they are complaining about, and on top of that they convert all your audio and video to 3gpp and downscale the heck out of it such that it's nearly worthless.

Who knows what's realistically possible but I wonder if Google is going about this the right way. Call up Tim Cook and say we'd like to give Android users a better messaging experience, how can we work with you to do that. Don't try to embarrass Apple into helping you - Apple very likely does not care - and certainly if you're going to point out where their choices lead to a worse experience for your users, make sure you're not doing the same thing.

DangitBobby(10000) 1 day ago [-]

Just to counter your anecdote, I've never had a single performance issue on the Android Messages app on my phone or the web interface. I have had friends with iPhones that fail to render images I've sent them, and they send this obnoxious 'so and so responded with a thumbs up' (still do, android phones just do the right thing now and show a thumbs up reaction instead of the text). So YMMV. Apple needs to fix their broken messaging app.

kriskrunch(10000) 2 days ago [-]

I setup a BlueBubbles server on a spare Mac and I installed the BlueBubbles App. Boom. iMessages on Android. Done.

Messaging is already extremely fragmented. BB is only the eighth messaging app on my phone. Considering getting on Beeper to consolodate this madness. Matrix/Element was too rough to utilize as is.

Almost everyone I text with is on iMessage in the US. 95% or more of my regular contacts. Many are often outspoken about their cult like allegiance to iMessage. Finally some respect.

The look on their face when I send them an iMessage from my Galaxy is priceless.

Now, nearly all of the SMS messages I get are spam. Google Messages and the phone companies are pathetic at stopping them.

alphabetting(10000) 2 days ago [-]

I have not had this experience. Google Messages has been incredible at stopping spam for me. Android is best in class on calls as well. I was seeing memes about car warranties and looked it up. Apparently a big problem with spam calls. Haven't seen a spam call on my pixels in years. All filtered into the ether.

mr_toad(10000) 2 days ago [-]

> The look on their face when I send them an iMessage from my Galaxy is priceless.

Well, technically it was relayed through your Mac. Not sure how many Galaxy users have Macs to send message through, so I'm not sure it's a workable solution for most.

thomasahle(10000) 2 days ago [-]

Can this be done without setting up a private Mac server?

GekkePrutser(10000) 2 days ago [-]

Yeah and there's also mautrix-imessage for those wanting to use element.

I know it's rough setting up but there's a really great ansible playbook that makes it easy to maintain. I really love having all my messaging consolidated and not having all those privacy-invasive apps on my phone. Having my chats all in one big database is another boon.

The playbook doesn't support mautrix-imessage but that's because that needs to be run on macOS.

milleramp(10000) 2 days ago [-]

Loved "It's time for Apple to fix texting"

monocularvision(10000) 2 days ago [-]


vlmutolo(10000) 2 days ago [-]

What's really ridiculous is that Android is now parsing this and displaying a reaction bubble.

baby(10000) 2 days ago [-]

I always thought iMessage was the dumbest thing. I'm abroad, I try to iMessage someone because I know it's going to be free: oops, it downgrade to text and I end up paying. Or I'm on a bad wifi connection: oops, I will not use your signal and just block.

In any case everybody I know uses whatsapp so I don't think it's a huge deal. Texts are for spams or restaurant waiting lists.

fancyham(10000) 2 days ago [-]

Then disable 'send as SMS if iMessage fails'. There's a toggle for it.

jesprenj(10000) 2 days ago [-]

What's wrong with continuing to use SMS exactly? Why do we need to use alternative incompatible protocols if existing protocols are reliable and work even of 20 years old devices?

Both iPhone and Android support SMS and I see no improvement in either using iMessage or RCS. I may be biased because I don't use neither of them.

thomasahle(10000) 2 days ago [-]

> What's wrong with continuing to use SMS exactly

You can't send images with SMS. People like to send each other images.

eftychis(10000) 2 days ago [-]

The only reason (from my POV) to stop using SMS, is that it is a security nightmare (no integrity or authentication, can be MITM) and for some crazy reason everyone has decided it is a splendid idea to use it for 2FA.

That is like using wolves to protect your sheep -- sure if you keep them well-fed it might work...

Otherwise, Google is complaining because iMessage is a moat Apple has, that Google can not break on its own.

m3kw9(10000) 2 days ago [-]

Apple needs to fix their keyboard auto correct. It sucks for texting

dbg31415(10000) 2 days ago [-]

You're ducking right they do!

partiallypro(10000) 2 days ago [-]

Surprised the awful videos from platform to platform aren't highlighted more in the comments here. It's cleary Apple nerfing the video to make it look bad. The site's top video goes over it. It's awful.

latexr(10000) 2 days ago [-]

> It's cleary Apple nerfing the video to make it look bad.

The site itself contradicts that claim. From the FAQ (emphasis added):

> When people with iPhones and Android phones text each other, Apple relies on SMS and MMS, outdated systems that cannot support large media files. That means photos and videos are often compressed and come through blurry. The severity varies by location and carrier based on compression and size limits.

moizici(10000) 2 days ago [-]

Why would Apple fix something that do not affect Apple users ?

summerlight(10000) 2 days ago [-]

Simple; regulators will come after if Apple refuses to do so. DMA is just one response.

Veuxdo(10000) 2 days ago [-]

Per the article, they do affect iPhone users.

ElijahLynn(10000) 2 days ago [-]

It does affect Apple users, very much so.

vzaliva(10000) 2 days ago [-]

Apple messaging is super annoying. I use Android phone but also have an iPad. Whenever I chat with someone on iPhone, is suddenly decides to route all messages via iMessage instead of SMS and I do not see them on my phone. You have to disable iMessage in iPad to avoid this.

isatty(10000) 2 days ago [-]

I don't see how this is a problem. If you want to use an inferior method then it should be opt out (like you are doing) instead of the other way around.

kelnos(10000) 2 days ago [-]

I don't love RCS[0], but Apple implementing it (including the E2EE extensions) would strike a huge blow to messaging fragmentation immediately, at least in the US.

Hell, Apple doesn't have to ditch iMessage; they just have to support RCS for messaging with Android users, or group messaging with mixed Android/iOS devices.

I would also (grudgingly) accept an opening of the iMessage protocol so Google could implement it in the Android Messages app. Not ideal by any measure, and I figure Apple would never do this (and I suspect Google would hypocritically not want to do this anyway), but it would at least improve things.

The thing that's sad overall is that the current state of affairs is just a result of an anti-consumer corporate pissing match. The only losers here are the users, both on iOS and Android. And meanwhile both Apple and Google get to tout the benefits of their preferred solution as if they're both the good guys, fighting for their users. When in reality they're merely fighting for their own market dominance.

[0] Tying messaging to your carrier is just a continuation of the crappy SMS 'portability' experience. Sure, most RCS backend implementations are currently provided by Google, but one thing I'd like to see would be the ability to select your RCS provider. Maybe others would crop up if this were an option, and if RCS were actually popular.

thomasahle(10000) 2 days ago [-]

> the current state of affairs is just a result of an anti-consumer corporate pissing match.

This would be true if Google didn't want to implement the iMessage protocol as you predict; but in the current state of affairs it's just Apple being anticompetitive.

Asdrubalini(10000) 2 days ago [-]

Side note: I wonder why they didn't put Telegram in the 'Other messaging apps.' section, instead of only Whatsapp and Signal.

sneak(10000) 1 day ago [-]

WhatsApp and Signal are E2EE. Telegram is not.

seydor(10000) 2 days ago [-]

This is not very smart, as apple doesn't do such things unless coerced by law. Instead , android should drop/cripple iphone support until they adopt RCS

thomasahle(10000) 2 days ago [-]

The problem is that in the markets that use SMS and iMessage heavily (the US) Android has the minority. In general minority players have to go for interoperability, while majority players can try to force everyone else out of the market.

otterley(10000) 2 days ago [-]

Apple, pointing to all the happy children in the iMessage pool: 'nah, we good, thanks'

kelnos(10000) 2 days ago [-]

Which is why regulation needs to step in. The 'free' market has failed.

> happy children

Are they, though? Sure, if you're an iPhone user and everyone you know is an iPhone user, things are great. But once one person uses Android (or, god forbid, a feature phone), your experience gets much worse. No more E2EE, no more read receipts, no more typing notifications, no more reliable delivery and message ordering, no more sending videos at a watchable resolution...

jes(10000) 2 days ago [-]

I wish Apple would give me a way to filter junk SMS texts via a regexp or something, without needing a third-party app.

aaaaaaaaaaab(10000) 2 days ago [-]

Settings → Messages → Filter Unknown Senders

vzaliva(10000) 2 days ago [-]

One way looking at it is that carrier job is only to provide data service. They should not be in the business of messaging. Users (and market) will chose to use whatever messaging service they want.

This even applies to voice. I rather do Signal voice call than carrier voice call with most of my friends. Better quality, encryption, etc.

asdff(10000) 2 days ago [-]

I think that is shortsighted because data service does not overlap cell service. Go into a grocery store and try and imessage a photo. If your stores are anything like mine it just won't work, until you defer to sending it via SMS, at which point it sends instantly no problem. At the end of the day I'm not paying for data service, I'm paying for data service plus cell service. I want cell service, it has benefits and is not replaced by data only service.

kelnos(10000) 2 days ago [-]

I like this in theory, but in practice it is very useful to have a universal way to contact people that doesn't depend on both parties having the same proprietary[0] app installed.

If I give someone my phone number (or email address, for that matter), I know they will be able to contact me without any more coordination required. That has a lot of value to me. If we eliminate phone numbers entirely (and thus SMS, MMS, and telco-mediated voice calling), we lose that ability.

I do think that, in the future (perhaps not in our lifetimes) we'll do away with phone numbers, but only after there is another universal way to contact via messaging or voice that involves only sharing some sort of identifier, without any other setup required.

[0] Yes, Signal is open source, but since they refuse to allow people to use modified clients, or build support for federation, they are a proprietary, closed system, just like WhatsApp or Telegram. It's just we can have more confidence in the privacy of our communications on Signal due to its open source nature.

sneak(10000) 1 day ago [-]

Those who do not understand the OSI model are doomed to repeat it.

lostgame(10000) 2 days ago [-]

I can't take this site seriously. It says it's 'not about' the green and blue bubbles.

It is, and it's largely that Apple has a vested interest in making their ecosystem look so much better in general.

If I'm texting my friends with an Android and group chat, etc; isn't working properly - I will automatically assume something about Android is broken, because it works perfectly to my other friends who use iPhones.

Apple will never - ever - 'fix' this, because it's not 'broken', it's a design meant to create the illusion that iOS is the better ecosystem.

iMessage is one of Apple's most valuable psychological tricks to keep people within the ecosystem, or convert others to join in.

This is a waste of a call to action.

It will be about as effective as praying to Rain Gods for rain. :P

Apple has a massive vested interest in not fixing this 'problem'.

There's also a ton of cross platform messaging apps that already have no issues when used with each other - including popular open source ones like Signal.

The websites' creator has their heart in the right place, but their mind is confused. This is all intentional on Apple's part. It's genius and they know it. They will never willingly stop a plan that is working so very well.

curious_cat_163(10000) 2 days ago [-]

> Apple has a vested interest in not fixing this 'problem'.

Perhaps, you are right. Their vested interest is in making more $ for AAPL shareholders. The sands may shift. There are incoming regulatory pressures and what not.

However, it is still fair game to point out what is broken though. The Internet (such as it is) is full of opinions. It is not a waste. It is a perspective.

londons_explore(10000) 2 days ago [-]

> This is a waste of a call to action.

Agreed. I can't imagine what the decision makers at Google thought this webpage would do? Will it suddenly make Apple implement RCS - I think not...

The only thing that might make Apple make open messaging in the near future is the threat of the EU mandating it via the Digital Services Act. And those platform rules apply equally to any app with more than 45 million people - so iMessage, Whatsapp, Messenger, Instagram, Tiktok, Twitter, etc.

gjm11(10000) 2 days ago [-]

It seems likely to me that the site isn't particularly intended to be effective in making Apple fix their integration with non-iMessage texting. It's intended to be effective in raising awareness that Apple is deliberately doing a lousy job of integration with non-iMessage texting.

rootusrootus(10000) 2 days ago [-]

> iMessage is one of Apple's most valuable psychological tricks to keep people within the ecosystem, or convert others to join in.

Perhaps a kernel of truth there, but the real success of iMessage is how it gives you all the features of a modern instant messaging platform without any hassle. Built in to the phone, same app as SMS with automatic fallback, available on MacOS, not limited to a phone#, etc.

Yeah, I can go download one of a number of other IM apps. A small fraction of people I interact with will be reachable on any given app, but a majority are reachable with iMessage. The network effect is very real.

Veuxdo(10000) 2 days ago [-]

You've explained the subtext of the article. Which means the article did need to be written.

kart23(10000) 2 days ago [-]

I and a ton of other kids switched to iphones in HS purely because of imessage. Google knows how big a lock-in factor it is. the wsj article from a while ago is actually the truth: group chats are terrible, mms just doesn't work, etc.


smoldesu(10000) 2 days ago [-]

Maybe deep down, a lot of people on HN did too but won't admit it. Not calling anyone out, but I know how the crowd around here values 'networking' and the vanity associated with it.

boesboes(10000) 2 days ago [-]

Who uses sms anymore these days?

I tried to go back to a non-smart phone, but it was impossible due to not having whatsapp. That might be a 'local' thing though, not sure.

Anyway, they should just release imessage for android; that would piss off meta too, which is a win in my book ;)

thefz(10000) 2 days ago [-]

Agree. SMS is relegated to 2FA and before today I did not even know that Apple had a special SMS application for its users.

pjmlp(10000) 2 days ago [-]

Plenty of people in Europe with our pre-pay SIM cards, having like 5 000 free SMS per month, minimum.

s17n(10000) 2 days ago [-]

Everyone in the US (if there is at least one android user in the chat)

wejick(10000) 2 days ago [-]

I dont remember when the last time sending message via SMS. You will not be able to find message app on my android launcher because I hide it, and many people in my circle never really open it other than to read spam message from operator.

So yeah most of the time whatsapp and telegram 100% of my circle. SMS is a thing from the past, I guess the gen z here don't even understand what's SMS.

(someone from SEA region)

mongol(10000) 2 days ago [-]

It's the only texting solution you can be sure to know works if you just have a phone number. So in these situations, it is the best choice.

mrweasel(10000) 2 days ago [-]

> Who uses sms anymore these days?

Most people? But yeah, it's a local thing. Denmark have had free SMS for something like 20 years, at least as an optional add-on to your subscription. So there where never a reason to move to something else. If you frequently used SMS you just paid the small free for a large number of SMS message, or even unlimited. Current subscriptions pretty much all have free SMS.

When smartphones arrived, most just use the built in messing app. On the iPhone that means that you use iMessage, but it's not something you think about. If you took the average Danish iMessage user and asked them, they'd just say it's SMS.

I don't know that I would want Apple to just dominate the messaging market, but iMessage on Android would kill of many of the existing platforms pretty quickly.

r2_pilot(10000) 2 days ago [-]

I use and prefer that others use SMS for messaging me. I do have other communication apps, but by far and away SMS is my daily driver.

PaulsWallet(10000) 2 days ago [-]

I absolutely use SMS. I use Android and don't have Facebook or Whatsapp so if you want to text me you are gonna use SMS.

sunsetandlabrea(10000) 2 days ago [-]

This is pretty disingenuous I think. Other than Android who is using RCS?

Why can't I message between WhatsApp and an RCS client. Or any other chat technology, how about Google Chat to RCS, or Slack to RCS, or anything else.

Their examples for 'the modern standard adopted by most of the mobile world': Motorola, OnePlus, Google Pixel, Samsung, Snapdragon are all providers of Android phones, so clearly they would use the default Android messaging service.

I have a few folk (mostly family) who uses Apple messaging, everyone else seems to be on WhatsApp.

bagacrap(10000) 2 days ago [-]

A lot of Android handset manufacturers do not in fact leave the default X in place for most X.

ocdtrekkie(10000) 2 days ago [-]

It's no different than half a dozen web 'standards' Google invented like Web Serial, WebUSB, Web MIDI, etc. Google implemented it on their monopoly platform, and then declared it a 'standard' and started getting their staff to start trying to shame everyone else for not adopting it as such.

kramerger(10000) 2 days ago [-]

> Other than Android who is using RCS?

Don't forget Android has over 80% world-wide market share.

yubiox(10000) 2 days ago [-]

It seems like apple phones ruin photo and video quality when messaging with android too, even over third party apps like whatsapp and signal. Do they think I will run out and buy an apple phone so this doesn't happen?

Gigachad(10000) 2 days ago [-]

That's the apps themselves compressing the shit out of media to save on costs. Apps have full ability to get lossless media out and send it to anyone.

ivoras(10000) 2 days ago [-]

This is a US thing, right?

Haven't received an SMS from a real person (in other words, all SMSes I get are 2FA etc) for, at least 5 years, maybe 10.

Even people who use iPhones don't send SMSes, MMSes or anything as obsolete (including RCS). Everyone just seems to use WhatsApp and Telegram (or if they don't know any better, Viber). Locale: Central Europe.

So, why would anyone stick to the obsolete stuff? Are there regions of the US which have cell phone signals but no Internet access?

leokennis(10000) 1 day ago [-]

Want to reply on:

> Everyone just seems to use WhatsApp and Telegram (or if they don't know any better, Viber).

To remark that Telegram by default is not E2E encrypted, you need to explicitly start a 'secret chat'. And group chats are not encrypted as well. And when you start a secret chat it uses Telegrams 'probably maybe secure, but possibly not because it's a non-standard in house built' encryption scheme with weird choices.

I absolutely love Telegram, but I will also definitely not use it for anything more confidential than mindless chatter and cat pictures'.

rootusrootus(10000) 2 days ago [-]

Network effect. SMS works everywhere, all phones support it out-of-the-box. WhatsApp is opt-in. Almost nobody I communicate with regularly has a WhatsApp account.

chizhik-pyzhik(10000) 2 days ago [-]

For some reason whatsapp/telegram/etc haven't taken off nearly as well in the US as they have in the rest of the world.

NYT had an article about this recently- https://www.nytimes.com/2022/02/02/technology/sms-whatsapp.h...

bagacrap(10000) 2 days ago [-]

basically it's the lowest common denominator. There are so many chat apps out there (signal, sms, fb, ig are popular in my circles) and the default app is the only one everyone has installed.

For people close to me, I insist on the use of signal, but I don't have that kind of social capital with every single acquaintance.

RussianCow(10000) 2 days ago [-]

The US market standardized on mobile plans with unlimited texting a long, long time ago, so I think this caused people to mostly stick to SMS/MMS for communication since it was the path of least resistance. I don't know what the situation in Europe is like now, but in the past I remember it being difficult to find plans without very small SMS caps when traveling. That could be why Europeans naturally gravitated towards other messaging platforms.

phantomathkg(10000) 2 days ago [-]

South East Asia has tons of SMS spam, WhatsApp spam, Telegram spam and basically <insert any messaging app here> spam.

SMS not in used a lot in Europe doesn't mean the world is not using it.

rodgerd(10000) 2 days ago [-]

> This is a US thing, right?

This is the leading surveillance capitalism company trying to lay the groundwork to break the privacy of Apple's messaging system, demanding that Apple give up the privacy that it provides its paying customers, because it is intolerable to Google that there exists data that it doesn't have access to.

The rest is noise from morons who think that you don't deserve privacy unless you sysadmin your phone to an NSA standard, and people who work in adtech.

oneplane(10000) 2 days ago [-]

This is indeed a US thing (culturally). Most countries seem to have chat culture revolve around Whatsapp, Telegram, Signal, WeChat or LINE.

On top of that, most people don't really care and read whatever comes in regardless of the format.

MMS was a failed concept, and so is RCS. Not because the technology is fundamentally bad, it's the implementation that is fundamentally flawed by keeping telcos in the loop. The only reason SMS didn't die is purely by accident: it was included as some sort of auxiliary technical channel, not really intended as a means of chatting with other people. Heck, it was almost not even included in the GSM standard and mostly thought of as a useless waste of protocol specification. This made it unattractive to market or monetise at first, and later on with the whole ringtone/bitmap mess around the 00's it only enjoyed a short bubble of commercial exploitation.

The cost, and the limited format then caused the likes of BBM and even MSN for mobile to be used as true chat replacements, except in the USA. That was around the same time as the flop that was MMS. Then WhatsApp (and others) came along and by then the whole telco legacy mindset finally caught up and it was way too late. Then Apple came around and a decade later finally RCS was invented at some sad endeavour to get back in the loop as a telco.

Similar things were tried to 'replace' email etc. in the AOL days, which also turned into a big flop.

h3mb3(10000) about 15 hours ago [-]

My assumption was always that, from the get-go, iPhone just had that much of a bigger market share in the US compared to e.g. Europe. I remember in Finland in the early 2010s most of the people in my age group (~20-25 years old) had Android phones while I've understood in the US I'd been in the majority. Not suprising that that situation lead to different networking effects in the US vs elsewhere.

asdff(10000) 2 days ago [-]

Because its better than chat apps. I can send a text all over the place. Cell coverage for nondata service is incredible, you'd have to be really remote at this point to not have it at which point you definitely don't have a data connection. Meanwhile there are places all over my city where I can't get a reliable enough LTE connection to open my chat apps let alone send a message, much less one with any attachments. Inside stores are especially bad with LTE. I can't even get an imessage out inside the grocery store. I have to defer to sms, but then it sends instantly.

lutoma(10000) 1 day ago [-]

Yes. I don't think I know a single person in Germany that seriously uses iMessage or text message. WhatsApp, Telegram, Signal, ... is the name of the game here.

I think a major factor is that Apple/iOS has a much lower market share vs. Android here, so iMessage was never a viable option unless you wanted to reach only 20% of your contacts.

LatteLazy(10000) 1 day ago [-]

I finally got my elderly relatives on Whatsapp about 3 years ago.

snowwrestler(10000) 2 days ago [-]

> Are there regions of the US which have cell phone signals but no Internet access?

Yes, there are huge swaths of the U.S. that are lightly covered and don't support Internet applications. I was in Marin County, CA (just north of San Francisco) last year and regularly saw 0 bars of 4g. At those times only SMS got through to friends and family (Messages app falls back to SMS if there is insufficient bandwidth for iMessage).

JohnFen(10000) 1 day ago [-]

SMS is convenient because everyone has it. With those other services, I have to find out what they use, install it, register for an account, etc. Simpler just to send a text.

LegitShady(10000) 2 days ago [-]

It works for every phone and doesn't require me to have an app installed. It doesn't change on which contact I have ("oh she uses WhatsApp, he uses some other app, this group chat is on facebook messenger, etc".

It's just one tech that works on all phones. I don't even mind if its missing five million emojis or things like that.

patja(10000) 2 days ago [-]

does WhatsApp still require you to hand over all your contacts to them when you sign up?

None of my contacts gave consent for me to share their private information.

Bayart(10000) 1 day ago [-]

France here. I use SMS every single day, moreso than any specific given messaging app. I also routinely use Whatsapp with North Africans and Live Messenger with old FB contacts.

These things are highly localized.

code_runner(10000) 1 day ago [-]

(From the US)

I have never even considered downloading an app to text people... because I just text them. I've never understood or needed something different. I'm only now piecing together that this is an American thing though.

throwayawya11(10000) 2 days ago [-]

Maybe Google should enable push notification support again for Mail.app Gmail users too.

staticfish(10000) 2 days ago [-]

I'm pretty sure last time I checked, Gmail supported IMAP's IDLE[1] extension just fine. I get instant notifications on Mail.app.


kitsunesoba(10000) 2 days ago [-]

Or even just make Gmail's IMAP support properly spec compliant instead of requiring third party clients to hack around its nonstandard behaviors.

LeoPanthera(10000) 2 days ago [-]

> Texts from iPhones can't always be sent to Android over Wi-Fi, leaving your messages unsent and convos hanging if you don't have cell service.

Yes they can? I have no cellular service at home but I have wifi, and my iPhone connects to 'T-Mobile Wi-Fi' via my home internet.

SMS messages are sent and received just fine.

throwaway67743(10000) 2 days ago [-]

Assuming you have an operator that supports wifi calling and a phone that both supports it and is 'whitelisted' (basically USA, Europe does not do such silly things)

willio58(10000) 2 days ago [-]

From what I'm seeing RCS just isn't a true solution. Apple and Google should come together to create a standard outside of the carriers.

kitsunesoba(10000) 2 days ago [-]

Absolutely agree. Carriers have no rightful place in the discussion, they're dumb data pipes and shouldn't be able to nickel and dime customers on messaging quotas and features, as RCS is designed to allow.

enaaem(10000) 2 days ago [-]

Do we really need a single standard? I and many others use multiple messaging services and it's fine. Each has their pros and cons. I can also contact people in multiple ways if one service fails.

bern4444(10000) 1 day ago [-]

I can understand Google's frustration but they have no one to blame but themselves.

Everyone is familiar with their graveyard of failed messaging applications (along with their graveyard of products generally).

When I had an android phone, I tried RCS with someone else on Android. It never worked. I'm sure it's improved, but as the common theme of this story goes, Google blew their chance.

I also don't trust google to abide by the 'standard' they've created. Their track record is abysmal. I don't want to use yet another messaging service that they've built. I don't trust it to exist in the future, receive support and updates, and for it to be maintained. If google abandons it that means the telecoms are stuck holding the bag and when we demand even more from the next iteration of messaging apps, RCS will go the way SMS has today.

This is nothing more than Google reaping the results of their own failures. It's a shame they squandered the opportunity over the last 15 years to develop a cohesive messaging app strategy across their products, but its their shame and now they have to pay the price.

In a last ditch effort they're trying to throw all the blame on Apple who was able to innovate and launch a successful messaging service years ago. Apple recognized that SMS could be improved, and they improved it. They didn't wait for anyone and they recognize the importance of continued support - a quality Google does not seem to foster.

Google had their chance over and over again but they blew it over and over again. I don't care about the little things RCS adds, message bubbles, delivery confirmation etc. These may be nice additions but they truly don't make a large difference. If an Android phone wants to send a high quality image, video etc they can share a link. That's good enough for me.

Google failed, miserably and publicly. This latest campaign is just embarrassing for them.

upbeat_general(10000) 1 day ago [-]

If you don't care about the additional RCS features, then aren't you saying essentially SMS is fine? Do you prefer iMessage just since it is more secure (compared to sms that is)?

raverbashing(10000) 1 day ago [-]

> Everyone is familiar with their graveyard of failed messaging applications (along with their graveyard of products generally).

I agree

And that's why Whatsapp and maybe Telegram/Signal/etc are the 'standards' today

thomasahle(10000) 1 day ago [-]

> I don't want to use yet another messaging service that they've built.

You literally wouldn't be. You would keep using iMessage. All that would be different is that you could now send images and videos to Android users. Right now you have to switch to another app to do that.

This is not a ''standard' they've created', this is a GSM Association standard, and it would be Apple, not Google, implementing it.

shellac(10000) 1 day ago [-]

> Their track record is abysmal.

And just to reinforce this, a decade ago Google supported a cross platform messaging standard: XMPP. And then they stopped it https://www.zdnet.com/home-and-office/networking/google-move....

obnauticus(10000) 2 days ago [-]

I would agree more if the RCS standard wasn't also hot garbage...

I would encourage anyone who is curious to read more about it. It's taken so long to gain traction that it has also become somewhat legacy. Also, it still requires a carrier sponsored phone plan? How is this "modern" in comparison to say every other carrier agnostic messaging app in existence?

Also this: https://twitter.com/RonAmadeo/status/1480679515298934786

resfirestar(10000) 2 days ago [-]

>There are zero benefits to phone identity over email

I can think of one: most people's email identity is subject to termination under Google's ToS. Same thing with identity tied to Facebook or other social networks. In the US, your ability to take your phone number to a different carrier is protected by federal regulations.

a2tech(10000) 2 days ago [-]

No one really wants to understand it, they just want to complain that Apple doesn't support it

arbirk(10000) 2 days ago [-]

Very interesting. I wonder what protocol and format the EU commission will point to in enforcing the Digital Markets Act

Hippocrates(10000) 2 days ago [-]

Agree. It sounds similar to the argument for USB-C charging, also a hot mess of a standard. But RCS is definitely more offensive.

lostgame(10000) 2 days ago [-]

I miss XMPP :(

2OEH8eoCRo0(10000) 2 days ago [-]

Google cannot legally ship, as part of Android, a carrier-agnostic messaging app like iMessage.

upbeat_general(10000) 1 day ago [-]

I don't love phone based identity but it's wrong to say it has no benefits.

While it does lock you out if you don't pay, at least you won't be locked out by accident since you can generally prove your identity to the carrier. This obviously is a con (sms hijacking) but for many people it's much more important.

Not to mention the importance of phone numbers being basically universal which is why 3rd party messaging apps haven't totally replaced sms. RCS has the potential to do so, or at least cut down on sms usage further.

equalsione(10000) 2 days ago [-]

It really is god-awful. RCS is a technology that benefits mobile operators, not users.

Also, Google really aren't in a position to lecture anyone on this topic, given their N+1 approach to messaging services.

kart23(10000) 2 days ago [-]

ehhhh, that twitter post is weird. It's like your phone number. You're free to switch carriers, and just like a phone number, you lose it if you stop paying. It's not designed to replace whatsapp, its replacing sms.

ElijahLynn(10000) 2 days ago [-]

It is actually light years better than SMS/RCS and has a huge value to end users. I can see if a message was read, I can send legit voice memos without size limits, I can send large high resolution photos.

It may not be perfect but it is better than what Apple is doing now.

blinkingled(10000) 2 days ago [-]

> iPhones make texts with Android phones difficult to read, by using white text on a bright green background.

Wow. I can't really come up with anything creative to blame Google for this one. Whatever you want to say about Google's messaging mess and RCS - Apple seems to go out of their way to make it inconvenient to text with Android users.

Also it doesn't sound like Google's asking Apple to give up iMessage - just that they use RCS instead of SMS/MMS to talk to Android users. Not an unreasonable ask given RCS is likely to be a widely adopted standard and an non-trivial improvement over SMS.

Edit: Color aside, the read receipts, MMS quality, Wifi send etc all seem worth fixing with RCS.

wincy(10000) 2 days ago [-]

What? I don't even notice the difference in color except that I know not to use the tapback stuff when I'm texting an Android user. Does the green on white actually bother anyone? This seems like grasping at straws to me.

Angostura(10000) 2 days ago [-]

This is the most trivial complaint I've ever read. I'm in my 50s and I have zero problems reading green bubbles - it just means that it hasn't been sent via iMessage - if I send to an iPhone and sending falls back to SMS it looks just the same. I can't believe people get that upset about green v blue.

nemothekid(10000) 2 days ago [-]

iMessage was released on iOS 5 with the release of the iPhone 4S. Before then, all messages had a green background. Somehow sticking with the default of more than 10+ years is intentional maleficence by Apple?

nomel(10000) 2 days ago [-]

The green used on the website is significantly brighter than on an iPhone. In fact, on the iPhone, I would say the green gives better contrast than the white text on a blue background.

For direct messages, the colored bubbles are only used on messages you send. Messages received are always white text on black background (dark mode) or black text on grey background (light mode).

edit: my bad. 'increase contrast' option is on, set years ago.

kingTug(10000) 2 days ago [-]

The puke-green text bubbles from android and calm-blue bubbles from iMessage always struck me as very intentional.

tomjakubowski(10000) 2 days ago [-]

The white on green is only used for the SMS messages you sent in the conversation.

Incoming messages are always black on grey, for SMS and iMessage both.

If the color scheme actually impacts legibility, it would only affect messages which you yourself wrote. It wouldn't have any effect on legibility of messages other people wrote, where that really matters a lot more.

etchalon(10000) 2 days ago [-]

The Green is historical, not a specific decision by Apple to hinder reading texts.

Before iOS 5, and the release of iMessage, all messages on iOS were green.

That Google is painting this as something else speaks to how disingenuous this whole conversation has gotten, in all corners.

NonNefarious(10000) 2 days ago [-]

Apple commits many UI offenses, but the alleged illegibility of SMS messages is BS.

Not to mention that Apple's messaging is hideously broken in more ways than Android integration. iMessage will simply delete your phone number from its 'can be reached at' list, which breaks years-long threads with a single (iPhone-using) friend into inexplicable new threads.

Ever go overseas? Try putting a local SIM into your USA phone somewhere else, and watch your phone 'forget' all of your contacts. Seriously: WTF? Suddenly all of your contacts are unrecognized by number. It's idiotic.

dan-robertson(10000) 2 days ago [-]

One thing to note is that received texts show up the same, it's only sent messages that are blue/green.

dataflow(10000) 2 days ago [-]

> I can't really come up with anything creative to blame Google for this one.

I got the impression RCS de-facto depends on Google servers in some way. Can someone confirm if that's the case?

isodev(10000) 2 days ago [-]

These days it's good marketing to blame Apple for everything.

First, RCS is not a very modern or practical standard. It was created in 2008 by carriers (GSMA!!) for their SMS/MMS centric (at the time) platforms. So is it a good idea to adopt this more than a decade later? I don't think so.

Second, it's really not Apple's fault that Google has failed to come up with a messaging solution. Google has released 13 separate messenger apps since 2011!

"Right now, Google runs three mainline messaging apps: Besides this Google Messages/RCS platform, there's also Google Chat, which is a more traditional over-the-top messaging service, and Google Voice, which is a Google-provided phone number with SMS. Google Hangouts is technically still around as the fourth messaging app, though that's shutting down in November. There are also siloed messaging apps built into Google Maps, Google Photos, Google Stadia, Google Pay, Google Assistant, and Google Phone, and none of them talk to each other.

Google's head of messaging also quit last month, so there's no telling what the future of Google messaging holds until someone takes the reins. I would suggest Google get its house in order before it starts throwing rocks at Apple." https://arstechnica.com/gadgets/2022/08/new-google-site-begs...

lostgame(10000) 2 days ago [-]

This point is a fallacy.

Prior to the invention of iMessage - (iOS 6?) - all iPhone texts were this colour of green. It was the same no matter who you texted.

This means Apple never intentionally designed the green bubbles to be more difficult to read.

firloop(10000) 2 days ago [-]

Feels like sort of a non issue, even the bottom of the page pushes people to apps like Whatsapp/Signal. If Google wants better iPhone messaging - can't it just ship its solution in the App Store? Not really sure why Apple must update iMessage for Google to get what it wants.

I personally love iMessage and use it and Signal primarily - I don't like the idea of Google dictating its feature set, especially considering its horrible messaging track record.

bagacrap(10000) 2 days ago [-]

No, Google isn't trying to ship another messaging app. It's trying to improve the interoperability of Android and iPhone when using phone number texting. Your experience in iMessage when texting with an Android user would be improved.

aquanext(10000) 2 days ago [-]

I use an iPhone and have never experienced any of these issues with blurriness. Do they have specific examples? As others have said, I think I'm good with the way things are right now.

thomasahle(10000) 1 day ago [-]

Ask a friend with an Android to text you a video, and see if it looks sharp or blurry. Using RCS as fall back would literally change nothing for you, except better quality conversations with Android users.

O__________O(10000) 2 days ago [-]

Neither SMS, nor RCS, have built-in end-to-end encryption — and both should die a quick and timely death. iMessage to use end-to-end encryption, but it's not open, and should also die.

What messaging needs is an open modern standards, nothing more, nothing less.

eftychis(10000) 2 days ago [-]

RCS is dead. The Cross Carrier Messaging Initiative (CCMI) has given up. Google is the one still pushing for it. iMessage is simply a big Apple moat. Why would Apple give it up to enhance Google's business position?

Google wants us to pick theirs over Apple's. Also note that to my knowledge, RCS is not available in all countries.

The other funny thing is that Google complains about SMS being insecure -- all while RCS does not support end to end encryption. Google Messages added that feature recently relatively (last year? please correct me below) and I still can't understand if it's on by default or not.

Here is a random article about RCS state. Feel free to google for more: https://linustechtips.com/topic/1327240-in-the-us-rcs-text-m...

I would love a common solution, but rationally I can not blame Apple for keeping a (to me from experiencing both) superior experience that brings in customers. And Google has just managed to catch up. Google will need to make their messaging an order or two better, to the point that Apple will have to join.

P.S. Also, I am skeptical any time the phrases privacy, end-to-end encryption, and Google cohabitate the same statement.

(Edit)P.S.2. Just use https://apps.microsoft.com/store/detail/bluebubbles/9P3XF8KJ... in the meantime.

dools(10000) 2 days ago [-]

I run a phone business in Australia which is, as far as I'm aware, the only product that supports voice, txt and picture messaging on a virtual number outside of North America.

Ever since I built the product, people have been telling me SMS is obsolete and RCS is coming (4 years now). Google bought Jibe Mobile in 2015, if you go to the Jibe website and try to submit their 'Get Started' form there is an error.

I have tried to get in touch with carriers to find out how to connect up RCS from my product (because hey, don't want to get behind the 8 ball) and haven't found any way to get it set up, even when asking my upstream providers.

I really don't think RCS is going anywhere, but if it is, it would be good to be able to build it into my product!

[0] http://www.benkophone.com/

jvolkman(10000) 2 days ago [-]

A lot has happened with RCS since that random article.

* All of the major US carriers announced that they'd ship Google Messages by default on android phones (including RCS). I believe Verizon was the last one [1].

* Google enabled end-to-end encryption by default for 1:1 chats [2]. They've said that e2e for group chats is coming later this year [3].

* Samsung replaced their own messaging client with a tweaked version of Google Messages on the S22 (edit: in the US) [4]. Samsung Messages already supported RCS, but I'm not sure if it supported Google's extensions like e2e.

And as others have mentioned: this isn't about Google wanting Apple to replace iMessage with RCS; it's about Google wanting Apple to support RCS as iMessage fallback in addition to the existing SMS support. Apple to Apple would certainly still be iMessage.

1: https://9to5google.com/2021/07/20/verizon-will-adopt-google-...

2: https://arstechnica.com/gadgets/2021/06/google-enables-end-t...

3: https://9to5google.com/2022/05/11/google-messages-rcs-group-...

4: https://www.androidpolice.com/samsung-galaxy-s22-series-ship...

morsch(10000) 1 day ago [-]

Maybe it's dead in the US, but German providers added support in 2021. Of course everyone here already uses cross platform messengers, so I guess it's dead, here, too.

__derek__(10000) 2 days ago [-]

First, Apple shaming Microsoft. Then, Microsoft shaming Google. Now, Google shaming Apple.

> missing read receipts and typing indicators

Life is better without both of these.

> no texting over Wi-Fi

This claim was odd. I visited Europe a few months ago and definitely sent/received SMS over wifi using my iPhone.

> When people with iPhones and Android phones text each other, Apple relies on SMS and MMS, outdated systems which do not always support texting over wi-fi. That means if you don't have a cellular network connection, depending on your carrier and situation, you may not be able to send and receive texts.

Oh, so the claim was deliberately misleading. That's not a good way to build trust.

egwynn(10000) 2 days ago [-]

> definitely sent/received SMS over wifi using my iPhone

Are you certain? From what I understand about how SMS works, I don't see how that's possible. Apple's own docs also appear to suggest that SMS-over-WiFi won't work: https://support.apple.com/en-us/HT207006

minhdanh72(10000) 2 days ago [-]

For me another frustrating thing when using iMessage on iPhone is that I often receive spam messages from unknown contacts, which I cannot block at all.

Jtsummers(10000) 2 days ago [-]

Click on the message thread, click on the sender at the top of the thread, select 'info' for the contact, at the bottom select 'block'. Not obvious, I'll agree, but you can definitely block it (it should be a much shorter path). This works for iMessages and SMS.

gravytron(10000) 1 day ago [-]

So we're just going start kicking and screaming because iOS users have a different experience than android users? Can anyone here help me understand why Android feels entitled to dictate non Android users' experience? This seems like a cultural failure within Android that is a byproduct of their inability to satisfy their users. They have completely run out of cards and all that they have left is low budget low effort marketing like this - or is that actually all that they ever had, to begin with after all? Haha.

Maybe focus on delivering a meaningful experience for your users and pull your nose out of the tail end of the iPhone?

upbeat_general(10000) 1 day ago [-]

It's a pain for iOS users as well. I have a friend with an android phone I want to message. We all have issues with the various 3rd party apps but her phone has solid RCS support. My iPhone does not (and cannot) until Apple implements RCS support.

sneak(10000) 2 days ago [-]

This is spam for the Android operating system, nothing more.

Google claiming that RCS includes end-to-end encryption here is misleading.

Encryption got explicitly axed from the RCS spec because carriers don't like it.

The end-to-end crypto they're talking about is a custom Google thing and not part of RCS.

Friends don't let friends use unencrypted everyday communications.

Reject RCS and reject Google platform marketing.

PS: Note also that iMessage has a crypto backdoor maintained by Apple for the FBI; Google should not be encouraging iMessage to become more useful/popular, as this reduces privacy and makes people less safe.

giantrobot(10000) 2 days ago [-]

> Note also that iMessage has a crypto backdoor maintained by Apple for the FBI

Describing iMessage as having an encryption back door is disingenuous at best. There's no known encryption back doors in iMessage. As far as can be determined iMessage is fully E2EE.

There is however a caveat that the unencrypted local messages backed up to iCloud can be turned over to LEAs/governments. This caveat holds for any unencrypted data synced to iCloud. It's no different than your e-mails sitting in your iCloud mail account. Disabling syncing of Messages with iCloud closes this hole.

Historical Discussions: We're improving search results when you use quotes (August 05, 2022: 738 points)
Google improving search results when you use quotes (August 04, 2022: 10 points)

(738) We're improving search results when you use quotes

738 points 7 days ago by Kortaggio in 10000th position

blog.google | Estimated reading time – 5 minutes | comments | anchor

In the past, we didn't always do this because sometimes the quoted material appears in areas of a document that don't lend themselves to creating helpful snippets. For example, a word or phrase might appear in the menu item of a page, where you'd navigate to different sections of the site. Creating a snippet around sections like that might not produce an easily readable description.

We've heard feedback that people doing quoted searches value seeing where the quoted material occurs on a page, rather than an overall description of the page. Our improvement is designed to help address this.

Things to keep in mind about quoted searches

For those doing quoted searches, here are some more tips, along with caveats on how quoted searching works.

Quoted searches may match content not readily visible on a page. As referenced above, sometimes quoted searches match content contained within a web page that isn't readily visible, making it seem like the content isn't on the page when it actually is present.

For example, content in a meta description tag is looked at for matches, even though that content isn't visible on the web page itself. ALT text that describes images is considered, as is the text within a page's URL. Material brought in through inline frames (iframes) is also matched. Google may also see content that doesn't initially load on a page when you go to it, such content rendered through JavaScript that only appears if you click to make it display.

Pro tip: Sometimes people use the standard Find command in a browser to jump to the phrase they want, after arriving on a page. If that doesn't work, though, you can try using a developer tools option. For instance, in Chrome, you can search from within Developer Tools to match against all rendered text, which would include the text in drop-down menus and other areas of the site.

Pages may have changed since Google last visited them. While Google revisits pages across the web regularly, they can change in between visits. This means quoted material might appear on a page when we saw it, but it no longer exists on the current page. If available, viewing the Google cached copy may show where the quoted content appeared on the version of the page we visited.

Quoted terms may only appear in title links and URLs. Quoted terms won't appear in web page snippets if they only appear within title links or URLs of a web page. We also do not bold matches that happen in title links and URLs.

Punctuation is sometimes seen as spaces. Our systems see some punctuation as spaces, which impacts quoted searches. For example, a search for ["don't doesn't"] tells our systems to find content that contains all these letters in this order:

don t doesn t

As a result, we'll match content like the ones below, where punctuation like commas or hyphens break up words — because when you remove the punctuation, the letter patterns are the same:

  • don't, doesn't
  • don't / doesn't
  • don't - doesn't

Snippets might not show multiple quoted terms. If a search involves multiple quoted terms, the snippet may not show all of them if they are far apart from each other. Similarly, if quoted material appears several times on a page, a snippet will show what seems to be the most relevant occurrence.

We mainly bold quoted content for web page snippets on desktop.

Our new bolding of quoted content generally only works for web page snippets on desktop. Bolding won't appear in snippets for recipe or video boxes, and it also won't appear when using some special modes such as image or news search. However, anything listed in these boxes or special modes will contain the quoted terms. Bolding also doesn't work for mobile results.

Quoted searches don't work for local results. Quote restriction does not work for results in our local box where listings usually appear with a map; we'll be looking more at this area in the future.

To quote or not to quote?

Using quotes can definitely be a great tool for power users. We generally recommend first doing any search in natural language without resorting to operators like quotation marks. Years ago, many people used operators because search engines sometimes needed additional guidance. Things have advanced since then, so operators are often no longer necessary.

By default, our systems are designed to look for both the exact words and phrases entered and related terms and concepts, which is often useful. If you use a quoted search, you might miss helpful content that uses closely related words.

Of course, there are those times when the exact word being on a page makes all the difference. For those situations, quoted searches remain available and are now even better.

All Comments: [-] | anchor

8bitsrule(10000) 4 days ago [-]

Typing two quotes is harder than just typing a comma after a multi-word string.

And if I type in a year ... 2020 ... at least make some effort to limit the top results to those that contain it.

dannysullivan(10000) 3 days ago [-]

We have date range operators such as before/after that do exactly that. See here: https://twitter.com/searchliaison/status/1115706765088182272

bitL(10000) 7 days ago [-]

With Google+ gone, will + make a return as well?

pipeline_peak(10000) 7 days ago [-]

Yeah, to get 2010 search performance, you'll have to buy the new Google+ Premium Service package for $5/month

copperx(10000) 7 days ago [-]

Google is the Clippy of this day and age.

Why can't we have a Google Advanced Search? Is it that hard? Is it rocket surgery? Is it forbidden by the powers that be? Would it start WW3?

I've always found it baffling that we have to play guessing games with a capricious search engine that tries to interpret what we want. There are some times when we know exactly what we're looking for!

jabits(10000) 7 days ago [-]

I'm pretty sure indexing and searching the entire internet satisfactorily to all is harder than rocketry or surgery...

collegeburner(10000) 7 days ago [-]

google literally starts captcha blocking me if i do too many quote or intext searches. there is obviously a business case for fucking over power users. 'you will look at the results The Algorithm surfaces and you will like it'.

flenserboy(10000) 7 days ago [-]

Those users probably don't give their clicks to ads, obvious or otherwise. Their focus on useful results means they cannot be easily monetized, and therefore need to be dissuaded from using G as much as possible.

O__________O(10000) 7 days ago [-]

First, I sincerely appreciate Google's efforts to acknowledge there are issues and fix them; it's amazing, thank you!


TLDR: It's impossible to report issues when Google makes it impossible to replicate the issues. In the below text, found four errors: (1) feature mentioned in Google's blog post is not working; (2) quoted search is not working; (3) + search operator does not work as it used to work; (4) it is impossible to replicate results. — Aware of likely hundreds of bugs like this, wish Google would listen.


This comment was a response to another comment, but given all the bugs I found, decided to make it a top level comment.


It does not, at least in one way, which was as a substitute for quotes when injected into a string where +'s replaced spaces; just confirmed this does not work, indifferent about it returning; see links below for proof.

My core concern is Google list ALL search operators OR operations AND publicly make SERP experiments per each that are reproducible without millions in resources AND whitelist for automated execution of these tests; if in unlikely chance Google's looking for strong opinions how to radically improve search quality, I am willing to do it for free or paid.

Here's the example searches that show + does not work in at least one way it once did as mentioned above:

- millions of results


- 3 results


* This is not the same as the +keyword syntax mentioned in thread; I will try to find an example.

EDIT: weird...

If I click this link:


I get zero result results, but if I click:


Then edit the a to z without making any other changes, like this:


I get 3 search results... which clearly is a bug.

EDIT: Grr... I was able to reproduce it, but then Google broke it. Using method above, here's proof it returned 3 results:


- Note: if you look at the screen shot above, you will see a SERP that breaks the feature Google announced that in the blog post; specifically that the query is not highlighted in the SERP description. More importantly, I searched for 'test test test test test z' but got results for 'text text text text text z' — which means quoted search itself is broken.

And proof I then got a bug (zero results) using the exact same search:



* Conclusion: To me, if you cannot share a search result and get the EXACT same search results (and possibly ability to see different ones AND annotations of why they are different) — how this not a bug; how can anyone independently test Google's search quality?

rnnr(10000) 6 days ago [-]

If you want the old stricter Google behavior be sure to check the 'Verbatim' option under 'tools', or add '&tbs=li:1' to your uri. Around 2008 Google started testing a new search engine logic, codenamed caffeine?, which eventually became the default. Without being sure, I think verbatim uses the old engine which wasn't trying to be smarter than you and your query, certainly is stricter and reminds the old Google a lot.

pipeline_peak(10000) 7 days ago [-]

They're being lazy and irresponsible. Non tech savvy people don't use delimiting quotation marks, nor should they. They are a technical feature for narrow searches.

Google Search has always been the dominant bridge between users and sites. Given the obvious complaints over search results, for a company of this stature to talk about explicit search performance during these times is pretty tone deaf.

To me it's a clear indicator that Google Search is many years away from being their flagship product. It's truly unfortunate to know there are thousands of people who'd like to change that by working together but simply can't because Google put their flag up first. And it's all because this once curious academic project has devolved into a surveil marketing machine.

Fuck Silicon Valley and brogrammer culture

dannysullivan(10000) 3 days ago [-]

The post specifically did NOT tell people to make use of quotes when searching. It said the opposite, and simply explained the change with snippets is to better help those who decide they do want to use quote searching for whatever reason. Here's what we said (I work for Google):

Using quotes can definitely be a great tool for power users. We generally recommend first doing any search in natural language without resorting to operators like quotation marks. Years ago, many people used operators because search engines sometimes needed additional guidance. Things have advanced since then, so operators are often no longer necessary.

By default, our systems are designed to look for both the exact words and phrases entered and related terms and concepts, which is often useful. If you use a quoted search, you might miss helpful content that uses closely related words.

Of course, there are those times when the exact word being on a page makes all the difference. For those situations, quoted searches remain available and are now even better.

lehi(10000) 7 days ago [-]

DuckDuckGo's absurd handling of quotes finally forced me back to Google. Adding quotes in DDG gives nonsense results: https://imgur.com/a/2SHpjPG

johnklos(10000) 7 days ago [-]

Same about DDG, but since I'll never go (directly) back to Google, I set up my own SearXNG instance. So much better!

wizofaus(10000) 7 days ago [-]

It seems DDG handles quotes somewhat similarly to how Bing does (I just tried 'input: dispatch' and got the same list of useless results from both. Google's are no better though.) But Bing is much better for 'life sucks and so do you'. Strikes me that DDG just isn't 'there' yet, which is the impression I've had almost every time I've used it.

marricks(10000) 7 days ago [-]

All my search results have that exact phrase in it, just some with non alpha characters in that phrase.

I have no clue what your DDG is doing but it's not behaving right.

emptyparadise(10000) 7 days ago [-]

I love DDG but it's so overzealous with rewriting queries to include more (usually irrelevant) results.

zild3d(10000) 7 days ago [-]

> Years ago, many people used operators because search engines sometimes needed additional guidance. Things have advanced since then, so operators are often no longer necessary.

Except it's been regressing to where operators are often more necessary again

alecco(10000) 7 days ago [-]

It's not entirely Google's fault. SEO-oriented content is ruining the web. It's a battle I'm not sure search engines can win.

Keyword indexing and PageRank worked for a while because the underlying data wasn't written trying to game them. Then came spam linking and keyword stuffing and more. I'm surprised search engines are still useful in spite of that.

wizofaus(10000) 7 days ago [-]

Funnily enough I just searched for 'input: dispatch' and couldn't see a single match that included the key colon character so I really hope they fix that.

carrotcarrot(10000) 7 days ago [-]

Google has replaced punctuation white white spaces for at least a decade.

boredemployee(10000) 7 days ago [-]

Lately I've been using books when I need information, instead of google, much better and less privacy issues. I recommend it!

klysm(10000) 7 days ago [-]

That's a very expensive alternative compared to a VPN

mcswell(10000) 7 days ago [-]

I recommend clay tablets. Fire them, and they'll last pretty much forever, or at least a few thousand years.

noduerme(10000) 7 days ago [-]

Okay, except the quote operator doesn't work as described and hasn't worked that way in at least 10 years. 'Did you mean x?' No, asshole, I put what I meant in quotes.

dannysullivan(10000) 7 days ago [-]

I work for Google search. It does work as described. If you have an example you're comfortable sharing where you feel it doesn't, I'd be happy to have the team look into it.

kgc(10000) 7 days ago [-]

So basically like how it used to work before.

fswd(10000) 7 days ago [-]

Yeah this was the standard in 2003-2005 if I recall. Then they removed it. Took them almost 20 years to reinvent it...

causality0(10000) 7 days ago [-]

I can't believe I was so naive as to think this was going to be Google apologizing for how it screws up most search operators by only obeying them if Google agrees with you. Most of the time it happily includes all sorts of SEO horseshit like synonyms, related words, and even companies who happen to be competitors of what I wanted exact results for. Words in quotations aren't supposed to be suggestions.

dannysullivan(10000) 3 days ago [-]

Words in quotes aren't suggestions. They tell us (I work for Google Search) to find only what's in the quotes. That's how they've worked for years. We haven't been 'only obeying them' if we agree. I'm not even sure what that's supposed to mean. We obey them as outlined in that post. The problem is that people sometimes can't find the quoted material on the page -- that we definitely did see and definitely did restrict to -- so we made it clearer in snippets to help them understand what we saw, where we saw it and hopefully guide them to the right locations in the doc.

mcswell(10000) 7 days ago [-]

Just to be persnickety: If you do a search for 'dog' (with the quotes), do you expect to find hits with only 'dogs'? What about 'doghouse' or 'Dogbert'? 'Dogged' (the past tense or past/passive participle of the verb 'dog')? If you search for 'goose', do you expect to also see 'geese'? (And for good measure, what should be done about languages where most nouns have case marking suffixes, like Russian, or languages with lots and lots of suffixes, like Turkish or Finnish.)

I'll agree about synonyms, if you use quotes you shouldn't see those.

throwntoday(10000) 7 days ago [-]

There was a moment in time where quotes and negeation hyphens were completely ignored. I have no idea why they ever thought that was a good idea.

wetpaws(10000) 7 days ago [-]

Cause they likely have some internal metrics we are not aware of that has a priority over the search quality.

colechristensen(10000) 7 days ago [-]

I still find that they often don't do what i intend them to

epistasis(10000) 7 days ago [-]

It could also be that the quoted term was in the page source, but not visible when doing a ctrl-f search from within a web browser. Whenever I have been frustrated that a term didn't appear in the visible text, it did show up in the source. So my searches were paying attention to the term, but not in a way that I cared about as a user.

notacoward(10000) 6 days ago [-]

Somebody got a good review for breaking it (but improving some other metric), somebody will get a good review for fixing it. Net result for users is zero, but all the churn looks like productivity in a 'metrics based' GAMMA review process. I've seen worse.

(Same problem with GDP as a measure of economic productivity BTW, but that's a topic for another day.)

throwaway30dc7(10000) 7 days ago [-]

Too little too late. I have used G for like maybe 5 of my last 1000ish searches.

There ARE between alternatives now. I like Kagi the most. Worth every penny.

disqard(10000) 7 days ago [-]

I too pay for, use, and swear by kagi now. It's great!

pmarreck(10000) 7 days ago [-]

Did Google remove the ability to do boolean searches? When I recently tried to do something like (word_a OR word_b), it tried to match on '(word_a' which was wrong

dannysullivan(10000) 3 days ago [-]

In a Boolean like that, the command would be to match either of the words, A or B. But sometimes our ranking systems might find that A seems more relevant, so it could get weighted more. That's still a Boolean match, though. And you don't need to do that. By default, we'll search for any of the words (OR) that are entered.

prometheus76(10000) 7 days ago [-]

I love where they said they've 'heard feedback'. Where? Where can you possibly give feedback to Google about their products? Is this just from their family members at holiday dinners?

taspeotis(10000) 7 days ago [-]

At the bottom of the search results:

    Australia | Brisbane, Queensland - From your IP address - Update location
    Help Send feedback Privacy Terms
If you click 'Send feedback' you can ... send feedback.
thaumasiotes(10000) 7 days ago [-]

> they said they've 'heard feedback'. Where? Where can you possibly give feedback to Google about their products?

This exact discussion is pretty common on HN. Some people complain that quoted search no longer works, and generally someone from Google will show up to tell us that it does.

It's very interesting to me that this change is specifically targeted at quieting the complaints rather than improving the search functionality. The results are the same as before, but they will now show you the match they found so you can't say 'but the string I searched for isn't on the page!'.

That will certainly address one half of the complaint. But I have also seen it mentioned that quoted search may fail to find pages that do contain the quoted text. The example given was along these lines:

Website: 'peering through reverent fingers <br> I watch them flourish and fall'

Query: 'peering through reverent fingers I watch'

Result: 'no results found for 'peering through reverent fingers I watch''

Unfortunately, highlighting where the quoted string occurs on the result page will do nothing to solve complaints that google fails to find pages that contain the quoted string.

scrollaway(10000) 7 days ago [-]

The PR people writing the article are rephrasing the words of the division lead who has aggregated the feedback from a bunch of project managers who are relaying what the data analysts in each team are interpreting from a host of metrics their engineers implemented.

It's an onion of nonsense. At no point is any user involved.

Edit: or as another sibling comment aptly points out, they also might have just heard feedback... from annoyed employees.

JacobThreeThree(10000) 7 days ago [-]

Lots of people have been complaining. Some HN examples:



Hopefully for Google's sake they've finally decided reverse the feature removal trend that's been happening for years.

endisneigh(10000) 7 days ago [-]

You would be surprised how many people who work at Google who hate Google products and say so in the internal groups. People have been complaining about this for years internally.

nextos(10000) 7 days ago [-]

You can do this in google.com > Settings (bottom right) > Send comments. My translation might not match exactly what you see in Google US.

I sent a comment about 3 months ago about a bug in night mode and it got fixed. No idea if my comment was the cause, though.

hnburnsy(10000) 7 days ago [-]

Yeah ironic considering this blog post (or any post at blog.google AFAICT) doesn't allow for comments. It is more like pressrelease.google or marketing.google than blog.google.

ancoron(10000) 7 days ago [-]

All Google products I have ever used provide the option to give feedback (including screenshots).

slater(10000) 7 days ago [-]

OK, now re-implement the '-[searchterm]' filtering that used to remove all results with that searchterm, up until about a year or so ago.

Had a search last week where i specifically stated '-[big european capital city]', and results were chock full of 'best [whatever] in [specified big european capital city i didn't want]'... what?!

dannysullivan(10000) 3 days ago [-]

If you don't want a city name being matched, then you need it inside of quotes and the - outside of those. From the examples you've given, it sounds like you might have it reversed. IE, there aren't correct

['-berlin'] ['-new york city']

These are the correct way to indicate you do not want (the - symbol) a word or phrase (the quote symbols:

[-'berlin'] [-'new york city']

de6u99er(10000) 7 days ago [-]

Remember when we could use '+' to tell Google it must include a term and '-' to define what the search result must not include? This was helpful to filter out advertisements and other useless nonsense while making sure to get what we want.

mancerayder(10000) 6 days ago [-]

It must have been important to Google to remove agency from the user. If the user is too specific in their search, it's more difficult to use information gathered about the user to fulfill what it appears their ultimate goal is: to spoonfeed.

That's what it seems like to me, moving to a world where technology tells you what you want, and you spend less effort telling it.

I think Netflix did a similar thing some time back, just with their front page. Originally it seemed like a place you went to look for things. At a later point, it seemed suddenly a bunch of things were shoved in your face by default, and looking for a thing seemed secondary.

Recommendation Engine, is that what Silicon Valley believes they're innovating on?

An Orwellian nightmare is what is being created.

nazka(10000) 7 days ago [-]

Hmm what do you mean by could? It's not possible anymore? So far it seems to be still there.

202206241203(10000) 7 days ago [-]

I got an idea: make search engine take useful results that you want and DALL-E them with ads to serve you mock websites with plausible content.

binwiederhier(10000) 7 days ago [-]

This is fascinating. Not because of the content of this article, but because it's the first glimpse behind the Google Search curtain that I've ever seen in an official Google post. You rarely see details about search explained. Or maybe I'm just ignorant.

Even that bit about 'don't' being turned into 'don t' was interesting. Again, not because I was amazed in any way. More so because Search has been so mysterious for many years.

joelthelion(10000) 7 days ago [-]

I also find it very interesting to hear them say that they're listening to user feedback. Perhaps the years of outrage are finally paying off.

ma2rten(10000) 7 days ago [-]

There was an excellent talk from Jeff Dean a while back which went into a lot of detail.

rob_c(10000) 7 days ago [-]

It's only 'myseterious' because google will have everyone believe so. They certainly have some of (if not the) best tuned algorithms for indexing and querying on the planet, but there's no angel dust or dark magic at work. (And frankly given how unreliable search in gmail is I'm amazed they keep their head above water)

mingusrude(10000) 7 days ago [-]

I love this old video from their 'weekly search meeting': https://www.youtube.com/watch?v=JtRJXnXgE-A

JacobThreeThree(10000) 7 days ago [-]

There is interesting information about Google Search on their blog and YouTube channels:



londons_explore(10000) 7 days ago [-]

'don't' being turned into 'don t' is due to the way search indexes work. It's a computer science problem, not Google specific.

The index is like a dictionary - you look things up by word. But you need to find some way to split up every page on the internet to decide what is a 'word' and what isn't. If you decide that quote marks are part of a word, then you'll end up with apple and apple' making different entries in the index which you probably didn't want.

mr-pink(10000) 7 days ago [-]

the key thing is they call it 'search' and not 'find' because they don't care if you actually find anything. at least it's not 'bing'

userbinator(10000) 7 days ago [-]

I think for many of us, the ultimate Google should just be some form of 'grep for the Web'. Defaulting to case-insensitive would be the only 'massaging' of the query that I'd consider a good idea.

robbrown451(10000) 7 days ago [-]

You don't even want it to spell correct? That's pretty extreme.

My 8 year old thinks I'm old fashioned because I don't just talk to the google lady casually like she is my friend, when doing a search/asking a question.

O__________O(10000) 6 days ago [-]

@dannysullivan (Google's public liaison of Search [1]; he is commenting in this thread)

- What is the best way to ping you and how do I format my search bugs/features in a way that makes it easy for them to be processed by Google?

- Already pointed out and documented replication issues in another comment in this thread [2].


[1] https://news.ycombinator.com/user?id=dannysullivan

[2] https://news.ycombinator.com/item?id=32354078

dannysullivan(10000) 6 days ago [-]

You can ping me here on Hacker News, probably best to reply to this, and I can watch for replies. I'm also on Twitter, @dannysullivan

For search issues, if people are comfortable, it's always best to share the exact query that was used. Putting it within brackets, as if those are the search box, can be helpful. Like if you searched for 'this that' as a quoted search, ['this that'] would represent it. Explain what the issue is, list any particular URLs if helpful, screenshots if those are also helpful.

treeman79(10000) 7 days ago [-]

Will google stop suppressing political positions they don't like?

jabits(10000) 7 days ago [-]

What are Google's political positions?

WatchDog(10000) 7 days ago [-]

I'm glad to hear that they are working on fixing this, but I would be keen to understand why quote searches have been so much worse over the last ~2 years.

solardev(10000) 7 days ago [-]

Ultimately they just want to show you ads for you to accidentally click on

II2II(10000) 7 days ago [-]

It soumds like they are only modifying the presentation of results, rather than the quality of the results. Useful, yes, but it's not necessarily what people are looking for.

uo21tp5hoyg(10000) 7 days ago [-]

Some search engines seem to just completely ignore quotes now, it's very frustrating especially when I know what I want to search and the search engine just isn't letting me.

kieckerjan(10000) 7 days ago [-]

Regular unquoted search has declined in the eyes of many people. Quoted search worked and works fine (if you take into account the caveats about tokenisation etc). They are improving the snippets. It is all in the fine article.

kurthr(10000) 7 days ago [-]

Me too.

My anecdote was that during the winter Olympics I was looking for a college friend who's name was several characters off of one of the competing athlete's. It didn't matter how much additional information I put in the search. Even quoted, if it included anything that looked like that popular athlete, then that was all that was returned.

When I finally found an old user name she used, then the searches worked. Her correct name and the other information I was searching with were in the page. So it was indexed, the information was simply ignored, because it wasn't news.

Interesting that DDG did not find the page even with the username. I do not know why, but it appeared it was not indexed there.

victor9000(10000) 7 days ago [-]

Doing some form of lossy tokenization would leave you in a pickle like this, but that's pure speculation on my part.

skybrian(10000) 7 days ago [-]

Maybe they're getting gamed somehow? With this change, it should be easier to see if that's the case.

Retr0id(10000) 7 days ago [-]

I hear this a lot, but it I haven't personally noticed any real problems with quoted search (other than the occasional difficulty actually finding the content on the resultant webpages).

I have noticed a general decline in search quality over the last couple of years, but nothing specific to quotes.

I realise I'm not saying anything particularly useful with this comment, but I just thought I'd add another datapoint.

Edit: Thinking more, my biggest issue is when the quoted text occurs in the 'recommended similar posts' section of a page (particularly common with reddit). That section gets re-rendered on each view, so it probably won't be there once I click the result.

dannysullivan(10000) 7 days ago [-]

I work for Google Search. Nothing changed in how they worked over the past two years. I think it's just that our snippets weren't reflecting where we were finding the quoted terms, causing people to think they were ignored. Hopefully this change will now help avoid that impression.

endisneigh(10000) 7 days ago [-]

Ultimately one thing people misunderstand is what it means philosophically to search.

Suppose I have some great content 'bt its writen lik dis'. One could argue saying searching for content with the query 'like this' should yield the previous statement. Others would disagree.

That's basically the crux of the problem. The more exactness you're demanding the fewer results you will receive. The fewer results that are available reduce the perceived utility of the search engine, Google in this case.

Case in point: I've been searching for some 'FoundationDB' related stuff. If you use HN's algolia for 'Foundation DB' (no quotes) it will show you queries where FoundationDB is a single word.

Is this good or bad?


DoingIsLearning(10000) 7 days ago [-]

> One could argue saying searching for content with the query 'like this' should yield the previous statement. Others would disagree.

My guess is the split will follow the lines of those who were used to do meta search in library indexes and databases and those who started using computers when we already had 'natural language' in computing.

Exactness is the whole point of search queries in my opinion. I am trying to filter out a whole universe of search space, last thing I need is second guessing heuristics.

To me excluding that 'great content' that slips through is the price to pay for accuracy.

karlding(10000) 7 days ago [-]

There's a 'Typo-tolerance' option [0] HN exposes in the search settings [1]. If you disable that, then those results no longer show up.

[0] https://www.algolia.com/doc/guides/managing-results/optimize...

[1] https://hn.algolia.com/settings

darkhorse222(10000) 7 days ago [-]

When I'm using quotes it's usually because the first, unquoted search ended up not being fruitful. In a world of algorithmic nonsense, quotes turn google back into what it should be best at: being an index.

makeitdouble(10000) 7 days ago [-]

> That's basically the crux of the problem. The more exactness you're demanding the fewer results you will receive. The fewer results that are available reduce the perceived utility of the search engine, Google in this case.

Isn't the use of quotes an explicit request from the user to have fewer results and more exactness ?

If you do a default search, the question of 'lik dis' being included or not is pertinent. Putting quotes puts you straight into the 'don't show me variations' camp.

jodrellblank(10000) 7 days ago [-]

> 'That's basically the crux of the problem. The more exactness you're demanding the fewer results you will receive. The fewer results that are available reduce the perceived utility of the search engine.'

I disagree, so I went to search for the Joel Spolsky blog post at joelonsoftware.com from antiquity where he complains that search engines prioritise finding 5,000,000 results instead of the result you want, and that's completely useless because you can never read that many results. DuckDuckGo changed my search because 'Not many results contain 'joelonsoftware' and then offered to search for what I typed in, if I arm twisted it. I only wanted one result, and think less of DDG for changing what I tried, in order to give me more results, instead of what I wanted.

It's here[1][2] from 22 years ago and says 'there are three important ideas from computer science which are, frankly, wrong, and people are starting to notice. I'm sure there are more, but these have been driving me to distraction: 1. The difficult part about searching is finding enough results,' then 'Most of the academic work on searching is positively obsessed with problems like "what happens if you search for 'car', and the document you want says 'automobile'". So when the big Internet search engines like Altavista first came out, they bragged about how they found zillions of results. An Altavista search for Joel on Software yields 1,033,555 pages. This is, of course, useless. The known Internet contains maybe a billion pages. By reducing the search from one billion to one million pages, Altavista has done absolutely nothing for me.'

The fewer, better, results that are available increase the perceived utility of the search engine.

[1] https://www.joelonsoftware.com/2000/08/22/three-wrong-ideas-...

[2] I tried a couple of DDG searches including inurl: and site: then switched to Google, went through 4 pages of results, and a couple of searches, before 'site:joelonsoftware.com 'useless' 'results'' got it at the top.

chrismcb(10000) 7 days ago [-]

I would argue that a search for 'like this'without quotes should maybe find that page. But the search with quotes should not find it.

hedora(10000) 7 days ago [-]

Now, if only they could solve the problem where the text they index does not match the text my browser displays.

I've seen a few searches recently (on DDG) where the indexed text is apparently hidden behind an authentication wall.

Google's cached page feature used to fix those sorts of shenanigans.

benreesman(10000) 7 days ago [-]

I'm still trying to forgive them for cutting the cached page feature. On the long slide from "the best thing on the Internet" to the eventual destination of "fuck this, fuck this in particular" that will probably for me be the moment the sign flipped.

MauranKilom(10000) 7 days ago [-]

> if only they could solve the problem where the text they index does not match the text my browser displays

Half the article is about this thing. I mean, they don't present solutions, but most of the reasons for 'Ctrl+F doesn't find the terms' are entirely reasonable to me.

MonkeyMalarky(10000) 7 days ago [-]

Ah the good ol 'whitelist the google bot IPs but pop-up paywalls for everyone else' trick. It'd be cool if google randomized their crawler's IPs and made their bot look like just another user, then we'd all get the same viewing experience.

klabb3(10000) 7 days ago [-]

The cache is actually a good feature, much more welcome than the forced amp-ification. If a website is bloated, paywalled, or offline, I can simply check the cache on an opt-in basis.

Seems like they've made the cache harder and harder to find over the years.

jrm4(10000) 7 days ago [-]

I'm amused at how an equally valid headline would be

'Search results will continue to suck for those who aren't tech-savvy enough to use quotes'

danuker(10000) 7 days ago [-]

They don't mention anything about result quality, just snippet quality.

fleddr(10000) 6 days ago [-]

Please now also improve search results without quotes. When I query a b c (without quotes) I want results exclusively containing all 3 words, not just one or two of them. There's a reason I typed it.

Next, remove content thieves like Pinterest entirely from the index.

Next, invest in recency + relevancy over just page reputation. A huge amount of high value recent content is written but it keeps losing against much worse old content which only ranks because it's old. As the typical example: the 15 year old Stackoverflow jQuery answer.

Invest in location awareness. Including location in your query gives laughable results.

Find a way to down rank sites gaming particular categories with shady tactics.

bel_marinaio(10000) 6 days ago [-]

You can probably use a browser plugin to get rid of pinterest results


systemvoltage(10000) 6 days ago [-]

All this hostility to users in last few years is because of $$$.

Google should just have a developer/engineer/expert mode that shows results using 2012 algorithms. I don't mind paying $5/month in exchange.

superbaconman(10000) 6 days ago [-]

I don't know... Recency sucks when searching for news related stuff. It feels next to impossible to look into the history of news commentary on any given topic.

dannysullivan(10000) 6 days ago [-]

By default, we (I work for Google Search). do tend to look for all the words indicated. It's just that sometimes, what seems to be the most relevant documents won't have all the exact words. If you want results exclusively to contain words, that's what the quotes command is for.

russh(10000) 5 days ago [-]

I would like a way to maintain a list of sites I never want to see in my search results.

nurettin(10000) 5 days ago [-]

20 years ago, search results were controlled via + and - prefixes. +typography -cat means 'I want typography to be present im the results, but cat is not allowed'. I checked and they still seem to work.

paintman252(10000) 6 days ago [-]

Wow, Google search sucks but first three tips is horrible advice. Like it would only make it worse

swyx(10000) 6 days ago [-]

Pinterest and Linkedin are the top culprits for rug-pulling search interest - they blatantly display different results to Googlebot than to humans. If I cant see the result you linked to without logging in, it shouldnt match on your search index.

The open web made you rich, please help keep it open.

estensen(10000) 4 days ago [-]

Staying in Italy for a while I got Italian ads (and I don't speak Italian), and was also often redirected to Italian pages when I explicitly wanted the English version. For example, looking up an error code for my watch I was redirected to the Italian product page for the watch instead of the forum with the answer I wanted.

I guess a VPN could be a workaround, but does the average Joe know what a VPN is?

YPPH(10000) 6 days ago [-]

As a non-American, I find Google's location awareness is already unmatched. It's what keeps me using Google over the alternatives. Googling a thing (business, place, location) near my house will give it as the first search result. The alternatives will often give me the title of something in the US.

DamnInteresting(10000) 6 days ago [-]

> Next, invest in recency + relevancy over just page reputation.

Counterpoint: Google already gives priority to recency, and it heavily rewards 'flip shop' sites that essentially plagiarize existing content. The original researchers/authors who did the leg work get buried in lazy rewrites.

belter(10000) 6 days ago [-]

Quality of search still has some improvements open. Searching right now with quotes for the title of this HN post:

'We're improving search results when you use quotes'

The first three results are interesting:

1- First result is the Google announcement but not this post. The title is 'How we're improving search results when you use quotes' not 'We're improving search results when you use quotes' and the search was with quotes, but to be accepted, maybe...

2- Second result is this post

3- Third result is a quackery page made by some SEO spammers. Now what is interesting is that is almost a copy-paste of Google with changes like this:

Google announcement text:

'...We've heard feedback that people doing quoted searches value seeing where the quoted material occurs on a page, rather than an overall description of the page. Our improvement is designed to help address this....'

SEO Spammers write on page ( with enhancement misspelled):

'...We've heard suggestions that folks doing quoted searches worth seeing the place the quoted materials happens on a web page, somewhat than an total description of the web page. Our enchancment is designed to assist handle this...'

And the page becomes the third link from Google results...

knodi123(10000) 6 days ago [-]

> folks doing quoted searches worth seeing the place

Ah, I remember this trick from my college days, using a thesaurus to beat exact match tests so you could copy from wikipedia et al. Glad to see it's still alive and well, and totally not being abused to make the internet a less useful place.

Pxtl(10000) 7 days ago [-]

If I can't ctrl-F and find the text I googled after I click through, using Google's browser, then either the browser is bad or the search engine is bad.

danuker(10000) 7 days ago [-]

Or the site is bad. Some sites show some content to Googlebot, and other content to mere mortals.

ldjkfkdsjnv(10000) 7 days ago [-]

The second Google starts to give in to the common complaints about search, its maybe over. Meaning, they have lost their edge, know it, and know they are vulnerable.

thehappypm(10000) 7 days ago [-]

To who, Bing? There are no serious competitors

chaosbolt(10000) 7 days ago [-]

Google and Youtube search are heavily censored, for example if you open Youtube and type 'JRE alex' then Alex Jones will be the last suggestion despite his episode having the most views, if you type 'JRE Robert' then Youtube will suggest Robert Downey Jr and other guests whose name starts with Robert, but it won't show Robert Malone, and if you write 'JRE Robert Malon' it still won't suggest it.

Now those episodes have been controversial, and I only bring them up because it's the example that came to my head (before someone misses the whole point and starts looking at the finger), and Google Search also censors them, now while I still use google mainly to access Stackoverflow and Reddit threads, I see no point in using it if I'm searching for anything I want a neutral conversation about that I can examine and make my own conclusion.

All in all the internet seems to be getting smaller and smaller, I don't use any social media apart from HN and Reddit, and I only use Reddit because I seem to still be addicted to it since it's probably one of the most censored of all of them.

10 years ago as a 20 year old I benefited greatly from how the internet was, here is an example: I grew up on the idea that there was nothing wrong with porn, and there isn't per se, and no one ever spoke about addiction like behavior when it came to watching it, then one day I discover a controversial post on Reddit and dove down the rabbit hole and lo and behold I had the same problems as this community of people trying to quit watching it, and I benefited from their experiences and knowledge, same about discovering communities against social media like Facebook, which pushed me to research the subject and deleting my account, etc. but now it seems like any controversial community is quickly banned or pushed aside in its own unfindable bubble and that to me is a great loss.

I want to see people have an opposite opinion than mine, and I want to be able to get into heated non censored discussions in comment sections and get suggestions about articles, studies and content to challenge my views.

bambax(10000) 7 days ago [-]

> Google and Youtube search are heavily censored, for example if you open Youtube and type 'JRE alex' then Alex Jones will be the last suggestion despite his episode having the most views

This is provably untrue. I just searched YT for 'JRE Alex' and the first three results are clips from the 'JRE' show (I didn't know what that acronym was before now) featuring Alex Jones [0].

Why are people repeating these complotist theories when it takes five seconds to disprove them.

Also, even if it was true (which it isn't), it still wouldn't be censoring. Censoring would mean not returning results when a match exist -- not massaging the SERP. It would also mean not publishing clips from said 'censored' content on Google's own video platform, where they have millions of views.

In France there is a controversial (to say the least) comedian, Dieudonné, a holocaust denier and an anti-semite. He's heavily censored by the French government who regularly bans his shows; but canceling his YT channel happened only recently, after a lot of interventions by the state and other non-governmental agencies.

It's been my experience that Google is quite resistant to censoring, in general.

[0] Here are the top three results on YT for 'JRE Alex':

    Alex Jones - God Doesn't Know Where He Came From | Joe Rogan
    2.6M views 3 years ago
    JRE Clips
    Taken from Joe Rogan Experience #1255 w/Alex Jones: https://youtu.be/-5yh2HcIlkU.
    Joe Rogan Experience #911 - Alex Jones & Eddie Bravo
    13M views Streamed 5 years ago
    Alex Jones is a radio show host, filmmaker, writer, and conspiracy theorist. Eddie Bravo is a jiujitsu black belt, music producer, and ...
    Joe Rogan Experience #1555 - Alex Jones & Tim Dillon
    23M views 1 year ago
    Tim Dillon is a standup comedian, actor, and host of the Tim Dillon Show. Alex Jones is a filmmaker, writer, and host of the Alex ...
gverrilla(10000) 7 days ago [-]

Google is like an abusive partner: it may be gifting you something nice, but in the end we all know he's doing it only to soften you and hurt you later. It started like a beautiful love story about knowledge and freedom, but now Google arrives drunk and hits us with a keyboard every night.

dymk(10000) 7 days ago [-]

You've clearly never been in an abusive relationship. At best it's a bad faux-dramatic attempt at humor that trivializes what abuse actually means.

threatripper(10000) 7 days ago [-]

If you don't like him then go to your other friends! What, you lost contact long ago? Do they even exist anymore? Too bad. Sounds like we're stuck together for a long long time.

hericium(10000) 7 days ago [-]

> Google is like an abusive partner: it may be gifting you something nice, but in the end we all know he's doing it only to soften you and hurt you later.

Wasn't this something nice just a normal, granted thing at the start of this relationship?

I'm not using their services for few years now but I remember the search engine working well with quotes in the past - before they started bubbling people in personalized results.

SllX(10000) 7 days ago [-]

For real. My first thought on seeing the headline was to click through and find out how they were ruining it. Probably most of my queries have quotes in them, just to try and enforce some sanity on the results I get.

Instead this appears to be restoring functionality that was lost. I swear Google used to actually do what this blog post says Google will now do, and I'm not sure when it disappeared, just that I stopped looking at the text blurbs since what they show usually isn't what I got. Maybe that will change now with this backpedaled change?

At least they're not ruining it.

disantlor(10000) 7 days ago [-]

i get your point but this is really pretty dramatic.

it's a website. not everything needs to be found, let alone searched for

stevage(10000) 7 days ago [-]

This seems like the tiniest of tiny improvements in a tiny aspect of Google Search.

Am I missing something?

How did this become the #1 story on HN?

CameronNemo(10000) 7 days ago [-]

I think developers and technologists use this feature often to locate people experiencing the same technical issues. Don't know what an error means? Google it. With quotes.

wraptile(10000) 7 days ago [-]

It really shows how frustrated google search users are when a minor improvement is celebrated to this extent.

robbrown451(10000) 7 days ago [-]

Something like that is about as key to using the web as anything. I've used quotes on searches to get an exact phrase since, what, 1995? The fact that has been broken for a long time and now fixed is absolutely worth top story of the day to me.

judge2020(10000) 7 days ago [-]

It's top HN because HN commonly sees complaints about quotes in Google Search not performing correctly; i'd link to some of these complaints, but the algolia search is needlessly fuzzy and doesn't seem to include all comments on HN.

jfoster(10000) 6 days ago [-]

> Years ago, many people used operators because search engines sometimes needed additional guidance. Things have advanced since then, so operators are often no longer necessary.

In my experience, things have degraded since then, so operators are increasingly necessary.

quitit(10000) 6 days ago [-]

Google was never good enough to remove the operators, and as you said it's gotten worse not better.

Seemingly people using operators would produce additional training for the algorithm which is currently not possible. This could also be a factor leading to the results decay everyone has experienced when trying to find something that isn't leading popular culture. It's that peculiar behaviour where tweaking the search terms didn't affect the results.

O__________O(10000) 6 days ago [-]

Which is the core issue, Google has likely gotten better for the average user, but worse for above average users; say this as an above average searcher that has helped average users search for years.

To my knowledge Google neither publishes search quality reports, nor acknowledges that there's a gap between their search quality for average users and above average users. In my experience even their "Search Quality Raters" (contractors paid to evaluate search quality) are rarely above average searchers, they are just good at evaluating what they evaluate.

SkyPuncher(10000) 6 days ago [-]

Yea, this is a case of not being customer centric.

I absolutely believe that Google has advanced technologically. Whether that's translated to improvements for users, it's not clear.

jjoonathan(10000) 6 days ago [-]

Ugh, yes. Google is still pretty good for navigation-by-title on popular content, but it is increasingly terrible at search. Breaking operators is part of that story, but I've seen more and more instances where Google will prove that it can return a page but won't return results for some of the phrases on the page. I half suspect that behind the scenes there is a one-hot encoding or embedding that just ignores tokens with low frequency, which completely wrecks Google's ability to navigate technical writing with abbreviations, part numbers, small brands, etc. Sometimes it does work, so I think of it like a popularity algorithm plus a decrepit fallback for the content it deems 'no longer necessary' to do a half decent job on.

cientifico(10000) 6 days ago [-]

I am willing to pay for the search experience of years ago.

In their defense, Google now gets right what my grandmother searches.

Ralfp(10000) 6 days ago [-]

My average search starts with „site:reddit.com" because otherwise first three pages of results will be SEO blogs where 90% of article follows „First, what is XYZ?" followed by „so we see why it serms hard, but if you pay us/our affiliate money we will do it for you nice and easy!"

mtlmtlmtlmtl(10000) 6 days ago [-]

Yeah this seemed completely bizzare to me, too. Not only are operators like site: and inurl: more necessary than ever, they're not even sufficient a lot of times, because if you don't know what website you're looking for you have to wade through oceans of blogspam that can't easily be filtered through operators since they're designed to match anything you search for. That's mainly a problem with the indexing; Google should be ignoring these sites. But it's probably padding their bottom line, so I guess they're not incentivised to.

I'll keep saying it again and again. When companies have almost no incentives other than maximising profit, this kind of BS will invariably be what we end up with. Dark patterns are just too good for business if not kept in check through regulatory or other means.

giancarlostoro(10000) 6 days ago [-]

Yeah, I suspect when they added all the 'filter out all the piracy sites' features, and anything else that's bad on the internet, it really ruined what was once the most powerful search engine I had seen to date. I don't think we'll ever see that again. Probably also when they 'personalized' the search results more and more.

What Google needs is a built-in way to block domains, like idk... Pinterest.

ubermonkey(10000) 6 days ago [-]

Exactly right.

I hate modern 'search'. I understand nontechnical people probably like algorithm-based 'enhanced' search fine, but if I search for something specific I want THAT THING ONLY.

'Modern' search made it really hard to ask very specific, strict questions.

Someone1234(10000) 6 days ago [-]

If you want the old, good, Google back: Turn on Verbatim Mode.

Search Tools -> All Results -> Verbatim.

Instead of Google throwing away 90% of your search query and looking for the 'top hits' (and returning almost entirely auto-generated pages), it keeps every word you entered and returns actually relevant stuff.

It really is night and day.

emrah(10000) 6 days ago [-]

> In my experience, things have degraded since then

'In my experience' is exactly right, search results degraded for a fraction of users such as developers.

It's good to see Google is going back to making those users happy although that probably means bad news for DDG et al

HeavyStorm(10000) 6 days ago [-]


wodenokoto(10000) 7 days ago [-]

> 'Punctuation is sometimes seen as spaces. Our systems see some punctuation as spaces, which impacts quoted searches. For example, a search for ["don't doesn't"] tells our systems to find content that contains all these letters in this order: 'don t doesn t'

Others have pointed this bit out, but I'd like to ask specifically: How do you search for special characters?

I recently came across some R code that used %||% and you cannot search for this. As in, you cannot search the internet for this at all.

Google, DuckDuckGo, Bing, Yahoo, all of them will ignore special characters. Even Github search will strip out the special characters leaving you to search for nothing.

karlicoss(10000) 6 days ago [-]

httpa://grep.app often helps for obscure code searches on GitHub

alecco(10000) 7 days ago [-]

Try 'percent pipe pipe percent'. If somebody bothered to spell it, it might show up. Might.

godshatter(10000) 6 days ago [-]

I tried 'r operators pipe percent' (without the quotes) in search.brave.com, and it came back with this link as the second link: https://stackoverflow.com/questions/50970207/what-does-the-o...

I'm guessing that would work fine on google, but I don't go there so I can't be certain.

nneonneo(10000) 7 days ago [-]

Try the code: operator on StackOverflow: https://stackoverflow.com/search?q=code%3A%22%25%7C%7C%25%22

dekhn(10000) 7 days ago [-]

See also some context for Google search operators including ones that have been removed: https://ahrefs.com/blog/google-advanced-search-operators/ and changed: https://www.googleguide.com/quote_operator.html (IIRC this was due to Google+ and was an internally unpopular change, but my memory may be faulty).

modeless(10000) 7 days ago [-]

I don't understand why they never added + back after Google+ finally died. When I was at Google I actually went as far as looking for the code to re-enable it in superroot. If I recall correctly I actually found some of the code but never had time to learn enough about superroot to make and test a CL.

Of course I probably wouldn't have actually been able to get it changed back just by making a CL but at least I would have learned why it couldn't be changed back.

a3w(10000) 6 days ago [-]

. works the way _ does, pretty much. So one more operator?

gotbeans(10000) 7 days ago [-]

I'm not even mad anymore about the search quality.

Google search is such a shitfest right now, you have to scroll a full page in a 100% 4k screen to get to actual results, it's absolutely disgusting.

I don't care if they search is unoptimal or downright bad. The ad situation is so much worse IMO.

metadat(10000) 7 days ago [-]

Big-G only shows about 10 or fewer actual (non-advertisement) results on the first page, too. Regardless of screen size.

What's up with that? Seems a little stingy.. why go to the effort of HTTP2 to deliver so few hits?

jedwhite(10000) 7 days ago [-]

Note here - they aren't changing what results are returned when you search with quotes, they are just highlighting what you searched with quotes in bold in the snippet displays in the search results page.

ColinHayhurst(10000) 7 days ago [-]

best comment

Historical Discussions: 'Too many employees, but few work': Pichai, Zuckerberg sound the alarm (August 10, 2022: 687 points)

(689) 'Too many employees, but few work': Pichai, Zuckerberg sound the alarm

689 points 1 day ago by quaffapint in 10000th position

www.business-standard.com | Estimated reading time – 4 minutes | comments | anchor

With global recession fears looming, some of the biggest names in the technology world have some ominous words to spare: Big Tech has hired more people, but only some of them are doing the work.

Meta (earlier Facebook) Founder Mark Zuckerberg fired the first salvo. It was the weekly Q&A on June 30, and he had said that the economy was headed for the "worst downturns that we've seen in recent history".

Then he continued.

"Realistically, there are probably a bunch of people at the company who shouldn't be here," Zuckerberg said on the call, according to a Reuters report. "And part of my hope by raising expectations and having more aggressive goals, and just kind of turning up the heat a little bit, is that I think some of you might just say that this place isn't for you. And that self-selection is okay with me."

In addition to reducing hiring, he said, the firm was leaving certain positions unfilled.

Google and Alphabet chief Sundar Pichai echoed the sentiment when he told employees that productivity was not high enough, considering the number of people on the company's rolls, CNBC reported.

"There are real concerns that our productivity as a whole is not where it needs to be for the head count we have. [We need to] create a culture that is more mission-focused, more focused on our products, more customer-focused," he said. Pichai had recently said that the firm would cut hiring and investment through 2023, pushing staff to work with greater urgency and "more hunger" than demonstrated "on sunnier days". It came after the firm reported its second consecutive quarter of weaker-than-expected earnings and revenue. Revenue growth slowed to 13 per cent in the quarter from 62 per cent a year earlier.

Meanwhile, Zuckerberg noticed that it was getting harder to get all the employees to attend a meeting as they were sometimes taking time out in a day for personal work. So the Meta boss said that, in an effort to be "cost-conscious," he was freezing or reducing staffing for low-priority projects and slashing engineer-hiring plans for the year by 30 per cent, reports added.

To be sure, the Covid-induced pandemic saw Meta embark on a massive hiring spree, growing its number of full-time staff from 48,000 at the end of 2019 to more than 77,800 — a 62 per cent jump. But now the firm must "prioritise more ruthlessly" and "operate leaner, meaner, better executing teams," Meta Chief Product Officer Chris Cox wrote in a memo, which appeared on the company's internal discussion forum Workplace before the Q&A.

Dear Reader,

Business Standard has always strived hard to provide up-to-date information and commentary on developments that are of interest to you and have wider political and economic implications for the country and the world. Your encouragement and constant feedback on how to improve our offering have only made our resolve and commitment to these ideals stronger. Even during these difficult times arising out of Covid-19, we continue to remain committed to keeping you informed and updated with credible news, authoritative views and incisive commentary on topical issues of relevance. We, however, have a request.

As we battle the economic impact of the pandemic, we need your support even more, so that we can continue to offer you more quality content. Our subscription model has seen an encouraging response from many of you, who have subscribed to our online content. More subscription to our online content can only help us achieve the goals of offering you even better and more relevant content. We believe in free, fair and credible journalism. Your support through more subscriptions can help us practise the journalism to which we are committed.

Support quality journalism and subscribe to Business Standard.

Digital Editor

All Comments: [-] | anchor

bagacrap(10000) about 19 hours ago [-]

I work at one of the two mentioned companies. 20% of the employees do 80% of the work, and another 60% do maybe 50%. the last 20% account for -30%. It's demoralizing how incapable management seems to be at jettisoning the chaf, however it was explained to me once thusly: good engineers are hard to find and also very valuable. An unmotivated/lazy engineer is one step away from being a valuable engineer. It's worthwhile spending a lot of effort trying to find what motivates them.

Also, if you look at revenue and divide by employees, you will see that the strategy of 'employee 5 engineers and hope at least one is good' is still profitable.

habosa(10000) about 16 hours ago [-]

I love this framing, those percentages match up 100% with what I saw on the inside of a FAANG company.

cm277(10000) about 14 hours ago [-]

This is more or less exactly why Google/Meta need to be regulated and cut down to size. The fact that super-expensive engineers do not have to create value for these companies, but they are still 'affordable' while every other tech company is struggling to hire, means that they are capturing way too much value, i.e. rent.

fathrowaway12(10000) about 22 hours ago [-]

I have been working for about 4.5 years now across 3 big tech companies (Google included).

I would consider myself a hardworking and ambitious person who loves computer science. I graduated top of my class at a state school. I've gotten promoted, consistently 'exceed expectations', lots of positive feedback, etc. But I have not actually accomplished a damn thing or written a single interesting or useful component in my professional work.

The thing is, it's becoming clearer and clearer that so much of what goes on is bullshit. A pattern I have seen 3 times now is managers significantly over-hiring to build their little management moat of mediocre junior devs, then leaving on to brighter pastures with their shinier resume or promotion.

Most of the work is dealing with other people's code messes, operational gruntwork, and ticket grinding that will have little to no impact and is a complete waste of smart people's time. It has little to do with building software or solving hard problems, just maintaining and tweaking the existing systems. It does absolutely nothing for your career. You could be Jeff Dean stuck on worthless legacy grunt crap with no upside and I doubt you would get noticed.

On the flip side, people who actually get interesting, promotable work are the luckiest in the world. It is the difference between a rocket ship advance of career and skill or stagnation and frustration. But this is rare IMO.

And it's so hard to tell going in which you're going to get. I've now taken a gamble on three teams that looked good on paper but turned out to be legacy management empire crap. It pisses me off that once you choose you're basically stuck there for a year or two before you can try again. I don't want to waste any more of my life on this merry go round.

strikelaserclaw(10000) about 7 hours ago [-]

This is why you shouldn't be a heads down 'efficient' worker doing any and all tasks you are given, if you are in this position always do enough of the boring work to be employed but use the rest of the time to research interesting projects that are being worked on in the company, then when you have a decent understanding of where you wanna go, go there.

stillblue(10000) about 16 hours ago [-]

> I don't want to waste any more of my life on this merry go round.

And? Don't leave us hanging. Have you figured out a way to get out of this vicious cycle?

jeffbee(10000) about 20 hours ago [-]

Just wanted to say that you've hit on a real problem: the rich-get-richer feedback cycle in which all the good projects are taken by high ranking engineers. On my teams, including at Google, we intentional counteracted that by giving the interesting work to the person just barely qualified to do it, and biasing the crud toward the senior team members.

nokeya(10000) 1 day ago [-]

When I click "Select all" in Gmail my browser hangs for several seconds at least. And now they say there is too much people. Hello! There is not too much people, there is is misutilization of your workforce! Instead of doing 100500+ chat app that will not do it to production, maybe focus on real customer problems and not on metrics? There is a lot comments now and before from people saying they seeing no impact from their work in FAANG. Of course, when your code means nothing and changes nothing - you will loose all motivation to do something useful

wildrhythms(10000) about 23 hours ago [-]

Fixing the 'select all' hang doesn't get anyone promoted, and managers can't parade it around as a win before they springboard to some other org.

Ekaros(10000) 1 day ago [-]

Work is not equally distributed. I'm sure there is plenty of Engineers everywhere that have way too much on their plates. And on other hand there is also plenty of those who do little or have little to do.

And then there is question of how much of the work is actually even needed. Specially in companies with too much money.

Also why should all employees attend to the meetings? Certainly to some, but it clearly is job that can afford certain level of flexibility in most teams.

FactualActuals(10000) 1 day ago [-]

I am running into this problem where I currently work. I'm not a rockstar developer but I can finish my tasks pretty quickly. But when management won't allow me to help take over some tasks to alleviate my coworker's growing backlog, I can't do anything but twiddle my thumbs waiting for more tasks.

khazhoux(10000) about 20 hours ago [-]

In every other HN thread, people slam FAANGs for being so bloated, and we praise the small startup where everyone on the team gets shit done fast. And in every other thread, people ask 'What the hell does <BigTechCo> need 10,000 engineers for??'

And now Zuck says 'We've gotten complacent as a company, and we have to turn up the heat and sense of urgency, and get rid of people who aren't contributing'... and HN freaks out at this 'dystopian' missive.

Pick a lane, people.

okdood64(10000) about 18 hours ago [-]

Alternative view: When you go on such a massive hiring spree in the last 1-2 years from headcount X to headcount Y, how can you be shocked at you're not getting headcount Y amount of productivity?

Proper onboarding attention and tenure impacts productivity.

gorgoiler(10000) about 19 hours ago [-]

Well said. Slashdot was groundbreaking for many things, especially so for categories.

I wouldn't mind having the ability to filter HN replies to "+5 Insightful" and drop anything else marked "Contrarian".

potatolicious(10000) 1 day ago [-]

It's frustrating that this thread seems to be focused so heavily on people sitting around resting and vesting.

Having been inside Google (and multiple other FAANGs) this is generally untrue, and focusing on this element of the problem misses a much larger productivity problem:

Most engineers at Google aren't 'sitting around doing nothing', they are very busy shipping projects that do not matter. Their days are filled with doing work that will not move the needle on any metric that matters to the company, but they are far from idle.

The misallocation of labor is a far bigger problem than said labor slacking off, and management must own it.

Google doesn't need their engineers to fly into startup mode, work 12 hour days, or never surf Reddit on company time. Their labor is severely under-utilized because they are assigned to zero/negative-impact projects or duplicative projects (hey, somehow you gotta ship 5 chat apps at the same time, right?)

Part of the problem is that Google's upper management refuses to engage with the product at all. Entire orgs are given very broad OKRs like 'increase DAUs by 10%' without virtually no guidance as to what features management is interested in. Authority to ship features also rests close to the leaf nodes of direct line-managed teams. The expectation is that teams are entrepreneurial and invent features, implement them, and ship them all without direct upper management involvement.

The result is a bunch of bad product that doesn't do anything positive for the company, were never soberly evaluated by upper management prior to building, and would never have passed the smell test if it did. This, above all other factors, is why Google produces so much product that it then has to scrap. This is the main cause of Google's low labor productivity - not because people are sitting around drinking coffee and eating free food - but because they are assigned to projects that do not pass muster, and there is an almost-comical aversion to validating product ideas before they are implemented.

The single biggest thing Google can do to improve its labor productivity isn't cracking down on slackers, it's forcing its management to actually engage with product definition so entire orgs don't burn years on things that don't matter.

chocolatemario(10000) 1 day ago [-]

I feel that in real terms, you are absolutely correct. Big tech companies consciously over-hire and throw away work with impunity knowing it will not hurt their bottom line. Denouncing employee productivity like this just seems like an excuse to trim some fat indiscriminately during economic downturn instead of attacking the root of the problem as you suggest they should. They obviously should try to fix the wasted work problem, but that is undoubtedly more difficult than overprovisioning your workforce and dialing back during times of economic duress.

celestialcheese(10000) about 16 hours ago [-]

Seriously - the misallocation of resources in areas of Google is insane. And then there are niches of the Google Cloud product that are seemingly woefully understaffed, or just negligent in their maintenance.

I was a gcloud customer for years, happily using the App Engine Flexible for a PHP apo. It wasn't perfect, but it did it's job autoscaling and managing a decent sized consumer app. For years there wasn't much like it in PHP land where you could get away with doing very little sysadmin/cloud arch and get the features they offered.

But then support just _stopped_ around 2 years ago.

The official docker images are still stuck on PHP 7.3 and there has been no updates or support for nearly 18 months in upgrading to 7.4, 8.0 or 8.1.. [1]

AppEngine wasn't some vaporware at gcloud, it was one of the OG products, yet their perfectly fine letting it just rot and officially support a version of PHP that's a year outside of security patches.

I was spending nearly $80k/yr on that product + the rest of the cloud spend that comes from tying yourself to a cloud vendor. But this issue was so bad that I invested the time migrating off appengine and google cloud to another vendor.

There are people internally assigned to this, but either complacency or lazyness or poor focus the details cost them a customer.

1 - https://github.com/GoogleCloudPlatform/php-docker/issues/530

clusterhacks(10000) 1 day ago [-]

>Their days are filled with doing work that will not move the needle on any metric that matters to the company

This, 100%. I think this simple observation reverberates across the entire software engineering field and at many (most?) non-FAANG companies as well.

I am not confident there is a real solution to the problem of making sure people only work on things that matter. Medium to small organizations seem to struggle with having management even understand what 'good' product looks like or how to optimize for that outcome.

strongpigeon(10000) 1 day ago [-]

This resonates strongly with my experience at Google as well. Specifically in ads, you got the feelings that none of the product leadership used the product or tried to drive a direction for where the product should be going. The end-result was full-on Conway's law (every team having their own separate pages), weird overlaps between a bunch of things (P-Max, Smart and App campaigns) and no real goals except maximizing metrics such as $$$ and # of campaigns using automation.

Of course, when Google Ads is the only way to buy ads on Google Search, revenue will go up regardless of whether Google Ads is a good product. Advertiser will go through any hoops if those ads make them money.

But then revenue goes up, the leadership pats themselves on the back for a job well done, plays musical chair a bit and let the product turn into an even bigger pile of mush.

dev_0(10000) 1 day ago [-]

I have seen colleagues working long hours because they take 2 hours of lunch break and 1 hour of teabreak. And management think they are hardworking.

abledon(10000) 1 day ago [-]

tbf, 1 persons 2 hours of coding is equivalent to another persons 8 or 16 hours of coding. Work smart, not Hard

ccn0p(10000) 1 day ago [-]

There's a lot of pointing fingers here. At the risk of sounding crass, any company with more than 1000 employees (pick a number) has high performers and low performers. Yes, culture, management, and process all basically move the sides of the bell curve, but nothing 'fixes' human nature and organizational inefficiencies as companies grow.

This is why companies rate and rank employees and low performers find their way to the door and/or go through [bi]annual RIF processes to clean up the org. It's the natural growth process.

beezlebroxxxxxx(10000) 1 day ago [-]

It's also around that point where you start to get low and high performers that, in my opinion, the burden of productivity should shift quite a bit to managers rather than individual employees. Once a company gets to a certain size, certain bureaucratic workflows and systems become far more necessary and entrenched as 'the way we do business'. Some 'low performers' at that point, as a result of this internal dynamic and internal limitations in a business, often just have less work to do or they are limited in sign-offs to work on other projects/coordinate with other teams. At that point, the role of managing teams and individuals becomes much more important and consequential. What you often get though, is management that defers accountability as problems with individual performance with employees below them. This is, essentially, a way of ignoring how the way the company operates has changed.

analog31(10000) about 20 hours ago [-]

To me, this may signal an ominous culture shift. I don't work for Google or Facebook, and don't expect to. (Not sour grapes, just reality). But if a return to a more draconian culture becomes popular with investors, it could eventually trickle down to my little hamlet. Not something I look forward to.

nrb(10000) about 20 hours ago [-]

Draconian culture is an environment that spurs tech innovation by new entrants. If anything, it might cause a mass migration of employees from those outfits into great new companies.

tomatotomato37(10000) 1 day ago [-]

Wasn't the whole point of showing off 'Unlimited PTO' was the ability to run errands? Seems like that facade is dropping fast

bobharris(10000) 1 day ago [-]

Laughs. How does 'Unlimited PTO' even work? Mystery of the universe. For me it's 'Unlimited PTO' right up to being let go.

PaulHoule(10000) 1 day ago [-]

Google would have been a lot more productive if it had hired people to work on one good messaging app instead of 13+ bad messaging apps.

Google has long had an attitude of 'we hire the best so we can afford to have them stand on one leg and balance on a ball while holding a cane in their mouth and balancing a bunch of dishes on the end of the cane while typing with one hand on a chorded keyboard and looking at a monitor through a mirror.' I've heard stories that range from 'of course I am productive, I am shooting the s--t all day with the smartest people to' to 'I have no idea of how what I'm doing impacts the bottom line'.

unicornmama(10000) 1 day ago [-]

Writing a messaging app is a fool's errand. You either build a chat app with someone elses money, invest in all chat apps (1/n) and hope you score a big one - e.g. like textbook publisher, or you wait and M&A the successful ones.

The barrier to entry to write a chat app is zero. Even if you are brilliant you will compete against hundreds other chat apps one of which will beat out with pure luck. Never compete against luck.

dont__panic(10000) 1 day ago [-]

Blaming the employees smells like a smokescreen for poor management IMO.

Who's to blame for lowered employee productivity: employees who are disconnecting from work more to avoid burnout thanks to corporate BS like paperwork and constant report filing? Or the managers who impose those requirements on employees but fail to empower the individual contributors beneath them in the org chart?

I recently left a large-medium sized tech company that failed to address massive structural issues in my department for years. It's not like these were a secret -- I brought them up constantly in my 1on1s, and tried to brainstorm solutions with my management chain.

When I left, the head honcho begged me to stay, and when I brought up those issues... told me he had no idea that was such a problem! But also refused to address it because he had to 'gather information' about the issue.

I'm much happier at a smaller company without so much bureaucracy. At some point, managers are so disconnected from their underlings that they are completely incapable of improving work conditions. And when you need high-level approval to make a big decision... more often than not, the big decision just never gets made.

If you choose not to decide, you still have made a choice.

jrockway(10000) 1 day ago [-]

The 'while balancing on a ball while holding a cane in their mouth...' thing really resonates with me.

Something that really surprised me at Google is how many core services had very thin test suites. I'm the kind of person that sees 100% code coverage and thinks 'that's a good starting point'. If I don't have that, I'll definitely break something important in 6 months. There were a lot of people at Google, though, that definitely didn't need those guard rails. The entire team could read a changelist and know exactly what the consequences are; they could just read the diff and run the complete test suite in their head. So there was no need for them to spend the time actually typing in a test suite.

It wouldn't work for me but there were a lot of people at Google that absolutely didn't need to follow 'good engineering practices' to do good engineering. I was impressed. A lot of people less smart than them try this and fail, but they made it work.

bergenty(10000) 1 day ago [-]

We'll a lot of their products come out if individual side projects, Google is an incubator of sorts so I'm not surprised that's how their product gets made.

pradn(10000) about 24 hours ago [-]

As a Google employee, the profusion of chat apps is caused by:

* a genuine interest in trying new things and trying to see if they'd stick, without the baggage of established UX & customers - Allo/Duo are like this. I don't think people give the company enough credit for this.

* leadership downplaying the cost to the Google brand of shutting things down all the time. When brought up internally, execs shrug saying that we must be willing to try and see if things stick. This makes sense, but why are people particularly mad at Google for doing this? It must be for a good reason, not merely a meme.

* org silos. The org behind Google Docs / Chat has a different reason for a chat app (chat as a checkbox for enterprise office suite sales) than the one behind Google Maps (you can chat with restaurants or whatnot)

* a lack of a good 'design dictator', meaning our chat apps, as with other apps, falter for lack of great UX and don't gain traction. The biggest example I can think of is how Google Chat has a loading spinner for the emoji picker - this simple thing should be lightning quick, but it took a year for someone to even prioritize it.

* faulty marketing / branding. Taking the simple, beloved 'GChat', which was the dominant chat app between AIM and FB Messenger, and wringing it though 'Hangouts' and 'Allo/Duo' and 'Chat' - that's no fun for users.

I think the lesson here is that people want a simple, hyper-fast app that gets out of their way and slowly adds nice things on top. I'd say the apps that are most fun and fast to use are Messenger and iMessage. (I have plenty of problems with both - unremovable stories on Messenger, lack of archiving chats and general slowness on iMessage).

All these are my opinions.

metadat(10000) 1 day ago [-]

What's wrong with taking a break to run necessary errands or pick your kids up from school? As long as you're meeting performance expectations, it should be fine.

Is Zuck really slaving away at his desk 9-5 everyday? I don't think so.

Sounds like another case of 'One rule for thee and another for me'

VirusNewbie(10000) 1 day ago [-]

I agree 100%. However, if someone schedules a meeting for work hours and 50% of the people can't make it because they are grocery shopping I can understand the frustration. These aren't low paid employees. Meta is paying 300k+ for mid level engineers.

mikhael28(10000) 1 day ago [-]

Maybe it's not the employees fault, but the management who hired them... or maybe it's the fact that it takes forever to get anything done at FAANG nowadays.

Or maybe, just maybe, interviewing based on esoteric computer science problems isn't the best way to identify high performing builders.. but a great way of identifying people who can hack a process to secure maximal reward.

Look, if I can 'crack the coding interview', then I can certainly crack 'how to do as little work as possible and stack paper to the ceiling while my stock vests'.

I wonder when the last time was that Mark or Sundar actually wrote any code they pushed to prod.

MichaelMoser123(10000) about 24 hours ago [-]

>Maybe it's not the employees fault, but the management who hired them...

I think the managers are just putting up a straight face, as they need to respond to the changing circumstances.

I think it has more to do with the economy and the war of Russia against Ukraine. All of a sudden there is less money to go around, interest rates are rising and it got harder to raise money.

And they probably changed their plans, now it is less about 'new features' and more about 'maintenance of existing systems'. But that didn't get into the article, so it's all the fault of the people who will have to look for a new job.

Searching for a new job isn't a pleasant experience, if you ask me.

(I am not working at google or facebook, but I will probably get to feel the implications as well...)

FearlessNebula(10000) 1 day ago [-]

Did Sundar ever write code? Wasn't he a PM? I wouldn't be surprised if Mark still writes some code, he's a hacker at heart

Aunche(10000) 1 day ago [-]

>Or maybe, just maybe, interviewing based on esoteric computer science problems isn't the best way to identify high performing builders

The interview process at FAANGs isn't designed to hire the 'best' people. It's designed to hire people who are 'good enough' in a consistent manner. Any form of standardized interview can be gamed. More personalized interviews can be better in theory, but they also open the door to nepotism and discrimination.

Admittedly, I'm biased because I'm unusually good at Leetcode and a rather lousy in terms of development velocity. With that disclaimer out of the way, I think the last thing that FAANGs need are more 'high performing builders'. In my experience, a lot of them tend to create a lot of useless passion projects that work their way into being dependencies and end up causing more harm than good. I may be a rest'n'vester, but at least I make sure the work I get done creates positive value for the company.

aeternum(10000) 1 day ago [-]

Yes, it's pretty clear that humans were overfitting to their interview objective function: comp-sci algo problems.

For companies with such strong ML backgrounds, in addition to the sheer amount of content dedicated to discussing and solving tech interview questions hosted on their own platform, one would think they would have noticed earlier.

vecter(10000) 1 day ago [-]

> Or maybe, just maybe, interviewing based on esoteric computer science problems isn't the best way to identify high performing builders.. but a great way of identifying people who can hack a process to secure maximal reward.

I see this argument all the time, but I can't find any other place that it comes from other than disappointment from those that didn't or can't pass those interviews. (Disclaimer, outside of college internships I've never interviewed for a FAANG SWE position nor have I ever worked for one).

Is it an objectively good measure of being a software engineer? Hard to say honestly. I doubt you'll ever find a truly great measure that you can test for in an interview. When I was interviewing candidates for my company, did I ask those leetcode algorithm questions? Not really. Maybe at most one basic tree traversal question (probably would fall under leetcode 'easy' if I had to guess, but honestly the kind of thing a student would learn in AP computer science in high school). Most questions were system design and problem solving with a coding challenge (building something simple, not solving algorithmic puzzles). So by evidence of my own actions, I don't believe that they're the optimal questions for screening engineers.

That having been said, I don't understand why people are upset by these interviews. Who cares? If you really think it's suboptimal, then other companies who have 'better' interviewing practices should be better at identifying undiscovered talent and hiring them. Better for you if you're hiring in those cases. Let FAANG fail on their own hiring practices. FTR I don't think they're that bad either, they just filter for a bunch of left-brained people who are good at math. Maybe they do make good engineers also. And if results are anything, clearly it's been working for FAANG for the past decade so who's to say that they shouldn't keep doing it?

> Look, if I can 'crack the coding interview', then I can certainly crack 'how to do as little work as possible and stack paper to the ceiling while my stock vests'.

This is a reach (to put it mildly) and unfairly paints people who are good at algorithms as inherently unmotivated and whose primary goal is to cheat the system without any evidence. Are you saying another talented developer who isn't good at algorithms could not or would not hack the system as such? I don't see any reason to expect either to be the case. Hacking said system does not require you to be able to prove the runtimes of a Van Emde Boas queue, it just requires some common sense that any human being has.

> I wonder when the last time was that Mark or Sundar actually wrote any code they pushed to prod.

This is pure ad hominem and unrelated to whether or not these questions are good screening questions. I certainly hope that Mark or Sundar are not wasting even a millisecond of their time writing code and trying to get a PR out to production. It's one of the absolute worst uses of their time. But while we're on the topic, Mark literally built the first version of Facebook (to be fair, probably in a bad hacky way) and Sundar was a product manager so I certainly don't expect him to write code.

koverda(10000) 1 day ago [-]

managers are employees too

immigrantheart(10000) about 23 hours ago [-]

You just have to use this to push your ideology that leetcode style of interviews don't work, don't you.

ThrowawayR2(10000) 1 day ago [-]

If you have time to faff around at a FAANG, you have time to be cultivating your network to include some very influential people, you have time to be taking advantage of training resources or learning from the experts there that are completely free that most ordinary developers would have to pay thousands to get access to, you have time to work on side projects either for the company or, if you dare, for your own personal benefit, you have time to be hunting around for internal transfers that will boost your career, etc.

If you want to rest and vest, hey, more power to you but the smart ones are taking advantage of the gigantic cornucopia of opportunity presented to them by merely getting in the door of an obscenely wealthy FAANG to catapult their careers ahead.

outworlder(10000) 1 day ago [-]

> maybe it's the fact that it takes forever to get anything done at FAANG nowadays.

At any large company. Tiny changes that should take an afternoon end up taking 6 months once all the red tape is done and all involved stakeholders have signed off.

shtopointo(10000) 1 day ago [-]

> Or maybe, just maybe, interviewing based on esoteric computer science problems isn't the best way to identify high performing builders.. but a great way of identifying people who can hack a process to secure maximal reward.

If anything, that might be the best way to identify someone that fits in a large corp like Google. Someone that doesn't mind going thru the drudge of studying esoteric CS problems probably will be more attuned to go thru the drudge of working for a large company like Google.

I'm thinking most of the time spent at Large Corp. Inc. is doing menial work, rather than hot projects where you learn and get to work on the cutting edge.

iroh2727(10000) about 21 hours ago [-]

Yeah at least several years ago I had an explanation for this (though I'm not sure if it still applies). Basically, I think one reason for this weird type of interview is that it was an indirect way to bias towards young hires.

Young people have that energy and naïveté to do a lot of the grunt work. And most work at any established company is kind've grunt work. Anyways, just a random theory but nowadays it may be backfiring.

DmitryOlshansky(10000) 1 day ago [-]

I got hired by Google in 2016 and I could tell you the interview was a series of interesting tasks all having to do with what I was hired for - working on compilers and related tools.

Though after that I was asked for additional interviews on basic algorithmic stuff cause Google thought original interviews to be too narrow in the scope, anyway hardly any esoteric stuff.

z9znz(10000) 1 day ago [-]

> when the last time was that Mark or Sundar actually wrote any code they pushed to prod

That would be a surpreme waste of company money, and probably they have engineers working for them who are far better developers than they are.

rajeshp1986(10000) 1 day ago [-]

+1, why blame employees? blame the management. In my previous job, our manager quickly grew team and hired 3x more people just cos he wanted to manage a larger team and get to hire managers under him so that he gets promoted to Sr. Manager.

dilyevsky(10000) about 19 hours ago [-]

Not sure about Mark wouldn't be surprised if he still hacks php on the side but Pichai joined google as a manager I think from mckinsey of all places... so Im going with "never"

galdosdi(10000) 1 day ago [-]

> Look, if I can 'crack the coding interview', then I can certainly crack 'how to do as little work as possible and stack paper to the ceiling while my stock vests'.

What's worse, many of the jaded people going through the motions probably started out gung-ho but then got frustrated to see how little impact they were really able to have and eventually became checked out. These kinds of things are self fulfilling prophecies in organizations.

throw8383833jj(10000) about 23 hours ago [-]

Productivity is limited by the scope of your products. If you have 100 Work years of work available to do but have 500 employees then each employee can only do 0.2 work years worth of work per year. People want to be productive but if there's not enough productive work do then people start doing unproductive work because it's all that is available.

Solution: You need to either increase the scope of products: get into new industrys like Tesla and Amazon are doing OR cut head count massively.

luckydata(10000) about 23 hours ago [-]

There's tons of stuff to do, the problem is management creates enormous hurdles for anyone to do anything within Google. The amount of red tape and disorganization is hard to believe if not witnessed first hand.

jjslocum3(10000) 1 day ago [-]

Glad these guys seem to finally be noticing.

I was a software engineering manager at a lean, high-margin, profitable start-up based in the NYC area starting in the late 2000s. We were acquired in 2014 by a very typical (for the time) SV-based competitor that had raised hundreds of millions in an IPO a few years earlier. Our acquirers had yet to see a single quarter of profit, of course.

I and my team had so many good laughs at the attitudes of our CA counterparts. One especially strong memory is when, a week after a particularly dismal quarterly earnings report, a junior engineer based in the HQ of our new corporate overlords sent out a team-wide email complaining about the corporate decision to no longer stock the refrigerators with free fresh blueberries. They bemoaned the lack of respect for the 'talent,' and tossed in gratis the ubiquitous pseudo-threat 'if you don't treat us right, we can always go down the road to an employer who will.'

On visits to HQ in Redwood City, I marveled at the paradisaical campus-like setting (several buildings around a 'quad,' with parks, a tennis court, swimming pool, gyms, etc. etc.) and noted the amount of time the local staff spent taking advantage of these amenities. I remember the engineers on my team from HQ explaining to me that my proposed stand-up meeting schedule wouldn't work beacuse their intramural basketball league scheduled their games for that time. Meanwhile, in our low-perqs atmosphere in NY, distractions were limited and productivity was high. We also all made money.

Since that was Silicon Valley during one of the many gold rushes, I thought that I must have been 'missing something.' What seemed like common sense to me was clearly heresy to the golden people there. The explanation I arrived at was that such perqs were the necessary counterpart to an expectation that your employees have no life other than work.

I came to realize I wasn't missing anything, they were. That company did end up burning through their cash stockpile, and had to sell a few years later for less than 1/4 of what they paid to acquire us.

mikhael28(10000) 1 day ago [-]

Ironically, they were smart to acquihire you.

It seems like management was aware their employees were bums, and needed your companies energy to infuse some productivity into their lifestyle.

Looks like it failed though.

allenu(10000) about 22 hours ago [-]

I agree that there is definitely a sense of entitlement in this industry, especially with these larger companies. On the other hand, if you're coming from a startup, you're going to be way more incentivized to put in the work since that ultimately determines your future (and likely, your pay). These guys playing basketball in the middle of the day are getting paid either way, regardless of how well their projects turn out. I don't blame them. I've been at those cushy jobs and doing extra work often doesn't get rewarded anyway, so you may as well maximize your quality of life.

kortilla(10000) about 23 hours ago [-]

That is a big pile of specious reasoning. All of the things you listed could have had effectively no meaningful impact on the bottom line.

There are tech companies that absolutely print money and have those perks. There are also companies that grind and don't turn out shit.

If fresh blueberries for software engineers are gonna wreck you, you aren't in a business worth doing.

excitom(10000) 1 day ago [-]

I remember a story, perhaps apocryphal, of an Austin-based startup that crashed in 2000. At the company meeting where layoffs were announced, the floor was opened for questions. Someone asked 'does this mean the rock climbing wall in the cafeteria won't be completed?'

strikelaserclaw(10000) 1 day ago [-]

A lot of SV engineers in the last 10 years have had it good. Like how people born in America have it good compared to people born in Africa (irrespective of intelligence, hard work, talent etc...). I think that will change.

015a(10000) 1 day ago [-]

You know; there are two sides to view this coin from: either 'those tech people are insane with all their beautiful buildings, great perks, and fantastic work-life balance' or 'those tech people are forward-looking with how we could just make work less shitty for everyone, if only other industries would catch on'.

I'm sad that even many on here seem to be opting for the 'insane' line of thinking, and not recognizing that Work Should Be This Way For Everyone. Its not insane to want to work 20 hour weeks. Its not insane to think working in a concrete windowless office building is uninspiring (our species built twenty story cathedrals to celebrate God; architecture matters; outdoor space matters). Its not insane to want some snacks & drinks throughout the 8+ hour work day (at least until we solve, you know, that pesky human drive called Hunger).

Some of y'all would rather wrestle with pigs in the mud than recognize that, maybe, there shouldn't be any mud at all. But, after all, capitalism is brain worms which convince you the system is optimal when everything sucks for the very people who keep it going. Rest assured, the CEO has a secretary who will go buy fresh blueberries on the company card the moment he desires them.

mgfist(10000) 1 day ago [-]

Sure but the big faang stocks literally print more money than every other company (idk maybe aramco or berkshire can compete, but nothing else). So something's working there.

shagie(10000) 1 day ago [-]

> The explanation I arrived at was that such perqs were the necessary counterpart to an expectation that your employees have no life other than work.


I believe a lot of companies are trying to establish a third place ( https://en.wikipedia.org/wiki/Third_place and https://www.joelonsoftware.com/2003/02/28/20030228/ ) to help transition new grads and young adults from a college atmosphere to a professional atmosphere... but putting a lot of emphasis on having that third place. Having it _also_ means that employees tend to stay later at work.

Things like https://www.woodworkingnetwork.com/custom-woodworking/cabine...

These are ways to use excess money in a way that rewards employees and makes some of the aspects hard to leave ('I could switch companies but then I'd lose the woodshop!') but it also sets up another set of problems in the nature of the third place - that its not work. The coffee shop that you show up to outside of work shouldn't have a manager / employee relationship between the patrons, but the coffee shop on the campus of a big company - that's harder.

It is those third space encroachments where the company is sponsoring it and yet the company wanting to not be political / social / getting into those HR issues, but yet the invariably show up there that lead to articles about how the company is going to be not political, or that half the staff is leaving because the company took a certain stance in a not-3rd space.

These third space encroachments where company life is used as a substitute for one's own hobbies and stepping beyond the college life atmosphere is where companies have social problems.


racl101(10000) 1 day ago [-]

> Meanwhile, in our low-perqs atmosphere in NY, distractions were limited and productivity was high. We also all made money.

Yes, that's how it usually works out.

By the way, 'perqs' is a peculiar word. English is my second language but I'm used to seeing the word 'perks'.

quantumsequoia(10000) 1 day ago [-]

I am very curious which company this can be, can't think of any companies in Redwood City that match that description

esoterica(10000) 1 day ago [-]

Unless you got paid more money than your lazy peers it seems like they got a much better deal than you did. Why are you bragging about working harder and getting treated worse than your coworkers?

pugworthy(10000) 1 day ago [-]

I work for a large company (2 letter name) that has none of those perks, and never really did (at least at most sites). I just perceive the company as being cheap rather than forward looking.

jrockway(10000) 1 day ago [-]

I see where you're coming from. One of the pieces of cognitive dissonance I had at Google was that I always had so much work to do, and there were just so many people around the office chilling out; waiting in long lines for free food, playing ping pong, making themselves an espresso. I never really felt like I had time for that; I got a grab and go sandwich and drip coffee and then hung out at my desk for 8 hours. I started the day with an infinite amount of work, and ended the day with an infinite amount of work. The melancholy of a good idea is that working on it just yields more good ideas; no matter how much work you get done, you'll always be making more.

The downside to my approach is that I super burned out. I had 'strongly exceeding expectations' for 2 quarters, then my project was cancelled so I switched teams and went on a PIP. Indeed, I flat up stopped showing up to work. (I was so bitter about the fact that I lined up a new job immediately, but people that didn't do that got 6 months of paid vacation to explore other teams. I got nothing, and I needed it bad. The company doctor did give me antidepressants and some unpaid leave though. Thanks for that, turns out antidepressants don't treat burnout.)

I didn't even know that burnout was a thing back then, but if I did, I would know that making sure that you jam in 40 hours of programming and meetings into every week without taking a break isn't that healthy or productive over the long term. All these people chatting in the lunch line or playing ping pong or doing an aggressive workout and then showering in the middle of the day were optimizing for their long-term productivity. 1 hour less task-doing today, 10 extra years in their career. Not a bad tradeoff at all.

At a startup, you might not be able to afford that; by the time you're burned out, you've already sold your company and are retired, so it's all good. But at a big company, it makes a lot of sense; talent acquisition is expensive and if you can get 10 years out of someone instead of 6 months, you're going to be a lot more successful. And there's that uncomfortable medium where that extreme productivity didn't actually make a business that can afford to not burn people out, but now everyone's burned out. A lot of companies are in that state, and there isn't an easy way out of that without a time machine.

Engineers that call you out on you burning them out are absolutely right to complain. The basketball game is a much better use of their time than the standup. Standups only matter to people organizing the project; the meeting is only for your benefit. It saves you the time of reading their commits and design docs, sitting in on their engineering discussions, soliciting feedback when writing performance reviews, etc. The actual creative work of software engineering is done when your head is free from distractions and anything you don't need to know about. A walk around the quad or a basketball game is a great way to chew on the ideas, discard all that's unnecessary, and set you up for the 4 hours where you physically translate a quarter's worth of thinking into code that can be checked in.

At the end of the day, it's not really the software engineer's fault for the company losing money. Businesses fail because there is not a plan for making money and the actual engineering tasks are irrelevant. 'Sprint 12323: rearrange the deck chairs on the Titanic.' is what 90% of software engineers are doing right now. They are right to go elsewhere when your business plan is so bad that the company can't even afford blueberries. Do you really think that if people just sat in front of their computer for 30 more minutes a day, or provided better updates in their standup, that the bad idea of a company would be saved? Some companies just weren't meant to be. VCs are very bad at not giving these companies money, though, so there are a lot of people running in circles doing nothing as they slowly realize they never should have started the company. Ultimately, you can't blame the nice campus or intramural basketball league for that.

Salgat(10000) 1 day ago [-]

A lot of companies use those perks as an excuse to get their workers to stick around an extra 4+ hours at the office. Of course this doesn't actually help productivity (they simply drag their day and work out longer), but to simple minded managers it sure seems like a huge win.

onlyrealcuzzo(10000) 1 day ago [-]

> I remember the engineers on my team from HQ explaining to me that my proposed stand-up meeting schedule wouldn't work beacuse their intramural basketball league scheduled their games for that time. Meanwhile, in our low-perqs atmosphere in NY, distractions were limited and productivity was high. We also all made money.

Your standup meeting could've been an email. Their immovable basketball game (quality of life) is far more important than a meeting that can happen at any time - and probably doesn't even need to exist in the first place.

Other than that, your points stand.

nrmitchi(10000) 1 day ago [-]

I thought it was a widely-known-secret that at least part of the insane hiring of engineers without clear teams, projects, etc for them to work on initially was to prevent individuals working for current-or-potential competitors.

Purposely over-hiring to prevent work being done elsewhere, and then claiming there is not enough work to be done, feels like it shouldn't be surprising to anyone.

Hell, Google has created ~18 (I think?) different messenger/chat apps at this point. If you wanted a clue that there wasn't enough work to go around (and that your promotion incentives may not be aligned with the business), this should have been the first clue.

ThrowawayR2(10000) 1 day ago [-]

> 'I thought it was a widely-known-secret that at least part of the insane hiring of engineers without clear teams, projects, etc for them to work on initially was to prevent individuals working for current-or-potential competitors.'

It's widely known among the sort of person who tends to believe in conspiracy theories, I suppose. The oppressive bureaucracy and misaligned incentives that allow senior leaders to destructively compete among themselves is more than enough to explain why ill-conceived and ill-run projects are common at FAANG-level megacorporations without resorting to making things up.

mytailorisrich(10000) 1 day ago [-]

That does not make sense.

On the other hand, if no-one stops it, there are always incentives to grow your team as much as possible.

As leader this increases your status both in absolute terms (100 vs 10 people under you makes a difference on your CV and on the title you can claim) and in relative terms (your team is larger than the teams of your peers and you can get ahead that way).

And so every leader at every level tries to expand their team.

mc32(10000) 1 day ago [-]

Google et al. cargo culted SGI culture -maybe it works for a class of geeks. Anyway they often coddled employees and treated them 'like family' as they like to say and tell them they are special and the lucky few. You can bring your pet to work (if no one has allergies to it), you can waltz in late, go get a snack, log in, chat with your friends, play with new gizmos, then go to a meeting, get lunch, then work out, then have another snack and then the last meeting of the day before you cut out early to get in the (Co.)-bus home before traffic gets bad.

Where the hell did they think productivity would go?

Ferrotin(10000) 1 day ago [-]

That's just something people said on the internet with no sound basis for it.

amzn-throw(10000) 1 day ago [-]

That's not a conspiracy theory,

I work for Amazon - for a decade. I love it - best job I've ever had. And historically, while it's been a tough place to work, we've always been able to attract top talent. Partially - impactful work. Partially - stock doubles every year.

Well guess what happened in 2020/2021? Despite incredible perseverance through the Pandemic, the stock stopped doubling.

Meanwhile, Microsoft, Meta, and others figured out that they can poach our engineers with a promise of way more base salary, and a less intense work environment.

We've had SDE1s (Juniors) leave Amazon for Meta because they got more money than our SDE3s (Seniors) were getting.

SDE2s (Intermediate) looked at their status quo thought 'I COULD bust my ass and get promoted to Senior...or I could go to Microsoft TODAY, get a Senior offer for what I'm already doing, and for more money than my raise would be'. (No offense to any of my friends at Microsoft, but https://www.levels.fyi/?compare=Amazon,Microsoft&track=Softw... doesn't lie)

I've talked to a few acquaintances that have left and the universal responses is: 'My job is so boring now. I miss Amazon. But It's not stressful (because there is no pressure on me), and I get paid more money'.

How can anyone think there is anything wrong with that? You can't. You can speak about Mission and Impact, and some engineers will be attracted to that - I work on building Forever APIs in the AWS Cloud that gets millions of transactions per second. That to me is WAY more interesting than working on Chat app 15/18.

But for most people they just want to make money and live their lives. Fair enough!

The result? Even though Amazon has adapted somewhat by bumping salaries, they've still lost an ocean of people to nothing particularly ambitious or interesting. They're being parked by Microsoft/Google/Facebook to work on boring unimpactful projects so they can't help Amazon kick their asses.

Sometimes one way to make your house nicer is by breaking the windows in the neighbor's house.

fdr(10000) 1 day ago [-]

I personally don't believe this at all. I think it's almost entirely bureaucratic inertia, and a prisoner's dilemma among the management. One who bloats, floats.

onlyrealcuzzo(10000) 1 day ago [-]

> If you wanted a clue that there wasn't enough work to go around (and that your promotion incentives may not be aligned with the business), this should have been the first clue.

There is definitely enough work to go around at Google, Amazon, and Apple.

Whether promotion makes any sense, and whether people are working on the things that actually move the needle is a different question.

phendrenad2(10000) about 6 hours ago [-]

Then why is their hiring geared toward brain teasers and Bigoh notation? If they want to keep people from building the next Facebook (costing them M&A money, because they'd have to acquire it), why not hire based on ability to get things done?

BobbyJo(10000) 1 day ago [-]

100% this. The clearest basis on which to measure productivity is product, and Google's scattershot approach is obviously not efficient.

api(10000) 1 day ago [-]

> I thought it was a widely-known-secret that at least part of the insane hiring of engineers without clear teams, projects, etc for them to work on initially was to prevent individuals working for current-or-potential competitors.

Wow, I've suspected this for many years and people told me it was nutty.

itsdrewmiller(10000) 1 day ago [-]

>I thought it was a widely-known-secret that at least part of the insane hiring of engineers without clear teams, projects, etc for them to work on initially was to prevent individuals working for current-or-potential competitors.

I've heard this claimed but not sourced, and it doesn't really make sense - there are millions of software engineers out there and Google or Meta only employ a tiny fraction of them.

tagami(10000) 1 day ago [-]

A haircut (10-20%?) would be good for all involved. There are great employees and there are slackers in all fields.

ReflectedImage(10000) about 1 hour ago [-]

Well I'm in a corp style place. The haircut was done by rolling dice. And whilst my number didn't come up, I wouldn't be so sure that's it's good for any involved.

tpmoney(10000) 1 day ago [-]

This seems like something that should be expected? Every time the WFH battle has come up over the last few years, there are always people talking about how they're able to do all their days work in 4 hours and spend the rest of the time idle "pretending to work". Is it really surprising that as a result of this companies are reevaluating how much slack time their employees have? Especially as wages and demand for wages due to inflation have spiked, you can probably shore up some of that demand just by dropping some of those 4 hour employees and using their wages to pay others to become 6 hour or 8 hour employees. Sure it's unrealistic for a company to expect every employee is 100% engaged 40 hours a week, especially in knowledge / creative work we're sometimes unplugging and downtime is exactly what the job requires. But it seems equally unrealistic to crow about how the pandemic has demonstrated that WFH is perfectly fine and had no negative impacts because everyone was already only putting in 20 hours a week and not expect that to have caused companies to make a shift.

MAGZine(10000) 1 day ago [-]

I think we've built companies and cultures that are incompatible with long-term employment and happiness.

Anyone who joins a company can crank full 8+ hours a day for a while to establish themselves (and a reputation).

The 'problem' is, as people establish themselves, the problem domain becomes less exciting. There's less urgency to crank indefinitely. They settle into a pattern that involves fewer hours, though those hours are more productive because they know the ropes.

There is a sweet spot where someone knows enough to be productive but isn't yet complacent. This is the spot that every employer dreams of: employees cranking, full speed, productively, for 8 hours.

It's just not sustainable. You can fire people and try and keep turning over staff such that everyone stays in that sweet spot, but you'll eventually end up with a different sort of headache when your staff has no organizational memory for why decisions were made. The people who built things and have the long-term visions have left, and those who pick up the torch try will never have the same big-picture in their head.

The challenging bit is how do you separate someone who works 3h a day because that's all they can sustain (and they're just being realistic), and those who work 3h a day, could work more, but chooses not to? I'm not sure you want to force either out, but can you incentivize the latter to produce more?

akmarinov(10000) 1 day ago [-]

There's no way you can sustain 8 hours/day of productive work 5 days a week as a developer. It's not working a field or packing boxes, there's a mental component that gets exhausted over time.

csa(10000) about 23 hours ago [-]

> you can probably shore up some of that demand just by dropping some of those 4 hour employees and using their wages to pay others to become 6 hour or 8 hour employees.

Sure. Do this if you want to kill morale and be chronically understaffed (either not enough bodies or not enough qualified bodies) for the rest of your existence.

This type of mentality breeds mediocrity for a number of reasons, the main one being that A-players will run away from teams/companies structured around these heuristics. Furthermore, they will make sure all of their A-player friends are aware of this environment.

Good managers and good management teams have no issues with productivity of in-office or remote workers. If your team or company is actually having productivity issues (rather than using productivity as a precursor to a rif), then point that finger at the management and management culture.

marcus_holmes(10000) 1 day ago [-]

The difference between working from home and working in the office is not how many hours of productive work you do, it's in what you do with the rest of the day.

Every single study done on it shows that creative staff (including engineers) are more productive working where they are less disturbed, that open-plan offices are the least productive environment, etc. So it's utterly unsurprising that people get more productive working from home and can do 8 hour's office work in 5 hours at home.

But even aside from that, if you can complete your work in 6 hours, but can't leave the office for another 2-4 hours because of the office culture, then you'll spend those 2-4 hours doing random stuff in the office. If you're at home, you can leave Slack on and go do something useful. It's not only that WFH gives people more time, it's that it removes the 'you must pretend to be busy for 25% of your workday' restriction.

As always, a negative reaction to WFH is a sign of bad management culture. Good managers are happy that their people are getting more done and happier about it. Bad managers see 'they're only doing 20 hours a week if they work from home!' and are angry about it.

fefe23(10000) 1 day ago [-]

As a consultant, I come around a bit.

I have seen many companies with very poor productivity, and in zero of those cases was it laziness of the employees. In fact they usually would have loved to be more productive. Nobody wants to spend their life being dead weight.

But as companies grow they install more and more rules and regulations that end up making sure nothing ever gets done. It is not unusual to meet 'developers' whose company calendar is 80% filled with meetings. Well no wonder they don't get anything done!

Also remember that this is only half the problem. The other half is that agile makes you iterate through pseudo productivity before you actually understood the problem, accumulating cruft that you need to maintain and extend as you go on. I wouldn't be surprised if of the productivity that is left, more than half gets wasted on crufty software structures and writing code before you understood the problem.

And then nobody wants to throw code away that turned out to be not what we need. Wasting yet more productivity on working around bad decisions from before we knew what we are actually building.

queuebert(10000) about 22 hours ago [-]

I feel this in my bones, as I'm having to fill out a 27-page document to ask permission to use a new piece of software on the intranet.

gonzo41(10000) 1 day ago [-]

I've seen companies where the leaders will only trust the opinions of the consultants. Even if they are the same conclusions of existing employees.

Hired talent isn't magical but for some businesses the consultant workers have an glow about them. The result is the business effectively making their own workforce redundant because they fear relying on them. And then morale tanks, and people leave.

999900000999(10000) 1 day ago [-]

>But as companies grow they install more and more rules and regulations that end up making sure nothing ever gets done. It is not unusual to meet 'developers' whose company calendar is 80% filled with meetings. Well no wonder they don't get anything done!

As long as you're not mean, you can hang out at most companies for at least 6 months just doing nothing.

I've been reprimanded before , when I took the initiative to try and start building out a framework. I literally had nothing else to do, but I was later told I should have waited until a committee could be formed.

Even if you barely do anything, at least you're not causing trouble. In my career. I've worked with several abrasive angry people, I've seen folks confront C level employees.

Developers who cry about having to use a PC to write some.net code and throw a temper tantrum. Threaten to just walk out because some legacy code needed updating and they're so used to having a precious Mac to code on.

That said, I actually really like him how limited social interactions are with remote work. I don't need to know your political beliefs, I don't need to be your friend, I don't want to get drinks with you, I want to do what is necessary for my job.

Corporate fluff plays a role. I imagine Google develops products that will never be profitable just so they can look at their shareholders and say, looky we do stuff aside from search.

hef19898(10000) 1 day ago [-]

Not getting rid of 'legacy' stuff that doesn't work is a, IMHO, a version of throwing good moneybafter bad money. Instead of acknowledging that the unusable code, or whatever, was a crucial part of understanding the problem, and throw it out once the problem was understood, people tend to build upon those not fit for purpose things...

michaelcampbell(10000) about 11 hours ago [-]

> And then nobody wants to throw code away that turned out to be not what we need.

Not entirely true. I don't mind that one bit. I can voice my opinion on what 'we need', but ultimately that's not my decision and there are people hired to do that. I get paid to write it, I'm happy in that spot, and if I end up not having to deploy it, go through whatever baroque testing cycles are in place, or do the job of 3 with the salary of 1 by having to do sysadmin, DevOps, or whatever other fad du jour is sweeping the industry with fancy terms just trying to keep the CEO's in their millions, fine with me.

coffeeisyummy(10000) 1 day ago [-]

This iteration through pseudo productivity comes from management's real world problem of demonstrating progress on their projects. The promises of visibility on your development team's productivity always turns Agile into a steaming pile of burn-downs and story points.

'No one has done true Agile' is the 'No one has done true Communism' for software engineering. Because, in the real world, no one uses Agile in an ideal environment free of pressures like deadlines or budgets.

wonderwonder(10000) about 23 hours ago [-]

I spent about a year and a half being dead weight. I was so completely burned after working months of 70 to 90 hour weeks I just couldn't do anything. Things that used to take me an hour to code now took me days. Complete mental block. Luckily I had built up such a good reputation prior I was able to coast and it was a weird project. In a new job / role now and it's better. Only work 40 hours max. Still not back to normal but 75% of the way there.

likortera(10000) about 18 hours ago [-]

Well said. This matches my experience 100%.

A pretty well known ticketing company bought our startup a few years ago, and after the first week of parties, raising salaries and hyping us the reality struck us very hard. It was impossible to do any work at all. Anything you wanted to do would require tons of meetings, there was always a few people blocking any initiative you could have.

And then the freaking Agile By The book (with agile coaches and all!) I couldn't stand for the life of me. We'd have like 10 ritual meetings a week and the joke was that those meetings were to discuss 'What we're going to do, what we're not doing, and what we didn't do'.

Worst part, is that *everything* pushed you to just stay at your desk watching online courses or reading stuff on the internet and do nothing, and as long as you showed up to your scheduled meetings, all was good. You'd even get promotions by just smiling around and being nice to others.

I left that and now I'm at a company about 3 times as big. The difference is that here we're 100% remote and 100% async, written communication. Literally ZERO work meetings a week, just one 'hang out' to not forget about the faces of your coworkers. No Agile, no Jira, no bullshit. A shared 'to do' list to show others what you're on and weekly reports of your progress. I just can't believe how well this works.

robomartin(10000) about 21 hours ago [-]

Remote working has been around forever. The pandemic opened the doors to a larger set of candidates. None if it changed human nature. There are people who do well on their own. I think most don't.

I know someone who has had remote jobs for probably 35 years. How does he spend his time? Re-roofing his home, upgrading his bathrooms, fixing his cars, etc. Not working. And these are six figure jobs. Watching this first hand —for decades— has not made me a huge believer in remote work for everyone. Not sure how to define who does well and who does not.

bogota(10000) 1 day ago [-]

I just know of at least 20 people left my previous company because we had nothing to do. Every meeting was trying to figure out what the direction was. As an engineer when the company gets to the size of 1000+ you are largely not at all empowered to solve this problem but have to rely on your manager or in some cases your managers manager.

But come time for performance review you get bad marks. If you think that many people are just lazy for no reason you have no right to be managing or running a business.

Sitting around pretending to work all day is a recipe for depression and burnout. No one wants that.

YZF(10000) about 20 hours ago [-]

Another factor here is that companies will hire more people as a growth strategy without having any clear idea of how to deploy them. Even if they have a high level idea they may not know how to translate that high level idea to something actionable. They have a high P/E. There's cheap money. They need to somehow grow. The only way they know to grow is just to hire more people. As you say, no wonder they don't have anything to work on. Maybe the idea is that if they're working for you they're not working for the competition. I donno.

2muchcoffeeman(10000) about 24 hours ago [-]

I'm not sure why you needed to come around.

Leadership signs off on hiring. Leadership signs off on installing far reaching processes that inhibit devs from making contributions.

I'm sure some people try to find ways to cheat the system. But I find it hard to believe that it's a wide spread problem. Even people doing the minimum work possible probably have a ton of other interests or ideas and would rather be engaged with their work somewhat and learning things than idling.

rubiquity(10000) 1 day ago [-]

I'm surprised to see this is the top voted comment because it is completely off the mark in this case and anyone that has spent any amount of time reading Blind (a website dedicated to... I'm not sure what exactly) knows it.

While Facebook/Meta, Google, and others have always paid comparatively well, in the past 2-3 years the pay shot up even higher and the only price of admission is supreme obedience to 'grinding LeetCode.' This hysteria created an entire culture of pay chasers that congregate on that Blind website with little regard for anyhing other than compensation. These people, who I consider to be among the most toxic people in tech, have a singular focus on pay and it is not at all surprising that when put in minimal supervision environments they choose to merely exist and collect said paycheck. CEOs lamenting this are merely reaping what they sow.

harpiaharpyja(10000) 1 day ago [-]

I thought the point of iterating early is that sometimes writing code is the best way to gain understanding of the problem (depending on the kind of problem). You're supposed to throw that stuff away... it's iteration...

rgblambda(10000) about 23 hours ago [-]

>And then nobody wants to throw code away that turned out to be not what we need.

I once spent two months trying to get my technical lead to do a code review for a PR I raised. Eventually the business informed us they didn't actually need the feature that the PR implemented. At that point, my technical lead immediately approved the PR so it wouldn't be (seen as) a waste.

rr888(10000) about 24 hours ago [-]

> Nobody wants to spend their life being dead weight

I read loads of blogs and posts where people are loving WFH, doing very little and openly recommending tech career to others because its so great. They might not think they're a dead weight, they just think thats what modern working is like.

hcarvalhoalves(10000) 1 day ago [-]

> The other half is that agile makes you iterate through pseudo productivity before you actually understood the problem, accumulating cruft that you need to maintain and extend as you go on. I wouldn't be surprised if of the productivity that is left, more than half gets wasted on crufty software structures and writing code before you understood the problem.

I've seen this increase proportional to the number of employees. People start trying to worry more about perception of progress by tracking proxy metrics, because the large the company, the harder it is to prove how each one contributes directly to the bottom line.

icedchai(10000) 1 day ago [-]

I worked at a company where I'd have at least 2 or 3 days a week where we had 4 hours of meetings. It was pure hell. Half the time I wouldn't even pay attention. I'd be browsing reddit or HN.

You hit the nail on the head with agile. I remember writing some code only to have the whole thing ripped out 'next sprint' because nobody bothered to think a couple weeks ahead. Or starting an integration project with a third party, only to find out they're not ready, so we have no API that actually works. So we waste time mocking it out, only to find out the docs they gave us don't match reality.

throwaway292939(10000) about 23 hours ago [-]

That's one take, but if you hang around on Blind (which is an anonymous forum heavily populated by FAANG), you will find many who gloat about how little they work.

commandlinefan(10000) 1 day ago [-]

> company calendar is 80% filled with meetings

The typical expectation on salaried employees is that you spend your 8-5 in meetings and then you 5-midnight actually doing programming work.

dboreham(10000) 1 day ago [-]

> agile makes you iterate through pseudo productivity before you actually understood the problem

This kind of 'development process theater' causes terrible cognitive dissonance.

codegeek(10000) 1 day ago [-]

'Nobody wants to spend their life being dead weight.'

I disagree. There are plenty of people who would love to be dead weight just to float around in a company. The larger the team/company, the more chances of those people being around. They pretend to be always busy and doing something but don't actually get anything done. Seen it all for 18+ years.

Having said that, there are plenty of people as well who would LOVE to do something meaningful but are stuck with red tape. I was one of those and quit my high paying Investment Bank Tech Job to start my own thing. I was getting paid big as a consultant and once my main project finished, they just wanted me around because traders loved me. I literally had to find things to do every day otherwise it was soooo boring unless something broke.

adolph(10000) about 24 hours ago [-]

Material affluence for the majority has gradually shifted people's orientation toward work—from what Daniel Yankelovich called an "instrumental" view of work, where work was a means to an end, to a more "sacred" view, where people seek the "intrinsic" benefits of work. "Our grandfathers worked six days a week to earn what most of us now earn by Tuesday afternoon," says Bill O'Brien, former CEO of Hanover Insurance. "The ferment in management will continue until we build organizations that are more consistent with man's higher aspirations beyond food, shelter and belonging."

Senge, Peter M.. The Fifth Discipline (p. 16). Crown. Kindle Edition.

gofreddygo(10000) about 21 hours ago [-]

Measure the product before measuring productivity.

across the board execs complaining about productivity turn out to be poor at defining product ('its just a website, how long could it take to build, Jeez').

Any productivity comparisons between software and other manufacturing processes should begin with a few minutes spent to compare software specs and the said product's spec, see how hard it is to change its spec ('add a button to accept payments' v/s. 'add a knob on the car's dashboard')

provide a technical spec first, then we can talk about productivity.

giantg2(10000) about 24 hours ago [-]

Some company cultures will punish people for taking the initiative too.

devwastaken(10000) 1 day ago [-]

Large corps are propped up by intellectual property law and economies of scale. They do not hold their market positions on their own merit. If we remove IP laws, we will have another golden age of tech innovation tomorrow.

rajeshp1986(10000) 1 day ago [-]

What I noticed is it is not employee laziness but the FAANG companies have ton of dead weight in terms of future projects or project features which never get released. One of my co-workers was working on a feature which was shelved after working 2+ years on it. He lost motivation after that and coasted the rest of the time doing minimum work. I think FAANG companies have lot of PMs and top management who are as clueless and lazy as engineers.

osigurdson(10000) 1 day ago [-]

The sad part about excessive meetings is often they are not enough on their own. In between all of the the pointless meetings, smaller, less formal, often unscheduled, real meetings where actual decisions are made still need to happen.

dleslie(10000) 1 day ago [-]

> It is not unusual to meet 'developers' whose company calendar is 80% filled with meetings. Well no wonder they don't get anything done!

IMHO, if you're a developer and have more than 8h of meetings a week then you are no longer a developer. In the worst case, you are a body to fill a seat in a meeting to fluff the self-importance of your management. In the best case, you're on track to being management yourself.

asdjjsvnn(10000) 1 day ago [-]

Part of the problem is also the incentives and performance axes that are defined to evaluate work/productivity.

At a higher experience level, you are expected not just to churn out code but also to demonstrate performance on axes such as influence, scope, leadership etc. In fact, if you just churn out code and not perform on other axes, you are under performing under other axes. So, I could solve a particular problem for my team quickly with no dependencies with other teams/people, but I am now forced to go to other teams and look if they have similar problems to solve and then work on getting alignment on a common solution which would work as a common framework for both team's use cases. While this in theory is good to have one generic solution for a set of similar problems, once a huge company has incentivized this, lot of people are trying to build the next standard/framework and as you'd expect adoption becomes a problem because everyone is trying to evangelize their own framework. The end result, you suddenly have to work with x number of people and let everyone align with what you are doing, that takes time, then you implement something and now have to convince others to use your framework, which again takes time. Add these dependencies and you have what you currently have, a mechanism that moves slowly with most people involved feel helpless and think if it was just up to them they would have it all done in a few days.

PKop(10000) 1 day ago [-]

>Nobody wants to spend their life being dead weight.

Of course this isn't true.

g051051(10000) 1 day ago [-]

> The other half is that agile makes you iterate through pseudo productivity before you actually understood the problem, accumulating cruft that you need to maintain and extend as you go on.

Well said!

sibeliuss(10000) about 22 hours ago [-]

> 'and in zero of those cases was it laziness of the employees'

Come on! This is straight up impossible. Anyone who has worked for any length of time in the tech industry has come across people that simply don't do anything, and are totally fine with that. It is *very common* and its borderline dishonest to say otherwise.

cm2187(10000) 1 day ago [-]

It's not just meetings. I spend 80% of my energy fighting internal resistance, in the form of moronic decision, moronic policies, short signtedness and incompetence. It's not even bad will or people deliberately sabotaging the business. Just frictions grinding the organisation to a quasi standstill, people taking principled approaches to cover their own ass irrespective of the consequences, or being so far remote from the ground that they have no idea of the consequences of their decisions. And in the middle of that you have some courgageous busy bees trying to make things happen despite this internal resistance. Many have given up. I am somewhere in the middle.

goodpoint(10000) 1 day ago [-]

> agile makes you iterate through pseudo productivity

In most companies agile/scrum meetings are make-believe work.

barrenko(10000) 1 day ago [-]

The exact point of a big company is that nothing gets done.

jrsj(10000) about 18 hours ago [-]

I wish this were true but most of the people I have worked with in the past 3 years are just lazy and will pretend to be doing a days worth of work for 2 weeks whenever they can get away with it

pojzon(10000) about 23 hours ago [-]

I really have trouble understanding why ppl should dream about labor / fulfilment at work.

There are so many ways to find a real meaning.

Be a great person, help others, read a book, do yoga, help kids with homework, plant a tree, build something with own hands, grow food, clean-up trash.

So many things to keep you busy. Work is just a necessity to do something that actually matters in the longer run (for majority at least).

Ppl that deeply care about the company and product are such a tiny minority.

abledon(10000) 1 day ago [-]

> Nobody wants to spend their life being dead weight.

Have you worked in Government?

edit: see https://news.ycombinator.com/item?id=26727803 for an example

yuan43(10000) 1 day ago [-]

> To be sure, the Covid-induced pandemic saw Meta embark on a massive hiring spree, growing its number of full-time staff from 48,000 at the end of 2019 to more than 77,800 — a 62 per cent jump. But now the firm must "prioritise more ruthlessly" and "operate leaner, meaner, better executing teams," Meta Chief Product Officer Chris Cox wrote in a memo, which appeared on the company's internal discussion forum Workplace before the Q&A.

The article doesn't mention a different problem. Those new hires entered at extremely inflated salary levels due to literally every other company doing the same thing at the same time. Righting that ship means not just layoffs, but recalibrating salary expectations. The process is just starting.

nvarsj(10000) 1 day ago [-]

It already recalibrated, didn't it? Much of that compensation is stock - of which its value dropped 50% for Meta.

pipeline_peak(10000) 1 day ago [-]

I wonder if Zuckerberg realizes how undesirable of a place to work he's creating, or there's some big picture I'm not getting.

Either way, a lot of silicon valley roles outside of SWE are absolute fluff. It wouldn't surprise me if it's now becoming increasingly obvious as he can no longer afford it.

mola(10000) 1 day ago [-]

He wants ppl to quit. He said it. He wants other ppl to pay for his mistakes

waspight(10000) 1 day ago [-]

Why is Netflix part of FAANG? Isn't all the other ones much larger corporations?

sn41(10000) 1 day ago [-]

Because without Netflix, it would be a bit awkward.

I've always felt that leaving Microsoft out was a bit problematic. But FAAMG does not sound very threatening.

truffdog(10000) 1 day ago [-]

Their top tier compensation packages and stock performance over the past 15 years

ajross(10000) 1 day ago [-]

When the acronym was coined, Netflix was in the process of disrupting the entire entertainment industry and it looked for all the world like it was going to eat them all. As it happened, the industry (well, Disney and HBO at least) figured things out faster than expected, so much of the speculation on Netflix turned out to be wrong. But they absolutely were a Top of the World tech innovator for a while there.

But it's just an acronym, it's not perfect. The other big error is that, obviously, Microsoft needs to be in that list given their pay scale and hiring process.

bastardoperator(10000) 1 day ago [-]

I'm reading this as 'Executive leadership makes hiring and planning mistakes and punishes employees opposed to taking personal responsibility'

Also, I block everything Facebook at the router level with unbound.

tra3(10000) 1 day ago [-]

Not a Facebook or a google fan or an apologist.

But isn't this a business decision? 'Punishment' implies a fault, but the employees are not at fault here.

What would 'taking personal responsibility' look like for management?

colinmhayes(10000) 1 day ago [-]

Mistake is a big word here. Maybe they figured the pandemic would provide the company opportunities it hadn't foreseen and in order to be the first to capitalize they need to hire 30,000 people. Now those extra employees aren't necessary, but that doesn't mean they didn't provide value earlier. I guess it's a bit rude to hire someone knowing that their position will be eliminated in a few years, but that doesn't make it a mistake, just ruthless.

senttoschool(10000) 1 day ago [-]

This is inherently a problem with full-remote or hybrid work.

People will point to 'studies' showing how remote work improves productivity. Maybe it did initially but eventually, people will check out, feeling isolated, feeling less motivated.

Some people who worked remotely before covid swears that it helped their productivity. But these people are biased because they were probably one of the few who were disciplined enough to make it work and they gained the employer's trust over time.

There were a lot of reports of Zuckerberg bemoaning about productivity. Tim Cook wanted everyone back in the office full-time before Delta. Google also wanted everyone back in the office. Clearly, these CEOs aren't just making decisions on a whim and they have real data on productivity rather than some 3rd party studies.

This opinion is not popular here but this is how I see it.

matt_s(10000) 1 day ago [-]

I disagree personally but voted up because this is a valid opinion and I suspect this is the reason why it feels like we get less done. Personally I feel like I thrive remotely, probably work too much but I like it so there's that.

It comes down to some people thrive working remotely and some don't. At any level higher in mgmt than a single team there isn't really any way to determine who can thrive and who can't. Pretend its a 50/50 split across 100 people, the only way upper mgmt can see to get pre-covid productivity is to go back to the office.

I will say another unpopular related point on this: people with young children are more than likely to not thrive working remotely. Or at least they've probably never had the chance to see if working remotely is good for them because they may have had their kids home with them these past couple years. You know how we don't like distractions when trying to do focus work? I can't imagine trying to do focus work with a child or two under the age of 5 there with you all day.

moomoo11(10000) 1 day ago [-]

I'd agree with this. Anecdotally speaking I have never met anyone on my team and I honestly can say that I feel like I'm a mercenary whose job is to just destroy tickets and keep a lookout at our monitoring. It feels so impersonal and is it really my fault or my colleagues that they don't feel as invested?

Messing with k8s, looking at logs, or occasionally hopping into a zoom to discuss architecture for an upcoming project that I don't find any interest in beyond ensuring the stock goes up, it feels like I'm a cog and I just do things and somehow we keep going.

Three years ago I would be super engaged and going to conferences to show off our latest work. Maybe it's the combination of doing boring (to me) infrastructure and dev ops work along with zoom culture. Back in the day I was a mobile application developer so that was quite a different lifestyle compared to this. Idk man, I'm doing my best to do a good job but honestly it is the worst experience of my life so far. I've been spending my time outside of work in evenings and weekends hacking away on side projects. They give me far greater joy, which I used to find previously at work.

DharmaPolice(10000) 1 day ago [-]

>Clearly, these CEOs aren't just making decisions on a whim and they have real data on productivity rather than some 3rd party studies.

Asking people to 100% return to the office is unpopular (or at least controversial) to some, right? If there was 'real data', why wouldn't they mention that in their communications to staff? Instead, it's full of wooly statements like 'there's something missing' and vague stuff about collaboration.

This seems to be a more generalised fallacy - 'The <government/CEO/authority figure> don't do things on a whim, therefore they must have additional (secret) information on <controversial decision>. Based on this, they're obviously correct - after all, they've got that secret info!'.

dev_0(10000) 1 day ago [-]

The amount of meetings need to be cut down for engineers...

cletus(10000) 1 day ago [-]

Once again, companies blaming strategic problems on ICs rather than real culprit: leadership. Or, rather, the lack thereof.

Having worked at both Google and Facebook I can tell you it's contradictory because in some cases you have an embarrassment of riches, hundreds or even thousands of heads, virtually unlimited resources (CPU, storage, networking), etc. Some make sense like Google+. I mean it was a failure and probably came way too late to succeed no matter what Google did but I understand trying. Maps, Docs, Youtube, Photos, Drive, Chrome, Android... all of these make sense.

I also understand you can't necessarily predict 'winners' so to a certain extent you have to try things and expect failures.

Interestingly though every project I listed there (apart from Drive and Photos) was an acquisition.

On the other hand, you have projects desperate for people that turn into abandonware because they don't get sufficiently funded, even when they have PMF.

There are a ton of middle managers at big tech companies who exist only to get promoted and to empire build. You could, in my opinion, take everyone from L7 (M3 at Google, M2 at Facebook) to VP and fire 75% of them and be perfectly fine.

Both of these companies are now in what I call permanent reorg churn. Every few months you'll get an email saying your mananger's manager's manager's manager now reports to a new manager as part of a broad reorg. You've never met any of these people. This is a meme internally.

But what you have to understand is that reorgs are a way of avoiding the appearance of failure while appearing to be doing something. Don't get me wrong. Bad organizational structure can set you up for failure and a good org structure can help you succeed but reorg churn is none of this.

Reorg churn is simply changing the structure every 6 months. Nothing is ever in place long enough to determine if it succeeded or failed. People responsible for those decisions have probably moved on.

Additionally, at Google in particular, the amount of process required to do anything is insane. But don't worry. Bureaucracy busters has another 3 surveys for you to fill out to improve things. I once spent a quarter just babysitting a launch calendar entry.

The checklist to launch anything is insanely long. Even getting a small amount of resources requires Machiavellian machinations.

But sure, there are too many employees. Got it.

Willish42(10000) 1 day ago [-]

> There are a ton of middle managers at big tech companies who exist only to get promoted and to empire build. You could, in my opinion, take everyone from L7 (M3 at Google, M2 at Facebook) to VP and fire 75% of them and be perfectly fine.

> Both of these companies are now in what I call permanent reorg churn. Every few months you'll get an email saying your mananger's manager's manager's manager now reports to a new manager as part of a broad reorg. You've never met any of these people. This is a meme internally.

> But what you have to understand is that reorgs are a way of avoiding the appearance of failure while appearing to be doing something. Don't get me wrong. Bad organizational structure can set you up for failure and a good org structure can help you succeed but reorg churn is none of this.

So many nails being hit on heads. Bravo. I see a lot of discussion in these threads around how hard it is to measure IC productivity, but nearly nothing about how to measure middle manager productivity (spoiler: you can't because their credit is based on work done by the people below them). In the middle of this hiring freeze stuff I got yet another reorg email from my company about my great-great-grand-boss, who I've never met, switching around to add a new layer of middle management new hires. Each of these is worth at least 5 IC headcount, probably more. I don't see a lot of criticism aimed at how _that_ band of the headcount doesn't match productivity...

strikelaserclaw(10000) about 7 hours ago [-]

to be honest, you can never be an extremely efficient org with a huge head count like google and to be honest i don't even see what their problem is, record profits year over year, how long exactly is that trend supposed to continue? in perpetuity?

pradn(10000) about 7 hours ago [-]

The bureaucracy to launch a project can be high. I think some of it is essential. A company really does need to ensure every project launch adheres to regulatory/branding/legal guidelines. We do have to adhere to a high a11y/i18n standard. A smaller company can just choose to ignore a bunch of these things.

The problem is when these requirements mean you have to hunt down a lawyer who can flip the bit and they're busy with other things for the next few weeks. Same goes for security reviewers. Another thing is the ambiguity for what exactly is required for a11y/i18n. These things can be improved.

I work in Google Cloud. After the main technical work is done, a feature launch requires probers, integration tests, metrics, alerts, dashboards, updating the gcloud CLI tool, updating client libraries in several libraries, writing internal support playbooks, writing external docs, writing an external business-side blog post. This can all be construed as bureaucracy since it's not the main feature itself. But we do need all of these things to provide a good customer experience. I've written the main code for a feature that took about 2-3 weeks and then spend 2-3 months doing the rest of these things.

origin_path(10000) about 13 hours ago [-]

Chrome wasn't an acquisition, unless you count the company that only made sandboxes. But that hardly counts.

Photos wasn't really an acquisition. Yes they acquired Picasa but Google Photos has no real connection to Picasa.

neves(10000) 1 day ago [-]

> Zuckerberg noticed that it was getting harder to get all the employees to attend a meeting as they were sometimes taking time out in a day for personal work.

> "There are real concerns that our productivity as a whole is not where it needs to be for the head count we have. [We need to] create a culture that is more mission-focused, more focused on our products, more customer-focused,

Ohhh! Two declarations based in unfounded evidence that will instill fear in employees and prevent them to ask for raises.

What a shitty piece of journalism.

com2kid(10000) 1 day ago [-]

An IRL meeting with 100 participants, you can't tell who is there or not. You can audit online meeting attendees.

> as they were sometimes taking time out in a day for personal work.

People have been going to the dentist during work hours since forever. I used to have a dentist down the street from my office for just this reason. Now I have a dentist just down the street from my house, for the exact same reason.

Heck Microsoft used to encourage people to go to the gym during the work day, a shuttle would come by, pick you up, and take you to the gym! Possibly something about all those research studies showing high levels improvement in mental tasks for hours after exercise.

mikhael28(10000) 1 day ago [-]

Honestly, I'd be pissed too if I was Zuck.

I'm a goddamn billionaire - if I invite you to the meeting, idgaf what you are doing unless someone died - get on this goddamn Zoom call. And even then, they better have been important! Dogs, cats, idgaf, get on this goddamn Zoom call!

Then, because I'm a genius who singlehandedly started Facebook by myself, I extrapolate this thought to its logical extreme and start intimidating my employees based on this intensely personal feeling.

I'm Mark fucking Zuckerberg. Get on this goddamn Zoom call.

telchior(10000) 1 day ago [-]

Which companies with similar concerns have actually managed to increase productivity in a way that satisfies the C-suite?

A much older anecdote: I had a friend who worked at Yahoo around the time Marissa Mayer was coming on as CEO. At the time, they were allowing semi-WFH for certain positions.

I literally never saw this guy go to work, or actually do any work. He was part of a stand-up comedy workshop and spent 100% of his time there. He'd figured out how to keep his manager happy enough, pass performance reviews, collect a huge paycheck, and do exactly squat. Somehow during all the 'clean house' reviews, he passed. Everyone, including him, were shocked that somehow, nobody seemed to be able to figure out that he was essentially a ghost employee. What finally got him was a 'return to office' directive -- no more WFH, which he couldn't comply with.

This all took place a decade ago, and I've thought of it several times post-Covid as all these companies that 'discovered' WFH suddenly decided that employees need to return. But none of the extensive attempts to fix Yahoo's culture, management etc came to anything, the company continued to backslide despite all efforts and now basically no longer exists. Mark Zuckerberg's aggressive 'some people shouldn't be here' statements feel like a repeat of that whole Yahoo debacle (although I suppose Facebook probably isn't yet as dysfunctional as Yahoo was in 2012).

underdeserver(10000) 1 day ago [-]

Given her attitude toward WFH, I'd say Marissa Mayer knew. Maybe not about this specific person, but then he was likely not a special case.

jedberg(10000) 1 day ago [-]

I think Yahoo was a special case though. At that point in the company's life, they attracted the kinds of people that wanted a job they could phone in. I knew a bunch of Ex-Yahoos, and they all cited this fact as one of the main reasons they left.

I also knew some Yahoos at that time, who were not like that, but were frustrated so many of their coworkers were, especially since they had to carry the load. But they liked their job so they stayed anyway.

Marissa came into a terrible situation, and tried to make some big changes to fix it. She wasn't successful, but she did try.

Scoundreller(10000) 1 day ago [-]

I think yahoo found something like 35% of its wfh staff hadn't logged in for weeks or months. (I can't find a source for the number, so maybe I'm off, but vpn logs were used to justify ending wfh, which is... an imperfect approach for many reasons).

Overall, I don't think the plan at yahoo was to fix anything, but just asset-strip it, which worked well for stockholders.

itsautomatisch(10000) about 19 hours ago [-]

A lot of the comments seem to focus on ICs/devs, but for me the people who fit the category of person being discussed are managers, people who basically do nothing and don't contribute anything of value.

daxaxelrod(10000) about 19 hours ago [-]

Every single manager I've had continued to be a solid IC. The only difference is that they had a larger portion of the vote when we decided what we wanted to work on but we were all in the trenches together.

syntaxing(10000) 1 day ago [-]

Yeah you lazy fucks, how dare you work so little that we only have an annual NET INCOME of 39B (39B for Meta, 76B for Alphabet).

vbezhenar(10000) 1 day ago [-]

I guess they measure some KPIs and observe big difference between peers.

mberning(10000) 1 day ago [-]

This is not surprising to anybody that ever worked at a sufficiently large organization. Once you get a large number of employees, then layer in HR, legal, compliance, etc. considerations it creates quite a lot of opportunities for low performers to get in the door and never leave.

schlauerfox(10000) about 2 hours ago [-]

Convenient that they only worry about this during recessions, what purpose would management have to ignore these problems until now? It's a sleight-of-hand to shift blame come earnings time and give management something to do to 'fix the problem' (that their mismanagement created) they can point to, nothing more.

yalogin(10000) 1 day ago [-]

The speed at which the tone of these is changing is amazing. Just a few weeks ago everyone said its getting impossible to hire and so they need to expand to other tech hubs, pay exorbitant salaries and offer lot of perks to attract candidates. All of a sudden now, just in the span of a few weeks, these executives started realizing their headcount is too high, productivity too low and that employees should self select out of the company. Doesn't this show incompetence on the executive part? They just didn't see this till the recession flags were raised, it's almost as if they need to cut costs to cover up falling revenue and so blaming the employees.

kibwen(10000) about 24 hours ago [-]

> Doesn't this show incompetence on the executive part?

Sure, but since when has an executive ever faced consequences for incompetence?

MichaelMoser123(10000) about 24 hours ago [-]

Management is just reacting to a changed financial environment. But they can't admit that, so that they make it look as if it is about someones incompetence.

I think its the war of Russia against Ukraine. All of a sudden there is less money to go around, interest rates go up and it got harder to raise money. I think they are just putting up a straight face, as they respond to the changing circumstances.

And they probably changed their plans as well, now it is less about 'new features' and more about 'maintenance of existing systems'. But that didn't get into the article, so it's all the fault of the people who will have to look for a new job...

gorgoiler(10000) 1 day ago [-]

It's knives out time, I'm afraid, for any activist or negative employee. I am flabbergasted by the number of people I've worked with who are flat out ungrateful when it comes to their relationship with their employer either being outright miserable and surly, or constantly virtue signalling about hypothetical problems that just drag everyone down the purity spiral.

They get paid and they push code but they seem to think that's the be all and end all of the relationship. It would be like living with a partner who takes out the bins and cooks every other night but never gives you a birthday card and constantly complains about your behaviour.

I don't think there's anything at all wrong with wanting to have good social relationships between staff because the flip side is that every Eeyore, loner, and whiner chips away at morale bit by bit until they are the only people left.

How have you rewarded camaraderie, positive attitude, leadership, and goodwill today?

MisterBastahrd(10000) 1 day ago [-]

What the hell does that have to do with being employed?

My employer gives me money. I give them labor. I am friendly with my co-workers because I am generally a friendly person, but I don't owe the company any more than I give and I don't deserve any more than I demand for myself.

There's no 'grateful' to be had here. I'm not grateful to have a job. I have a job because I earn it.

OkayPhysicist(10000) 1 day ago [-]

Your employer isn't your partner, they're your John. They get what they pay for, and that's it. If they want you to perform gratitude to stroke their egos, then they can pay extra for it.

spamizbad(10000) 1 day ago [-]

Man, where have you worked? It sounds awful. Is this a SV specific type of personality? I feel like a low output dev who complains constantly wouldn't last 6 months before landing on a PIP.

My experience, most 'dead weight' employees tend to be quiet types who never rock the boat. They want to just keep flying under the radar. They say please and thank you, they show up to company events, but just.... don't produce. Which can make putting them in a PIP extremely awkward because you feel like the bad guy.

Meanwhile, the most proactive 'complainers' I've worked with have all been median to high output engineers. As a manager, I find my approach for them is to try and get them is to mature socially inside the org and work to break them of their bad habits. Results are mixed, but I've had some success.

dron57(10000) 1 day ago [-]

Your employer is not your partner, come on. The employee - employer relationship is just business. Why should you feel grateful for getting your market based compensation?

becquerel(10000) 1 day ago [-]

Uh... why should employees have a friendly relationship with their work? We don't work because we want to make friends. We work because otherwise we don't have money to buy food or clothe ourselves. This is not a voluntary arrangement. Expecting us to be grateful for it is absurd.

summerlight(10000) 1 day ago [-]

It's weird to say, but there's a genuine lack of headcount all over the company while Pichai (and probably Zukerberg) also does a correct assessment on the situation. This seems a contradiction, but if you take a deeper look into mid level managements there are a good number of teams responsible for billions of users/revenue and driving their growth but screaming for more headcounts. (Yeah, I'm in one of those teams) # increases to hundreds (or thousands?) at smaller scale. IMO, they deserve more headcounts for their scope. But it's also clear that the overall productivity of those companies begins to plateau.

Why does this happen? Of course I don't know. I've seen some clues on bigger structural issues but cannot say for sure. But the famous 'I just want to serve 5TB' video gives us some hints... Most of the particular issues mentioned in the video have been solved but its spirit hasn't gone away. And now back with a good reason. Which makes it much harder to solve.

Think about launching non-trivial but small features in their major products. At a small company, a competent junior engineer can usually do that within a quarter. In Google it's not that simple. There are so many stakeholders. Privacy and security. Legal. Downstream dependencies. Infrastructure team. PA wide modeling and quality review. They're also busy and might not like your launch. At least PM will likely be your side but they may have a different priority than yours. To navigate this organizational complexity, you probably want to have a good manager/tech lead. If you don't care? You're going to piss off them for sure and if the things go very wrong then you could get indivisible attention from the VP level...

And you're now dealing with several hundreds of millions of users so a minimum level of engineering quality should be ensured. You gotta deal with resource planners who also need to allocate finite hardware resources among unlimited demands. The service should have some level of reliability, scalability and redundancy. Thanks to all the works done by core and technical infrastructure team, this is easier than other places but the inherent complexities don't go away. Oh, did I mention that most of the complex infrastructures have integration tests that run over 1~2 hours with a good level of flakiness? If the build dashboard doesn't go green, you might miss your launch by 1 week. It's just a tip of iceberg for productionization, multiply the work by 10x. This is a death by thousand cuts and I don't see a silver bullet to solve everything at once.

jeffbee(10000) about 20 hours ago [-]

Having worked inside and outside Google I just want to say how happy I am that products can't be launched without privacy and security review. I've seen some outright garbage launched at other companies with obvious, gaping holes in their privacy and user-data security stories and it's completely terrifying.

wildrhythms(10000) about 23 hours ago [-]

>Why does this happen? Of course I don't know.

Let me make an observation... eventually every manager gets to a point where the only way to get promoted is to grow their reports. So they beg for additional headcount for their team with little (but important) work, hire a bunch of people who are better suited working in other areas, and repeat for 10+ years until the CEO notices.

xtat(10000) about 12 hours ago [-]

The thought that meeting attendance is the measure of work is so beautifully corpo cringe tech. Honestly FAANG is looking more b-list every day.

xtat(10000) about 12 hours ago [-]

Also how many employees does meta actually need? 1% of what they have currently? (70,000!)

dekhn(10000) 1 day ago [-]

By the end of my employment at Google I was not working very hard. Probably a few hours a day, mostly doing whatever I felt like doing. My managers consistently gave me 'meets expectations' regardless of how much I achieved or how hard I worked. However, any time there was an emergency related to my function, I had everything required to jump in, fix serious problems, and then get out of the way during the cleanup then contributing my bit to the postmortem. I could tell there were very few (fewer all the time) people who truly understand google prod, and in that sense, the company seems to be OK with paying top salaries to people who can prevent the company losing lots of money, or other critical prod issues.

nr2x(10000) 1 day ago [-]

I think it's actually a good thing to just have a pool of people who know how stuff actually works.

Otherwise there could be very key infra that only one or two people fully understand since the code is "mature", doesn't need modifications, and nobody wants to work on it.

In theory of course, I'm sure in reality the digital world isn't at the mercy of <200 SWEs who gave up on promo and live in the basement.

russellbeattie(10000) 1 day ago [-]

> My managers consistently gave me 'meets expectations'

That's because his bonus was probably tied to your performance. By making sure all his subordinates receive meets or exceeds expectations, then he looks good. His manager does the same, all the way up the chain.

They played the same game when I worked at Amazon. What's more, it became automated. They introduced non-optional surveys that popped up on your computer daily. At first I assume it was a well intentioned system to gauge general employee sentiment. It was annoying and stupid HR bullshit, so of course I immediately went in and disabled it. After a year or so, my manager finally notices and orders me to enable it again. I soon guessed why. Within a few months, we start having quarterly group meetings going over graphs of the answers. And of course, the surveys aren't anonymous, so he would call out the people who gave bad answers and start grilling them about their issue in front of everyone, if they didn't immediately recant, then they would 'schedule a meeting'. I assume his performance bonus had become tied to the results and everyone needed to tow the line. It was amusing to me how many of the younger employees didn't understand the game they were playing and would continue to answer honestly. I just glanced at the options, picked whatever made my manager look good and went on with my day.

You'd think those idiots in charge at the upper management levels would have heard of Goodhart's Law: 'When a measure becomes a target, it ceases to be a good measure.' But apparently not.

rrrrrrrrrrrryan(10000) 1 day ago [-]

That's how I think of it. You pay firemen for their ability to solve a problem quickly and efficiently, and for being able to execute when called upon.

Giant companies making money hand over fist pay a lot of 'don't fuck this up' salaries. The primary goal for everyone is to keep the money printer running smoothly; everything else is secondary.

jsemrau(10000) about 23 hours ago [-]

I have never worked for Google and likely never will. Stories like these make me wonder why we put X-Googlers on such a high pedestal. I don't mean that personally in an offence to anyone. It's just a general observation.

aliasxneo(10000) 1 day ago [-]

I have a feeling this extends to several areas in Google. I come from the GDC side of things and have the exact same experience. To keep my job requires very minimal effort on my part. In fact, nowadays I'm punishing myself by trying to do anything 'above and beyond.' This is mostly due to the rapid growth of committees and the struggle for power that has come out of it (i.e., I'm more likely to be denied by a change control board over political reasons).

Regardless, I'm on my way out despite people's shock that I would leave such a 'cushy' job. The fact of the matter is that the lack of challenge has actually caused me to spiral into a deep depression and the best decision for me personally is to move on.

ipsum2(10000) 1 day ago [-]

Were you an SRE? What you described sounds very similar to what I experienced.

chaosbutters314(10000) 1 day ago [-]

chrome is still buggy, the search bar moves my plugins a little after loading and I end up favoriting an empty page by clicking the star on the search bar. I think Google engineers are highly overrated for such a simple problem to still exist

SavageBeast(10000) 1 day ago [-]

I see this as more or less a ruse to justify ridding the companies of all the now remote people who moved away to live in Cheap Town during the pandemic. This is a pretext for the typical Corporate House Cleaning/Reduction In Force scenario. Some people do well working remote (Im one of them in fact) but I suspect and from what I've seen the majority of people simply cant handle the responsibility/self management of working remote.

Alternatively, the economic forecasters at these companies see trouble on the horizon economically and know that layoffs to boost stock price will be necessary. In such case, best develop a pretext for these layoffs thats not 'We're having financial trouble so we're laying people off'. Instead it's 'Nope, nothing to see here, THIS IS FINE - we're just cutting dead weight!'.

I don't work for either of these companies nor do I know anyone personally who does, but I have to wonder if a sort of entitled, country club culture developed there and this an effort to reign in that behavior. Maybe someone with some inside insight can comment here?

codefreeordie(10000) 1 day ago [-]

'Rest and vest' is a phrase that gets bandied about often -- including by people who are trying to do it.

I couldn't tell you what fraction of employees, but there are folks hiding in all of the big tech companies that are happy with their comp, aren't trying to advance, and have adopted the 'do the minimum to not get fired' approach to their work.

If too many of these get together in one org or on one team, the whole thing gets poisoned and everyone starts barely getting anything done.

dam_broke_it(10000) 1 day ago [-]

> Corporate House Cleaning/Reduction In Force scenario

HP did this back in 2013; be in the office or resign.

fred_is_fred(10000) 1 day ago [-]

'I suspect and from what I've seen the majority of people simply cant handle the responsibility/self management of working remote.' While this may be true, if this has to be solved by forcing everyone to move to SF/NY - then couldn't you just save more money by firing their managers?

bluedino(10000) 1 day ago [-]

"Realistically, there are probably a bunch of people at the company who shouldn't be here," Zuckerberg said on the call, according to a Reuters report. "And part of my hope by raising expectations and having more aggressive goals, and just kind of turning up the heat a little bit, is that I think some of you might just say that this place isn't for you. And that self-selection is okay with me."

dividefuel(10000) 1 day ago [-]

For companies that adjust salary for remote workers, those employees who moved to Cheap Town are now cheaper for the company to pay than those who work in Silicon Valley.

lordleft(10000) 1 day ago [-]

I'm a bit worried that this will trigger some kind of move to measure productivity in increasingly crude ways -- i.e. exhaustive, invasive telemetry that tracks every mouse click and keypress.

dymk(10000) 1 day ago [-]

Companies are welcome to do that, and see just how fast their staff quit.

gjvc(10000) 1 day ago [-]

google for 'employee monitoring software' to see how hard this is being pushed

sn41(10000) 1 day ago [-]

Snowcrash vibes. If you read an email too fast or too slow, you could be fired.

falcolas(10000) 1 day ago [-]

Probably. For every hard problem, there's a bad technological solution.

jollyllama(10000) 1 day ago [-]

IMO In-person five days a week is preferable to dystopian tracking automation, not that they're mutually exclusive.

Rayhem(10000) 1 day ago [-]

Technical people often believe using more/new tools will solve people problems. 'If only we could measure more by better decomposing tasks in Jira, then we'd know how to be more efficient! If only we could add micro-specific tags to or documentation, then anyone can search for what they want and find the resource! We just need to put every single process anyone has ever heard of into confluence; then anyone can look them up and follow them!'

Tools don't solve people problems because at the scale of people problems everyone has a different philosophy about the tool (and the problem). Communication is what solves people problems.

dev_0(10000) 1 day ago [-]

Developers should be 'lazy'? Hardworking developers tend to create tedious solution that are not optimized

macawfish(10000) 1 day ago [-]

Seriously, and it can sometimes make for frustrating amounts of tech debt and tangles.

arkitaip(10000) 1 day ago [-]

Either these CEOs are insanely incompetent leaders who have gone on obviously unnecessary hiring sprees as late as a few months ago, or something else is up. Either way, it takes a particularly dishonest leaders to suggest that the main problem is low employee productivity.

leishman(10000) 1 day ago [-]

It's no secret that most people at Google hardly work. It's been like this for years but you're right it's management's fault for allowing this culture.

bell-cot(10000) 1 day ago [-]

Doubtless those honest, selfless CEO's were completely duped by evil conspiracies of rotten & incompetent middle managers, who've spent the last decade or two building ever-larger pyramids of bloat to set their golden thrones on top of...

titzer(10000) 1 day ago [-]

Quoth Zuck:

> "And part of my hope by raising expectations and having more aggressive goals, and just kind of turning up the heat a little bit, is that I think some of you might just say that this place isn't for you. And that self-selection is okay with me."

Wow. Just. Wow.

Why not inject some more dysfunction into an already strained relationship with employees and callously but passively aggressively deal with a seriously broken hiring pipeline in the laziest way possible? If a company can't be bothered to set performance expectations that are measurable and actionable, but just expects to push people out by 'turning up the heat', that's an abject failure of a workplace. There used to be things like quarterly/yearly performance reviews, ratings, even 'performance improvement plans' for under-performing employees--you know, clear expectations, clear communications, criteria and steps and timelines put forward when someone is not meeting expectations.

You know, sometimes life happens to people and they slow down a quarter or two, maybe because of a family crisis, divorce, child, death in the family, traumatic event. Global pandemic? 2 years of isolation WFH? Yeah, there might be reasons...

But, from the top, the message 'these people will find their way to the door if we make work suck enough'--I couldn't imagine anything more demoralizing.

mlword(10000) about 24 hours ago [-]

I don't understand this either. He has to trust entire layers of useless middle management to get accurate performance numbers. All he'll get are invented numbers on a piece of paper (metaphorically speaking).

The ones who leave may be dissatisfied with the artificial goals.

frogpelt(10000) 1 day ago [-]

I don't think you can infer from this article that Meta isn't setting new, measurable and actionable performance expectations internally.

Though you could be inferring that from working there or from all the other news about them.

WatchDog(10000) about 20 hours ago [-]

Setting measurable performance expectations for software roles is notoriously difficult.

Setting quantitative targets often leads to developers optimizing for whatever metric you set, while compromising on the details that aren't quantifiable.

For all of the problems and biases that qualitative performance review has, I think it makes for a more enjoyable and engaging environment.

fiveyearsinarow(10000) about 17 hours ago [-]

There are people who were performing well but had temporary setbacks due to circumstances, and there are people who wanted to coast from day one. It's easy to tell them apart if you have worked with them closely.

cmollis(10000) 1 day ago [-]

'too many employees, but few work'.. this is misleading.. given the spin, you might think this indicates that they hired these people to do specific line-of-business things, and they didn't get done. However, what actually happened was they hired a bunch of people to do.. something.. but they weren't sure what.. all they were told is that they need to hire them.. then they realized they might have 'a down quarter or two'.. apple's killing their advertising business, and they're thinking.. 'hey wait a minute.. our headcount's gone up.. no one in middle-management seems to know what they're doing (which is actually our fault.. but we can't say that), so we'll call the people we hired lazy unmotivated clowns and get rid them that way.' Cue the high-fives.

twblalock(10000) 1 day ago [-]

I've been at places where I would love to hear the CEO say that. Being forced to work with poor performers, lazy people, and people who deliver poor quality results is frustrating and demoralizing.

Those kinds of people can stick around for years, especially in good times when the company is making so much money that leadership doesn't need to care. Netflix is one of the few large companies that has a culture of culling the herd even in good times, and I wish more large companies would take that approach.

antonymy(10000) 1 day ago [-]

Weird how Zuckerberg's red flag for low productivity was employees avoiding meetings to work on personal projects. In the first place, I was under the impression FAANG companies encouraged employees to pursue personal projects for the benefit of the company.

nso95(10000) 1 day ago [-]

Sounded more like errands than personal projects

scarmig(10000) 1 day ago [-]

Oh sweet summer child.

rajeshp1986(10000) 1 day ago [-]

It is amusing how the article says 'employees' but everyone in the comments are talking about developer productivity and laziness. It is shocking how no one points out about the laziness of PMs and management. I have seen PMs and managers taking generous time off and lazying around. Don't forget lot of hiring happens because managers and PMs do planning roadmaps and hire based on that. Some managers also hire more than necessary just cos they want to manage more people. In my previous company our manager hired 3x more engineers than available projects. He jumped ship recently and moved to another FAANG company while the team is now clueless and feeling scared that team might see layoffs.

wildrhythms(10000) about 23 hours ago [-]

My favorite is the PMs who I've never met in my life, apparently they work in my product area, I only hear from them 2x per year only when they're planning some crucial 'team summit' far away from their usual work location (and spouse/children) where they get shitfaced drunk at the company's expense. Nothing actionable ever comes from these 'summits' in case that's not obvious.

jbverschoor(10000) 1 day ago [-]

Maybe zuck expected 10x engineers

sgt(10000) 1 day ago [-]

Instead he added 10 times the amount of 1x engineers.

jstx1(10000) 1 day ago [-]

I don't think that this is about anyone's slacking off. If you're going to reduce staff, you make yourself look better by dressing it up as improved productivity and efficiency, that's why Zuckerberg is making these statements.

galdosdi(10000) 1 day ago [-]

Bingo. Nothing about a 'recession' makes it easier or harder for upper management to manage performance. If they were good at it before they're good at it now and if they were bad at it before, they're not going to suddenly get any better at identifying poor performers. But if layoffs are going to have to happen for unrelated reason, might as well try to spin it so that those that are left behind feel grateful and prideful rather than resentful and worried.

taylodl(10000) 1 day ago [-]

They increased their headcount by 62% during the pandemic and now are like - these people are deadweight and not productive? Really? There are a lot of logistics you have to have in place to hire that many people, especially when you're already a LARGE company, and keep them all working. It seems to me their hiring process is completely broken - hire everybody, see who works out, can the rest. This just confirms the horror stories I hear from people working at FAANGs. It's not anywhere I want to be.

colinmhayes(10000) 1 day ago [-]

I mean it's not unreasonable to think they needed those people then, but now they don't. If you've got 40,000 people worth of work and 70,000 people some of them aren't going to be productive. Just because you had 70,000 people worth of work last year doesn't mean they're still productive this year.

r00fus(10000) 1 day ago [-]

Looks like some cuts are coming to the FAANG workforce - perhaps it's a good time to poach people for your upcoming initiative.

Not necessarily just those who will be laid off, but the ones who don't like their coworkers getting laid off so they can do 1.5x the work for the same money.

Fire up those LinkedIn contacts!

shubb(10000) about 9 hours ago [-]

Do we want FAANG employees? From what I'm reading here great people started these companies and we've had 20 years of sleepy civil service style meeting people. They sound like they might fit at a bank but not at any company that needs to produce?

alexk307(10000) 1 day ago [-]

This is such an embarrassment for these two... Aren't YOU the CEO of your company? Didn't YOU approve hiring plans and corporate goals? If there's not enough work, why don't you find something for them to work on, or replace your managers with folks that will. Instead, it's the employee's fault for being so unproductive.

yibg(10000) 1 day ago [-]

Not to defend either of them as CEOs but isn't that what they're doing? Part of addressing a problem is surfacing it in the first place.

ken47(10000) 1 day ago [-]

Well, we're viewing these messages without context. Absent of context, I would say that Zuckerberg's messaging is too aggressive, and many otherwise decent employees would find it toxic. Now, if there was a proper build-up of messaging to this level of aggression, perhaps that context justifies it to a certain extent. But it looks quite bad in isolation.

Pichai's messaging is more reasonable, even without context. He's just saying the company needs people to work harder. I'm fine with that.

2OEH8eoCRo0(10000) 1 day ago [-]

They lack creativity and vision. The biggest cash cows at these companies are already built.

Ads and app stores

danschuller(10000) 1 day ago [-]

It just seems like setting a frame for upcoming layoffs nothing more or less than that.

StopHammoTime(10000) about 19 hours ago [-]

This is the REAL conversation that needs to be happening.

davidguetta(10000) about 15 hours ago [-]

Well so what ? Hes acknowledging the problem and plan to fix it. Isnt that the kind of responsability taking you are precisely asking for ?

outworlder(10000) 1 day ago [-]

> If there's not enough work, why don't you find something for them to work on

Or why don't they bring back the 20% rule and let the smart folks they have hired to come up with new projects? Some of them may end up bringing revenue.

hgomersall(10000) 1 day ago [-]

I'm convinced that much of this hand wringing is about self justification. I think the role of the CEO is essentially to pretend to be in charge so everyone actually doing the work doesn't lose faith.

raincom(10000) about 23 hours ago [-]

It's like a bull market, where 70% of the value of shitty companies is driven by the bull market and the vertical they are in. Same goes for employment: during good(bullish) times, one doesn't need to deliver much and still can coast.

Historical Discussions: I replaced all our blog thumbnails using DALL·E 2 (August 08, 2022: 652 points)

(659) I replaced all our blog thumbnails using DALL·E 2

659 points 3 days ago by dsmmcken in 10000th position

deephaven.io | Estimated reading time – 11 minutes | comments | anchor

Blog posts with images get 2.3x more engagement. Here's the problem - we make a query engine for streaming tables. How the heck are you supposed to pick images for technical topics like comparing the similarities between Deephaven and Materalize, viewing pandas DataFrames, gRPC tooling for getting data streams to the browser, using Kafka with Parquet for storage, or using Redpanda for streaming analytics?

As a small team of mostly engineers, we don't have the time or budget to commission custom artwork for every one of our blog posts.

Our approach so far has been to spend 10 minutes scrolling through tangentially related but ultimately ill-fitting images from stock photo sites, download something not terrible, slap it in the front matter and hit publish. Can AI generated images from DALL-E make better blog thumbnails, do it cheaper, and generally just be more fun? Yes, quant fans, it can.

I spent the weekend and $45 in OpenAi credits generating new thumbnails that better represent the content of all 100+ posts from our blog. For attribution, I've included the prompt used to create the image as the alt text on all our new thumbnails.

Preview: quick before and after

Here's a page of replaced images: view the full blog for all 100+ images.

My favourite is the image below for a post discussing some of our pre-built Docker containers:

Prompt: 'Blue whale with stacks of shipping containers on its back, cgsociety artstation trending 4k'

Here's 10 things I learned about AI image generation:

Prompt engineering is adjusting your input to the AI model to get the desired output, and it's hard. For technical topics, the first challenge is coming up with a creative idea. My approach was to quickly re-read each of our posts, make some notes on whatever images came into my head while reading, and also look up images and logos related to any of those topics. I tried to think of what comes to mind when reading, then brainstorm a creative take on the content or a metaphor. For example, our recent article is announcing a new Go client library. I came up with the idea of a blue gopher (like the Go mascot) looking at streams of tabular data on multiple computer monitors. Sounds cool, but getting an image that matched what I thought I wanted and getting that to actually appear on screen wasn't easy. It took me 4 tries just to get the gopher to actually be blue rather than the monitors, and another 5 just to get an image I liked. I learned that the more specific you are - to the point of being redundant - the better.

Prompt: 'a cute blue colored gopher with blue fur programming on multiple monitors displaying many spreadsheets, digital art'

Maybe it was because this was my first attempt, but with 100 more posts to go, I hoped I could get better with practice. It would be super cool to just feed DALL-E a whole blog post and have something great pop out, but even with some GPT-3 magic we probably aren't there yet.

When you create an account you get 50 credits. You can buy more credits, and 1 credit is 1 prompt ($0.13 per image), with each prompt giving you 4 images as output to choose from. While generous, in my opinion, this is not enough credits to get good at generating prompts. The first few took 6 or 7 tries to get something acceptable. Now that I've written hundreds of prompts, I can often get what I want in 2 or 3 tries.

First try! Prompt: 'a pipe coming out of the wall in a blue room with bitcoins pouring out of it, digital art 3d render'

A basic prompt without a style modifier usually looks pretty bland. It will either come out a little cartoony, like a bad photo, or like a poor collage. Adding stylistic cues will greatly improve your results. Some quick tips:

'Prompt: cottagecore robot reading a book on a porch'

I added "artstation", "cgsociety", "4k", and "digital art" to a lot of the pieces on this blog. DALL-E also includes helpful tips while you wait the 10 seconds for your output, showing you examples of adding style cues to your prompts.

After playing with it a bit, I realized practice is good, but I needed to get better faster. Studying some images on r/dalle2 gave me some inspiration and ideas on how to craft better prompts. I also found this PDF e-book helpful as well.

5. You may need to photoshop out gibberish text.

Sometimes my prompt resulted in output that included text. Unfortunately, DALL-E really struggles with text, and it's often nonsensical. I thought this would be distracting for blog thumbnails, so I would either photoshop it out, or photoshop in corrected text. Imagen from Google is supposedly better at text, and I look forward to trying that someday. I would appreciate any tips on prompts that hint I don't want any text in the output.

What's up with that text? At least it can be quickly photoshopped out.

6. Watch out for unexpected content violations.

A couple times I was warned about a content violation for my prompt, which gives you a warning and no output. An over-eager list of banned words can give you the occasional false-positive. Once I was using the word 'shooting' to describe a beam of light shooting through the sky. Sounds fine, but I guess they don't like the word shooting in any context. It would be better if the warning explicitly stated what word it didn't like, as sometimes I was left guessing. Another time, I was referring to a blood sugar monitor. I'm guessing DALL-E won't generate anything related to the word 'blood', even if the prompt itself is non-violent.

You might not get what you want all in one prompt, but maybe you can get the pieces you need individually and assemble it in photoshop, or combine multiple images. You can also upload an image back to DALL-E for editing with AI inpainting or crop differently. I intentionally did very little editing for our blog, limiting myself to removing gibberish text. If I was using it for a more serious purpose or to create art, I would have done more assembly of images. Using it as tool in a traditional photoshop-like workflow might be the real long-term value of AI image generation.

If you want twelve turkeys crossing a finish line, you are going to get anywhere between 4 and 20 turkeys. It doesn't matter if you say '12', 'twelve', 'a dozen', or say it multiple times in multiple ways. If you only want 2 or 3 of something that'll work fine, but DALL-E struggles with higher numbers. Maybe it is a bit like a young child, in that it can't count that high? If you just want 'hundreds' of something, it can do that, but the quality isn't great.

That's not twelve turkeys. Prompt: 'Film still, establishing shot of 12 turkeys in marathon crossing a checkered finish line on a street in a race, golden hour, low angle'

Having an AI image generator doesn't instantly make you a better artist, just like having a Canon 6D Mark II doesn't make you a better photographer. Curation and judging what looks good is still important. I am sure that when Photoshop debuted old school graphic artists lamented that it would kill the industry by making things too easy. It didn't. These are just tools that will feed well into any artist's process.

If I was the CEO of Adobe right now, I would be either pushing to train a top-tier competitive AI image generator or bootstrap it with an acquisition like midjourney, and then bet the farm building an editor around it. A future where I can open a canvas of any desired size (and not just 1024x1024) or use an existing photo, then start selecting arbitrary parts of it, and then prompt in what I want and where, would be one heck of a program. Figma was a huge shift and it's eating Adobe's lunch. I could see either an AI-driven image editor crushing Photoshop someday, or becoming it's best feature.

While the role of the artist isn't going away soon, the role of stock image sites might disappear. As someone who previously worked as a graphic designer, and has spent thousands of dollars on stock images, I definitely can see a future where I can ask for an alpha masked blue shark to use as a base for whatever photoshop client project needs it.

The shark on the right took me two seconds to get from DALL-E. We are just a few years (or months?) away from this being a thing.

The largest stock photo company, Getty Images, recently went public (actually they did a SPAC). I wouldn't bet on their long term success. Maybe it will stick around for just historical events of real people?

I think AI image generation is perfect for creating images for slide decks. It's so common to need an image metaphor to accompany a slide, and this is perfect for that. I've spent days building polished decks for presentations at conferences, for CEOs, and sales teams. I see a future where it could be more self-serve. Please bake imagen straight into Google slides. Make it as easy as creating a new slide, clicking on the image placeholder, and typing in a prompt.

I had a blast replacing our 100 or so blog posts with AI-generated images. Was it worth $45? I think so. On average I would say it took a couple of minutes and about 4-5 prompts per blog post to get something I was happy with. We were spending more time and money on stock images a month with a worse result. Not only was this swap fun, but having unique and memorable images will help you, the reader, remember and retain our content better.

I found that once I found ones I liked, I tended to reuse a lot of the same stylistic modifiers along the way. It made me wonder if we should develop a consistent style for our blog, so all our images look like a related set, or have a signature style. But how do you even have a signature style when it is the AI generating your images? How will this change art? Will it make news photos impossible to trust? I don't know the answer to any of those questions.

If you like this post, and want to see what other images we come up with for our blog topics, consider subscribing.

All Comments: [-] | anchor

Beaver117(10000) 3 days ago [-]

Serious question, although everyone seems to avoid it, when will this or a similarly advanced system allow porn? In fact, the porn companies have been awfully quiet for a while. What are they doing? Usually they are the ones on the brink of new technology.

donkarma(10000) 3 days ago [-]

Already out

sbf501(10000) 3 days ago [-]

You should've invested in CloudFront first because your site isn't loading.

lmarcos(10000) 3 days ago [-]

Would nginx (caching everything) work on a $5/month VPS?

dsmmcken(10000) 3 days ago [-]

I know. I wasn't actually expecting to hit front-page. Trying to reinforce it best I can.

mym1990(10000) 3 days ago [-]

Just curious, is there any general number range for how much traffic a front page post might get? Less than 10k, 10-100,000, 100,000+, etc

superchroma(10000) 3 days ago [-]

When asked about what jobs the robots would come for first, I would have had to say that digital artist was pretty low on my ranking before now.

Bjorkbat(10000) 3 days ago [-]

And digital artists aren't still pretty low on your ranking?

I don't know, after all the predictions about self-driving cars, I'm cautious. Especially considering that back then, it almost seemed obvious that we'd have self-driving cars by now. Cars were certainly capable of driving themselves back in 2016, it just seemed like we needed to iron out a few kinks. How long could that possibly take?

Now, I have no idea when it'll happen.

I'm not necessarily saying that it'll take AI forever to do what humans can do. Rather, I think its very hard to make good predictions with all the hype slightly deceptive marketing.

smugma(10000) 3 days ago [-]

Same. I would have thought the arts would be the last to move to AI. What do you put at the end of the list now? It's not trucking. Or ordering. Or anything related to porn. Eliza is 50+ years old, not actually a good replacement for a psychotherapist yet but I would imagine it could go a long way.

I'm biased from the terrible experience I had trying to get my kids to learn online in the pandemic, but I think schoolteacher might be one of the mass professions that is least susceptible to being AI Engineered away.

Ethicist is probably a safe career path too, but there aren't that many of those. And Politicians will of course prevent robots from taking over their jobs.

ravenstine(10000) 3 days ago [-]

It's coming for us, too.

It won't be long before most software engineer positions are eliminated while some are replaced by software 'technicians' with enough expertise to command AI to generate working code. Perhaps the technicians will be tasked with building tests and some automation, but even that stuff can be delegated to AI to an extent.

This may seem far off because the present economy is accustomed to paying engineers large sums of money to write apps. Even with the retractions we've been seeing in hiring and venture capital, there's just enough easy money still there and the capabilities of code-writing AIs isn't quite there yet.

All we need is a significant market correction and the next generation of AI to wipe out a large swath of tech jobs.

The next step regardless is applying technologies like DALL-E to web design, and for said technology to be widely used, open and affordable. We won't need web designers or even UXD.

Then we won't need as many engineers when AI can solve a lot of common problems in building software. AI can do it better because it won't spend inordinate amounts of time dillydallying over next-gen frameworks, toolchains, and preprocessors. AI won't even have to worry about writing 'clean' and maintainable code because those things will no longer matter.

gitgud(10000) 3 days ago [-]

The reason it's focused so much is that art has incredibly low stakes... and people don't want AI making any seemingly important decisions...

Dylan16807(10000) 3 days ago [-]

Maybe, but this is replacing fifty cents of stock photo, not digital artists.

mastazi(10000) 3 days ago [-]

The 'Bonus killer feature request for Google' kinda exists in PowerPoint in Office365, except that it's based on stock pictures, rather than synthetic images. It selects a picture based on the text currently present in the slide. So, unlike the 'feature request', you don't even need to type a description. The feature is called 'Design Ideas'.

rg111(10000) 3 days ago [-]

I used this feature so much when I was in school.

And I was thinking exactly this.

akamaka(10000) 3 days ago [-]

Am I one of the few people who finds these generated pictures really bad? They often have weird and unsettling details when you look closely.

I mean, it's an incredible achievement in AI that we can generate images at this level, but I don't want them shown to me on a daily basis while I'm reading blogs.

lxe(10000) 3 days ago [-]

For maximum coherency, you have to make batches of 50 - 100 and pick the best one. Which can be time consuming and expensive.

ebjaas_2022(10000) 3 days ago [-]

I agree. They're pretty in the same way as fractals are pretty, but still boring and bland.

I would not have any of the ones that I've seen this far on my wall, or as my blog icons.

WheelsAtLarge(10000) 3 days ago [-]

I'm with you. I would hate to see these images all over the place -many are just unpleasant to see.

The cover image generated for the cosmopolitan cover is stunning at first but after seeing it a few times it begins to feel uncomfortable to look at. The uncanny valley is alive and well in many of these images.

l33tman(10000) 3 days ago [-]

I would suggest scanning through the r/dalle2 sub-reddit, as the submissions there are rated. There are limitations in the way the current crop of AI generators work, but in the hands of someone who knows these and know what prompts to specify you get completely amazing results that you as a layman can't tell is AI generated (without an expert investigation maybe into pixel-level artifacts).

amelius(10000) 3 days ago [-]

The pictures are certainly deep down in the uncanny valley, but I think they would be great for nightmarish games. In fact, game developers (and especially game artists) might be the next profession on the line, to be automated by AI.

thrownaway561(10000) 3 days ago [-]

It's not about being perfect, it's about having something that doesn't take time to produce. like the article says, searching google and stock image sites looking for a picture that very few people are going to ingest is a huge waste of time.

sgtFloyd(10000) 3 days ago [-]

I sunk ~20 hours and $100 playing with DALL-E since last week and I've had a very different experience. Sure--my first dozen attempts with the engine gave bad results, but once I learned to 'speak its language' it got easier to generate highly-polished images. The most realistic results come by appending things like 'realistic photograph, 4k, in the style of a fashion magazine' to prompts. I suppose any style would work, as long as the body of source material in that style is (mostly) high-quality.

Here's a couple examples I produced with just a little trial and error. FWIW I have an engineering background and zero design experience.

'Frida Kahlo crossed with Julia Child, 4k realistic, expressive photo, hdr' https://labs.openai.com/s/hvFClrAMCXN6zwqJUJwsmYSB

'John Lennon crossed with Paul McCartney, 4k photograph' https://labs.openai.com/s/lb7qw07tdvRPZ9nmkrCmU0RA

Maybe they're not perfect, but I'm impressed as hell. Exploring what's possible by wording prompts differently feels very much like using a search engine for the first time. Give it a year. This technology is going places.

viburnum(10000) 3 days ago [-]

Yeah, maybe it's okay for a tech company to be weirdly robotic, but I'd be happier without them random illustrations.

Kranar(10000) 3 days ago [-]

I find the images to be incredible, but it's very unsettling when you focus on certain details like hands, feet, and eyes. The hands and feet that it draws are almost always mangled, and while it does a good job of drawing an individual eye, it doesn't seem to draw two eyes in a well coordinated manner, either one eye is bigger than the other, or there is something weirdly unsymmetrical about the eyes that makes the image look creepy.

Alex3917(10000) 3 days ago [-]

> Am I one of the few people who finds these generated pictures really bad?

Bad compared with what? They certainly convey a lot more information than a randomly generated gravatar.

mrtksn(10000) 3 days ago [-]

The results tend to be residents in the uncanny valley. They are nice if you want something unsettling. They are very impressive, can be very aesthetically pleasing(especially with midjourney) but they look very alien.

Maybe part of the reason we are so impressed with those is because they break our perception of reality. It looks like the renaissance statues that are made from marble but looks like cloth.

ryanSrich(10000) 3 days ago [-]

> Am I one of the few people who finds these generated pictures really bad?

Well they're bad at not looking like AI generated art. It's impressive, but I've yet to come across an example that doesn't look like AI generated art. A few seconds of surface level inspection and you can see the weird AI psychodelic circling effect (no idea what the technical name is - eye-ball-ification?)

upupandup(10000) 3 days ago [-]

They are good enough for most people and over time those details will get better until we have no need for illustrators.

Already I see website agencies and bloggers using DALL-E. What I do see is that it is easy to pick out DALL-E generated images, in that its too fantastic. Way over the top to a fault.

ajqreh(10000) 3 days ago [-]

The article isn't loading for me, so I can't really comment on the images it contains, but I've found telling the ai to apply an impressionistic filter does wonders for removing the unsettling aspect. Obviously that limits you to a specific style of image, but I imagine there are other stylistic filters you might apply that achieve the same goal.

I could spend all day looking at the output of 'impressionist cats' and similar queries.

yieldcrv(10000) 3 days ago [-]

I had MUCH lower expectations about this article's images, once I got it to load I was surprised, no, amazed!

boredemployee(10000) 3 days ago [-]

I tried it a lot and I think it works ok for simple, mundane, trivial prompts, but when you start to ask for sophisticated stuff it gets weird.

hackernewds(10000) 2 days ago [-]

You might be looking at the limited launched product. The fully powered product has amazing results


throw_m239339(10000) 3 days ago [-]

These are good enough for 99% of blogs out there.

Just like AI generated articles are good enough for 99% of content farms out there.

jedberg(10000) 3 days ago [-]

> Maybe it was because this was my first attempt, but with 100 more posts to go, I hoped I could get better with practice. It would be super cool to just feed DALL-E a whole blog post and have something great pop out, but even with some GPT-3 magic we probably aren't there yet.

I see a business opportunity here. Feed text into GPT-3 and have it generate DALL-E prompts to make appropriate images.

Then you have it do the same thing for a children's book.

danielbln(10000) 3 days ago [-]

I've done something similar a while ago, turned out great (according to my toddler): https://www.reddit.com/r/dalle2/comments/ueizwz/i_printed_a_...

bitL(10000) 3 days ago [-]

Did anybody try to generate desktop icon sets for Linux?

frozencell(10000) 2 days ago [-]

I haven't seen anybody generating "useful creations" except cover albums and thumbnails, yet.

glofish(10000) 3 days ago [-]

Visit their main blog page to see a subset of the resulting images all at once




Guest9081239812(10000) 3 days ago [-]

I find the images fairly distracting and they take all the focus on the above page. The artwork also isn't very consistent which makes it feel like a jumbled mess. When you click a post the image takes the entire screen and pushes all the content below the fold. It's more like browsing a community art portfolio instead of a tech blog.

I'd prefer smaller thumbnails or icons that give more context to the actual post. This way they could add some benefit, such as helping to visually categorize the content. As of now, they're just a bunch of random illustrations taking up valuable screen real estate.

That being said, thanks for sharing, it's interesting to see an example of someone integrating DALL·E 2 into their workflow.

system2(10000) 3 days ago [-]

Looks like a hipster LA artist drew them all.

bradgranath(10000) 3 days ago [-]

Gety will become a trainingset vendor.

imhoguy(10000) 3 days ago [-]

Exactly, 'We don't have exctly what you ask for but maybe you would like these GettyAI(tm) generated images:'

Halan(10000) 3 days ago [-]

Suggestion for the next blog entry: How my blog post going viral on Hacker News pushed me to finally use a CDN

l00sed(10000) 3 days ago [-]

I wrote that one, but haven't gotten enough traffic again to see if it works... https://l-o-o-s-e-d.net/hug-of-death

motoboi(10000) 3 days ago [-]

How I generated static html for my static site and it got served and cached.

amsterdorn(10000) 3 days ago [-]

this. I couldn't even load the page.

egypturnash(10000) 3 days ago [-]

God it's so fucking depressing to see all you techies debating whether or not the skill I have dedicated my life to getting good enough at to earn a quiet, modest living with should be automated away or not. And insisting that surely your jobs are too special and complex for this to ever happen to you.

At least I can take solace in the fact that for now these things aren't gonna be taking a bite out of the furry art commissions I like to take, since that's way too associated with crazy cartoon porn for them to not censor relevant keywords.

dmitriid(10000) 3 days ago [-]

News at 11: jobs are taken away and replaced by advances in technology.

You are not special. I am not special.

boredtofears(10000) 3 days ago [-]

Wait until the world becomes oversaturated with AI generated imagery and then make a killing when people realize that real artists can actually produce something original.

ramblerman(10000) 3 days ago [-]

Thats a bit unfair. The discussion is not about what should happen. But a prediction of what will happen, given these technologies exist. This is a tech forum after all, and speculation about tech and society is a big part of that.

Personally my societal concern is yet another industry where we had multiple small jobs will be ruled by a few conglomerates.

Small business are like democracy in a free market. And we keep evolving to it all ending in the hands of a few.

alickz(10000) 2 days ago [-]

i feel for ya mate, i can see it coming for me too

zackmorris(10000) 3 days ago [-]

Trust me, we (the techies) are in strong denial about how much time we have left. I've been programming since the late 1980s and have learned everything from VHDL to Clojure and give us till no later than 2040. Realistically, more like 2030, due to the billions of dollars being thrown at AGI for finance and other monetizations. Of course, money won't be worth anything after that, but I doubt that will stop anyone.

In my own life, I've decided to transition away from shared truth towards manifesting the reality that I want to live in. I try to help people now, I meditate a lot about humans becoming aliens, I try to be in the moment whenever possible and be thankful for consciousness. But I no longer put my energy into the ego-based materialism that captured tech. Since wealth inequality can't be stopped, I feel that the only salvation lies in nonattachment.

Traubenfuchs(10000) 2 days ago [-]

> furry art

I can't wait for a dalle2 level model trained on all of e621, rule34, x-hentai, furaffinity, sofurry, inkbunny and u18chan. It's gonna be great.

julianlam(10000) 3 days ago [-]

Don't worry, while the rest of the industry is busy making artsy abstract images of chimpanzees surfing on a wave of coronavirii, you can quietly work on UX design I guess.

roughly(10000) 3 days ago [-]

Not to mention that the whole goddamn trick here is basically taking the output you and your peers have created, anonymizing it, throwing it in a blender, and then acting like the AI has generated something new and humans aren't needed anymore.

hackerlight(10000) 3 days ago [-]

Of course it should be. You aren't entitled to my money for your services if a computer can do a good enough job for free. And human history is full of examples of innovations displacing humans and temporarily unemploying them until they found other jobs. That's basically what 'productivity' is about in economics, and overall it's a good thing we're not still in the stone age in the name of protecting people's jobs.

nluken(10000) 3 days ago [-]

> insisting that surely your jobs are too special and complex for this to ever happen to you

This insistence bugs the crap out of me, and shows just how arrogant tech-types (and even more so, business-types) can be. The late David Graeber noted this attitude, and while he's talking about investors/entrepreneurs/financiers, it could equally apply to the cocky software engineers I've met over the years:

'It's possible for futurologists to imagine robots relpacing sports editors, sociologists, or real estate agents, for example, yet I have yet to see one suggest that the basic functions that capitalists are supposed to perform, which mainly consist of figuring out the optimal way to invest resources in order to answer current or potential consumer demand, could possibly be performed by a machine. Why not?'

I optimistically believe that art will never be automated away like software engineering has the potential to be. The human creative element is so core to what art is that replacing it with a machine misses the point entirely, even if a machine could fully replicate an aesthetic.

rg111(10000) 3 days ago [-]

As companies in spotlight make it impossible to generate gore or porn, will more commission artists' specialization be in that area?

There are already enough offbeat companies dedicated in generating porn, I am sure.

croes(10000) 3 days ago [-]

I find it quite ironic that a AI company like OpenAI seems to use word lists to prevent inappropriate content creation.

frozencell(10000) 2 days ago [-]

Private word lists right.

fiat_fandango(10000) 2 days ago [-]

I assume those who currently have access to DALL.E 2 are legally prevented from selling second-hand use of their access? I'm currently on the waitlist and I've been curious if there's any way to bill access like this as 'design consulting' or something?

Posts like this are absolutely fascinating!

stainablesteel(10000) 2 days ago [-]

is there a way to know for sure if something was generated by it?

hamasho(10000) 3 days ago [-]

First time to hear 'Prompt engineering' but I feel familiar with this term. It's my daily job to adjust google search queries to get the information I need, even though sometimes I don't know what I need.First time to hear 'Prompt engineering' but I feel really familiar with this term. It's my daily job to engineer google search queries to get the information I nee.

Valakas_(10000) 3 days ago [-]

Prompt 'engineering' will last a couple of years, until the UI improves to the point any dummy can do it, just like with instagram filters.

brundolf(10000) 3 days ago [-]

Honestly, this is an awesome use of DALL-E and I'll probably start doing the same for my blog

It's perfect because:

- The images just need to get across a vibe, they don't need to be perfect

- It's a low-value enough use of images that you'd probably never commission a human artist to do them; instead you'd either use stock photos, or skip having images completely

- The nature of header images for a tech blog tends toward the abstract/surreal, which means it's either hard to find the right stock images, or the ones you do find will be super abstract to the point of being boring

All of these make it a great use of the technology

april_22(10000) 2 days ago [-]

Agree, especially since it could bring some image variety into blogging. Sometimes I keep seeing the same, overused image again and again. Now you can create super cool looking, unique images and tailor them to how you'd like them to look like.

throwaway0x7E6(10000) 3 days ago [-]

>The images just need to get across a vibe

I can't speak for everyone, but with my own experience of reading (at least partially) a dozen or two technical articles almost every day for many years, pointless media is a hallmark of low quality. these days, I just immediately bail with Ctrl+W as soon as I encounter a twitter-pop-culture meme/gif in the header or anywhere near the top. sure, it does mean I skip the 5 out of 100 that were worth reading, I save a lot of time by skipping the 95 out of 100 that weren't.

antioppressor(10000) 2 days ago [-]

Oh jesus. 99,5 percent of sites use terrible, bland, uninteresting images. Like people have no sense of beauty at all. Now people will use these horrible abominations that DALLE shits out, well, a very nice and bright future is ahead of us :D.

Morgawr(10000) 3 days ago [-]

I've recently started dabbling in short story narrative writing as a hobby and I found a super interesting usage of dall-e is to generate certain art works or art style to draw inspiration from.

For example I came up with [0] after writing a draft about an old warrior/mercenary in a fantasy-like setting and then put something into dall-e and built upon that just to get the right 'vibe'. Or if you're into more 'cosmic horror' kind of stuff I generated artworks like these [1] which gave me a lot of inspiration for future short stories I'm planning to draft.

I only spent about $15 so far, and a lot of it was just experimenting with artstyle (mostly to get some interesting discord profile pictures and logos) but I feel like I learned a lot. I can't stress how ridiculously cheap it is for the amount of quality artwork you get out of it.

[0] https://twitter.com/xMorgawr/status/1555728353780310017

[1] https://twitter.com/xMorgawr/status/1556667345443049473

powersnail(10000) 3 days ago [-]

> The images just need to get across a vibe, they don't need to be perfect

I dislike super generic stock photo at the beginning of an article. It's completely pointless, sometimes aesthetically unpleasant, often disconnected with the actual content, and hence a distraction.

If neither you nor the reader cares about the stock photo, why not just forsake the thumbnail or use your website's logo?

dreadlordbone(10000) 3 days ago [-]

Archived link: https://archive.ph/uNfeK

robocat(10000) 3 days ago [-]

That is missing the images.

thenerdhead(10000) 3 days ago [-]

What service did they use for $45? Is this legal to include in published works too like a book?

jdminhbg(10000) 3 days ago [-]

From the Dall-E terms of use:

> Use of Images. Subject to your compliance with these terms and our Content Policy, you may use Generations for any legal purpose, including for commercial use. This means you may sell your rights to the Generations you create, incorporate them into works such as books, websites, and presentations, and otherwise commercialize them.

wardour(10000) 2 days ago [-]

What about royalties to the designers / artists that created the art being used to train these systems?

personalidea(10000) 2 days ago [-]

Every time an artist looks at a painting, gets inspiration from that experience and creates an image themselves, they need to pay royalties?

WheelsAtLarge(10000) 3 days ago [-]

>'While the role of the artist isn't going away soon, the role of stock image sites might disappear. '

Not, yet. While it's cheap relative to stock images, it's time consuming to generate exactly what you want. Prices for stock images will collapse for the common quick to use images but the price for the specialized high end images will hold their value or even increase in value. Those historical and such images will continue to be valuable.

It will be interesting to see if a specialized job will rise where people will get paid to generate just the right image. It might be called 'A.I. image artist ' This individual will generate an image with an A.I. but use graphic tools to finalize it for use.

saghm(10000) 2 days ago [-]

OTOH, sometimes stock pictures made by humans aren't even that relevant if the entity using them is mostly just looking for filler: https://www.reddit.com/r/weirdwikihow/

pigtailgirl(10000) 3 days ago [-]

-- I agree with you - however it's not that time consuming to get what you want - it's pretty easy once you get used to DALL-E - so far there isn't anything I've not been able to get on a couple tries (granted after spending ~$30 learning the prompt system)- however once you're used to it - it's fairly easy - I agree that the market for very custom work will go through the roof - but 'I need a burger' or 'I need an American looking hot dog' eeeek!!! =) --



flir(10000) 3 days ago [-]

Vocabularies are starting to appear. Something like a pattern language.

mirekrusin(10000) 3 days ago [-]


nutanc(10000) 3 days ago [-]

You are right. Humans will move on to building more high quality images. But for regular run of the mill stock images, AI is already there. I had done a small experiment to create stock images[1] [1] https://medium.com/ozonetel-ai/generating-a-landing-page-wit...

andreyk(10000) 3 days ago [-]

Agreed - having played around with DALL-E 2 a fair bit and having made a lot of usage of stock images over the years (for blog posts with specific subjects), I would say the former takes more work/time than the latter. With stock images I can just do a quick search on Shutterstock and find a lot of high quality options (usually), whereas with DALL-E 2 I need to figure out the exact prompt I want and iterate on it for a while. Stock images are not that expensive -- if you buy many it's as low as $2 per image, or on the high end (if you pay to just download a few per month) it's more like $10 per image. It does cost more, but time is money, so...

deltree7(10000) 3 days ago [-]

Also there is nothing preventing Stock Image Sites themselves using Dall-E to generate additional images. Heck they can use their own existing images for training (which the other's can't due to copyright issues) to increase the portfolio, but the Stock Image Sites can access free public images.

So, counter-intuitively it may strengthen Stock Image Sites value

aantix(10000) 3 days ago [-]

Here's a vast catalog of Dall-E images and the prompts used to generate them. https://www.krea.ai/

Oh, and if you generate an image with Dall-E and there's a face that is distorted, you can use this tool to restore the facial features. https://arc.tencent.com/en/ai-demos/faceRestoration

rkuykendall-com(10000) 2 days ago [-]

The future of jobs is Celery Man


zionic(10000) 3 days ago [-]

>Prices for stock horses will collapse for the common quick to use horses but the price for the specialized high end horses will hold their value or even increase in value.

This was all true when cars became a thing. What's the market cap of horse production companies before and after?

jstummbillig(10000) 3 days ago [-]

> While it's cheap relative to stock images, it's time consuming to generate exactly what you want

My dall-e experience is very limited but looking for the right photo out of many is a very time consuming process, at least at designer level.

hackernewds(10000) 2 days ago [-]

The #jobs decreasing is an existential threat to any sector

texaslonghorn5(10000) 3 days ago [-]

Would that be similar to a 'prompt engineer' role?

Sebb767(10000) 3 days ago [-]

> It will be interesting to see if a specialized job will rise where people will get paid to generate just the right image.

I don't think so. People just need the result, so the AI will simply become a tool of the trade and you won't have any more AI image engineers than you have dedicated Photoshop artists right now.

bayesian_horse(10000) 3 days ago [-]

I signed up months ago for the beta and I'm still not in.

madduci(10000) 3 days ago [-]

Me too, that's awkward

sfink(10000) 3 days ago [-]

Ugh, this immediately drives home the realization that representative images are soon going to be devalued and useless, to the point that we'll all be ignoring them soon. Possibly even stripping them with ad blockers or similar tools.

I actually think it's really awesome to be able to do this with a series of blog posts, and even if you look past the stylistic inconsistencies and oddities, this particular usage is good and adds value.

Which is kind of the problem. Relatively low cost, currently high benefit? It's going to be driven into the ground.

We've seen this over and over again. Some reliable form of signal, or of value, becomes inexpensive enough to produce that it gets commoditized, monetized, and weaponized against us all.

Email is a major productivity advance that gives a low-friction way of communicating for mutual gain? Well, now we're drowning in spam and phishing attempts and people won't read random unsolicited messages—if they even make it pass the automated filters. Same for text messages. Bold images and lettering used to be good for highlighting and accentuating important information. Now we don't see them, even if they make it past our ad blockers, because the neural networks living in our skulls know to filter them out as negative-value advertisements.

The same thing will happen here. Nearly all blogs will soon be sprouting cutesy images to go along with the posts. Initially, many of them will be useful and add value, suggesting a metaphor or analogy or simply providing a visual anchor to make the content more memorable.

But they'll quickly become expected and necessary and we'll have the usual race to the bottom. Everyone will have some image because it boosts engagement by 8%... wait now 6%... oops it's too common, we're awash in crappy irrelevant images just added for the boost, which is down to 2%... oh crap, now the absence of an image is a good signal for content quality, we're at -1%!

(If you put work into the prompt and curate carefully, it will still be a net positive to your content. But it won't matter for long in terms of traffic/engagement, because everyone will be mentally ignoring it.)

iroh2727(10000) 3 days ago [-]

Yep. My analogy for this kind of effect is McDonald's (and other fast or prepackaged food). Fast food never eclipsed good food because, well, it's not that good. But it spreads. It's just the