Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

August 07, 2020 17:10

Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: Germany plans to dim lights at night to save insects (August 06, 2020: 1103 points)

(1102) Germany plans to dim lights at night to save insects

1102 points 1 day ago by Shivetya in 10000th position

www.msn.com | Estimated reading time – 3 minutes | comments | anchor

Germany is planning to ban floodlights from dusk for much of the year as part of its bid to fight a dramatic decline in insect populations, it emerged Wednesday.

© Yuri KADOBNOV Bavarians were unexpectedly enthusiastic about saving bees in a 2019 petition.

In a draft law seen by AFP, the country's environment ministry has drawn up a number of new measures to protect insects, ranging from partially outlawing spotlights to increased protection of natural habitats.

'Insects play an important role in the ecosystem...but in Germany, their numbers and their diversity has severely declined in recent years,' reads the draft law, for which the ministry hopes to get cabinet approval by October.

© David GANNON Sundown could mean bright lights must go out in future for German cities like capital Berlin.

The changes put forward in the law include stricter controls on both lighting and the use of insecticides.

Light traps for insects are to be banned outdoors, while searchlights and sky spotlights would be outlawed from dusk to dawn for ten months of the year.

The draft also demands that any new streetlights and other outdoor lights be installed in such a way as to minimise the effect on plants, insects and other animals.

The use of weed-killers and insecticides would also be banned in national parks and within five to ten metres of major bodies of water, while orchards and dry-stone walls are to be protected as natural habitats for insects.

The proposed reforms are part of the German government's more general 'insect protection action plan', which was announced last September under growing pressure from environmental and conservation activists.

- Weedkiller -

Attention will now turn to the agriculture ministry, which is under pressure to deliver on promises such as an overall reduction in the use of pesticides.

Most notably, Germany said last September that it would phase out the controversial weed killer glyphosate as part of the insect action plan.

On Wednesday, environmentalists welcomed the draft law but urged further action from the agriculture ministry.

'We will not stop insect decline with tinkering alone,' said Rolf Sommer, a director at the German chapter of the World Wildlife Fund (WWF).

The environment ministry's proposals were 'a starting block for more insect protection', but more reforms were needed to pesticide regulations he added.

The German Nature Conservation Association (DNR) meanwhile called on agriculture minister Julia Kloeckner to 'do her homework' and deliver on the promise to phase out glyphosate by 2023.

In the past year, Germany has repeatedly made headlines with its efforts to protect insects.

Last April, the state government of Bavaria was caught off guard by a wildly popular petition calling for greater protection of bees.

Rather than putting the petition to a referendum, the state simply passed it straight into law after 1.75 million people signed it in a matter of months.

Earlier this year, electric car giant Tesla faced delays to the construction of its new 'gigafactory' outside Berlin due to the relocation of several ant colonies away from the building site.


Continue Reading Show full articles without 'Continue Reading' button for 24 hours.

All Comments: [-] | anchor

Solstinox(10000) 1 day ago [-]

...and get better sleep.

WarOnPrivacy(10000) 1 day ago [-]

Amen. I would kill for this. I keep my own outside lights off, so there's a bit less blindy light pollution on my end of the street. It's the same reason I never report the nearby street lights when they go out.

The best outdoor light is less.

k__(10000) 1 day ago [-]


I had a big orange street lamp hanging right in front of my bedroom for years.

shekharshan(10000) 1 day ago [-]

'Every mammal on this planet instinctively develops a natural equilibrium with the surrounding environment; but you humans do not. Instead you multiply, and multiply, until every resource is consumed. The only way for you to survive is to spread to another area. There is another organism on this planet that follows the same pattern... a virus' -- Agent Smith (The Matrix)

kharak(10000) about 22 hours ago [-]

This so called equilibrium is created in nature by starvation or death by predators. Let's all go back to starvation and killing ourselves, for equilibrium.

_Microft(10000) about 22 hours ago [-]

Sure, if there is one thing history has taught us it's that humans will not be contained. Humans break free, expand to new territories and crash through barriers, painfully, maybe even dangerously, but, uh... well, there it is.

Humans will find a way.

quattrofan(10000) about 20 hours ago [-]

Germany saves the insects, China empties the oceans.

dang(10000) about 19 hours ago [-]

Please don't take HN threads further into nationalistic flamewar, regardless of which nations are at issue.


blakesterz(10000) 1 day ago [-]

I always just assumed that there are few insects now because there are so many more insecticides in use, this is the first time I've heard of lights. There's not much explaination in this article, but here's one with some more:

Nighttime Light Pollution May Be Cause of Insect Population Decline


'An analysis of the effects of artificial light at night on insects shows that there is strong evidence to suggest a link between nighttime light pollution and declines in insect populations.'

hinkley(10000) about 22 hours ago [-]

It's not just kind but application.

It's heartbreaking and happens at least once a year, that someone near conventional agriculture/horticulture is posting a desperate for advice in every forum they know about because some asshole upwind let a cloud of 'cide drift over their horticultural masterclass of a lot, and everything is shriveling.

Recompense can be hard to get. Maybe we need an ACLU meets EFF for biodiversity.

Let's ignore for a moment any agenda to ban monocultures, and look at not just the kind but the density. Pollinator species would be doing better if the monocultures weren't so seemless. We need a mosaic of habitat and to reduce the distances between the ones we have. But it's an uphill battle to maintain them in the face of drift and runoff (really I think we shouldn't be cultivating hilltops, that would be a start, and simply dodges the runoff problem)

TheAdamAndChe(10000) 1 day ago [-]

It's a combination of factors. Insecticides, pollution, loss of habitat, light pollution(especially from daylight LEDs), global warming(throwing off reproduction timing), ubiquitous roadways and vehicles, flat surfaces(beetles get stuck on their back).

ChuckMcM(10000) 1 day ago [-]

Wow. I have to say that I have never considered the case where insects needed to be saved. It seems to me that as lifeforms go, they have been the most durable form of multi-cellular life on the planet.

That said, that there are negative systemic effects on the planet's ecosystems that are directly related to the massive production and use of herbicides and pesticides is not as surprising.

Light though? Even after reading that article and looking through scholar.google to find papers that associate lighting with insect decline, I'm not convinced the use of lighting at night has a material impact. The ratio of unlit/lit habitat seems wildly in favor of unlit.

And yes, I understand that unlit areas are often agricultural, and those areas contain the most insecticide/herbicides. It still appears to me that the root cause is the use of those products in farms and not that cities are poor replacement habitats.

fwsgonzo(10000) 1 day ago [-]

It very well could be less of a problem than roundup, but at the same time it's good for everyone I think, that we turn off the lights at night. If only to see the stars.

pgeorgi(10000) about 10 hours ago [-]

> The ratio of unlit/lit habitat seems wildly in favor of unlit.

Depends on where. Germany has a very different structure due to its population density than most parts of the US, for example, with comparably less unlit habitat.

So that may be a less important issue in some places but having enough suitable space for themselves in the US doesn't help insects in Germany.

boudin(10000) 1 day ago [-]

Sound is also actually quite problematic. Flowers can use vibrations to detect the presence of a nearby pollinator insect and makes itself more appealing. https://www.smithsonianmag.com/smart-news/flowers-sweeten-wh...

The sound pollution might have the adverse effect of causing some plants to lose this ability to preserve energy for reproduction, indirectly impacting insects via the food chain.

Currently, due to the massive use of chemicals in the countryside, cities start to be havens for insects. There is a lot of beehives in London for example (too much actually according to a beekeeper I talked with). Until we manage to massively shift the way we do agriculture it might make sense to make it as safe as possible for insects in urban areas.

ChuckMcM(10000) about 14 hours ago [-]

Just to be clear, I'm a fan of cutting down light pollution for legitimate reasons (energy conservation, astronomical observation, and wildlife protection) but I twitch at reasons that are either exaggerated or misleading. I felt that throwing in lights along with the pesticide limitations was not a well motivated argument.

My experience has shown that when people are skeptical of your argument they are less likely to comply voluntarily and so the good that you hoped to do is lost and you have damaged your credibility with the people you need to support these ideas.

A small example of this in action was back in 2010 when Google decided they were going to stop stocking bottled water in the fridges around campus because the 'energy waste and plastic disposal problem.' The only problem was that Google regularly achieved over 95% capture of recyclable trash out of their bins and the energy saved was on the order of maybe half a dozen employees not commuting from San Francisco.

The real reason they wanted to do this was of course cost. They could replace bottled water with filtered dispensers and that would save them a few pennies per employee per month. I had a good conversation with some folks in charge that they could have said, 'We're giving every employee an insulated water bottle in your choice of size and color (12, 20, 32, or 40 oz) that you can keep at the office and use to get water from our fancy new dispensers. This will save us money, AND its better for the environment.' That would have been honest and gotten a good reception I believe with the rank and file. But dressing up the reason in a cover story that everyone easily poking holes in simply lowered morale and confidence in management.

tksb(10000) about 15 hours ago [-]

Even if the light doesn't have a quantifiable effect on insects, would it not be wise to reconsider the extended use in absolute 'off-hours'?

pvaldes(10000) 1 day ago [-]

Ironically this would create a better surveillance net. With a high light in the background you obtain black silhouettes. With low or none light in the background UV cameras are perfectly able to take photos in pitch black, and a low level of light gave you faces of the suspects instead.

So perhaps, maybe, 'speculation in action', this is not only useful or exclusively about insects

allendoerfer(10000) 1 day ago [-]

No, this is actually about insects.

rspoerri(10000) 1 day ago [-]

Do you remember the time when you had to clean your windshield after driving into holidays?

oever(10000) 1 day ago [-]

Insects are spread out over more cars now.

efa(10000) about 23 hours ago [-]

Yes, last week. Driving from Texas to Wyoming and back. Tons of splattered insects.

foepys(10000) about 24 hours ago [-]

Cars are actually more aerodynamically shaped nowadays. But riding a motorcycle will give you the same result today as 20 years ago. To which I must say: yes, the insects seem to be back in larger numbers this year. At least in Germany.

But not everywhere: the less farm land, the more insects. Like, from literally none on farm land to 'I have to completely clean my helmet because of all that yellow goo' in larger forests.

elwell(10000) about 23 hours ago [-]

> The windshield phenomenon is the observation that recently fewer dead insects accumulate on the windshields of people's cars. It has been attributed to a global decline in insect populations caused by human activity.

> The research also found that modern cars, with a more aerodynamic body shape, killed more insects than boxier vintage cars up to 70 years old.


Dahoon(10000) 1 day ago [-]

Yes it happened two days ago.

42droids(10000) 1 day ago [-]

Yap, this is the saddest thing. After a month long road trip around Europe the windshield & front air vent were almost clean.

theobeers(10000) 1 day ago [-]

I just finished a two-week road trip from Berlin to the Black Forest and back (with other stops in both directions). The windshield was gunked up by the end—more than I expected, but nothing like what I remember from my childhood.

OkGoDoIt(10000) about 23 hours ago [-]

I did a one week roadtrip from California to Colorado this summer and had to clean my windshield at every gas station due to splattered bugs. It's my first real roadtrip so I don't have any basis for comparison.

jjcon(10000) 1 day ago [-]

Quite a bit of this is owed to traffic patterns and windshield glass changes though

moneytide1(10000) 1 day ago [-]

Light pollution seems to be a good thing to suppress in general (a bonus for astrological sight-seeing). Could one factor be that constant light negatively affects insect (and plant) circadian rhythms?

veddox(10000) 1 day ago [-]

I think you meant 'astronomical' sight-seeing? ;-)

Yes, constant light does have a strong effect on animal circadian rhythms, but for insects, the problem is rather that they exhaust themselves from flying around the light source too long.

thatguy0900(10000) 1 day ago [-]

I actually was sitting outside a few nights ago and realized that there was no more fireflies, and I couldn't remember the last time I had seen one. I used to go out all the time during summer as a kid to catch them, they would light up whole fields. That was just like 15 years ago. Pretty sobering realization

linuxftw(10000) 1 day ago [-]

Used to be 100's or 1000's of them during the peak in summer where I lived. Now it's a handful. This is out in a rural area, so urban lighting doesn't explain the decline.

If I had to wager money, I'd say it was a blight introduced by another insect. Around the same year I saw an invasion of Asian lady bugs, the firefly memories stop. There were literally millions and millions of those Asian lady bugs released or hatched one year.

sethammons(10000) 1 day ago [-]

Fireflies get killed by insecticides used for mosquitoes

fasteddie31003(10000) about 23 hours ago [-]

I was driving though Nebraska a few months ago and there were so many fireflies it looked like a scene out of Avatar.

echelon(10000) 1 day ago [-]

It's also past peak firefly season, which is mid May to early June in North America.

Georgia has lots of them. Or at least we did last year. I haven't gotten out much in the COVID-19 world.

A year without fireflies.

toss1(10000) about 16 hours ago [-]

Yup, North of Boston here, fairly woodsy area, too many years seeing too few firefiles, although this summer there were a few more than past years - I hope it's a growth trend!

abstractbarista(10000) 1 day ago [-]

I saw more than I've seen in many years the other night. Seems to just depend on the area.

wenc(10000) 1 day ago [-]

I'm in Chicago (in a neighborhood close to downtown) and in previous years I'd hardly ever seen any fireflies. This year, they're everywhere -- in parks and even in trees on neighborhood streets.

I understand fireflies are sensitive to light pollution. I can't imagine Chicago's any different from other major cities. COVID has people staying home a lot more but that has nothing to do with street lighting. I wonder what's going on.

I do think that living organisms are extraordinarily adaptable if the changes are gradual and bounded. Most of us have this idea that an ecosystem is this delicate and fragile thing, but nature actually has a lot of robustness and redundancies built-in (which are inefficient but increase chances of group survival).

gre(10000) 1 day ago [-]

For me it's red wasps. Growing up in Texas there used to be so many, you could see them flying low searching for spiders in the grass. Last Sunday, in the heart of summer, I couldn't find a single red wasp where there used to be hundreds.

darkstar_16(10000) about 6 hours ago [-]

So true. We used to have fireflies everywhere when we were growing up. My daughter hasn't seen a single one in her 9 years in this beautiful world.

SuoDuanDao(10000) about 2 hours ago [-]

I'm probably a bit North of you (Southern Canada) and this is the first year I've seen lots of fireflies at night. So there's a good chance it's more a case of migratory patterns changing than a mass die-off. Don't mean to make light of the matter, but it might not be as grim as it appears right now either.

jungletime(10000) about 13 hours ago [-]

Lots of fireflies in my backyard every night, probably because it is partially flooded due to the record high water levels in the Great lakes. Have been trying to film them. But subpar results so far. I have a vintage 70-200 lens adopted to m4/3. It says macro on the lens, but I can't focus on anything close at all.

mattoxic(10000) about 18 hours ago [-]

It hit me a couple of years back, a night time drive through the Australian countryside and your car would be plastered with insects - not anymore. And fireflies, I haven't seen one in years,

gcheong(10000) about 17 hours ago [-]

I used to work summers out of college in an AK cannery. Some years there were barely any fish and the season would be closed early. Other years there was so much fish the cannery could barely keep up. Unless you're monitoring things regularly its hard to say whether there is an actual decline or not.

DSingularity(10000) 1 day ago [-]

Yeah I feel the same. I recently felt pretty astonished after seeing them again once I went out in the woods.

novaleaf(10000) 1 day ago [-]

and to your children, it will be the way nature always was.

disambiguation(10000) 1 day ago [-]

i live in brooklyn, the nearby parks have plenty of fireflies around dusk

iofiiiiiiiii(10000) about 7 hours ago [-]

I have had similar realization with regard to simply mosquitoes. We used to have to put bug-nets on doors and windows to stop bugs flying in during the summer. Now, not needed anymore - at most we get one bug flying in per week.

jriot(10000) 1 day ago [-]

My backyard in Louisiana has quite a few of them at night. I see probably 20 - 50 a night back there.

formerly_proven(10000) 1 day ago [-]

Last time I've seen a firefly must have been at least ten years ago.

nemo44x(10000) about 13 hours ago [-]

Unless there is actual data it's probably best to not trust your memory. Every time you remember something the reality of it changes. Memory is lossy. Bias and selection are favored over improvement or precision of recall. Every recollection is inferior to the previous one.

I have a few memories of extensive fire fly experiences back in the day. But when I really think about it, I don't remember all the days and years without a dramatic memory. And the ones I have may be distorted.

epanchin(10000) about 24 hours ago [-]

Driving through Europe one finds the front of the car covered in squashed insects. Drive through the UK and there are none. Depressing.

at_a_remove(10000) about 23 hours ago [-]

This is one of those areas where I actually am in favor of some pretty tight regulation; photons spraying about leads to a tragedy of the commons rather quickly.

Drive around -- billboards are lit up at night. Headlights might as well be pointed into the sky and are blinding. Stores are scarcely less well-lit when they are closed than during operating hours. I can go for a walk at night and read a paperback the whole way. Porchlights are left on all night, for no good reason.

Sometimes, on a my regular trips to visit a friend, we go out at around one in the morning to a road far from other towns and I can actually see the faint haze of the Milky Way, but in the suburbs, I am lucky to pick out thirty stars and/or planets. I don't even think that all of this constant light makes us any 'safer.'

ozim(10000) about 9 hours ago [-]

Funny thing is that we mostly turn on the lights to keep burglars at bay. I don't have the link but there was some article pointing out that burglars also don't really like dark places. Like using flashlight inside of dark building can easier give out break in. Burglars are also people so they kind of scared of what might be in the dark, like you might miss something on the floor and break your leg or arm and then you would be stuck at the place you broke in.

kyuudou(10000) about 21 hours ago [-]

I used to point this out to hipster trendies who illuminate their trees at night. Don't they consider nocturnal life that are dependent on solar cycles for proper circadian rhythms? Seems disruptive. I got the impression the trees were a bit miffed.

entropie(10000) about 17 hours ago [-]

Having light polution in your habitat during night cycles can make certain plants mutate or make them hermaphrodite while growing indoors. The light from a socket distributor might be enough.

duaoebg(10000) 1 day ago [-]

I always wonder how much of the reduction in cleaning of glass can be attributed to better aerodynamics and better glass.

LargoLasskhyfv(10000) 1 day ago [-]

Motor cyclists and bicyclists tell the same. I as a bicyclist do too. Because apart from the sting of the wind in the eyes it was necessary to drive with sports glasses/lenses/some visor because otherwise you'd be almost certainly hit by insects in the face or eye. That isn't the case anymore. There simply are less of them in the air.

fortran77(10000) 1 day ago [-]

There's technology now to put streetlamps on motion detectors. They can have them very dim until they're needed.

astrea(10000) 1 day ago [-]

I feel like that would be nauseating for residents on heavily trafficked roads

black_puppydog(10000) 1 day ago [-]

We have one street like this here in Grenoble. I was so freaked out the first time I walked there, since the lights dimmed up gradually, so it took me a couple of times to be sure I'm not imagining things. :P

auganov(10000) 1 day ago [-]

What's the benefit for humans and why can't it be done any other way?

I'm not a big fan of encountering insects. Very weird to see someone would actually want to protect them in cities.

> Light traps for insects are to be banned outdoors

So you won't be able to kill mosquitoes outside?

asutekku(10000) 1 day ago [-]

Insects are a pretty damn important part of the ecosystem that provides us our nutritions.

danparsonson(10000) 1 day ago [-]

Insects are a vital part of Earth's ecosystem, providing food for larger animals and (especially crucial for us) pollination.

> So you won't be able to kill mosquitoes outside?

There are plenty of other, more specific ways to do that, no? The problem with light traps is they are indiscriminate.

Not to mention that if you apply some repellent, you don't need to kill them at all.

tass(10000) 1 day ago [-]

One issue with these is that they kill many more moths and other insects than they kill mosquitoes

tpmx(10000) about 21 hours ago [-]

Also Germany: Shuts down nuclear plants and lights up more coal plants.

qayxc(10000) about 3 hours ago [-]

More coal plants is incorrect and the shutdown of the nuclear power plants was planned anyway. Since the mid 1980s no less...

Basically most of the plants that were shut down simply didn't get their runtime extended again and would've been shut down in a few years anyway.

And the utility companies owning them get billions in compensation and don't have o pay for waste disposal of all the irradiated parts either...

Facts that nuclear proponents always conveniently ignore...

Shivetya(10000) 1 day ago [-]

Well as I have posted before, my neighborhood went full LED last year with street lights and so every night is like a full moon. This has become widespread not just in subdivisions but everywhere as people see LED lighting as free.

Perhaps they were a bit oversold.

ip26(10000) about 23 hours ago [-]

The low cost can definitely drive the wrong design. But if a redo is being done, it's an opportunity to turn down power levels, use better shielding, etc. Modern LED lights can provide the same visual acuity at both less power as well as less lux, and with less light spillover.

zackmorris(10000) 1 day ago [-]

The best thing we can do on a personal level is to stop using broadleaf weedkillers like Scott's Weed and Feed and other herbicides like glyphosate (Roundup).

I've been doing handyman work at my boss's house during COVID-19 and he has clover growing in his yard. He has a fair number of bees, certainly more than on my end of town which is all chemlawns.

It's arguably ok to use a little weed killer by hand on dandelions if you must (better to spray 2-3 courses of vinegar instead). But killing all clover, morning glories, and other broadleaf flowers in lawns has led to the near-extinction of bees in residential areas. Herbicides and pesticides in agriculture wiped out the rest.

Edit: fixed the first sentence to distinguish between broadleaf herbicides and glyphosate (which kills everything).

bluntfang(10000) 1 day ago [-]

any suggestions for a poison ivy patch? I'm doing the rounds of vinegar and placing tarps/boxes over the area, trying to avoid the nuclear option.

baix777(10000) about 23 hours ago [-]

Clover is great for lawn health as it fixes (adds) nitrogen to the soil.

Before the widespread use of herbicides having some clover in the lawn was viewed as a good thing because of this. Lawn seeds even included a bit of clover.

When herbicides came out that killed clover as a side effect then we saw people saying (marketing) that clover is bad.

kickout(10000) 1 day ago [-]

Agreed. Ending the chemical era in modern agriculture will be big (and enabled by SV) https://thinkingagriculture.io/innovation-efficiency-chemica...

ssully(10000) about 22 hours ago [-]

I don't use any chemicals in my yard mainly because I don't want to expose my dogs to that stuff. The result is a yard similar to your boss's. When the clover flowers we get plenty of bees, and there are other tiny flying insects, which attracts birds. Never really gave much mind to birds, but it's really nice going out and seeing cardinals on a daily basis. We've even had multiple robin nests around the house, which is fun to see their hatching cycle.

throwaway9870(10000) about 24 hours ago [-]

Old lawns (1950s) had clover by design. Even the lawn seed had clover seed already in it. Clover produces nitrogen which grass consumes, so it is a nice synergy. Clover eventually fell out of style because kids stepping on bees isn't desirable, and fertilizer and herbicides made an all grass lawn feasible.

tengbretson(10000) 1 day ago [-]

Was reserving lawns for humans and all of the countryside for bees not already a pretty sweet deal? How are we being out-negotiated by insects?

kickout(10000) about 24 hours ago [-]

Other comment would be the agriculture use of chemicals dwarfs residential (and also land mass too). Need to figure out agriculture use of chemicals before we worry about residential use (although its easier to 'see' the results)

saityi(10000) about 24 hours ago [-]

My backyard this year is an experiment in doing my best to encourage something that takes care of itself and is more supportive of local fauna. I laid down clover and what the local university's agriculture dept recommends for native-friendly grass seed. I've used no weed killer back there. I'm growing some ornamental grasses and flowers native to the area for decoration.

I feel like it's been nice. I think the clover is pretty, and I feel like we have many more butterflies this year than any of the previous years. The grass has been thriving. Dandelions don't seem to enjoy growing with clover very much, so there's less of them this year, although goosefoots seem more than happy to coexist with clover.

The clover has held up well to dogs, too, which has been a nice bonus.

I think I'd like to add some creeping thyme next year.

upofadown(10000) 1 day ago [-]

>... other broadleaf herbicides like glyphosate (Roundup).

Glyphosate kills everything. If you use it as a broadleaf herbicide you are going to have a very bad week...

xyzzy_plugh(10000) 1 day ago [-]

My understanding is that these are not legal in most first world countries anymore -- I know they are banned in Canada and parts of Europe at least.

It sounds like Germany is already on their way to banning glyphosate[0]

0: https://www.theguardian.com/environment/2019/sep/04/germany-...

wirrbel(10000) about 23 hours ago [-]

I would say in Germany that balance is a little different. Herbicides and pesticides in lawns and gardens aren't so widely used as in the US (agriculture of course different story). At the same time, light pollution by streetlights is a lot worse because streetlights are in every street and overall the streets are more brightly lit. In the US its comparably fewer streetlights (and people often leave on their porch lights to light the street).

GordonS(10000) 1 day ago [-]

I never going the obsession with pristine lawns without clover, daisies, moss and such. I actually think it looks quite nice with some variation.

My wife is militant about this, insisting that our lawn is sprayed with horrible chemicals at least once a year to kill off the clover, daisies and moss.

Seems such a pointless thing to do, and very obviously not good for wildlife.

alexdumitru(10000) about 21 hours ago [-]

I don't know why people hate clover. I'm planting clover along with grass, because it looks nice and helps the soil.

duxup(10000) 1 day ago [-]

>But killing all clover, morning glories, and other broadleaf flowers in lawns has led to the near-extinction of bees in residential areas. Herbicides and pesticides in agriculture wiped out the rest.

Is that the case for sure?

I thought the state of the bee population wasn't quite that well understood...

I have no doubt that various herbicides hurt, but I'm not sure that just that science means their current population is entirely due to such use.

cgs(10000) about 19 hours ago [-]

I live in the high desert of central Oregon and finally got sick and tired of trying to maintain a lawn in the desert. I removed the lawn then planted a ton of zone 4/5 draught tolerant native plants, most of which flower. There are bees everywhere now. And drip irrigation so much less water used. Feels great!

rubber_duck(10000) 1 day ago [-]

>has led to the near-extinction of bees in residential areas

Why do you want bees in residential areas ? I think that it's illegal to keep bees in some proximity to residential areas around here (my grandparents used to keep bees).

odiroot(10000) 1 day ago [-]

Can you get even darker? Streetlights in Berlin (for example) are already uncomfortably dim.

csunbird(10000) about 7 hours ago [-]

Exactly, I hate driving at night in Berlin, because of the insufficient lighting.

fock(10000) 1 day ago [-]

if you don't have to expect a thug with a gun around the corner like in the US, maybe it's completely ok?

dcgudeman(10000) about 22 hours ago [-]

Posts like this are really disappointing to me. I come to hacker news to read about technology and startups. If I wanted to read about general news topics (bees, insects, glyphosate, general monsanto hate, etc...) I would frequent google news, CNN, or reddit.

rat9988(10000) about 19 hours ago [-]

As you can see, you'll always have someone replying with the guidelines and act as if we all come here to have the opinion of software people on topics they don't master at all.

some-guy(10000) about 21 hours ago [-]

You must be new to Hacker News.

jmknoll(10000) about 21 hours ago [-]

Forgive the snark, but if you don't think using technology to make the world a better place is an appropriate topic for HN, you could consider getting your news from techcrunch.

yelloworangefog(10000) about 22 hours ago [-]


> On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.

ashtonkem(10000) 1 day ago [-]

This is good; we've badly underestimated the cost of excess light pollution.

Also good is using more LED street lamps that focus the light where we need it, rather than all over the place. Light pollution is bad, and street lights are very power hungry, with the old sodium lights consuming up to 0.25kWh per hour.

efa(10000) about 23 hours ago [-]

North Korea must be swimming in insects. https://i.imgur.com/S1mrMKN.jpg

kibwen(10000) about 23 hours ago [-]

Here's a quite good video about the pros and cons of LED vs high/low-pressure sodium street lamps: https://youtube.com/watch?v=wIC-iGDTU40 . It goes into lamp efficiency, how our eyes see differently at night, response time under different sorts of light, color temperature, circadian rhythm disruption, and light pollution.

bredren(10000) 1 day ago [-]

I think the impact of light and sound pollution is vastly underestimated. Sound and light pollution not only causes hidden environmental issues but also does damage to human health.

Suffering sound and light pollution is a hidden additional burden placed on lower economic status people.

Poor people experience health inequities due to increased exposure to light and sound, but more pressingly air pollution. [0]

The status of enjoying quiet is even clear in the middle class. You can even see the emerging divide on transit in who has noise cancelling headphones and who does not.

[0] https://ec.europa.eu/environment/integration/research/newsal...

masswerk(10000) 1 day ago [-]

Actually, cold LED light is part of the problem. Better focusing only mitigates the problem they are posing.

emptybits(10000) about 18 hours ago [-]

> Also good is using more LED street lamps that focus the light where we need it,

Focusing light where needed is good. Using LEDs, as we do now, has downsides though. LED lighting in cities is making astronomy and astrophotography and even animal life worse in some cases.[1]

Traditional sodium vapour lights pollute around a wavelength of 589 nm.[2] That's the yellow/orange sky you used to see in the sky around cities. In theory, if you filtered out that wavelength 10 years ago, you'd be able to eliminate a lot of light pollution. But today, with more LED lighting, the pollution tends toward white/blue. This is 'nicer' in some objective and subjective ways. e.g. colour accuracy. But it's a bitch to filter out, without also removing all the white and broad spectrum we want to see in the features of the night sky.

Animals are also sensitive to white and blue light emitted by modern LEDs, and the NatGeo link and others go into why this can be a bad thing, compared to 'old fashioned' lighting methods, inefficient as they are.

[1] https://www.nationalgeographic.com/news/2017/11/light-pollut...

[2] https://www.peterzelinka.com/blog/2019/8/light-pollution-fil...

tyho(10000) 1 day ago [-]


rokweom(10000) about 24 hours ago [-]

Sodium lamps are very efficient. Saying that they consume up to 250W doesn't mean anything. There are LED lamps that consume much more.

craig_asp(10000) 1 day ago [-]

This would be great for amateur astronomers too.

trentnix(10000) 1 day ago [-]

And the sales of insect repellent.

thiago_fm(10000) 1 day ago [-]

I wonder how many centuries it might take for the US to start to care about insects or the environment like some European countries do with the recent regress in policies overall that isn't about ONLY increasing overall GDP in the US.

And I say this even not considering Trump, which is a big menace. But democrat candidates in the US also don't care about the environment, not showing much promise. So do the Americans in general with their huge SUVs and completely unhealthy lifestyles(for them and the environment). I wonder how much this shitshow can go on.

gamblor956(10000) 1 day ago [-]

The states of California and Alaska each individually have more wild forest lands than the entire European continent.

The US created the concept of state-protected wilderness reserves and national parks.

The US was the driving force behind the Kyoto Protocol, and the originating nation of solar power, wind power, and nuclear power.

California leads the world in automobile fuel economy targets, and the US as a whole has cleaner-burning fuel than the EU, which for bizarre reasons settled on diesel as their standard even though it's far worse for the air than gasoline.

Europe's pretty far behind in caring about the environment.

tijuco2(10000) 1 day ago [-]

Have you been to US? I visited some states and I'm now living in Massachusetts for a little more than three years. I've never seen so much green in my life like see here in the US. It's beautiful. The article doesn't mention security measurements that will be taken in order to keep security levels where it is.

johnmarinelli(10000) 1 day ago [-]

to be fair, every country i've visited/lived in pales in comparison to Germany when it comes to environmental friendliness.

dang(10000) about 19 hours ago [-]

Please don't take HN threads further into nationalistic flamewar, regardless of which nations are at issue.


burlesona(10000) 1 day ago [-]

I often hear Europeans make sweeping judgements about the US that reflect a lack of understanding of the country and continent. In part I think this is because most visitors from abroad only travel to NYC, SF, or LA, which are great cities, but not representative of the majority of the country.

Incredible as this may sound, around half of the US land area is completely uninhabited. Fully 1/3 of the land is owned by the federal government, and most of this is managed with environmental considerations at the forefront. The only part of the US that is even vaguely like European population density is the northeast corridor from Boston to DC, which represents a small percentage of the country.

People in the US care a lot about the environment, but the issue has been politicized of course and pits different factions against each other. We also have a housing crisis, and it's very difficult to get humans to worry about saving anything else when they can't get a roof over their head.

The US has myriad environmental problems and needs to do better, but it also has a very different situation on the ground compared to Europe, and thus requires different approaches in some areas.

pointillistic(10000) 1 day ago [-]

Some European countries, named here, only very recently cared more about insects than people.

42droids(10000) 1 day ago [-]

German side streets are fairly dark already. This change will affect 'large' polluters, eg. floodlights in car parks, outside store areas, brightly lit buildings, etc. It is a very welcomed change. In our neighbourhood ppl knowingly plant flowers and leave grass grow 'wild' to support insects.

cheez(10000) 1 day ago [-]

I do this too but I get bylaw visits from the city. Which cities?

m_fayer(10000) about 6 hours ago [-]

I love this. Of course it's different if you're in a dangerous neighborhood, but if you're not, dark streets punctuated by orbs of light from reasonably bright street lights are romantic, mysterious, sometimes soothing. It makes that evening stroll a bit more magical.

Ma8ee(10000) about 22 hours ago [-]

I've left about a third of my garden as a wild "meadow" and it has been full of flowers and bumblebees all summer. Even the municipality has left some lawns in the parks uncut for the benefit of wild flowers and bees. (Sweden btw)

nicbou(10000) 1 day ago [-]

Yes, German streets are quite dark. Some villages even close the lights at midnight.

komali2(10000) 1 day ago [-]

This will be great for stargazing as well then!

codeduck(10000) 1 day ago [-]

I wish more places would do this. It would be amazing for plantlife, wildlife and amateur astronomy.

LXanb(10000) 1 day ago [-]

There are loads of foxes and bats in big cities in Germany already. There are tons of insects as well.

Humans are already not allowed to build anywhere because of environmental regulations (unless you are a corrupt politician) and to keep the rents high.

Germany is getting more and more ridiculous. People must use energy saving bulbs to increase the profit for Philips, but one can legally burn 100l of fuel per day in a car.

Noise laws are lagging behind but insects need to be protected.

masswerk(10000) 1 day ago [-]

Insects are kind of vital to us. Without them, you can't have crops or fruits, no agriculture. The drop in population is already concerning.

balozi(10000) 1 day ago [-]

The cynic in me views this as yet another attempt at fethishizing nature by people who choose to live in sterile urban environments. The problem with this type of fetishization is that it eventually combines with class privilege to produce narrow public policy that seldom promotes the public good.

Furthermore, those that live in less-urbanized less-sterile locales are painfully aware that nature and her army of insects always has the upper hand.

travisporter(10000) 1 day ago [-]

Inherent in your statement is the idea that privileged people choose to live in urban environments, which is probably false half the time. I choose to live closer to work in an apartment to minimize my costs and travel time. I would love a backyard.

lostmyoldone(10000) about 20 hours ago [-]

Europe is quite densely populated in some areas, and insect populations in these areas have dropped to a fraction of past numbers in some areas, over only a few decades. It's quite reasonable to try to do something about it, as we're actually dependent on insects, to a likely significant but somewhat unknown extent.

That there are insects in some places doesn't mean the issues with insect, and general biodiversity loss suddenly disappears.

Bugs are everywhere, but there are species disappearing every at a rate much much higher than they should disappear for natural reasons.

There certainly are cases where fetishization can be part of it environmental policy, but usually it's then about something you can brag about, not about the impact light has on the sex life of bugs.

Even with the possible fetishization of nature, most people still don't care enough to be willing to lower their standard of living, even quite superficially. Generally, I would say cynicism regarding environmental policy is often misplaced, as even terribly inefficient policy makes the existence of the larger issue apparent, and opens the door to better and more informed policy the next time around.

Look at almost any are of legislation, the first plodding steps into a new area is often hilariously misplaced, or terribly expressed with the 20/20 of hindsight. Regardless, they were necessary steps. In some areas, biodiversity arguably one of them, the first plodding steps are taken just now. It's progress, if small.

macinjosh(10000) about 21 hours ago [-]

What is more important? Your nan walking home from the corner store after dark under a street light or a beetle?

Germany: beetles!

Scottopherson(10000) about 19 hours ago [-]

If we decimate the insect population there won't be any nans or corner stores. If a street light is the only thing keeping your nan safe then maybe your area has bigger safety issues to address.

Regardless, for any issue there is usually middle ground to be found and snark comments like yours do nothing to help find or support it.

Historical Discussions: Laws of UX (August 02, 2020: 1033 points)
Laws of UX (January 19, 2018: 535 points)
Laws of UX (September 02, 2019: 5 points)
Laws of UX: Fitts's Law (November 19, 2018: 3 points)
Laws of UX (January 18, 2018: 3 points)
Laws of UX (January 15, 2018: 3 points)
Laws of UX (July 21, 2020: 2 points)
Zeigarnik Effect (January 14, 2019: 2 points)
Laws of UX (April 21, 2020: 1 points)
Laws of UX (December 07, 2019: 1 points)

(1033) Laws of UX

1033 points 5 days ago by seesawtron in 10000th position

lawsofux.com | Estimated reading time – 1 minutes | comments | anchor


Laws of UX is a collection of the maxims and principles that designers can consider when building user interfaces. It was created by Jon Yablonski.

The Book

An expansion of the ideas found on this site is now available in book form, titled Laws of UX: Using Psychology to Design Better Products & Services.


Posters on this site are designed to be printed at 11×17 inches. You can download the entire set here. Posters are licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.'


Tools used to create this site include paper, pencil and Sketch App for design, Gulp for development workflow automation, Sass for CSS preprocessing, and Nunjucks for templating. Typography is set in IBM Plex Sans and Plex Mono, an open-source typeface by IBM.

©Jon Yablonski 2020 | All Rights Reserved

All Comments: [-] | anchor

Animats(10000) 5 days ago [-]

This is on user interfaces for sites that are selling something. It's not for user interfaces that are for doing something.

csours(10000) 5 days ago [-]

My basic definition of UX is providing users data and decisions in a certain context.

In some contexts (selling), certain things are more important, like having a beautiful appearance and no data overload. Your first time user may be your most common user, because they may only make one purchase per [lifetime, year, etc]

In other contexts, for instance industrial controls, data density is more important, and the user may be expected to take more time to learn the interface. There is still some need to accommodate new users, because everyone is a new user at some point.

In any case, there is a decision to be made, aided by some data presentation.

nojvek(10000) 4 days ago [-]

Why are they mostly named after white men? I'm sure other people practice and research UX too right ? Like some laws have multiple researchers who wrote the original paper but the law takes on the white man's name rather than the other minority person. Why?

It would be a lot more welcoming if we changed the laws to be named what they mean.

E.g Fitt's Law -> Size & Distance Law. Doherty's Threshold - Respond to input within 400ms


noir_lord(10000) 4 days ago [-]

Why is that relevant here?

Other than igniting the inevitable firestorm.

blatter2016(10000) 4 days ago [-]

Oh god, not this trite BS again.

ivan_gammel(10000) 5 days ago [-]

#13, the Pareto Principle, is the main source of digital inequality nowadays. Lots of modern UIs do not perform well in terms of accessibility, because backlog is never prioritized for minority groups of users. The art of inclusive communications will never exist if we continue applying this as a law, instead of focusing on reducing the costs of accessibility.

realtalk_sp(10000) 5 days ago [-]

Accessibility is the sort of thing that has to be regulated into existence because the economic cost-benefit often cannot be justified. Similar to handicap parking, ramps, etc.

punnerud(10000) 5 days ago [-]

Copy-paste highlight from every page, for easier read:

1. Aesthetic Usability Effect - Users often perceive aesthetically pleasing design as design that's more usable

2. Doherty Threshold - Productivity soars when a computer and its users interact at a pace (<400ms) that ensures that neither has to wait on the other

3. Fitts's Law - The time to acquire a target is a function of the distance to and size of the target

4. Hick's Law - The time it takes to make a decision increases with the number and complexity of choices

5. Jakob's Law - Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know

6. Law of Common Region - Elements tend to be perceived into groups if they are sharing an area with a clearly defined boundary

7. Law of Prägnanz - People will perceive and interpret ambiguous or complex images as the simplest form possible, because it is the interpretation that requires the least cognitive effort of us

8. Law of Proximity - Objects that are near, or proximate to each other, tend to be grouped together

9. Law of Similarity - The human eye tends to perceive similar elements in a design as a complete picture, shape, or group, even if those elements are separated

10. Law of Uniform Connectedness - Elements that are visually connected are perceived as more related than elements with no connection

11. Miller's Law - The average person can only keep 7 (plus or minus 2) items in their working memory

12. Occam's Razor - Among competing hypotheses that predict equally well, the one with the fewest assumptions should be selected

13. Pareto Principle - The Pareto principle states that, for many events, roughly 80% of the effects come from 20% of the causes

14. Parkinson's Law - Any task will inflate until all of the available time is spent

15. Peak-End Rule - People judge an experience largely based on how they felt at its peak and at its end, rather than the total sum or average of every moment of the experience

16. Postel's Law - Be liberal in what you accept, and conservative in what you send

17. Serial Position Effect - Users have a propensity to best remember the first and last items in a series

18. Tesler's Law - Tesler's Law, also known as The Law of Conservation of Complexity, states that for any system there is a certain amount of complexity which cannot be reduced

19. Von Restorff Effect - The Von Restorff effect, also known as The Isolation Effect, predicts that when multiple similar objects are present, the one that differs from the rest is most likely to be remembered

20. Zeigarnik Effect - People remember uncompleted or interrupted tasks better than completed task

eplanit(10000) 5 days ago [-]

Thank you! I was thinking that the site should offer a simple list as an option.

Ironically, in the strict (and best) sense of the notion 'user interface' (i.e. being an effective, efficient, easy conveyance of information to/from computer and user), yours is better than the fancy site.

dathinab(10000) 5 days ago [-]

> Jakob's Law

I have a problem with that law.

EDIT: TL;DR/Clarification: What matters is what users believe as 'intuitive and easily usable' which is not implied by something being familiar nor by what most other sites or apps do.

People don't want your site to work like other sites at all.

They want your site to work 'intuitive and easily usable' for them.

As such they only want your site to behave like other sites if what they perceive as 'intuitive and easily usable' was coined bye such 'other' sites.

But in reality what user perceive as 'intuitive and easily usable' is often largely coined bey the UX patterns around which they 'learned' to use tech initially.

Which means what is 'intuitive and easily usable' is often displaced in time and depending on your audience might MAJORLY differ from what most other sites you can find today in the internet do!!

A good example is sourcehut (e.g. look at `https://git.sr.ht/~sircmpwn/scdoc`).

It's visually completely different from what most other github like sites do and probably for some people it might look bad.

But for their target audience it's supper appealing as it appeals to what that audience originally learned to be 'intuitive and easily usable'.

So I would strongly argue that Jakob's Law is a harmful over simplification of the actual effect which might be ok for some target audiences but literally might brake your business if applied blindly.

Izkata(10000) 5 days ago [-]

> (e.g. look at `https://git.sr.ht/~sircmpwn/scdoc`).

HN doesn't have backticks syntax, so the trailing one is getting inculded as part of the link and 404ing. Fixed link: https://git.sr.ht/~sircmpwn/scdoc

jakear(10000) 5 days ago [-]

I cannot even begin to fathom how a ux website that literally touts 'don't make your users wait more than 400ms' as it's #2 rule also has

- 400ms css transition

- network load of new page

- scrolling down to view content

- another 400ms transition to go back to main content

as its primary mechanism of interaction... is the irony totally lost on them? Or did they drink the #1 kool aid to the extent that they believe their 'design' will outweigh their myriad UX failures?

csours(10000) 5 days ago [-]

The point is, don't have a static screen for 400ms after an interaction. If I click on something, and nothing at all happens for half a second, I feel a disconnect. A 400ms animation is a choice that you may not make, but does not 'feel' the same as a page load, or other stall.

dahart(10000) 5 days ago [-]

None of those things you listed are in conflict with rule #2.

The length of an animation is not the same as making your users wait for an initial response to your input. The rule is talking about how fast the computer appears to acknowledge and begin responding to a user input, it is not talking about how fast any given task is completed. The wait is the length of time between clicking the button and when the screen starts to animate.

Animation you have to wait for can sometimes be a bad design choice on a page, and I'm not a big fan of overuse of css transitions. But a navigation transition helps the user keep mental track of where they are, and it specifically signals that the computer is responding to your input. The animation itself here doesn't break the Doherty threshold, it does the opposite, it meets the Doherty threshold.

Network loading of a new page is unavoidable (out of the control of the page author), is absolutely standard practice for internet apps (see Jakob's law), and your browser (not the page) is what handles the interaction. The browsers adhere to the rules listed here to every extent possible, and they respond to page loads instantly by showing you a loading spinner, a blank page, and allowing interactions like cancelling the load while loading. Again, the interaction criteria here is that the computer acknowledges your input, not that it is required to finish something that can take time.

Scrolling is another interaction that meets rule #2, it doesn't break the rule. The rule is not about design or layout. When scrolling, if the page is moving in response to your input, then it's meeting the rule. And scrolling is one of the things browsers bend over backward to make as smooth and fast as possible.

GuB-42(10000) 5 days ago [-]

The 400ms rule is that if the user does something, the UI must do something, anything.

It can mean 'display the linked page within 400ms' or 'start the animation within 400ms'. You can't expect 1 TB of data to be transferred within 400ms but you can display a progress bar within 400ms.

Apple used that to great effect. iPhones, at least in the early days always felt more responsive than Android phones. That because they hid latency using smooth animation. And it worked. It was also apparent when I compared Chrome and Firefox a few years ago. Chrome felt faster but mostly because Chrome was faster at showing something on screen. They took about the same time to fully load the page.

quickthrower2(10000) 5 days ago [-]

Ahhh! You had to scroll down. I didn't realise that. Thanks!

arcturus17(10000) 5 days ago [-]

> scrolling down to view content

Shouldn't this be normal?

pretty_lorelei(10000) 5 days ago [-]

The button in the bottom left corner of individual rule pages is almost sadistic: 'Do you want to scroll down a screen to view actual content you already clicked for - or find and click the button in the place your cursor spends the least time at?' Edit: also I will never click the social links on the right, because they are absolutely unreadable to me.

imvetri(10000) 4 days ago [-]

'Dont cut papers'. Gets printed on papers, delivers a message. Look at the message not the paper.

petepete(10000) 5 days ago [-]

Contrast ratio of 1.19 on the numbers isn't a good start either.

steinuil(10000) 4 days ago [-]

> Users often perceive aesthetically pleasing design as design that's more usable.

Is that right? Personally I feel like the opposite is true; I'll take an ugly looking but very usable UI over one that has all the bells and whistles but is actually a nightmare to use. Surely when the UI is bad and the UX is also bad the effect is compounded, and first time users might very well perceive the combination of bad UI/good UX as much worse than good UI/bad UX, but I really don't think this should be given out as good advice.

thiht(10000) 4 days ago [-]

That's not what the 'law' says at all.

If you have two websites with very similar UX, both very usable with the same quirks, one beautiful and one ugly, users will be more forgiving towards the beautiful one.

tlow(10000) 5 days ago [-]

This is not new and is almost entirely covered in two reference resources I use as a designer.

1. Universal Principles of Design https://g.co/kgs/X19MeR

2. The Humane Interface by Jef Raskin https://g.co/kgs/JfdBkB

peferron(10000) 5 days ago [-]

I use Universal Principles of Design as an engineer as well. Before starting a new project, I'll flip through the pages and compile a list of principles that are particularly relevant for the project. Then as the project progresses I'll check that list from time to time to help me steer it in the right direction.

Eugeleo(10000) 4 days ago [-]

Thanks for the links! Do you happen to have any resources specific to web-design as well? I was recently given the task to program a frontend and the design-part proved to be the most complicated one. The more the better; not only geneal guidelines, but also common design patterns etc — I can utilize them all.

Edit: Specifically, I think I can make the website easy to use, but I have no idea how to make it look good.

jhardy54(10000) 5 days ago [-]

Please just post the URL instead of using a redirect link, especially if you're just doing a google search for the words you posted.

1. https://www.google.com/search?q=Universal+Principles+of+Desi...

2. https://www.google.com/search?q=The+Humane+Interface

taphangum(10000) 5 days ago [-]

This is great. But IMO, the real fundamental aspect of UX, from which all others emerge, is the law of having a great information architecture.

I wrote more about this here: https://simpleprogrammer.com/information-architecture-develo....

jhardy54(10000) 5 days ago [-]

> Want To Earn More... And Receive Extra Recognition? Take this self assement and see how you can boost your software development career today! Yes, Show Me How!

I wanted to read your thoughts but your pop-up modal prevented me from reading your article and also decreased the probability that I'll ever visit your website again.

spanhandler(10000) 5 days ago [-]

The only site my tech-challenged dad can kind-of use well is Craigslist. Plain text, accessible, good hierarchies, nothing buried or nested, fast, and, crucially, everything is in the same place every time he visits. Some of the best UX on the Web.

FridgeSeal(10000) 5 days ago [-]

It feels like modern website design conflates "better UX" with "surface level attractiveness"

Craigslist is a great example, original reddit is another example: my UI/UX designer friend considers original reddit to be quote "ugly and horrible", and while there definitely could be some improvements, the reddit redesign (which I know my friend would come up with something similar to) is quite literally orders of magnitude worse, but is aesthetically "nicer".

Original reddit looks ugly, but everything you want from an interface is there once you get through a 3 minute learning curve: information dense, enough white space (but not too much), consistent behaviour, fast, respects scrolling, etc etc.

Where did we go "wrong" with web design that what we have now is seemingly worse? And what does a good balance of "actually functionally useful" and "aesthetically pleasing" look like?

jakear(10000) 5 days ago [-]

Not to mention you can use it without any sort of account. Truly fantastic website design.

muzani(10000) 4 days ago [-]

Here's the straightforward plaintext version:

1. UI > UX.

2. Respond in less than 400 ms.

3. Make buttons clickable.

4. Make it simpler.

5. Copy functionality and UX off other sites.

6. Draw borders around similar functionality.

7. Simpler imagery is better.

8. Users think objects next to each other do similar things.

9. Similar items close together look like one big thing.

10. Things that look the same (color, font, etc) will look like they do the same thing.

11. The average person can remember 5-9 things at once.

12. Remove all unnecessary elements.

13. Focus on the 20% that does 80% of things.

14. Any task will inflate until all of the available time is spent.

15. People judge the experience by its beginning and end.

16. Be tolerant to whatever actions the user may take.

17. People remember the first and last items in a series.

18. You can't reduce all the complexity.

19. When one thing stands out from others in a group, it will be remembered.

20. People remember incomplete tasks, i.e. use progress bars.

4 and 12 are identical. 14 has nothing to do with UX or UI. 15, 11, and 17 are the same. 18 is an excuse masquerading as a Law. From 11-18, it feels a lot of these are pulled in from some Tim Ferriss book or some generic self-help Seven Secrets of Highly Influential People. Perhaps it believes its 15th and 17th law so much that it thinks it can hide fluff in the middle of the list.

It establishes credibility with a lawsofux.com domain them then proceeds to wreck it by violating its 7th law. It does solidly prove its first law, that if you have a pretty enough site, everyone will believe it.

prox(10000) 4 days ago [-]

I highly recommend in the ux threads to read "about face : the essentials of interaction design"

It is a comprehensive look at ux and goes over the history, the context and how to incorporate in a project.

pasiaj(10000) 4 days ago [-]

People judge an experience by its peak and end, not beginning and end.

The best (or worst) part and the last part, that is.

yoz-y(10000) 4 days ago [-]

Worst thing is, that taking 1, 4, 11, 12 and 13 as a gospel ends up with stuff like macOS Big Sur. Power users are left wanting because features that are useful, but a tiny bit niche are getting either removed or hidden in shelves.

read_if_gay_(10000) 4 days ago [-]

Your critique is based on your interpretation of the laws which is in many cases clearly incorrect.

Take for example 15. It doesn't say people judge by the beginning and the end, but by the peak and the end. Or 4 - representing Hick's law as "make it simpler" misses so much nuance that the key point is completely missed.

It rather seems like you barely skimmed the page before deciding to shit on it for arbitrary reasons.

SilasX(10000) 5 days ago [-]

Great compilation! But 12-14 don't seem to have direct relevance to to UX. (Occam's Razor, 80/20 rule, and 'work expands to fill available time'.)

avodonosov(10000) 4 days ago [-]

Yes, I also don't see how to apply the #14 'Any task will inflate until all of the available time is spent.' to UX. It's more form the area of management.

csours(10000) 5 days ago [-]

Are you thinking of UI?

Some of these may be more pertinent while analyzing User Experience compared to composing User Experience.

Occam's Razor: What is this control meant to do?

Pareto: does this UX cover at least 80% of the user's needs? Are there critical cases in the remaining 20%?

Parkinson's Law is more of a bank shot, but think of it along with the Pareto Principle: Organize the UX so that the user can accomplish the most important tasks in the least time.

spaetzleesser(10000) 5 days ago [-]

'Jakob's Law

Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.'

A lot of designers should write this down 100 times before they start working. An extension should be

'users prefer your site to work the same way as it did yesterday since that's what they already know.'

FridgeSeal(10000) 5 days ago [-]

looks pointedly at Spotify

Can we also have some rule about the maximum frequency of changes? Alternatively can I go one further and suggest that for UI/UX changes, agile style sprints are absolutely not the way they should be done? Rushing out change after change in the UX won't let you actually see what works and, and let you iterate, it just disrupts users. These changes should be thoroughly thought through, distilled, tested properly - not wildly deployed onto a subset of users, and if necessary, grouped up and deployed altogether so that there is only one UI change to adapt to, not 30.

physicles(10000) 3 days ago [-]

Another related aphorism, I believe from Joel Spolsky:

"Users don't want to use your software, they want to _have used_ your software."

Users come in with a task they want to perform, not so they can bask in the glory of your wonderful UI.

wgx(10000) 5 days ago [-]

<ad> Shameless plug: If you want 81 more: https://uxbook.io </ad>

50(10000) 5 days ago [-]

Added to cart! But in similar vein, do you know if UX writing is a viable career? If so, how would one get into it? What should be on one's portfolio?

myth2018(10000) 5 days ago [-]

I often see the second law being broken by the first (or a misinterpretation of it): in the eagerness to make the site look beautiful, the designer make it slow, either deliberately with animations, either inadvertently by increasing load times.

Also, I commonly see Miller's and Zeigarnik Effect being disregarded: although our working memories are not so great, we possess some and we can retain some basic information about our workflow. Then, IMHO there's more harm than good in putting a lot of visual clues, drawers, panes over panes etc. so that navigation is 'improved'.

Speaking particularly about form-based enterprise applications, I keep thinking that some important empirically learned lessons from the last 40 years are simply being thrown away. Simple, correct and, most important, FAST interfaces trumps everything else, even if it's a TUI.

Oh, and regular users also can and ENJOY using the keyboard -- a tool which is being deprecated in this mistake of trying to create a uniform experience between desktop and mobile.

JamesBarney(10000) 5 days ago [-]

I'm not entirely convinced simple is better for enterprise apps. I'm a consultant who has to jump around between a lot of project management tools like Asana, Jira, and Azure devops and the simpler the interfaces make me far more frustrated.

They all have pretty much the same functionality, but some hide functionality for UI's sake. Then anytime I need to do something new, instead of looking around a busy screen, I'm sifting through a bunch of websites I found on google.

Akronymus(10000) 5 days ago [-]

One of the reasons I like HN for is the lack of ANY animation. It just does what you want it to do.

Also: At work I make internal software for managing basically everything. We do not use any animations, besides one for dropdowns. And those are only .1 second long.

So, I kinda tend to think that animations/trying to be clever in interactions are actively harmful.

Speaking of being clever, the new reddit layout seems to try to be a single page application. I hate those quite a bit, due to making it much harder to browse.

projektfu(10000) 5 days ago [-]

What do the shapes mean?

test7777(10000) 5 days ago [-]

you can scroll down to get the actual content. but get this, while that site is about UX, its UX is so bad you didn't figure that out right away.

1f60c(10000) 5 days ago [-]

I think they were just meant to make the design more interesting :)

jmole(10000) 5 days ago [-]

the shapes are the raison d'etre for the article.

'How many UX concepts can I attempt to illustrate with basic shapes and shading?'

KineticLensman(10000) 5 days ago [-]

Some of them seem completely arbitrary. Others vaguely relate to the law involved - e.g. #6 where the circles share a common boundary.

I prefer the rules in my copy of Design Elements - A Graphic Style Manual. Its rule #2 make a comment that is apt here: communicate don't decorate.

mekoka(10000) 5 days ago [-]

1. Users often perceive aesthetically pleasing design as design that's more usable.

As a non-designer who's had to fight many designers on this, I'd like to draw your attention to the fact that this is about perception. It's about a user's first impression regarding usability prior to actually using the system. It's not an actual impression based on their experience after interacting with it. That impression can quickly evaporate the moment they try to do anything with the interface. If time and money have to be allocated to develop a product, use them wisely. The properties of this principle seem to be more relevant to marketing and advertising than actual UX.

I believe that the wrong take-away from reading this law would be that improved esthetics implies improved usability. Attempting to place esthetics and function on the same pedestal would be a mistake. When it comes to usability, function is superior to esthetics and should almost always be given priority, no matter how ugly.

I agree that esthetics can and should be used in a way that it supports function, but as you add to it, there will always be a point of negative return (functionally speaking). If you misunderstand what that law is saying you might be misled into thinking that esthetic climax and usability climax are on the same locus. I think we can agree that something can often still be made even more beautiful way past the point where it's almost completely unusable. Adversely, you might be called into making something uglier in order for it to become useful again. Better get comfortable with the idea.

627467(10000) 5 days ago [-]

I'm a designer (maybe of the less designry sort) and I agree that I would have moved the 1st law further down and added more clarifying context here to avoid the misunderstandings you listed.

peferron(10000) 5 days ago [-]

I find this principle useful when users come with negative preconceptions against the usability of your product, perhaps because of previous iterations that weren't very usable or because they tried other products in the same space that weren't very usable.

Putting additional effort on aesthetics can encourage these users to consider your product with fresh eyes, or give it a chance when they otherwise wouldn't. Of course, past that initial help you need 'real' usability to keep them.

For example, I would pay a lot of attention to this principle if I was working on a product in the payment space (e.g. Stripe), taxes (e.g. TurboTax), etc.

bobbles(10000) 4 days ago [-]

I feel like #1 is the reason that absolutely every useful web app is slowly becoming less useful to the people that use it every day by filling all the actual usable space with white to make their screenshots look better while all the power users get more and more irritated.

Even 'compact' views which seem to be a throwaway effort to alleviate these are becoming more and more spacious and forcing scrolling even on huge screens.

kabacha(10000) 5 days ago [-]

It's missing by far the most important law: if the user uses the program the same way after a year of usage it's a failed program - in other words good program design needs to allow the user to grow.

TwoBit(10000) 5 days ago [-]

'By far the most important law'? What are you smoking?

ourmandave(10000) 5 days ago [-]


Aesthetic Usability Effect 01 Users often perceive aesthetically pleasing design as design that's more usable.

Doherty Threshold 02 Productivity soars when a computer and its users interact at a pace (<400ms) that ensures that neither has to wait on the other.

Fitts's Law 03 The time to acquire a target is a function of the distance to and size of the target.

Hick's Law 04 The time it takes to make a decision increases with the number and complexity of choices.

Jakob's Law 05 Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.

Law of Common Region 06 Elements tend to be perceived into groups if they are sharing an area with a clearly defined boundary.

Law of Prägnanz 07 People will perceive and interpret ambiguous or complex images as the simplest form possible, because it is the interpretation that requires the least cognitive effort of us.

Law of Proximity 08 Objects that are near, or proximate to each other, tend to be grouped together.

Law of Similarity 09 The human eye tends to perceive similar elements in a design as a complete picture, shape, or group, even if those elements are separated.

Law of Uniform Connectedness 10 Elements that are visually connected are perceived as more related than elements with no connection.

Miller's Law 11 The average person can only keep 7 (plus or minus 2) items in their working memory.

Occam's Razor 12 Among competing hypotheses that predict equally well, the one with the fewest assumptions should be selected.

Pareto Principle 13 The Pareto principle states that, for many events, roughly 80% of the effects come from 20% of the causes.

Parkinson's Law 14 Any task will inflate until all of the available time is spent.

Peak-End Rule 15 People judge an experience largely based on how they felt at its peak and at its end, rather than the total sum or average of every moment of the experience.

Postel's Law 16 Be liberal in what you accept, and conservative in what you send.

Serial Position Effect 17 Users have a propensity to best remember the first and last items in a series.

Tesler's Law 18 Tesler's Law, also known as The Law of Conservation of Complexity, states that for any system there is a certain amount of complexity which cannot be reduced.

Von Restorff Effect 19 The Von Restorff effect, also known as The Isolation Effect, predicts that when multiple similar objects are present, the one that differs from the rest is most likely to be remembered.

Zeigarnik Effect 20 People remember uncompleted or interrupted tasks better than completed tasks.

abvdasker(10000) 4 days ago [-]

This transcription of the website into a single quickly and easily readable comment is the best criticism possible of the source.

TeMPOraL(10000) 5 days ago [-]

And this is indeed the optimal form of listing these 'laws'. Because that's what that site really is. A list of 20 sentences.

romanows(10000) 5 days ago [-]

Let me give it a try:

    <h1>Laws for good User Experience (UX)</h1>
        <dt><a href='https://en.wikipedia.org/wiki/Aesthetic%E2%80%93usability_effect'>Aesthetic Usability Effect</a></dt>
        <dd>Users often perceive aesthetically pleasing design as more usable.</dd>
HTML-ified, tweaked the text, expanded the initialism :P
ggrrhh_ta(10000) 5 days ago [-]

You improved my UX experience by a factor of infinite.

isatty(10000) 5 days ago [-]

Thank you so much.

Rule 21 should be: don't make the user click 20 times.

karaterobot(10000) 4 days ago [-]

Many people in this thread are criticizing the website's design, as though to criticize the content itself. In case it's not clear from the website: they did not make any of these observations themselves, they just curated and presented a list of observations other people made, usually decades ago. It doesn't follow that any ineffective design or unclear wording on the presenting website reflects on the validity of the source content.

A stronger criticism, at least in my opinion, is that presenting observations about user behavior as 'laws' is misleading, since humans aren't particles. You can have laws in physical science, but not in design. If you thought of these as laws rather than wise words to bear in mind, you'd be an ineffective designer.

JamesSwift(10000) 4 days ago [-]

To be fair to the critics, there is a difference between posting a blog post about laws of UX and creating a site with a domain of 'lawsofux.com' dedicated to the practice. In this case, I would assume that the site I am interacting with is trying to adhere to its rules how it interprets them.

scraft(10000) 5 days ago [-]

This is how it looks for me in the Materialistic Hackernews app


Certainly some fine UI laws in action.

mimsee(10000) 5 days ago [-]

maybe update android webview?

Waterluvian(10000) 5 days ago [-]

The site is beautiful and I couldn't do it. But I'll still be a bit judgemental. It breaks my first law of UX: don't be self serving.

The first screen is a long scroll of 20 elegant looking blocks and titles none of which mean anything on their own. I have to click on each and sit through some superfluous animation before I can even get access to the explanation.

Because of the animation, t he natural flow of swiping back on my phone results in a really janky transition.

I have to say, I love the style of these graphics. Minimalism I guess? Reminds me of the University of Waterloo during my childhood.

ardy42(10000) 5 days ago [-]

> The first screen is a long scroll of 20 elegant looking blocks and titles none of which mean anything on their own. I have to click on each and sit through some superfluous animation before I can even get access to the explanation.

Bad user! You're not following the first law of UX:

> Users often perceive aesthetically pleasing design as design that's more usable.

> Aesthetically pleasing design can make users more tolerant of minor usability issues.

> Aesthetically pleasing design creates a positive response in people's brains and leads them to believe the design actually works better.

> Aesthetically pleasing design can mask usability problems and prevent issues from being discovered during usability testing.

You're supposed to go 'oh, pretty!' and forget how crappy it really is! /s

Seriously now: the first law of UX should be that aesthetics are secondary to usability. If your focus is on making something 'eye catching' or 'beautiful' then you're doing bad UX. If designers followed that, we wouldn't have so many UX trainwreaks.

renewiltord(10000) 5 days ago [-]

Found the short summaries sufficient for everything except Law of Prägnanz, which I can't see how to apply. Much preferred the name+blurb style over a bigger description.

andai(10000) 5 days ago [-]

Yeah, the site breaks the fifth rule, Jakob's Law:

> Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.

Side note, this text was hard to copy because the site changes the text highlight color to black, the same as the background.

agumonkey(10000) 5 days ago [-]

rule 21: favorize balanced density of information rather than aesthetics ;)

bensfrankl(10000) 4 days ago [-]

> The first screen is a long scroll of 20 elegant looking blocks and titles none of which mean anything on their own.

The index page is even worse. It sits under the burger/help menu. Non of the titles make sense, you'll have to click each and read to have a clue.

paulie_a(10000) 5 days ago [-]

The site is completely broken on my mobile.

Paulie law: make a website you would like to use

The other 25 are irrelevant

amelius(10000) 5 days ago [-]

> I have to say, I love the style of these graphics. Minimalism I guess?

I'd say if you practice Minimalism, then also make sure it applies to the total amount of space you're using for it ;)

II2II(10000) 5 days ago [-]

Being judgmental about the UX is fair in this case since it is a site about UX.

Quite frankly, it has a terrible UX. It took me three visits to understand what you meant by access to the explanation. I could only find a bunch of uninformative slides and a blurb promoting a book on the first visit two visits. On visit number three, I found the explanations.


I originally interpreted the scrolling indicator as an advance to next slide button. When I clicked on the button it presented the new slide and the advance button disappeared, so I further explored the site by clicking on the learn more button. That presented another uninformative slide so I ignored the advance button. After all, why would I want to repeat an action that failed to produce the desired result the last time I did it? Noticing the next link, I clicked on that in hopes that it would produce a more desirable result. Unfortunately, it produced the same sort of result as the advance button. At that point, I just gave up and clicked on the Info button and saw the blurb promoting the book. I tried clicking on the links to the left, saw the uninformative slides, and concluded that it was a gawd awful promotion for a book that included nothing but chapter headings.

Those were my first two visits. On visit three, convinced that there must be something more, I actually tried the advance button on the right slide and discovered that it was a scroll indicator. By that point, I was already so frustrated with the experience that I just gave up on the site without seeing what the creator had to say.

As for the graphics style, sure it's great. On the other hand, taking the minimalist presentation too far produced a site that was more difficult to interpret.

Edit: trying to keep terminology consistent as my understanding of the site evolved.

smrk007(10000) 5 days ago [-]

It's much better from desktop view, I found.

mcbishop(10000) 5 days ago [-]

It seems the site is designed for someone that wants to take ample time to meditate on and memorize the 'laws' (it's basically flash cards).

So it's not designed for me — a person on lunch break, hoping to read at least 5 articles via Hacker News before my burrito is gone.

While I was annoyed there, I still appreciate it. It's a free resource. It's pretty. I gleaned a few useful takeaways (especially the idea of putting more important things on the edges).

pbreit(10000) 5 days ago [-]

First law quite ironic.

dathinab(10000) 5 days ago [-]

It also brakes the rule that all text should be selectable (so that people can e.g. copy past it to an search engine).

(The laws titles are not selectable on desktop/linux/firefox.)

Furthermore the short description of the laws does sometimes subtle differ from the actual long descriptions in ways which make major differences about how 'correct' that law is.

hinkley(10000) 4 days ago [-]

Bit low on contrast and flirting with over-saturation.

joeyspn(10000) 5 days ago [-]

> I love the style of these graphics. Minimalism I guess?

It's a 'Memphis style' variant. Pretty trendy these days. Now the industry is starting to move to 'Neumorphism'.

cosmotic(10000) 5 days ago [-]

Haha, I didn't even realize you could click to get more info until you said something.

alangpierce(10000) 5 days ago [-]

In case others are also confused by:

> The first screen is a long scroll of 20 elegant looking blocks and titles none of which mean anything on their own.

The page behaves differently on mobile vs desktop, and that quote is a description of the mobile experience. On desktop, it also includes a one-sentence explanation of each law without needing to click through.

sfifs(10000) 4 days ago [-]

The irony is this designer made an excellent demonstration of poor design :-)

i agree none of the graphics use actually seem to mean anything intuitively to me. Looks like just pretty placeholder pictures.

Also if one find oneself writing down 20 general rules for anything, it's an excellent indication one may not have actually grokked the topic because it's very very unlikely you'll apply 20 rules at the same time. It's likely in some situations, 3-4 are important and in other situations, some others are more important and this ability to classify and recognise what applies where is an indication of knowledge and pedagogical strength.

keithnz(10000) 5 days ago [-]

There's also some 'dark patterns' like in the info section, if you click the twitter link, it trys to make you tweet the link to the webpage.

northwest65(10000) 5 days ago [-]

Wait, there's animation?

mdoms(10000) 5 days ago [-]

It breaks my number one rule of UX for the web: don't fuck with scrolling. Native scrolling is one of the most natural interactions we have with our computers. Even a slight deviation is noticeable and annoying.

galaxyLogic(10000) 5 days ago [-]

Ironically the first law says 'Users perceive aesthetically pleasing design as design that's more usable'.

So now you say 'I love the style of these graphics'. So if the 1st law is correct it means you perceive this design as usable.

Then here's the irony: Just because you PERCEIVE it as more usable does not necessarily mean it is more usable. Which seems to be the case as per your experience.

But so what is the point about trying to make users PERCEIVE a design as usable? Shouldn't the goal be to make it actually usable whether users perceive it as such or not?

shrimpx(10000) 4 days ago [-]

I found the organization to be great, and there's a "next" button at the bottom of every page so you can consume the whole thing linearly.

mrob(10000) 5 days ago [-]

Postel's Law ('Be liberal in what you accept, and conservative in what you send') is a bad idea. You should be conservative in both, which will result in broken software, but broken software gets fixed. Postel's Law adds avoidable complexity, and complexity is a major source of security vulnerabilities. Unlike broken software, insecure software often stays insecure until it's being actively exploited.

freehunter(10000) 5 days ago [-]

As someone who has spent their entire career in infosec, one of my most important rules is "security should break your company". The alternative is, someone else will break your company for you. If you can't produce a document that explains what this server does and every external interaction it could possibly have, you're failing. Be extremely conservative in what you accept.

If you're allowing basically anything in, eventually the wrong person with the wrong code will get in. And that might very well be the last day you have a business.

noobermin(10000) 5 days ago [-]

It seems better to try to ship software that works than ship software that will intentionally break. I get 'everything will break' but ensuring it will break sooner and more spectacularly seems like a headache for users.

MaxBarraclough(10000) 4 days ago [-]

Depends on context. Web browsers are required to cope with old malformed HTML.

david422(10000) 5 days ago [-]

It's good when dealing with humans, it's bad when dealing with machines.

Allow humans a fudge factor for ease of use, but not at the cost of integrity.

For instance, allow phone numbers with separators, or spacing, or shortened versions etc. It's aggravating when a form asks for a cc number and allows a user to type spaces but the field has a max length so you have to backtrack and delete all the spaces.

Make apis conservative in what they accept. This reduces complexity and the likelihood of bugs.

kazinator(10000) 5 days ago [-]

> Users often perceive aesthetically pleasing design as design that's more usable.

Figures that would be rule number one. Everywhere you go there are pretty UI's that are garbage, functionally.

Animation, transparent fading, no distracting scroll-bars or borders, wrapped around a 1-2-3 menu selection.

> Productivity soars when a computer and its users interact at a pace (<400ms) that ensures that neither has to wait on the other.

That is just nonsense. Productivity soars the faster the computer is. Ideally, the user should never waits for the computer; everything should be instant.

Whether the user waits for the computer is neither here nor there.

The ultimate computer-does-not-wait-for-the-user is to have the users prepare job specifications entirely outside of the computer and then submit them to a clerk at a job window. The computer never waits for a user because the job queue is busy from numerous users. It is not very good UI.

In an actual interactive UI, a needless delay could be caused by, for instance, collecting the necessary parameters for an operation through a cascade of multiple dialogs instead of batching them together. Though the computer waits more, too, that's irrelevant; the user's time is what was wasted.

I'm amazed anyone pretending to be a UI person would even feign being interested in how much time the machine spends waiting for the user.

wtracy(10000) 5 days ago [-]

The 400 ms threshold cited for the Doherty threshold is from a 1982 research paper. I imagine that in 1982, 400 ms response times felt pretty darn good.

Also, the nature of the tasks that users performed in 1982 may have meant that there was no benefit to <400ms response times. (If it takes twenty seconds to read and understand the output, and another ten to key in your next command, then a 400 ms delay is not a meaningful productivity drag.)

Today, I would guess that the threshold is closer to 50 ms.

(Also, 'time spent by the computer waiting for the user' makes sense in the context of a 1982 research paper. That has changed in the last 28 years, to say the least.)

All of this is a long way off saying that if you are making UX decisions based on 28 year old research, you are on shaky ground!

stblack(10000) 5 days ago [-]



01. Aesthetic Usability Effect: Users often perceive aesthetically pleasing design as design that's more usable.

02. Doherty Threshold: Productivity soars when a computer and its users interact at a pace (<400ms) that ensures that neither has to wait on the other.

03. Fitts's Law: The time to acquire a target is a function of the distance to and size of the target.

04. Hick's Law: The time it takes to make a decision increases with the number and complexity of choices.

05. Jakob's Law: Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.

06. Law of Common Region: Elements tend to be perceived into groups if they are sharing an area with a clearly defined boundary.

07. Law of Prägnanz: People will perceive and interpret ambiguous or complex images as the simplest form possible, because it is the interpretation that requires the least cognitive effort of us.

08. Law of Proximity: Objects that are near, or proximate to each other, tend to be grouped together.

09. Law of Similarity: The human eye tends to perceive similar elements in a design as a complete picture, shape, or group, even if those elements are separated.

10. Law of Uniform Connectedness: Elements that are visually connected are perceived as more related than elements with no connection.

11. Miller's Law: The average person can only keep 7 (plus or minus 2) items in their working memory.

12. Occam's Razor: Among competing hypotheses that predict equally well, the one with the fewest assumptions should be selected.

13. Pareto Principle: The Pareto principle states that, for many events, roughly 80% of the effects come from 20% of the causes.

14. Parkinson's Law: Any task will inflate until all of the available time is spent.

15. Peak-End Rule: People judge an experience largely based on how they felt at its peak and at its end, rather than the total sum or average of every moment of the experience.

16. Postel's Law: Be liberal in what you accept, and conservative in what you send.

17. Serial Position Effect: Users have a propensity to best remember the first and last items in a series.

18. Tesler's Law: Tesler's Law, also known as The Law of Conservation of Complexity, states that for any system there is a certain amount of complexity which cannot be reduced.

19. Von Restorff Effect: The Von Restorff effect, also known as The Isolation Effect, predicts that when multiple similar objects are present, the one that differs from the rest is most likely to be remembered.

20. Zeigarnik Effect: People remember uncompleted or interrupted tasks better than completed tasks.

markdown(10000) 5 days ago [-]

Edited to take into account the laws of UX:


01. Users often perceive aesthetically pleasing design as design that's more usable.

02. Productivity soars when a computer and its users interact at a pace (<400ms) that ensures that neither has to wait on the other.

03. The time to acquire a target is a function of the distance to and size of the target.

04. The time it takes to make a decision increases with the number and complexity of choices.

05. Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.

06. Elements tend to be perceived into groups if they are sharing an area with a clearly defined boundary.

07. People will perceive and interpret ambiguous or complex images as the simplest form possible, because it is the interpretation that requires the least cognitive effort.

08. Objects that are near, or proximate to each other, tend to be grouped together.

09. The human eye tends to perceive similar elements in a design as a complete picture, shape, or group, even if those elements are separated.

10. Elements that are visually connected are perceived as more related than elements with no connection.

11. The average person can only keep 7 items in their working memory.

12. Among competing hypotheses that predict equally well, the one with the fewest assumptions should be selected.

13. For many events, roughly 80% of the effects come from 20% of the causes.

14. Any task will inflate until all of the available time is spent.

15. People judge an experience largely based on how they felt at its peak and at its end, rather than the total sum or average of every moment of the experience.

16. Be liberal in what you accept, and conservative in what you send.

17. Users have a propensity to best remember the first and last items in a series.

18. For any system there is a certain amount of complexity which cannot be reduced.

19. When multiple similar objects are present, the one that differs from the rest is most likely to be remembered.

20. People remember uncompleted or interrupted tasks better than completed tasks.

enriquto(10000) 5 days ago [-]

Cannot take this article seriously. The best UX for this content is clearly a single page of text with 20 sections.

nwallin(10000) 5 days ago [-]

This is, without a doubt, one of the worst UXs I've ever used.

This cobbler's children have no shoes. Not because he doesn't care about his children, or can't be bothered after a long day. No; this cobbler simply doesn't know how to make shoes.

edit: is the site satire? Poe's law is strong with this one.

ggcdn(10000) 5 days ago [-]

The 0th law of UX: Any website that attempts to describe good UX will itself be criticized for bad UX.

hmmazoids(10000) 4 days ago [-]

Well yeah you better preach what you teach or look like an idiot

Historical Discussions: Moved a server from one building to another with zero downtime (August 05, 2020: 1016 points)

(1019) Moved a server from one building to another with zero downtime

1019 points 2 days ago by huhtenberg in 10000th position

www.reddit.com | Estimated reading time – 6 minutes | comments | anchor

We're a smallish eating disorder and chemical dependency healthcare provider. When the company was founded 5 years ago, a lot of dumb choices were made, expensive dumb choices. The idea was the company would be HUGE in a few years, but the leadership at the time forgot that takes work and competence. One bad decision was to embed Salesforce deeply into our patient acquisition/intake/follow-up processes from day one.


One thing about COVID19 is that people having to social distance made some people realize how bad they or their family members were, with respect to their disorder. No distracting events/activities outside the home let people see their own behaviors, or their family members, up close. We do about half in-patient work, and half IOP/PHP programs.

Our admissions stayed stead when the shutdowns started, and have stayed strong throughout the pandemic. When you're talking about patients who are literally near death because of substance abuse or eating disorders, it's not an exaggeration to say we're essential healthcare. Our patients' disorders can definitely cause them to be immune compromised, so we stayed open and took massive precautions. We haven't had a single patient in our facilities who was positive yet, fingers crossed.

But while patients were still coming in, insurers have been sloooow to pay. Collections are tight. Charges are being accepted, but insurers are just taking their sweet time paying the accepted charges.

The Main Event:

Our Salesforce renewal came up, and we knew it was going to be a problem. We engaged with Salesforce about creating a payment schedule, because coming up with about $160k in one lump sum by the due date wasn't going to happen. We had 75 licenses for all our various departments.

SF absolutely refused to work out a payment scheme for us on the contract payment, which was separated into three product invoices, $12k, $32k, and $123k, but all due on the same date. Literally refused, I got one email response that was just 'No.' Weeks of emails back and forth seemed to be going nowhere.

I thought we got somewhere when they finally set up a conference call to discuss it, but then the finance guy for Salesforce basically said 'just pay all three invoices by the due date and then we'll be fine.' I said, 'While technically, yes, that is a payment plan, but I assumed you understood that the request was to break this up across several months. With COVID19, our patient count is stable but insurers are paying very slowly right now. We simply can't afford that as a lump sum.' The finance guy literally said nothing.

After 20 or so seconds our sales account manager stepped in and mentioned a couple programs they're introducing for new customers to make payments easier during the COVID crisis. I asked if we were applicable to use them, he said, 'Oh, well, no, I guess not, they're just for new customers.' I said, 'So you'll help new people but not existing customers.' he replied, 'Well, that's why we're on the call today.' I said, 'yes, it is, but I'm not hearing anything from Finance.'

The finance guy piped up, 'Like I said, all I can do is suggest you pay the three individual invoices one at a time, and as long as they're paid by the due date, then we'll be all caught up.'

I said, 'So this call has basically been just so you can say you had the meeting and made an offer, isn't it? You never had any intention of actually working with us, but now you can, on paper, say you tried. Look, if we don't work out a deal, I have to move us to another product. You can have the money over a 6 month period, or you can shut us off, and lose us permanently.'

He said, 'We'll see.'

I said, 'This is why people hates Salseforce. You have us over the barrel, and you're going to make sure you fuck us. Thanks.' I hung up.

I researched other CRMs to see what would fit our company, and luckily found one. They had all the features we need, in one way or another, they were honest about what their product did and didn't do, and they had a team that was eager to help us import our data and work out the process workflows. We went into overdrive.

30 days later, SalesForce sent us a letter that we hadn't paid, and asked when they could expect payment? I said, 'I have been telling you for months we can't make the full payment at once. I wasn't kidding, exaggerating, or bluffing. I told you $160k wasn't going to appear in our bank accounts on the due date.' They sent an email warning us that we'd be cut off in 7 days. I didn't reply. They sent another email a day before the shut off, I didn't reply. They sent a final email the day they shut us off. I ignored it.


11am, SalesForce shut us off. We were already on the new product, had all of Intake trained, had our processes fairly well worked out, and had begun training after-care people. We had managed to cut the seat count down to 40 people by simply not letting people have accounts 'just in case.' The next week, I swear to Ba'al, Intake had the most admissions in company history. Not by a huge margin, but a record is a record. It wasn't due to the new CRM, we just had a lot of people willing to jump in, but our intake team HANDLED IT with the brand new/ CRM system they had just started using. I can't take credit for that, our intake people are fantastic.


It's been over two months. People like the new system more, we've ironed out the bugs, people are used to it. CEO is thrilled we pay $1,900 per month rather than the equivalent of $13k/month. Intake has a smarter system, one we designed for what we NEED rather than what we might want.

And SalesForce? Fuck'em. Don't ever think you have me in a hostage situation. I'll shoot the hostage and ask you, 'now what?'

Addendum: I didn't name the new CRM because I didn't want this to sound like an ad for them. They're great, I like them, they're a small company in mid-Michigan that's been around for a decade or so, and they're good honest folks, so far. Spoiler: They're Nutshell and we're happy with them, but this is not an ad.

All Comments: [-] | anchor

arethuza(10000) 2 days ago [-]

This reminds me of a small company I joined many years ago that did deployments by RAID - find a working server (possibly at a customer site) swap in a blank HD, wait for it to rebuild then take it and put in a new server and repeat the process.

Like finding people who argue against revision control systems, it's really quite a challenge convincing people why things like this are a bad idea - after all 'it works!'.

yjftsjthsd-h(10000) 2 days ago [-]

That's... actually fascinating, if in a slightly insane way. There's pets, there's cattle... and apparently there's a herd of cloned pets, which I'd somehow never considered before:)

exabrial(10000) 2 days ago [-]

Reminds me of the 'hot slide' technique used for old telephone switches

ansible(10000) 2 days ago [-]

They did some crazy stuff in the old days. Like when they moved a telephone exchange live... the whole building.

walrus01(10000) 2 days ago [-]

Sort of on the subject, i've seen a brochure for a specialty product marketed to law enforcement. It's meant for use with the seizure of live, powered on desktop PCs and similar that have a high likelihood of full disk encryption.

Essentially it's a medium sized double conversion ups, with a really high quality sine wave inverter, and some electronics that can match phase with a live 120vac 60Hz circuit. And a tool kit which consists of the insulated electrical hand tools needed to do a midspan removal of the cable jacket and splice into the wires in an ordinary PC power cable. The person using it is of course supposed to be trained in advance, and competent at the process of attaching the UPS to the live circuit.

throwaway744678(10000) 2 days ago [-]

Wouldn't it be safer to open the case and connect some kind of battery + adapter after the power supply?

bob1029(10000) 2 days ago [-]

How do they deal with the loss of network connectivity?

I could pretty easily write a script that forces my machine to reboot and do all manner of other things if some sort of network change is detected.

miles(10000) 2 days ago [-]

HotPlug Field Kit https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

'With the CRU WiebeTech HotPlug you can transport a computer without shutting it down.

'The HotPlug allows hot seizure and removal of computers from the field to anywhere else. The HotPlug's patented technology keeps power flowing to the computer while transferring the computer's power input from one A/C source (such as a wall outlet or power strip) to another (a portable UPS) and back again.

'We created this product for our Government/Forensic customers, but it has IT uses as well. Need to move a server without powering it down? The HotPlug can do it.

'It's great for digital forensic investigators and techs who can't risk losing access to data on a running computer. With many computers now employing full-disk encryption, shutting them down poses the risk of having to crack a password after moving the computer to a lab for analysis, which can greatly increase the time and expense of an investigation. When combined with a WiebeTech Mouse Jiggler, you also won't have to worry about the computer entering password-protected screensaver or sleep modes.'

jlgaddis(10000) 2 days ago [-]

Search for 'HotPlug' on YouTube.

mercora(10000) 2 days ago [-]

linked below is an old advertisement/demo video of a similar device or maybe even the one you mentioned :)


jimmaswell(10000) 2 days ago [-]

I'd have thought plugging something into the outlet and unscrewing the outlet to take with you would be more convenient than carefully splicing wires just enough not to disconnect them. All the easier if it's on a power strip.

huhtenberg(10000) 2 days ago [-]

In a similar vein, there are USB gadgets that emulate a mouse that keeps on jiggling, to prevent the machine from locking out on user inactivity.

However, there are anti-jigglers too that lock the machine when any new human input device is plugged in.


jcrawfordor(10000) 2 days ago [-]

I once basically spent a summer doing this, not over a parking lot but to consolidate the remaining equipment in a large number of racks into a few new ones - this was a former sales office of a megacorporation that had been built to have its 1970s-era computer room proudly displayed through windows into the main conference room, a very weird setup without the context that in said '70s that conference room was used to pitch prospective customers on business automation.

Anyway, by the time I was there it was still a '70s-vintage large computer room but now massively overprovisioned on space, cooling, etc, particularly with most IT functions having moved to corporate. A decision was made to repurpose part of it as a test lab and move all the actual remaining equipment to three racks in the corner.

I'd do about two servers a day in between other things, taking advantage of redundant power supplies to transfer the PSUs one at a time to extension cords, swap to a long network cable fast enough that TCP sessions probably didn't time out, and then unrack onto a hydraulic lift card and do the same procedure the other way.

I presented this at the start as far from a guaranteed strategy - that it would minimize downtime but there would inevitably be some due to mistakes. None of this was really that critical. There were a few devices that were pretty old and poorly maintained, we agreed up front that if these lost power for some reason and then failed to boot, we would just say they'd lived long lives and purchase replacements.

I guess the point is that this whole situation was kind of unusual and I would generally not recommend doing this, we were lucky that all the equipment left had stakeholders that acknowledged it was legacy stuff and they could tolerate losing it.

The irony is, of course, that it went perfectly. So far as I know there was not a single problem experienced through the whole thing. I even managed to swap the phone lines to the (surprisingly busy!) legacy fax server when each was out of use.

groby_b(10000) 1 day ago [-]

> There were a few devices that were pretty old and poorly maintained, > if these lost power for some reason and then failed[... we'd] purchase replacements.

And the sysadmin let that opportunity pass??

michaelmior(10000) 1 day ago [-]

> swap to a long network cable fast enough that TCP sessions probably didn't time out

I'm just picturing the networking equivalent Indiana Jones swapping the idol with sand in Raiders Of The Lost Ark.

nobrains(10000) 2 days ago [-]

I don't know. If the 'boss' was charged '4.5 hours of work, 2 hours of consultancy, and 4.5 hours of consultant', and assuming he would have been charged half of that with downtime, maybe the boss did get a good deal. We don't know the cost of downtime for him.

I mean if he had access to technical resources who were willing and capable to do this for him, he chose to do it.

notwhereyouare(10000) 2 days ago [-]

I personally find it hard to believe that a rough estimate of $450 for the job (spitballing $45/hr for 10 hours) is less than 5 minutes of downtime and they only have 1 server.

Then again, could easily be wrong

moduspol(10000) 2 days ago [-]

It's also possible that 'downtime' has different meanings to different people. The client may be seeing 'downtime' as the net result of what happened the last few times the server was 'down,' which could have been for any number of reasons (potentially even unrelated to the server itself).

When you get clients describing things like this, it's possible they've been promised things about this server before by other consultants that didn't pan out. They don't want to give you the full details because then you'll recommend a different route that they don't want to take (justifiably or not).

It's easier for them to frame the problem to a consultant in a way that allows for only one potential solution, even if perhaps better ones exist, because the guy in charge of making the decision isn't technically skilled enough to assess whether others proposed by consultants are as viable.

And, of course, one might read a little into why there exists a 'boss' with such a highly-critical IT need that is hiring a consultant to do work like this, and thinks that threatening to not pay at all if there is any downtime is the best way to do it.

I mean, what if they opened the door to this closet and it grazed a power cable on the floor and the machine just shut off? Why even bother staying around to bring things back up? It wasn't your fault and there's already downtime: you're not getting paid.

D895n9o33436N42(10000) 2 days ago [-]

This reminds me of a famously obtuse and obdurate boss who asked for things that were utterly impossible. He had delusions of grandeur which left him convinced that he and only he was qualified to challenge the "cheap, fast, good - pick any two" triangle.

Naturally, I did my best to explain the laws of physics to him, but he wouldn't hear it. In a spectacular display of Stockholm syndrome I did my best to appease him for four years, but, as many of you can surely predict by this point in the story, I failed in every possible way and eventually gave up. Just wish I could have my four years back.

I was glad to read that OP at least got paid well for his efforts.

Tade0(10000) 2 days ago [-]

I applaud you for being able to stand four years of this.

I usually get fired from such positions in less than two.

jtbayly(10000) 2 days ago [-]

Seems very risky. Not something I'd want to do if minimum downtime was the goal. One wrong piece of gravel ends up with catastrophic failure instead of 5 minutes of downtime.

dmurray(10000) 2 days ago [-]

But the goal was zero downtime, not minimum downtime. The client made it clear that 5 minutes of downtime was equivalent to catastrophic failure. So they correctly found a solution that reduced the chance of '5 minutes of downtime', at the expense of an increased risk of catastrophic failure.

xyst(10000) 2 days ago [-]

Why wouldn't cloning the VMs to a second server, then split the traffic between the primary and secondary server work? Once traffic to the second server is confirmed, you could shut off the second server and haul it off to the new location.

I would probably still charge a much higher rate since the owner was an arse, but at least you would get back your 7-8 hours.

AnIdiotOnTheNet(10000) 2 days ago [-]

Not all services can be load balanced in this way

Live migration of VMs would have been a better option, which was brought up in the reddit comments and dismissed because HyperV live migration is spotty. While I'd have to agree with that assessment, it isn't so spotty that what they actually did was less risky.

blcArmadillo(10000) 2 days ago [-]

It sounds like there was no second server.

morphogenesis(10000) 2 days ago [-]

Database inconsistency for one thing. This works for frontend web services but how do you reconcile the writes between the two servers?

viraptor(10000) 2 days ago [-]

You're making assumptions about what's running on the servers. Let's say it's a VoIP conference server with a shared dedicated room - effectively you have an ongoing session shared between multiple connection and you cannot stop it. Or you have stateful local processing so you can't 'split the traffic'. Or a number of other limitations...

fooblat(10000) 2 days ago [-]

> Stupidest thing I've ever had to do.

I don't really understand the 'ranty' tone. The client had very specific requirements and the author came up with an effective solution and was fully paid to deliver it. Sounds like a win for everyone.

some_random(10000) 2 days ago [-]

The client was an asshole who demanded 100% uptime and stated that they wouldn't pay if there was any downtime at all. The rant is entirely justified.

swarnie_(10000) 2 days ago [-]

I see reddit so i assume this is the sysadmin subreddit?

They're famous for not being a cheery bunch. Because reddit's demographic does swing younger the sub used to be filled with endless posts about being socially incompetent or possessing 0 business craft.

Does anyone know if it improved?

ianhawes(10000) 2 days ago [-]

I believe the expectations of having 0 downtime was not expressed until the day of the transfer.

intpx(10000) 2 days ago [-]

probably because proper architecture (clustering, HA etc) and planning would have never made this an issue. This is still an extremely risky operation, hot swapping power and switching interfaces on the fly all while sitting on a cart in a corridor. In any disruptive work there is never a guarantee of no downtime for affected assets. I know the OP came in as a consultant, but If I was the MSP tech, I would have demanded a paper trail a mile long to cover my ass if this went sideways and If i was the account manager for the client, I would have refused the work. Its not good business to agree to do work where you know there is a better than good chance there will be an outage and your client is saying they wont pay if there is an outage. Even agreeing to it puts you in a bad spot for future work. I guess as an outside consultant, bewilderment is a better reaction than ranting, but this is the kind of shit that drives ops folks crazy

jedimastert(10000) 2 days ago [-]

> Me: You didn't notify them of scheduled maintenance like we discussed on Friday?

It appears that the client didn't have the specific requirements on initial consult.

throwaway0a5e(10000) 2 days ago [-]

Reddit (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints. Following best practices and rules is one of those things. This server migration clearly runs counter to established wisdom so OP using a writing style of 'look how terrible and asinine this was' will be rewarded and gain traction much more than a 'look how interesting this was' writing style.

social_quotient(10000) 2 days ago [-]

Slight topic drift - Any thoughts on how the pandemic might materially change assumptions about an onsite/onprem being better than cloud or manage data center when the code people are now actually remote to the "Local" infrastructure. Something specific to the reality of the pandemic strikes me as something that would make the die hard local only folks have to start rethinking the position.

(Not to suggest it's bad, just different now that a primary assumption about people work in the office is less true)

AnIdiotOnTheNet(10000) 2 days ago [-]

As someone who works in a very anti-cloud company culture (which I happen to agree with), this incident has had no effect whatsoever on that mindset. We don't dislike cloud because it is accessed remotely, we dislike cloud because of the lack of control we have over everything running there. If something happens and our local systems have a problem, there are people here, like myself, who's highest priority will be fixing it and second highest priority will be communicating the status of that. Your problems are never a priority to a cloud vendor and communicating with you is even less of a priority. That's before we even get into the absurd expenses and reliance on big fat pipes.

ocdtrekkie(10000) 2 days ago [-]

I feel a lot safer knowing I'm controlling all the variables during a global crisis, actually.

This article provides an example of how when you operate on prem, literally any crazy option remains on the table for you. If you asked your cloud provider to do this, it'd be a no.

macintux(10000) 2 days ago [-]

At my first job we were starting up the company and didn't really know what we were doing; one early server was sitting on a folding table and its power cord was wrapped around a leg, so just to replace the table with something more robust involved downtime.

em-bee(10000) 2 days ago [-]

the careful application of a saw or an angle grinder would have made it possible to remove the folding table without unplugging the power cord. :-)

crumpled(10000) 2 days ago [-]

I heard an anecdote about a company splicing some fiber cable in the middle of a utility van and having to cut the van apart at the end.

elliotpage(10000) 2 days ago [-]

I've been there, solidarity for cheap furniture based maintainence windows.

Humphrey(10000) 2 days ago [-]

I haven't read the article, but I'm reminded of that episode of Seinfeld and the frogger arcade game

dfsegoat(10000) 2 days ago [-]

I have pondered this exact scenario (server move w/0 downtime) - because of watching that episode - wouldn't have thought about it otherwise.

..It's interesting how pop-culture and your chosen profession intersect, at times.

devchix(10000) 2 days ago [-]

I recall sometime in the mid 2000s there was a fever for achieving five-9s (99.999% uptime, I think -- it became fodder for a few episodes of Mr. Robot). Not that the metric ever went away, but back then a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service. Sun 15K and 25K series (Gilfoyle had a used one in the garage running his network) were behemoths whose mem/cpu boards you could swap out wholesale while the entire frame and backplane was powered on, and while the OS the board came out of remains functioning. There were many caveats around the procedure but it worked. Execs and sales guys loved those demos. These monsters were expensive and banks and energy conglomerates were buying them by the dozens. There was also a big todo about hot swappable drives. The idea that you could be doing hardware maintenance while the machine was still running was a novelty, something like brain surgery while the patient was not only awake, but awake and eating, driving his car, talking on the phone, etc.

A decade later I look back with deep surprise that we didn't think to abstract out the service instead of the hardware. I don't know how many of those behemoths are still being bought, now I work almost exclusively with small server instances that can come and go on the fly. Micro services and AWS have taken five-9s in a different direction. I frequently think of Sun as a failed Hephaestus, in a Christopher Nolan film he would be brilliant but could only turn out clumsy tools because of his deformity, he hates the things he makes so he throws them away before completion. Men find these cast-offs and temper and refine them.

chasd00(10000) 2 days ago [-]

i remember those. A friend of mine was a network engineer at a local datacenter ( UUNET then MCI pre-scandal ) and said companies were buying Suns for everything no matter how trivial.

He worked a night shift and i use to go hang out with him in the noc and download movies (residential bandwidth was not what it is today). Odd he nor i ever get in any trouble for that heh.

tlack(10000) 2 days ago [-]

Well as I recall there were a few reasons that people focused on reliability in hardware in the late 90s:

1. Shared state storage systems that supported replication were rare (I think Oracle and Informix maybe?)

2. Virtualization software was in its infancy (did SunOS have something before Solaris?)

3. RAM and hardware were waaaaay more expensive, meaning you often had to buy more pure metal just to answer questions fast enough

At least that's my take on it based on my dim faded memories

bcrosby95(10000) 2 days ago [-]

AWS single region SLA isn't 5 9s though. If you want 5 9s in the cloud, not even multiple AZ is enough - you need to go multi region or even multi cloud.

theevilsharpie(10000) 2 days ago [-]

> [In the mid-2000's], a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service... A decade later I look back with deep surprise that we didn't think to abstract out the service instead of the hardware.... Micro services and AWS have taken five-9s in a different direction.

In the mid-2000s, enterprises were (and in many cases, still are) running proprietary software with proprietary RPC protocols that had no available source code or other means of modification, and most had no support for application-level high availability, access control, or any other operational quality-of-life feature that people take for granted today. Rather, that functionality was handled at the infrastructure level, through things like the aforementioned Big Iron.

The world looks different today, but those machines made sense for the environment at the time.

larrik(10000) 2 days ago [-]

> a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service.

AS/400's were capable of that in the 90's (possibly the 80's as well). Heck, they'd call IBM for replacement parts on their own. You'd show up for work and there'd be an IBM guy waiting to be let in. He'd swap out a part with no downtime, and be gone. I've seen machines with uptimes of over a decade with zero on-site IT.

techslave(10000) 1 day ago [-]

> a novelty

not a novelty. the true bigiron vendors (not sun) had been doing this for decades.

mainframe reliability puts the upstart unix systems to shame.

g051051(10000) 2 days ago [-]

I guess servers have gotten a lot more robust in the last decade...there's no way any server I ever managed would survive something like that.

maeln(10000) 2 days ago [-]

A lot of server are SSD-only these days which make them less fragile. Still, I really wouldn't see myself pushing a running server in a cart.

anfractuosity(10000) 2 days ago [-]

Reminds me of this - https://www.youtube.com/watch?v=vQ5MA685ApE

'Moving online webserver using public transport'

salzig(10000) 2 days ago [-]

had the same thought :)

zomglings(10000) 2 days ago [-]

In the rain!

tyingq(10000) 2 days ago [-]

The Indiana Bell building move is pretty impressive. http://www.paul-f.com/ibmove.html

cpuguy83(10000) 2 days ago [-]

I'm picturing that Seinfeld episode where George tries to move the Frogger arcade from a restaurant that is shutting down but doesn't want to lose his high score.

will_pseudonym(10000) 1 day ago [-]

HOLES! I need HOLES! :)

JeroenKnoops1(10000) 2 days ago [-]

Reminds me of the OpenVMS clusters.. Police in Amsterdam celebrated in 2007 an uptime of 10 years of their cluster. In this period, all hardware was replace, and half of it was moved to another location 7 km away. All data moved from DAS disks to SAN without one application needed to be stopped. Also VMS was upgraded from 6.2 to 7.3-2. The VMS cluster did not go down during all of these changes. I <3 OpenVMS

tandr(10000) 2 days ago [-]

Would be interesting to know if it is still up and running?

JeroenKnoops1(10000) 2 days ago [-]

During Y2K I've also had to shutdown various OpenVMS servers with uptime over 10 years... Only because of company policies, not because OpenVMS required the reboot.

umarniz(10000) 2 days ago [-]

Interesting read, makes me wonder as a thought experiment if it counts as downtime if the latency of commands on the machines rises to 5 minutes?

You could clone the VM to another instance and record commands going to VM1 and replay them to VM2 after 5 minutes.

This whole brain fart of mine doesn't make much sense but if you play along with it, does it still count as a downtime or just very high latency?

NateEag(10000) 2 days ago [-]

It depends on how downtime is defined in the contract.

That sounds like I'm being snarky but I mean it - whether an actual legal contract or just the documentation given to users, any system where downtime matters should have some discussion of what impacts downtime can have and how it's measured and managed.

That documentation is what defines 'downtime'.

I'll add that what you've described is a sort of low-fi manual version of DB replication (https://en.m.wikipedia.org/wiki/Replication_(computing)).

pc86(10000) 2 days ago [-]

Wouldn't requests time out on the client side long before five minutes?

tyingq(10000) 2 days ago [-]

That works where you have control over all of the timeouts and failure detection at every level and layer. TCP keepalives, for example, could thwart you. Or client side timeouts, or firewall connection state tables, etc.

5 minutes of unplanned downtime in a pub/sub setup could easily go unnoticed, since that setup is typically tuned for long timeouts and/or repeated retries.

tobyhinloopen(10000) 2 days ago [-]

10 hours investment for no downtime seems like a good deal for the owner

topkai22(10000) 2 days ago [-]

Depends on if he really has customers accessing the system 'all the time.'

Besides, as pretty much everyone has noted, running a zero-downtime system on a single physical machine in what sounds like is just a normal cable room is kind of nuts. Those 10h would have been much better spent to move that puppy to someone else's data center and get some redundancy.

Although reading between the lines, maybe the lease was up and they were waiting to the last minute to move it.

user5994461(10000) 2 days ago [-]

I am so scared to imagine what would happen if there was any issue during the move (very likely when dragging live cables and powers over hundreds of meters).

The client would immediately refuse to pay anything because he was very clear he wouldn't pay a thing if there is downtime.

Then, the next contractor would be super quick to judge you and the situation, reinforcing that you were an incompetent idiot and the client was right to kick you away on the spot and not pay a dime.

Glad it went well in the end. There is so much to lose for the person trying to help.

willcipriano(10000) 2 days ago [-]

This is a junior sysadmin I suspect. With a bit more experience you'd learn to say something along the lines of 'no downtime, sure, that will be 30 grand' and the ability for downtime will suddenly materialize. Him and his friend did this big song and dance, took a huge risk and only got paid for ten hours worth of work in the end.

gnopgnip(10000) 2 days ago [-]

4.5 hours of a consultants billing rate can be much more than 10 hours of your regular hourly rate working a similar job. A good consultant will have a contract. The client saying I won't pay if XX happens doesn't mean anything unless it was in the contract.

Networking/spanning tree loops, arp table mismatch/corruption, the switches at the destination being misconfigured are all realistic problems that would result in downtime here. The normal way you do this is with live migration from hyper-v or vmotion from ESXI. If the initial migration is not successful, you just leave the server powered on while you address the issues. Once the VM has been migrated you can do whatever you want with the original server without worrying about downtime.

ThePowerOfFuet(10000) 2 days ago [-]

Also, moving a server with spinning disks? What could possibly go wrong.

sleepybrett(10000) 2 days ago [-]

If there was downtime during the move and the client was there and declaring that they would not pay, you just walk away. You'd be surprised at how fast they can cut a check in that situation.

linsomniac(10000) 2 days ago [-]

Lower stakes, but ~15 years ago a friend had a Linux box in the corner that had huge uptime. I want to say the uptime started shortly after the kernel patch that fixed the 400-ish day overflow of the uptime counter. He moved to a new home and very carefully moved the running server using it's UPS. He didn't have to worry about keeping networking up though.

I used to be all about long uptimes. I eventually started seeing long uptimes as a negative though. A long uptime probably means patches have not been applied.

jccooper(10000) 2 days ago [-]

I also did that once, about the same timeframe, specifically to preserve an uptime.

I think the cult of runtime came about simply because it was impressive that a personal computer could stay running for more than a few days when most of the world ran Win95. And because development cycles were longer and there weren't a lot of network threats.

panpanna(10000) 2 days ago [-]

Really disappointed they didn't use a wireless network of some kind.

lordnacho(10000) 2 days ago [-]

My first thought as well. Set up WiFi along the path, basically turn the machine into a laptop. But I think there might be a disconnect when you change base stations? At least when I move my laptop between rooms in my house there's often a momentary problem while on video call.

The other way I'd do it is more similar to described. Create redundant network paths to the server, then cut one.

dbalatero(10000) 2 days ago [-]

I wouldn't, the risk of disconnects is high.

neilv(10000) 2 days ago [-]

Good thing the server had two power supplies. There was a YouTube video (which I can't immediately find) of people moving a server across town, on the train, without powering it off, and, IIRC, they had to splice the UPS into the power cable.

When it's done for pay rather than for fun, and payment is conditioned on zero downtime, I hope they charged a premium to make up for the risk of no pay. Offhand, I don't know what's a good way to do that -- I've never had a consulting client demand terms like that for billed-by-the-hour work.

kijin(10000) 2 days ago [-]

Effective hourly rate = base hourly rate * risk.

Risk = client risk * task risk.

Client risk is based on your past experience with the same client. If they're prone to demand last-minute changes or stupid stuff, they get charged a higher rate on every project afterward. Jacking up the client risk factor is also a nice way to fire a client you don't want.

digitalsushi(10000) 2 days ago [-]

I had to search the reddit commits for 'vmotion'. They have it covered.

This anecdote is an amazingly good story for telling at the pub over a few beers. It's a terrible story for a strategy.

If this is a mountain, my molehill is that one night in the late 90s, I got paged cause the SMTP outbound server was overheating. At midnight I drive across sleepy NH backroads, and stopped at a Wendys to get a chicken sandwich and iced tea, for the caffeine.

When I got to the server room, I pulled the 2U Dell server out of the rack and discovered the CPU cooling fan had seized up. Mind you, this is a New Hampshire data center in 1999, and it has a filing cabinet with manilla folders, and carpeted floors. This thing was never prepared for any disasters.

A half hour later, the SMTP server was up and running cool again.

I greased the fan with the mayonnaise from my sandwich.

Spooky23(10000) 2 days ago [-]

The real lesson is that the teller of the tale sort of did initally -- fire the customer.

If the story is true, the client is a stereotypical know-it-all small business owner who gets by on bullying. You see them frequently in businesses that pay low-skill workers a small premium that is hard to replace. (ex: cleaning services, pool guys, mechanical contractors that do low-end maintenance work, etc)

As a contracted SME, taking a job like this is dumb. The chances of failure, where 'failure' == the server going down is high, and the customer will just stiff you.

jjice(10000) 2 days ago [-]

I'm a young developer, so I've never had the chance to work with on prem servers (and the chances that I will are looking slim), but I've always loved these 'war' stories.

m3kw9(10000) 2 days ago [-]

Mayo is around 1 part water to 1 part oil ratio..

y_tho(10000) 2 days ago [-]

It ́s 2008. A manager that just doesn ́t care anymore tells the new IT person to replenish the fan mayo.

-'Why? I don ́t know why. It just works.'

-'Is Hellmann's ok?'

IT person documents that Hellmann's is preferred.

Nextgrid(10000) 2 days ago [-]

I've once used cooking oil as a thermal paste substitute. Worked well enough and nothing went wrong.

dsr_(10000) 2 days ago [-]

The (probably soybean) oil is a fine lubricant, but the constant motion should cause the egg proteins to coagulate. How long did it operate before you replaced the fan properly?

Milank(10000) 2 days ago [-]

No downtime is acceptable, but they have only one server?

What if a technical failure happen? What if there's a fire in the server room? What if there is an earthquake and the building collapses? What if... many things can happen that can result in a long, long downtime with this tactics.

If uptime is so crucial, the system should be setup in such way that moving one server should be a peace of cake, not a spec-ops mission.

walrus01(10000) 2 days ago [-]

From an ISP perspective this seems like the sort of company that orders one $250 a month business DIA circuit (at a price point where there is no ISP ROI for building a true ring topology to feed a stub customer) and has no backup circuit. Then the inevitable happens like a dump truck 2km away with a raised dump driving through aerial fiber and causing an 18 hour outage.

Some circuits might average 5 to 7 nines of uptime over a year, but the next year is dump truck time... You can never truly be certain.

icedchai(10000) 2 days ago [-]

Never mind these less common scenarios... What do they do about Windows updates?

momokoko(10000) 2 days ago [-]

You'd be shocked how rare downtime is with modern hardware. A redundant power supply and SSDs in the right RAID configuration typically will not have any issues for years until it can be replaced by a newer model. Also, hardware monitoring is significantly improved to the point where you'll typically know if something will fail and can schedule the maintenance.

In the past power supplies and spinning disc hard drives would fail much more often.

It's basically a solved problem, outside of extremely mission critical, 5 nines kind of stuff, that we all forgot because of AWS.

HN ran, and may still run, on a single bare metal server.

YetAnotherNick(10000) 2 days ago [-]

You wrote one server but describe the failure modes of having one data center. I think it is very very uncommon and hard to allow for data center level issue. After all Instagram and 100 other site failed when one AWS data center went down. I would interested to know how/whether anyone's backend will work if any data center and its databases completely fails due to fire/earthquake/networking etc.

Second thing is having multiple machines for server. In theory it might help in increasing the availability but in practice I haven't seen any random issue due to machine which occurs just based on probability. I think almost all failure modes that exist, they are correlated between machines. eg suppose you have data loss on one machine, you could more likely than not, blame it on code and it would be similar across machines.

wastedhours(10000) 2 days ago [-]

We used to have one server for a website I was a content guy on - it was in a standard PC case, plugged into a switch in the IT team's office (this was not a tech-centered org).

The main IT guy went on holiday and one of the cover guys from another office decided to tidy up. He unplugged the server and thought (and told me after his thought process) 'if anyone was using it, they'll let us know'.

This was the one, single box for the whole website - no one else was monitoring (even though the central office had a proper, dedicated web team) and the assumption was I was sysadmin.

An hour later I'm sprinting down the corridor to find out what the hell happened and why I can't even SSH into the box.

We put a sticker on the case saying not to unplug it after that...

coldcode(10000) 2 days ago [-]

I worked at my last job for a place with a single rack mounted set of Windows servers at a data center - with no backup power supply, no backups of any kind for that matter, no UPS and no redundancy of any system, plus they didn't even have an admin for 6 months. The CEO refused to spend money on a 2nd anything. The company has 2000 employees. One server held all of the companies photos (which is basically the core of the business) and of course was not backed up.

misiti3780(10000) 2 days ago [-]

or even better, how do they apply OS patches?

galoisgirl(10000) 2 days ago [-]

> Should have been a 5 minute job if done correctly. Owner ended up paying for over 10 hours of work. Stupidest thing I've ever had to do.

You can see the common sense ship has sailed.

redwood(10000) 2 days ago [-]

Remind me of how IBM positions mainframes: they are so highly available that you simply never let them shut down.

gear54rus(10000) 2 days ago [-]

He should have taken it offline without notifying this brain-dead manager. Probably wouldn't have noticed lol.

And then charge for those 5 hours for good measure.

In general, this stupid trend of wanting 0 downtime makes no sense to me. If you're not NASA, police or other emergency service you 100% can afford a few hours of downtime with scheduling it be forehead.

PinguTS(10000) 2 days ago [-]

It's been done 7 years ago even using public transport.


jagermo(10000) 2 days ago [-]

The most dangerous part is them expecting the 3G to be available during the subway ride.

csours(10000) 2 days ago [-]

I really thought this post on HN was going to be that story. Thanks for digging it up.

jfcorbett(10000) 2 days ago [-]

Reminds me of the time where IT at a previous employer told us that due to a 'new IT strategy', our production cluster that had been sitting comfortably in the basement for years had to be moved to an 'approved IT hub facility'... in another office 500 km away and across the North Sea.

There was downtime.

Promptly after our cluster settled into this wonderful new facility, a cooling pipe in the ceiling leaked on it, frying 1/3 of our nodes.

yjftsjthsd-h(10000) 2 days ago [-]

On a personal selfish level I was quite happy to see our workloads moving to datacenters that we couldn't (reasonably) physically access, because it replaced 'can you go drive to the DC and replace a failing disk' with 'we put in the request for smart hands to replace the failing disk'. Of course, there's some notable tradeoffs, but it makes me feel better when the business decides to do such things...

Historical Discussions: Unbricking a $2k bike with a $10 Raspberry Pi (August 01, 2020: 962 points)

(962) Unbricking a $2k bike with a $10 Raspberry Pi

962 points 6 days ago by ptx2 in 10000th position

ptx2.net | Estimated reading time – 40 minutes | comments | anchor

A few years ago I splurged on an exercise bike. It's a pretty expensive item imho, both in upfront and ongoing subscription costs. But I was able to justify it by riding it all the time so that per-ride it wasn't that expensive. I'm happy to say this was a huge boon for my fitness and it was worth every penny.

The bike is the Flywheel Home Bike by Flywheel Sports. You open up the app, pick a class, and start riding. The app shows you a live video stream of an instructor, your position on the leaderboard so you can compete with the other people in the class, and your realtime stats like power (watts) and cadence (rpm). There's all the usual motivational badges that come with fitness apps these days, and each ride is logged so you can track your progress over time.

Or, that's how it used to work anyway.

Flywheel recently and abruptly shut down the Home Bike service following a legal battle with their competitor, Peloton. The bike does still work in that you can still pedal and adjust the resistance and technically get a workout. But the app is no longer so there are no classes, no competition, and no stats.

Flywheel customers left wanting a bit more had a few options:

  1. Swap it for a free refurbished Peloton. Join Peloton for the same monthly fee and take their classes instead. Not a bad deal and arguably an upgrade.
  2. Use the free LifeFitness ICG Training App. The Flywheel Home Bike is a rebranded LifeFitness IC5 so it just happens to work with this app. This gives you live stats like power and cadence and the ability to track your progress but doesn't provide any live classes or competition, nor is it officially supported.
  3. Add a set of power meter pedals ($). Use the bike with the massively-multiplayer online cycling simulator Zwift and other training apps.
  4. Reverse engineer the bike. Set its data free to be used with Zwift and other training apps. No power meter pedals required.

The rest of the post is a walk-through of my experience writing some code that enables the Flywheel Home Bike to work with Zwift and other training apps. It likely also works for the LifeFitness IC5 and support for other bikes should be easy to add.

The finished program is called Gymnasticon and the code is open-source on GitHub.

Project goals

The primary goal is to make the Flywheel Home Bike work with Zwift. It would be great if it also works with other cycling apps like TrainerRoad and Rouvy.

The solution should be easy to use for a non-developer and non-destructive to the bike.

The plan to get there is broken down into 3 parts:

  1. Write some code to get power and cadence data out of the bike's proprietary protocol
  2. Write some code to send power and cadence data to Zwift emulating a Bluetooth bike
  3. Put the two together into a final working solution

Part 1 - Getting data out of the bike

Only two pieces of information are needed to make a bike work with Zwift: power (watts) and cadence (rpm). Power enables Zwift to calculate your speed and position in the game. Cadence improves the experience of cadence-based workouts and enables Zwift to accurately animate your character.

We know from experience that this bike is capable of producing that information and that it communicates with the official Flywheel app using Bluetooth so the first step is to open up a Bluetooth service explorer and see what's available.

Bluetility on macOS shows the services offered by the Flywheel Home Bike.

The bike advertises a single service with two characteristics (aka data values). A web search for the service UUID tells us this is Nordic UART Service which is a custom service defined by Nordic Semiconductor. It allows sending arbitrary data back and forth, emulating a serial port.

The characteristics are named from the bike's perspective. To transmit data to the bike we write to the receive (RX) characteristic. To receive data from the bike we subscribe to the transmit (TX) characteristic.

Clicking subscribe on the (TX) characteristic shows that the bike is already sending some data. It is a bit hard to see the data here though. The short JavaScript program below uses the noble Bluetooth client library to connect to the bike and dump all the received data to the console so we can begin to analyze it.

 * Connect to the Flywheel Home Bike's Bluetooth UART service and log
 * received data to the console.
import noble from '@abandonware/noble';
import {on, once} from 'events';
(async () => {
  // nordic uart service and characteristics
  const uuid = '6e400001b5a3f393e0a9e50e24dcca9e';
  const rxUuid = '6e400002b5a3f393e0a9e50e24dcca9e';
  const txUuid = '6e400003b5a3f393e0a9e50e24dcca9e';
  // wait for adapter
  const [state] = await once(noble, 'stateChange');
  if (state !== 'poweredOn') {
    throw new Error(`bluetooth adapter state ${state}`);
  // scan
  await noble.startScanningAsync([uuid], false);
  const [peripheral] = await once(noble, 'discover');
  await noble.stopScanningAsync();
  // connect
  await peripheral.connectAsync();
  const {characteristics: [tx, rx]} = await
      [txUuid, rxUuid]);
  // start receiving
  await tx.subscribeAsync();
  const packets = on(tx, 'read');
  // exit on ctrl-c
  let exit = false;
  process.on('SIGINT', () => { exit = true; });
  // print all received data
  const start = new Date();
  for await (const [packet] of packets) {
    if (exit)
    const t = new Date() - start;
    const mm = `${Math.floor(t/60)}`.padStart(2, '0');
    const ss = `${t % 60}`.padStart(2, '0');
    const mmss = `${mm}:${ss}`;
    console.log(mmss, packet);
  // stop receiving
  await tx.unsubscribeAsync();
  // disconnect
  await peripheral.disconnectAsync();

Running the program, we get our first clear look at the data the bike is sending:

00:00 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 26 00 00 00 00>
00:00 <Buffer 00 00 00 00 00 00 00 0c 00 00 00 01 38 55>
00:01 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 26 00 00 00 00>
00:01 <Buffer 00 00 00 00 00 00 00 0c 00 00 00 01 38 55>
00:02 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 26 00 00 00 00>
00:02 <Buffer 00 00 00 00 00 00 00 0c 00 00 00 01 38 55>
00:03 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 26 00 00 00 00>
00:03 <Buffer 00 00 00 00 00 00 00 0c 00 00 00 01 38 55>
00:04 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 26 00 00 00 00>
00:04 <Buffer 00 00 00 00 00 00 00 0c 00 00 00 01 38 55>

Some initial observations on the data:

  • The bike sends two packets of data every second.
  • First packet is 20 bytes long, second is 14 bytes long.

Bluetooth LE 4.0 and 4.1 allow at most 20 bytes of application data per packet so this is likely a single 34-byte message sent in two chunks. A small change to the program joins the two chunks back together and adds a heading to make the output easier to read and reference.

Updated program output:

Offset         0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
00:23 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0c 00 00 00 01 1e 55>
00:24 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0c 00 00 00 01 1e 55>
00:25 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0c 00 00 00 01 1e 55>
00:26 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0c 00 00 00 01 1e 55>
00:27 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0c 00 00 00 01 1e 55>
(started pedaling)
00:28 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0d 00 00 00 01 2e 55>
00:29 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0e 00 00 00 01 2d 55>
00:30 <Buffer ff 1f 0c 00 21 00 14 00 00 00 00 04 58 00 95 00 00 00 00 00 00 00 00 00 00 00 00 0f 00 00 00 01 e1 55>
00:31 <Buffer ff 1f 0c 00 21 00 14 00 00 00 00 04 58 00 95 00 00 00 00 00 00 00 00 00 00 00 00 10 00 00 00 01 fe 55>
00:32 <Buffer ff 1f 0c 00 21 00 14 00 00 00 00 04 60 00 95 00 00 00 00 00 00 00 00 00 00 00 00 11 00 00 00 01 c7 55>
00:33 <Buffer ff 1f 0c 00 21 00 14 00 00 00 00 04 62 00 95 00 00 00 00 00 00 00 00 00 00 00 00 12 00 00 00 01 c6 55>
00:34 <Buffer ff 1f 0c 00 21 00 14 00 00 00 00 04 62 00 95 00 00 00 00 00 00 00 00 00 00 00 00 13 00 00 00 01 c7 55>
00:35 <Buffer ff 1f 0c 00 20 00 14 00 00 00 00 04 5e 00 93 00 00 00 00 00 00 00 00 00 00 00 00 14 00 00 00 01 fb 55>
00:36 <Buffer ff 1f 0c 00 1f 00 13 00 00 00 00 04 5e 00 90 00 00 00 00 00 00 00 00 00 00 00 00 15 00 01 00 01 c0 55>
00:37 <Buffer ff 1f 0c 00 1d 00 12 00 00 00 00 04 54 00 8c 00 00 00 00 00 00 00 00 00 00 00 00 16 00 01 00 01 d6 55>
00:38 <Buffer ff 1f 0c 00 1c 00 11 00 00 00 00 04 54 00 89 00 00 00 00 00 00 00 00 00 00 00 00 17 00 01 00 01 d0 55>
00:39 <Buffer ff 1f 0c 00 1c 00 11 00 00 00 00 04 4f 00 89 00 00 00 00 00 00 00 00 00 00 00 00 18 00 01 00 02 c7 55>
00:40 <Buffer ff 1f 0c 00 1c 00 11 00 00 00 00 04 4d 00 89 00 00 00 00 00 00 00 00 00 00 00 00 19 00 01 00 02 c4 55>
00:41 <Buffer ff 1f 0c 00 1b 00 11 00 00 00 00 04 4d 00 87 00 00 00 00 00 00 00 00 00 00 00 00 1a 00 01 00 02 ce 55>
00:42 <Buffer ff 1f 0c 00 1b 00 11 00 00 00 00 04 4b 00 87 00 00 00 00 00 00 00 00 00 00 00 00 1b 00 01 00 02 c9 55>
00:43 <Buffer ff 1f 0c 00 1a 00 10 00 00 00 00 03 4b 00 84 00 00 00 00 00 00 00 00 00 00 00 00 1c 00 01 00 02 ca 55>
00:44 <Buffer ff 1f 0c 00 19 00 0f 00 00 00 00 03 45 00 82 00 00 00 00 00 00 00 00 00 00 00 00 1d 00 01 00 02 df 55>
00:45 <Buffer ff 1f 0c 00 18 00 0f 00 00 00 00 03 45 00 7f 00 00 00 00 00 00 00 00 00 00 00 00 1e 00 01 00 02 20 55>
00:46 <Buffer ff 1f 0c 00 18 00 0f 00 00 00 00 03 43 00 7f 00 00 00 00 00 00 00 00 00 00 00 00 1f 00 01 00 02 27 55>
00:47 <Buffer ff 1f 0c 00 17 00 0e 00 00 00 00 03 3e 00 7d 00 00 00 00 00 00 00 00 00 00 00 00 20 00 01 00 02 69 55>
00:48 <Buffer ff 1f 0c 00 16 00 0e 00 00 00 00 03 3e 00 7a 00 00 00 00 00 00 00 00 00 00 00 00 21 00 01 00 02 6e 55>
00:49 <Buffer ff 1f 0c 00 15 00 0d 00 00 00 00 03 37 00 77 00 00 00 00 00 00 00 00 00 00 00 00 22 00 01 00 02 69 55>
00:50 <Buffer ff 1f 0c 00 15 00 0d 00 00 00 00 03 37 00 77 00 00 00 00 00 00 00 00 00 00 00 00 23 00 01 00 02 68 55>
00:51 <Buffer ff 1f 0c 00 16 00 0e 00 00 00 00 03 3c 00 7a 00 00 00 00 00 00 00 00 00 00 00 00 24 00 01 00 02 69 55>
00:52 <Buffer ff 1f 0c 00 16 00 0e 00 00 00 00 03 3c 00 7a 00 00 00 00 00 00 00 00 00 00 00 00 25 00 01 00 02 68 55>
00:53 <Buffer ff 1f 0c 00 17 00 0e 00 00 00 00 03 42 00 7d 00 00 00 00 00 00 00 00 00 00 00 00 26 00 01 00 02 13 55>
00:54 <Buffer ff 1f 0c 00 17 00 0e 00 00 00 00 03 42 00 7d 00 00 00 00 00 00 00 00 00 00 00 00 27 00 01 00 03 13 55>
00:55 <Buffer ff 1f 0c 00 17 00 0e 00 00 00 00 03 42 00 7d 00 00 00 00 00 00 00 00 00 00 00 00 28 00 01 00 03 1c 55>
00:56 <Buffer ff 1f 0c 00 17 00 0e 00 00 00 00 03 43 00 7d 00 00 00 00 00 00 00 00 00 00 00 00 29 00 01 00 03 1c 55>
00:57 <Buffer ff 1f 0c 00 17 00 0e 00 00 00 00 03 43 00 7d 00 00 00 00 00 00 00 00 00 00 00 00 2a 00 01 00 03 1f 55>
00:58 <Buffer ff 1f 0c 00 17 00 0e 00 00 00 00 03 43 00 7d 00 00 00 00 00 00 00 00 00 00 00 00 2b 00 01 00 03 1e 55>

Some more observations:

  • When not pedaling, it's almost all zeroes.
  • Upon pedaling, a few of the zero values change – a promising sign.
  • Some non-zero values (0, 1, 2, 33) remain constant.
  • Some values (27, 29, 31) are increasing monotonically at different rates.

Power and cadence are hopefully some of those values that become non-zero upon pedaling.

Finding the cadence

Our approach here is to record some data while riding at a known steady cadence (60 rpm) for 30-60 seconds and then search for that value in the data. We can repeat this process at a couple different cadences to verify and clear up any coincidences.

This metronome came in handy for keeping a steady cadence.

The first few seconds of data:

Offset         0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
00:00 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 35 00 00 00 00 00 00 00 00 00 00 02 d0 00 2c 00 5f 87 55>
00:01 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 2a 00 00 00 00 00 00 00 00 00 00 02 d0 00 2c 00 5f 98 55>
00:02 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 24 00 00 00 00 00 00 00 00 00 00 02 d0 00 2c 00 5f 96 55>
00:03 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 25 00 00 00 00 00 00 00 00 00 00 02 d0 00 2c 00 5f 97 55>
00:04 <Buffer ff 1f 0c 00 58 00 36 00 00 00 00 0c 35 00 f2 25 00 00 00 00 00 00 00 00 00 00 02 d1 00 2c 00 5f 33 55>
00:05 <Buffer ff 1f 0c 00 5a 00 38 00 00 00 00 0c 35 00 f5 25 00 00 00 00 00 00 00 00 00 00 02 d2 00 2c 00 5f 3b 55>
00:06 <Buffer ff 1f 0c 00 57 00 36 00 00 00 00 0c 49 00 f1 25 00 00 00 00 00 00 00 00 00 00 02 d3 00 2c 00 5f 41 55>
00:07 <Buffer ff 1f 0c 00 52 00 33 00 00 00 00 0b 49 00 ea 24 00 00 00 00 00 00 00 00 00 00 02 d4 00 2c 00 5f 5b 55>
00:08 <Buffer ff 1f 0c 00 4f 00 31 00 00 00 00 0b 40 00 e6 26 00 00 00 00 00 00 00 00 00 00 02 d5 00 2c 00 5f 42 55>

The recording begins and ends at standstill and otherwise should be approximately 60 rpm. Any offset with a max above 80rpm or below 50rpm or that doesn't start and end with 0 is a non-match and can be discarded. We're left with only two candidates:

It is clear from the chart that cadence is the uint8 at offset 12. A few extra tests at different cadences confirm this. There are non-zero values at offset 11 and 13 which confirms cadence is a uint8 and not the lower byte of a uint16.

This means the bike can report a maximum cadence of 255 rpm. What happens at 256 rpm? Does it wrap to 0 or clamp to 255? It turns out it's pretty hard to pedal that fast and so this remains a mystery.

Francois Pervis @ 260 RPM

Finding the power

The exact same approach could work here: ride at a steady known wattage for 30-60 seconds and then look for that value in the data. However I don't have a good way to know exactly what wattage I'm doing.

So this time we'll keep a steady cadence and just keep adding resistance every few seconds. The data should show a series of plateaus each higher than the last. The absolute values should also be within a reasonble range starting out around 100 watts and staying well under 1000 watts.

We are most likely looking for a 16-bit integer this time and we'll have to consider both big and little endian encodings.

The first few seconds of data:

Offset         0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
00:00 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 2d 00 00 00 00 00 00 00 00 00 00 01 b2 00 12 00 25 ba 55>
00:01 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 00 00 00 2d 00 00 00 00 00 00 00 00 00 00 01 b2 00 12 00 25 ba 55>
00:02 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 28 00 00 2d 00 00 00 00 00 00 00 00 00 00 01 b3 00 12 00 25 93 55>
00:03 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 28 00 00 2d 00 00 00 00 00 00 00 00 00 00 01 b4 00 12 00 25 94 55>
00:04 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 25 00 00 2d 00 00 00 00 00 00 00 00 00 00 01 b5 00 12 00 25 98 55>
00:05 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 25 00 00 2d 00 00 00 00 00 00 00 00 00 00 01 b6 00 12 00 25 9b 55>
00:06 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 29 00 00 2d 00 00 00 00 00 00 00 00 00 00 01 b7 00 12 00 25 96 55>
00:07 <Buffer ff 1f 0c 00 56 00 35 00 00 00 00 0b 35 00 f0 2c 00 00 00 00 00 00 00 00 00 00 01 b8 00 12 00 25 1c 55>
00:08 <Buffer ff 1f 0c 00 5a 00 38 00 00 00 00 0c 35 00 f5 2d 00 00 00 00 00 00 00 00 00 00 01 b9 00 12 00 26 1c 55>

Any offset with a max below 100W or above 1000W, or that doesn't start and end with 0 can be discarded. This time we're left with three possibilities: offset 3, 5 and 13.

The absolute values suggest that offset 3 is the current power in watts. Based on my effort, offset 13 starts out too high and offset 5 doesn't peak high enough.

So, what is the data at offset 5 and 13?

After some experimentation with the ICG Training App, I discovered that the bike has some hidden features not exposed by the Flywheel app. One of those features is that you can take a test to see the maximum power you can sustain over one hour of riding, known as Functional Threshold Power (FTP). The FTP result is stored on the bike and it reports your power as both watts (offset 3) and percentage of FTP (offset 5). The FTP% is frequently used in training programs. The default FTP stored in the bike appears to be 160 watts as when offset 3 ≈ 160, offset 5 ≈ 100.

The value at offset 13 is a very optimistic estimate of speed in km/h × 10. It could be that the rider weight defaults to 0.

As a bonus, when I was recording data I noticed that offset 15 is the position of the resistance dial ranging from 0 (easy) to 100 (hard). This was easy to see in the program's output when adjusting the resistance while not pedaling as it was one of the only values changing. It's not necessary for making the bike work with Zwift.

A bug in the bike?

Another thing I noticed during this testing is what appears to be a bug in the bike:

Offset         0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33  
02:21 <Buffer ff 1f 0c 01 7b 00 ea 00 00 00 00 33 6c 01 f6 3a 00 00 00 00 00 00 00 00 00 00 00 7e 00 06 00 0e 67 55>
02:22 <Buffer ff 1f 0c 01 92 00 f8 00 00 00 00 36 6b 02 05 3b 00 00 00 00 00 00 00 00 00 00 00 7f 00 06 00 0e 6e 55>
02:23 <Buffer ff 1f 0c 01 96 00 fb 00 00 00 00 36 6b 02 08 3b 00 00 00 00 00 00 00 00 00 00 00 80 00 07 00 0f 9b 55>
02:24 <Buffer ff 1f 0c 00 00 00 00 00 00 00 00 00 6b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 81 00 07 00 0f f1 55>
02:25 <Buffer ff 1f 0c 01 a2 01 02 00 00 00 00 38 6c 02 0f 3b 00 00 00 00 00 00 00 00 00 00 00 82 00 07 00 0f 5b 55>
02:26 <Buffer ff 1f 0c 01 97 00 fb 00 00 00 00 36 6c 02 08 3b 00 00 00 00 00 00 00 00 00 00 00 83 00 07 00 10 81 55>
                         |                        |        |
                       power                   cadence   resistance

The data at 2:24 shows a blip where power, resistance and some other values briefly drop to 0, while cadence is unaffected. Then at 2:25 they're back. It turns out there are several instances of this throughout the data.

What this means is that you could be riding at a very high effort and all of a sudden your power drops to 0 for a second. Not the end of the world but it has the potential to be annoying. It seems to happen once every few minutes.

Flywheel doesn't publish any details on how the bike works internally but the original manufacturer does hint at how power is calculated. From the LifeFitness IC5 page:

WattRate® Power Meter Displays a precise measurement of the user's effort in watts. This precision is achieved by a positioning sensor that measures the resistance applied to the magnetic brake system.

So the bike doesn't actually measure power like a power meter would. Instead, it maps the position of the magnetic brake to a point on some factory calibrated curve or lookup table.

One possible explanation then is that there's a hardware problem with the positioning sensor or, perhaps more likely, a firmware bug causing resistance to occasionally incorrectly read 0. The power value is derived from the resistance reading and so it also ends up as 0.

A simple fix is to just use the previous power value if ever: the cadence is non-zero and the previous power is non-zero and the current power is zero. A slight improvement is to keep track of the slope and factor it in when calculating the predicted value. If the bug occurs during a fast acceleration or deceleration that should give a slightly better result.

The message format

What we know about the message so far:

Offset         0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33  
02:23 <Buffer ff 1f 0c 01 96 00 fb 00 00 00 00 36 6b 02 08 3b 00 00 00 00 00 00 00 00 00 00 00 80 00 07 00 0f 9b 55>
                         |     |                  |    |   |
                       power   |             cadence   |  resistance
                             power%                  speed
Offset Description Type
3 Power (watts) uint16
5 Power (percentage of FTP) uint16
12 Cadence (rpm) uint8
13 Speed (km/h × 10) uint16
15 Resistance (percentage) uint8

We already have what we need but we can make some educated guesses about some of the rest of the packet.

The name "Nordic UART Service" hints that this protocol was intended for use on a real UART and if so the first and last few bytes would be for serial frame synchronization and error detection.

Offset Description Type
0 Start of packet marker (?) uint8
1 Length (of bytes to follow, excl. end) (?) uint8
2 Type of payload (?) uint8
26 Active duration (seconds) (confirmed) uint16
28 16-bit counter (distance?) uint16
30 16-bit counter (distance?) uint16
32 Checksum (xor of 0 and 1..31 inclusive) (confirmed) uint8
33 End of packet marker (?) uint8

One last small change to our program's output gives us a really primitive replacement for the Flywheel app's "free ride" function.

power cadence resistance
  23W   63rpm   0%
  24W   73rpm   0%
  26W   73rpm   0%
  27W   82rpm   0%
  29W   82rpm   0%
  30W   89rpm   0%
  32W   89rpm   0%
  33W   99rpm   0%
  35W  110rpm   0%
  41W  110rpm  15%
  47W  119rpm  15%
  57W  119rpm  20%
  66W  121rpm  19%
  74W  121rpm  24%
  80W  119rpm  23%
  84W  118rpm  24%
 100W  118rpm  32%
 115W  117rpm  35%
 136W  117rpm  40%
 160W  107rpm  41%
 177W  107rpm  41%

Part 2 - Getting data into Zwift

Now that we can get realtime Power and Cadence data from the bike, it's time to figure out how to communicate with Zwift.

The plan is to pick a bike that Zwift already supports and mimic it. The first step is to download Zwift and have a look over the website.

It turns out Zwift supports a lot of devices. It took some research to decide which one to emulate. We want to pick one that is easy to emulate and widely supported by other apps too.

From the Zwift website supported devices page:

There are many indoor bikes on the market that use proprietary communication channels. Zwift supports indoor bikes that broadcast power (watts) via ANT+ or Bluetooth Smart (BLE) using open standards.

After looking into ANT+ and Bluetooth LE, I decided to go with Bluetooth LE. ANT+ would have required extra hardware for no obvious benefit in this case.

Bluetooth LE Cycling Power Service

The Cycling Power Service is what we need to implement. The spec is available on the official Bluetooth website. It's very detailed. Here is the relevant summary:

Cycling Power Service (UUID 0x1818):
Cycling Power Feature (UUID 0x2a65) (Read)
Cycling Power Measurement (UUID 0x2a63) (Notify)
Sensor Location (UUID 0x2a5d) (Read)

The Cycling Power Feature and Sensor Location characteristics are constants that tell Zwift what extra features we have (cadence) and where the power measurement is being taken.

The Cycling Power Measurement characteristic is where the realtime data from the Flywheel bike goes. Power can go in as-is. Cadence needs to be provided in a different format. Rather than periodically telling Zwift an instantaneous cadence like "60 rpm" we need to send two values: the total count of crank revolutions (pedal strokes), and the timestamp of the last crank revolution, then Zwift will calculate the rpm itself.

noble, the Bluetooth client library we used earlier to talk to the bike has a companion library, bleno, for writing Bluetooth servers/peripherals, which we'll use here.

After some trial and error we end up at the following screen:

Sending random power and cadence data to Zwift over Bluetooth.

Part 3 - Putting it all together

We can talk to the bike and we can talk to Zwift. Now all that's left to do is combine the code from parts 1 and 2 and get live data streaming from the bike directly into Zwift.

The final program:

For the most part this went as planned however I did run into one issue.

Bluetooth LE connection parameters

Shortly into the first test ride on the Raspberry Pi the bike lost its Bluetooth connection. I figured maybe it was some wireless interference but it continued to happen.

Debugging at the application level didn't reveal any insights or useful error messages so I captured a btsnoop trace with btmon.

The excerpt below shows the disconnection reason. There is a relative timestamp (seconds) on the top-right of log entry.

> HCI Event: Disconnect Complete (0x05) plen 4                                #449 [hci0] 118.546006
        Status: Success (0x00)
        Handle: 64
        Reason: Unacceptable Connection Parameters (0x3b)

Scrolling back, there is about 30 seconds of successful communication:

> ACL Data RX: Handle 64 flags 0x02 dlen 27                                    #384 [hci0] 87.541516
      ATT: Handle Value Notification (0x1b) len 22
        Handle: 0x000b
          Data: ff1f0c0000000000000000000000002200000000
> ACL Data RX: Handle 64 flags 0x02 dlen 21                                    #385 [hci0] 87.543353
      ATT: Handle Value Notification (0x1b) len 16
        Handle: 0x000b
          Data: 0000000000000000000000003155

Then the bike asks to update connection parameters and the system rejects the request. The bike continues to send data for 30 more seconds then gives up and disconnects.

> ACL Data RX: Handle 64 flags 0x02 dlen 16                                    #386 [hci0] 88.440929
      LE L2CAP: Connection Parameter Update Request (0x12) ident 6 len 8
        Min interval: 16
        Max interval: 60
        Slave latency: 0
        Timeout multiplier: 400
< ACL Data TX: Handle 64 flags 0x00 dlen 10                                    #387 [hci0] 88.441103
      LE L2CAP: Connection Parameter Update Response (0x13) ident 6 len 2
        Result: Connection Parameters rejected (0x0001)

The connection parameters are used to control the tradeoff between data throughput and power consumption. The Texas Instruments BLE-Stack docs gives a good breakdown of how each parameter works.

So why is BlueZ rejecting these parameters and can we get it to accept them?

There are some debugfs endpoints that imply it is possible to control the range of acceptable connection parameters:


However I was unable to get them to have any impact. The connections continued to be created with unfavorable parameters and the update requests continued to be rejected.

Next, I tried an option in noble (HCI_CHANNEL_USER) to allow the application to handle these requests directly. This fixed the immediate problem but presented a new problem. HCI_CHANNEL_USER gives noble exclusive access to the Bluetooth adapter and prevents bleno from working. After reading over the noble and bleno code it didn't seem like there was an easy way to have them co-operate on a single exclusive socket.

After reading over the linux-bluetooth mailing list and some searching, I stumbled upon this GitHub issue where a poster mentioned they had success using hcitool to update the parameters on a connection.

For the Flywheel bike the command used is:

hcitool lecup --handle 64 --min 16 --max 60 --latency 0 --timeout 400

(Note: the min and max are in units of 1.25ms and timeout is in units of 10ms.)

The log below shows the above command successfully updates the connection parameters:

@ RAW Open: hcitool (privileged) version 2.22                                     {0x0006} 81.324864
@ RAW Close: hcitool                                                              {0x0006} 81.328082
@ RAW Open: hcitool (privileged) version 2.22                              {0x0006} [hci0] 81.329693
< HCI Command: LE Connection Update (0x08|0x0013) plen 14                      #388 [hci0] 81.331279
        Handle: 64
        Min connection interval: 20.00 msec (0x0010)
        Max connection interval: 75.00 msec (0x003c)
        Connection latency: 0 (0x0000)
        Supervision timeout: 4000 msec (0x0190)
        Min connection length: 0.625 msec (0x0001)
        Max connection length: 0.625 msec (0x0001)
> HCI Event: Command Status (0x0f) plen 4                                      #389 [hci0] 81.333532
      LE Connection Update (0x08|0x0013) ncmd 1
        Status: Success (0x00)
> HCI Event: LE Meta Event (0x3e) plen 10                                      #392 [hci0] 81.841057
      LE Connection Update Complete (0x03)
        Status: Success (0x00)
        Handle: 64
        Connection interval: 75.00 msec (0x003c)
        Connection latency: 0 (0x0000)
        Supervision timeout: 4000 msec (0x0190)
@ RAW Close: hcitool                                                       {0x0006} [hci0] 81.842147
> ACL Data RX: Handle 64 flags 0x02 dlen 27                                    #393 [hci0] 82.441717
      ATT: Handle Value Notification (0x1b) len 22
        Handle: 0x000b
          Data: ff1f0c0000000000000000000000002100000000
> ACL Data RX: Handle 64 flags 0x02 dlen 21                                    #394 [hci0] 82.591155
      ATT: Handle Value Notification (0x1b) len 16
        Handle: 0x000b
          Data: 0000000000000000000000003255

30 seconds later we're still connected and receiving data from the bike...

> ACL Data RX: Handle 64 flags 0x02 dlen 27                                   #453 [hci0] 112.441630
      ATT: Handle Value Notification (0x1b) len 22
        Handle: 0x000b
          Data: ff1f0c0000000000000000000000002100000000
> ACL Data RX: Handle 64 flags 0x02 dlen 21                                   #454 [hci0] 112.516077
      ATT: Handle Value Notification (0x1b) len 16
        Handle: 0x000b
          Data: 0000000000000000000000003255
> ACL Data RX: Handle 64 flags 0x02 dlen 27                                   #455 [hci0] 113.416650
      ATT: Handle Value Notification (0x1b) len 22
        Handle: 0x000b
          Data: ff1f0c0000000000000000000000002100000000
> ACL Data RX: Handle 64 flags 0x02 dlen 21                                   #456 [hci0] 113.566099
      ATT: Handle Value Notification (0x1b) len 16
        Handle: 0x000b
          Data: 0000000000000000000000003255

60 seconds later we're still connected and receiving data from the bike...

> ACL Data RX: Handle 64 flags 0x02 dlen 27                                   #525 [hci0] 148.516576
      ATT: Handle Value Notification (0x1b) len 22
        Handle: 0x000b
          Data: ff1f0c0000000000000000000000002100000000
> ACL Data RX: Handle 64 flags 0x02 dlen 21                                   #526 [hci0] 148.666003
      ATT: Handle Value Notification (0x1b) len 16
        Handle: 0x000b
          Data: 0000000000000000000000003255

Problem solved! Fortunately this was the end of the disconnects.

The dependency on hcitool is not ideal. I'd still like to know what was going on with the debugfs endpoints or how to otherwise influence BlueZ to accept these connection parameters. If you have any ideas let me know.

It is worth noting that the HCI socket interface used by noble and bleno is deprecated by BlueZ in favor of the D-Bus API.


The final step is to set this up on the Raspberry Pi Zero W to start on boot and restart on failure. Whenever the Raspberry Pi is plugged-in and within range, we can jump on the bike and start the Zwift app and everything should just work.

First, the node binary needs permission to advertise Bluetooth services:

sudo setcap cap_net_raw+eip /usr/local/bin/node

And the following systemd unit file takes care of starting the application at boot and keeping it running:


The first test ride in action. Note the bike has a convenient USB charging port that can power the Pi.

Zwift working with the Flywheel Home Bike with the help of a Raspberry Pi.


This has been in daily use for over 2,000 miles and works really well.

Zwift is a lot of fun.

TrainerRoad and Rouvy work but I haven't used them much.

The Raspberry Pi Zero W is very versatile and affordable. It's great that it can be powered by USB.

The Flywheel Home Bike is a very solid build quality bike. I'm glad to have found a way to reuse it.


  1. Figure out the Bluetooth messages that need to be sent to calibrate the bike. The first pass at this would be to use the ICG Training App to perform the calibration procedure while using Bluetooth developer tools on the device or a sniffing proxy.

  2. Add a motor to the resistance dial and implement the Bluetooth Fitness Machine Service so Zwift can control the resistance.

All Comments: [-] | anchor

gojomo(10000) 6 days ago [-]

Drat, I was hoping this would also have a recipe for unlocking/reinitializing the attached Android-based tablet, on some of these bikes. Then it could run other biking apps - or just provide reading/music while riding. (Maybe even: it could do the BLE translation?)

The community hasn't figured out how to root the tablet, yet, but there are some hints as to the manufacturer/boot-launch-software – https://www.reddit.com/r/FlywheelAnywhere/comments/gexqte/ha... – if anyone has any ideas or is interested in a challenge.

sagz(10000) 4 days ago [-]

I'm trying to create an Android recovery for a peloton, which is similar https://github.com/Goayandi/mediatek_mt8176_development/issu...

gkoberger(10000) 6 days ago [-]

That's horrible that Peloton was able to do this. Maybe there's more to the story, but it seems like the court allowed them to shut down a competitor for simply building a similar (relatively obvious) product?

In fact, the CEO of Peloton had the idea for Peloton... IN A FLYWHEEL CLASS. That's insane.

xsmasher(10000) 6 days ago [-]

Clicking through the Verge links it sounds like Peloton had a patent that Flywheel infringed on.

We'd need to do a lot more reading to have on opinion on that case.

syshum(10000) 6 days ago [-]

It is a classic example of 'on the internet' patent,

The problem is not Peloton persay, the problem is the Patent office allowing these obvious 'Thing in the real world -> On the internet' patents to pass right on through

gojomo(10000) 6 days ago [-]

I've heard from a Flywheel customer who's been following the case closely that the discovery phase was very damning for Flywheel – blatant theft of Peloton's corporate documents, including things like proprietary operating details, internal financials, projections, & future plans, with direction & understanding of Flywheel management/investors, feeding directly into Flywheel's plans.

So it was bigger than just a patent case, and Flywheel, caught red-handed, had no choice but to settle to Peloton's satisfaction.

sneak(10000) 6 days ago [-]

Peloton also engage in all manner of shady marketing, heavily astroturfing all purportedly "community" forums, and silencing criticism or negative comments.

I had one of their bikes, and sold it because the service was crap. I encourage people not to do business with this shady company.

mikece(10000) 6 days ago [-]

Awesome project. I wish more smart devices offered a "run it locally on a Pi if you're paranoid or a control freak." I avoid IoT devices in general because I have no idea where the data is being stored, if it's stored securely, or if it's being sold. I prefer to manage this all myself.

foxrob92(10000) 4 days ago [-]

>I wish more smart devices offered a "run it locally on a Pi if you're paranoid or a control freak."

Wouldn't that defeat the purpose of the 'smart device' (from the manufacturer's point of view)? The business model often seems to be locking you in to a subscription (rent seeking) or selling your personal information (surveillance capitalism).

vikramkr(10000) 6 days ago [-]

Reading into this - flywheel apparently stole patented ideas from peloton including streaming on demand exercise classes? And that's why their service is shutting down? How is that supposed to be a protectable/patentable concept? That sounds like some first class patent trolling at face value.

ponker(10000) 6 days ago [-]

It wasn't just patent infringement — they stole corporate documents and assets from Peloton Levandowski-style. There are other companies doing exercise streaming like Mirror which haven't had to shut down so I don't think this is some kind of massive patent roadblock for the industry.

ggm(10000) 6 days ago [-]

Was it really 'bricked'

askvictor(10000) 6 days ago [-]

No; this is hyperbole, and cheapens the meaning of the word bricked. It's core function still worked perfectly. The app no longer works. The app was always a subscription service. There was an offer to swap (for free) with another bike, which continues to let you pay to another subscription service for similar or better functionality.

I'm all for hacking your gadgets, and open APIs, but let's get a sense of perspective.

Johnny555(10000) 6 days ago [-]

This is the reason I bought a Keiser bike instead of a Peloton, it has a simple display that works with Bluetooth sensors (I use my watch to record my sessions). I use the Peloton app in an iPad (I could pair the bike with the iPad app, but don't).

The bikes are about the same price, but the Keiser screen is a simple LCD display that's much cheaper to replace if it gets broken and is fully usuable without a subscription.

Plus, I think the bike looks better.


troydavis(10000) 6 days ago [-]

I did the same and my Keiser M3i has been fantastic. Their GitHub repos are current and their lead developer actually answers implementation questions: https://github.com/KeiserCorp

When I'm considering buying a smart device that I'd be uncomfortable throwing away if/when the company ceases support, I ask them beforehand: Is there an open API and/or SDK? Does code or protocol escrow exist? This saved me from buying a smart watch and a smart ring. The manufacturers seemed healthy at the time but were bankrupt or acqui-hired and shut down within a year.

mttjj(10000) 6 days ago [-]

Hmmm. Both of you are getting me very close to pulling the trigger on the M3i. Just so I'm clear, you have to buy the "M Series Converter" bluetooth device as well so it can connect to 3rd party apps, correct? Or I could just record an Indoor Cycling workout manually on my Apple Watch, right?

userbinator(10000) 6 days ago [-]

It's interesting to see the discussion here focusing mainly on 'open from the producer/manufacturer' side, when what I think is the really important point here is that 'opening from the consumer/user' side can be easy and empowering: companies and services will come and go, but your ability to take control effectively depends only on your willingness to discover and explore.

I avoid a lot of 'smart' products in general, but feel comfortable with working on the equipment I do have --- whether it's maintenance, repair, or modification --- and I think that's the most important thing to keep in mind; to not be scared of treating things as anything other than mysterious black boxes. It seems that a lot of people treat 'reverse engineering' as some equally mysterious and imposing idea, when it's really just about problem solving or figuring out how something works.

Also, I don't think the RPi is necessary here; the bike is a Bluetooth device, so any computer with a Bluetooth interface can receive its data and process it. I'm not an RF expert, but rebroadcasting BT seems like it would create more interference.

ohazi(10000) 6 days ago [-]

> your ability to take control effectively depends only on your willingness to discover and explore.

This should be the limiting factor, and this is often the case when a product was designed sensibly and the interface just wasn't documented. Reverse engineering a device like this is relatively straightforward.

But more often, companies deliberately obfuscate, encrypt, or booby-trap their interfaces in order to actively prevent reverse engineering, and this is the reason for the frustration you're seeing in the other comments.

noisy_boy(10000) 5 days ago [-]

> your ability to take control effectively depends only on your willingness to discover and explore.

Lots of folks have additional constraints of time due to family/other responsibilities etc. I can afford to spend an hour or two if the API is open from the manufacturer's side to write a script or setup a simple service; I definitely cannot afford to spend time on packet sniffing or such low-level reverse-engineering.

usrusr(10000) 6 days ago [-]

> Also, I don't think the RPi is necessary here; the bike is a Bluetooth device, so any computer with a Bluetooth interface can receive its data and process it. I'm not an RF expert, but rebroadcasting BT seems like it would create more interference.

The rebroadcasting is done because the goal was to get the data into a proprietary piece of software that expects data to come in over Bluetooth.

And rebroadcasting will actually be just fine on the air medium because right after receiving is exactly the time when the bike won't send another message.

The RPi is complete overkill of course, a tiny $3 nRF52 module could do that job just fine.

jdechko(10000) 6 days ago [-]

Some interesting info here. I have a Garmin speed sensor and a "dumb" mag trainer (CycleOps Mag). Given the trainers known resistance curve, I was thinking about trying to come up with a script to approximate power based on speed.

Basically I want a home brew alternative for Zwift and TrainerRoad virtual power.

rconti(10000) 5 days ago [-]

That would be a cool project. Do you already know the resistance curve, or you're just saying SOMEONE knows it? I'd worry that it would be impacted by heat, but perhaps not significantly.

I've got a Peloton, have a Stages power crank on my road bike, and previously had Favero Assioma power meter pedals (highly recommended product BTW). I ran the pedals on the Peloton for a few rides to get a feeling for how close the calibration was. (surprisingly close, in my case).

noodlesUK(10000) 6 days ago [-]

It's a shame that more exercise bikes don't have open APIs. Zwift is an absolutely awesome way of keeping in shape if you like cycling, but the barrier to entry is that you need to own a bike and a bike trainer that are both pretty expensive. Maker projects like this one always make me happy, because it's repurposing an old piece of equipment to function just as well as a new one. Next step would be adding smart controls to the resistance ;)

emb-fit(10000) 6 days ago [-]

It's a $2k training bike. You could spend $1k and get a really nice brand new road bike and a smart trainer that controls resistance like a Wahoo Kickr Snap (or several others) for ~$500.

I just don't get why the Peloton thing is so popular when you can get a smart trainer and a bike you can actually take outside for sooo much cheaper. You could even sign up for Zwift and a Trainer Road subscription and come out waaay ahead of $50/month.

The protocols coming out of these things have become pretty much a standard as well. Get an ANT dongle for your computer and the data can be consumed from so many apps, even an open source project like Golden Cheetah. Or just read the data from a head unit that already supports it.

soared(10000) 6 days ago [-]

Open apis pretty much ruin honest competition, leaderboards, live races, etc. People already game metrics on their pelotons to get top 10 in races, and I don't think companies want to deal with that on a larger scale.

It's a great idea, but I don't think it's always the right idea.

fortran77(10000) 6 days ago [-]

It may have been possible or the company to post the API publicly after Peloton shut them down without incurring any further legal liability. That would have been a responsible thing to do...and a way of getting back at Peloton.

einpoklum(10000) 6 days ago [-]

> a $2k bike

Maybe that's your problem right there. Get a simple bike, leave the house (ok, I know there's Corona, I didn't say congregate), find someplace planar, and cycle in the real world.

In Amsterdam, a decent used bike will cost you the equivalent of 80 USD, maybe less.

jasonvorhe(10000) 5 days ago [-]

If he's using such a training bike with the intent of using a service such as Zwift, I'd expect the author to already have a decent road bike and that the training bike is more of bad weather, winter and easy access to fitness workouts without having to deal with traffic.

But even if he doesn't have a bike yet, he most likely would ride a road bike and you don't that for bucks.

I agree though, riding bikes is extremely underrated, no matter where you live.

syshum(10000) 6 days ago [-]

Amsterdam also has a large public cycling culture, with a public infrastructure to support public cycling

It also has a climate geared more towards outdoor cycling.

Not everyone has one or both of those factors, making indoor cycling more reliable, safer, and more enjoyable

arkanciscan(10000) 6 days ago [-]

An escalator can never break: it can only become stairs. You should never see an Escalator Temporarily Out Of Order sign, just Escalator Temporarily Stairs. Sorry for the convenience.

Mitch Hedberg

mschuetz(10000) 6 days ago [-]

They can fail quite spectacularly: https://www.youtube.com/watch?v=o1SjQfwLieU

anticensor(10000) 5 days ago [-]

We have a much shorter expression in my mother tongue, Turkish: we call stairs 'merdiven' and escalators 'yürüyen merdiven', so 'Escalator Temporarily Stairs' becomes 'yürümeyen merdiven'.

ohazi(10000) 6 days ago [-]

Products like this make me furious.

Selling a product whose interface/API/whatever is deliberately obfuscated so that the manufacturer also has a monopoly on a subscription service or an app for said product is blatantly anti-consumer, anti-competitive, anti-environment, and should be illegal.

Fuck Peloton. Fuck Flywheel. Fuck all the proprietary IoT companies.

And apparently fuck me for having the gall to want to control my air conditioner from my computer rather than GE Android app #12 that has God-knows-what baked in and that's going to be abandoned in two years anyway.

Nobody should ever feel like they have to throw out an otherwise functional refrigerator-sized appliance because of software obsolescence.

I am absolutely willing to die on this hill. We need a GDPR-sized hammer to fix this.

aranibatta(10000) 6 days ago [-]

trying to dig into the root of this frustration, i think it mostly stems from the misalignment for how we expect the internet to behave (open, accessible, all the other fun Tim BL stuff, etc). it's extremely unfortunate that companies like peloton, john deere, skydio, on some level DJI, and others as we cross software into the real world haven't kept up with those expectations. huge loss, and concerning behavior as the internet starts to expand further IRL.

bigmattystyles(10000) 6 days ago [-]

Just like the wink hub. They recently switched from free always to we now need 60 bucks a year from you. Talk about bait and switch.

Too(10000) 5 days ago [-]

The bike is not even close to being bricked. You can use it without the app, there are multiple replacement apps; both a free version from the same vendor, a equvalent subscription from the new owner and some other third party apps.

Otherwise yes, agree that subscription dependent IoT needs regulation.

DebtDeflation(10000) 6 days ago [-]

>GE app #12 that's going to be abandoned in two years anyway

Not merely abandoned, but abandoned with gaping security flaws, no other way to update the device, and an always-on internet connection. We really have not thought this IOT stuff through.

flingo(10000) 6 days ago [-]

> I am absolutely willing to die on this hill. We need a GDPR-sized hammer to fix this.

Any thoughts on a labelling system? Like, 're-flashable open source firmware', 'cannot be remotely bricked', 'API-spec and code examples included' stuck on the side of the box.

I'm surprised some hardware hacker supply company, like adafruit/sparkfun hasn't gotten on this yet.

lostlogin(10000) 6 days ago [-]

> app #12 that has God-knows-what baked in and that's going to be abandoned in two years anyway.

It was a revelation putting a Pihole on the network and making a firewall rule that forced any non-Pihole port 53 traffic back to the Pihole. Samsung and Google make a lot of connections to home.

CorruptVoices(10000) 6 days ago [-]

You must also hate Apple right?

squarefoot(10000) 6 days ago [-]

> Fuck Peloton. Fuck Flywheel. Fuck all the proprietary IoT companies.

And not just in the IoT field. Pretty much every company that could use their closed software/firmware/designs plus lawyers to enforce their right to render a product useless or obsolete, or simply becoming the only authorized to repair it then refuse the repair, so the user must buy a newer one, will eventually do that if allowed by the law.

John Deere has been really hostile to their customers, for example. https://www.vice.com/en_us/article/xykkkd/why-american-farme...

gorgoiler(10000) 5 days ago [-]

It has to be fixed through regulation. The free market does not support long term, expensive solutions, such as APIs.

If you provide the bike with a public API, it's going to cost more. You now need to actually test the API works, instead of simply testing the bike works, end-to-end, with your own software.

A stable API with supported version changes and backwards compatibility is a much more challenging engineering prospect.

Forever 'twas this. Maybe they can start by using something like an automotive CAN bus.

1vuio0pswjnm7(10000) 6 days ago [-]

The scariest thing IMO is that this strategy seems to work for the seller. (Besides John Deere oweners and the HN crowd) How many consumers are protesting this kind of nonsense? I would bet most are dazzled by the prospect of some 'futuristic' product feature.

To me, this stuff actually devalues the product. It has the potential to be useful but the way it is being used is to deny the buyer full ownership and reserve additional rights for the seller.

lsllc(10000) 6 days ago [-]

Agreed. Right to repair, which I believe includes software should become law. I have a pile of bricked stuff, mostly electronics that I can't use (hello Pebble Watch). I hesitate to buy 'connected' products unless they are open (e.g. Z-wave switches), never proprietary stuff!

userbinator(10000) 6 days ago [-]

I agree with your points, but in this case it was not obfuscated at all and as the article shows, not really difficult to figure out. Furthermore, this knowledge can now be shared with others.

As long as RE is legal (and even when it isn't...), people will figure things out.

We need a GDPR-sized hammer to fix this.

AFAIK, RE for interoperability (which this is absolutely an instance of) has always been and is likely to remain legal in the EU.

Balgair(10000) 6 days ago [-]

This twitter account is well aligned to this sentiment, proof included: https://twitter.com/internetofshit

pentae(10000) 6 days ago [-]

My condo building has these fancy new Technogym 'MyRun' cardio machines which are basically regular workout machines with giant iPads on them. So now instead of turning them on and starting a workout I must wait 3 1/2 minutes for the touchscreen to boot every morning and every other week theres a failed software update causing the machine to shut down mid workout to restart the software in an endless loop. Or it just won't turn on due to the software on/off switch not working and we have to call the technician. It's unbelievably bad. Unless you actually care about connecting your workout to an app (i dont) a prehistoric cheap workout machine from 20 years ago has a better experience and is actually maintainable

twodave(10000) 6 days ago [-]

I might even take it a step further and say no product should be rendered unusable without an Internet connection unless the Internet is an absolute requirement of the product's basic function. I.e even IoT devices ought to be able to be air-gapped and on their own closed network (unless, for instance, it's a clock or something and relies specifically on things like weather/traffic APIs). And even in cases where there is a legitimate product feature that requires an internet connection, not having one should NOT render the other parts of that product unusable.

juskrey(10000) 5 days ago [-]

Hey, we both know the answer - you don't need fancy app to be fit on your bike. Neither for making sense from your aircon.

m463(10000) 5 days ago [-]

I just return stuff.

I bought a segway mini years ago, only to find out you had to download an app before you could ride it.

Same with with a DJI mavic quadcopter.. you could not just turn it on and fly it with the included RC remote controller - it would just say 'see app' or some nonsense.

both got sent back. Seemed a little silly at the moment (I just got this cool new thing -- cave.. cave..) but I'm way better off.

(turns out the mavic sends details of EVERYTHING to lots of sites willy-nilly)

sokoloff(10000) 6 days ago [-]

Is it really that different than the razors-and-blades, inkjet printers, games consoles, or electric toothbrush business model? Or even the new cars and service model [wherein more money is made in the service bays than on the showroom floor].

rubber_duck(10000) 6 days ago [-]

>And apparently fuck me for having the gall to want to control my air conditioner from my computer rather than GE Android app #12 that has God-knows-what baked in and that's going to be abandoned in two years anyway.

Do you have IR controller for your AC ? If so you could get one of those smart plugs that you can control remotely (zigbee, wifi service, whatever) and it will communicate with your AC via IR, you plug it in to the AC socket and then plug AC into it.

dfox(10000) 6 days ago [-]

One issue with open interfaces for these kinds of applications is BT-LE itself. It is nominally designed as open standard for transferring exactly this kind of data, but it cannot do anything else than SNMP-like get/put/trap on some attribute, so everybody ends up designing their own proprietary L4 TCP-like protocol on top of attribute writes and reads to accomplish anything interesting.

pabs3(10000) 6 days ago [-]

What does the GDPR-sized solution look like?

Maybe 'every IoT vendor must put their source code in escrow and when they go out of business the source code becomes GPL'? Of course they could just escrow incomplete software with parts just being sourceless blobs and no-one would have verified it is complete by the time they go out of business. It also doesn't work when they buy proprietary software bits from other companies.

FpUser(10000) 6 days ago [-]

Generally yeah, fuck'em all. There are however cases when playing with the device using 3rd party software could lead to bypassing safety features and if people got hurt who is to answer?

postit(10000) 5 days ago [-]

And to think I'm used to make fun of RMS for complaining about how he wanted to run his own software on his microwave.

pabs3(10000) 6 days ago [-]

I would like to see a 'Right to Repair' for software.

thereisnospork(10000) 6 days ago [-]

Its a pretty good hill to die on. Everything IOT should be forced to communicate over transparent and self-documenting protocols so a. anyone can write an app to control said device and b.[0] Alexa/Cortana/Siri can query to set up a voice control interface.

[0] Maybe in the future when their capabilities get a bit better, but the gist is every device should respond to a 'hello' ping with a list of commands and NLP'able descriptions such that for an air conditioner 'alexa set temp 67 degrees F' just works.

aiisjustanif(10000) 6 days ago [-]

Just wait until you here about this company called Tesla that is paving the way for "your car is an IoT subscription". Oh yeah and this company called John Deere with their subscription model for their DMCA protected tractor that you "own".

dvcrn(10000) 6 days ago [-]

> And apparently fuck me for having the gall to want to control my air conditioner from my computer rather than GE Android app #12 that has God-knows-what baked in and that's going to be abandoned in two years anyway.

My aircon is infrared based so I bought a $20 infrared emitter, recorded all the button signals from the remote control and plugged it into HomeKit. Now I can voicecontrol/Remote control my antique aircon better than current models that still don't support HomeKit

Bonus points: I put the emitter in line with both aircon and TV and use the same device to control both

thedanbob(10000) 6 days ago [-]

I'm looking into mini splits for an apartment and project room I'm building in my shop. I really like the Mitsubishi ones partly because they have a documented port to connect to them and people have already written code and schematics. Unfortunately, they are twice the price of the cheap ones that only have a proprietary wifi dongle that I'd have to reverse engineer myself.

Zigurd(10000) 6 days ago [-]

There are other machines, like rowing machines, for which an open solution is hard to incentivise, since the machine, the app, and the sensors will all come from a single vendor.

I'm a little surprised, however, that Peloton, and Peloton-alikes ever happened because cyclists have training devices with open interfaces: Bikes, mounted on smart trainers that have standardized wireless links and protocols, connected to a choice of apps.

All it takes is taking the rear wheel off a bike. Or not even that for the most basic trainers, which clamp the rear axle and provide resistance to the rear tire. A fascinating case of a market segmentation that is less susceptible to being breached than one might think.

jasonvorhe(10000) 5 days ago [-]

I have one of those Tacx trainers where you just put your bike in. It's a hassle. You need to use their own quick release to hold the bike in place. The cord with the gear shifter (which sets the resistance) is always somewhere where it's annoying the rider and the shifting control itself doesn't fit on all handle bars. It's also really noisy.

I have been tempted very often to just buy one of these ready to use training bikes because of the frustration of: put some yoga mat cut-outs underneath the Tacx to suppress the vibration of the spinning weight, get the bike from the cellar, exchange the quick release, store the old one somewhere where it's not getting lost, put the bike in place by its rear wheel, place the front wheel in a fixed position, get a fan to put in front of the bike trainer, mount the gear shifter, which clumsily hangs on the handle bar because my handle bar is too big. (It's a standard road bike handlebar)

Did I mention this thing was really loud?

I'm also sometimes worried about the fixed pressure on my carbon frame but I think that's actually unreasonable.

Historical Discussions: Frances Allen has died (August 05, 2020: 819 points)
Remembering Frances E. Allen, first woman to win Turing Award (August 05, 2020: 2 points)

(863) Frances Allen has died

863 points 1 day ago by ntumlin in 10000th position

www.ibm.com | Estimated reading time – 3 minutes | comments | anchor

An Experimental Compiler for IBM's Advanced Computing System (ACS) became her next project. Fran designed and built the machine-independent, language-independent optimizing component of the compiler. The result was a tool to help drive the hardware design and a new way to analyze and transform programs.

This work led to Fran's seminal paper on Program Optimization, first published in 1966, describing a robust new framework for implementing program analysis and optimization as well as a powerful set of new algorithms. Fran's 1970 paper on Control Flow analysis introduced the notion of "intervals" and node dominance relations, important improvements over the control flow abstractions given in her earlier paper. Her 1972 paper, "A Catalog of Optimizing Transformations," identified and discussed many of the transformations commonly used today.

As important as distinguishing her work in the world of computing and programming, Fran was also committed to her team by embracing their ideas and synergies and, in particular, supporting women. She spent many years as a mentor through IBM's mentor program.

In addition to the Turing Award, Fran was awarded with scores of accolades and honors. Earlier this year, the Institute of Electrical and Electronics Engineers (IEEE) announced it will honor Fran with the IEEE Frances E. Allen Medal, to be awarded for the first time at the IEEE Honors Ceremony in 2022. IBM was instrumental in working with IEEE to create the medal in her honor. Fran would join dozens of other science luminaries who have been honored with eponymous IEEE Medals, IEEE's highest level of awards. "Professionally, Fran spent a lifetime working to advance the field of computing and pioneer new breakthroughs. Personally, she was equally focused on inspiring and motivating young people – especially women – to do the same," said Fran's nephew, Ryan McKee, on the IEEE honor.

In addition, Fran was a member of the National Academy of Engineering, the American Philosophical Society, and the American Academy of Arts and Sciences. She was a Fellow of the Association of Computing Machinery (ACM), the IEEE, and the Computer History Museum and has two honorary doctorate degrees as well as several awards for her work for women in computing. She has been inducted into the Women in Technology International (WITI) Hall of Fame and received the Augusta Ada Lovelace Award from the Association for Women in Computing.

When she wasn't exploring new computing opportunities, Fran's passions were climbing mountains and studying environmental issues. She was a member of the American Alpine Club and the Alpine Club of Canada, participating in exploratory expeditions to the Artic and on the Chinese/Tibet border.

In an interview with author Janet Abbate who wrote a book on female computer scientists, Fran reflected on her love for hiking and the mountains, and equated it to her career: "And, you know, it's somewhat of the same sort of thing: it's kind of challenging, and interesting; and how does one involve oneself in it? What capabilities does one bring to it that will make a difference?"

Frances E. Allen, an oral history conducted on August 2, 2001 by Janet Abbate, IEEE History Center, Hoboken, NJ, U.S.A.

The IEEE History Center has a collection of more than 800 oral histories in electrical and computer technology which can be accessed, here: http://ethw.org/Oral-History:List_of_all_Oral_Histories

All Comments: [-] | anchor

jzig(10000) about 22 hours ago [-]

Why doesn't the black bar link to the related submission?

RankingMember(10000) about 22 hours ago [-]

That's a great idea

tus88(10000) about 21 hours ago [-]

That's too Web 2.0 for HN XD

filereaper(10000) 1 day ago [-]

@dang can we get a black bar in her honor?

LargoLasskhyfv(10000) 1 day ago [-]


hardwaregeek(10000) 1 day ago [-]

I'd recommend reading her interview in Coders At Work. I never realized that compilers were already a flourishing field by the time C came around, and that C actually ended up having some negative effects in compiler dev.

AnimalMuppet(10000) about 18 hours ago [-]

How did C have negative effects on compiler dev? Did everyone else get a lobotomy, and forget how to do what they already knew?

wglb(10000) 1 day ago [-]

It was more that C hindered optimization research. The interview was awesome as you said.

gavreh(10000) 1 day ago [-]

Looks like a great book, will read - thanks for the recommendation.

chaostheory(10000) about 23 hours ago [-]

Going on a tangent, I think most of us missed Bill English's passing a few days ago since I think because people only noticed it on the weekend when there are less HN users active.

dang(10000) about 23 hours ago [-]

It was mentioned in this article: https://news.ycombinator.com/item?id=24059231, but not in an obvious way.

sanxiyn(10000) 1 day ago [-]

Frances Allen wrote 'A Catalogue of Optimizing Transformations' in 1971. 50 years(!!!) later, they are still the backbone of optimizing compilers. I think the only major thing missing is autovectorization.

She was in her 30s when she wrote it.

sanxiyn(10000) 1 day ago [-]

I think you should read it if you work on compilers, even now. But the reason is a bit different.

It was the survey of the state of the art at the time, but obviously it is not the state of the art now. Then why should you read it? Because it is written in two layers.

The first layer goes, we tried many optimization ideas, but only these were effective in practice: inlining, register allocation, etc. Others were not. Surprisingly, this layer is still mostly true today! This is both happy and sad depending on your view. Personally I think it testifies that compiler is a mature field, and it matured by 1970. (And that Frances Allen did lion's share of work maturing it.)

The second layer is, so here is how you should do inlining, register allocation, etc. While this layer is also full of gems, it is necessarily badly outdated. The paper predates graph coloring register allocation, for example. On the other hand, ironically, the state of the art 1970s algorithms are often a good choice today when you want an algorithm that is fast and low memory. (Ironic, because they were slow and high memory at the time!) This doesn't apply when there is an important new concern, for example cache locality, but happily it mostly doesn't affect compiler.

I think there should be a project to write the-state-of-the-art-in-1970s compiler. It would be a great simple and minimal alternative to GCC and LLVM, and it would also work great as a JIT code generator. We probably should name it Fran.

raverbashing(10000) 1 day ago [-]

Did the concept of SIMD even exist in 1971?

sitkack(10000) 1 day ago [-]


Optimizing Compilers for Parallel Computers, lecture by Frances E. Allen https://www.youtube.com/watch?v=Qv-wXcUxrmE

Frances Allen, 2006, ACM A.M. Turing Award Lecture, 'Compiling for Performance: A Personal Tour' https://www.youtube.com/watch?v=NjoU-MjCws4

musicale(10000) 1 day ago [-]

She also spoke at the acm turing100 symposium in 2012.


relaunched(10000) 1 day ago [-]

I'd love to see a black banner / bar in her honor.

fb03(10000) 1 day ago [-]

I am rooting for this also.

hardwaregeek(10000) 1 day ago [-]

Agreed. Any Turing Award laureate, let alone the first female Turing Award laureate, deserves a black banner

anotherevan(10000) 1 day ago [-]

As an aside, when there is a black bar, I would also love it to link to the reason it is there.

DoreenMichele(10000) 1 day ago [-]

First female IBM fellow.

First female winner of the Turing Award.

Lots of other notable stuff.

Sadly, this is the first I've heard of her. Hopefully all that means is I'm not a real programmer.

Edit: To be clear, I really meant 'I hope other people here are familiar with her work, even though I am not because I'm not a real programmer.' I'm happy to see that some people are, in fact, familiar with her and her work.

globular-toast(10000) 1 day ago [-]

Interesting. If you search for 'died' on HN in the last month, say, you'll find many examples. Most have 0 or 1 comments, many have around the same as this has now, but none are about it being sad to not have heard about that person. Any idea why this one in particular made you feel that way?

CarbyAu(10000) 1 day ago [-]

And it doesn't mean even that.

Many worthwhile people in the world. Not all of them are famous to the people who would love to know them.

At least we can both get to know her more starting now.

grappler(10000) 1 day ago [-]

That sounds like black bar level stuff

belorn(10000) 1 day ago [-]

I create programs professionally and I know very few winner of the Turing Award by name. If someone said 'who was the first native-English speaking Turing Award winner' I would not be able to answer unless I went through each winner and manually checked. I suspect quite few programmers in the world (and people on HN) are familiar enough with Turing Award winners that they can answer questions like that.

I would guess that people who work on building compilers and compiler optimization are more likely to know her name than programmers.

mumblemumble(10000) 1 day ago [-]

> Hopefully all that means is I'm not a real programmer.

Sadly, no, it's not just that. Most my immediate colleagues, for example, don't know of her or her work, either. It's one heck of a field.

She had expressed some dismay, in interviews, at being the first woman to win the Turing Award. Not the Turing Award part, of course, the 'first woman' part. She was far from being the first deserving candidate who didn't happen to be a dude. So I hope she wouldn't mind linking this here, even today: https://www.hillelwayne.com/important-women-in-cs/

graycat(10000) 1 day ago [-]

Fran and I were on the same floor at IBM's Watson lab. I was in an AI project doing applied math, e.g., some optimization, and mathematical statistics (for the AI work we were doing, system monitoring, i.e., anomaly detection, better than our AI work!). She was regarded as a major expert in compiling and numerical codes for scientific computing.

I heard that she was working on a software product that among many other things would do fast matrix multiplication using some parallelism.

So, just for the heck of it, I wrote and ran a little routine in PL/I that used PL/I's feature of multi-tasking to get some parallelism and showed my code to her. She was a little surprised I'd written the code, had a smile, and explained why her work closer to some hardware features (I don't recall the details) would be faster!

I wasn't surprised or disappointed that my little PL/I tasking code would be slower than what she was doing, but at least I got her to explain the hardware she was using and how she was exploiting it!

As I recall, she was married to Jack Schwartz at Courant Institute of NYU and as in

Nelson Dunford and Jacob T. Schwartz, Linear Operators.

flippinburgers(10000) about 12 hours ago [-]

Thank you for sharing! I always enjoy hearing stories about how great minds act when interacted with! She sounds like she was a pleasure to work with.

jameshart(10000) 1 day ago [-]

I'm sure you are just trying to share a memory of interacting with an illustrious colleague here - and my sympathies go to you as it's always a shock to learn of the passing of someone you worked with.

But I think you maybe need to work on how you present this anecdote - as it is, it reads like you tried to mansplain her own research to a Turing Award winner. I hope you approached with more humility than this telling suggests?

Also, you should be aware that contextualizing professional women in terms of who their husband is or what his credentials are has long been used to underplay women's individual achievements. Again, I don't think that's your intent, but you could consider whether, in the case of talking about Jacob Schwartz, you would have been moved to drop in the detail of who he was married to?

Historical Discussions: Show HN: I built an After Effects for dummies (August 04, 2020: 832 points)

(837) Show HN: I built an After Effects for dummies

837 points 3 days ago by michaelaubry in 10000th position

storycreatorapp.com | | comments | anchor


Your podcasts need attention. You have no way of promoting on social media without video. Story Creator makes it easy to create a beautiful audiogram.


You found an awesome product that you want to sell, awesome. Now you need to drive sales. Use video content to drive traffic and make sales.


You have a message and you want to impact lives. There is a formula for grabbing attention and communicating your message. Video is a large part of that formula.

Course Creators

Promote your courses or share teasers from existing courses. Using video can also help you offer your audience value through daily tips.

Start Ups

You're moving fast and you need to gain attention. Everytime you have a feature update you can engage your audience and keep them up to date with stylish video content.

All Comments: [-] | anchor

switchstance(10000) 3 days ago [-]

Any major differentiators from https://offeo.com?

michaelaubry(10000) 3 days ago [-]

Never heard of them. Just checked out the site, it looks awesome.

Are you a user?

dna_polymerase(10000) 2 days ago [-]

Looks great, but looking at the pricing page makes me wonder, what is this Video limit. Do I get to edit 20 videos for $19 per month, or what? Why would you limit the number of videos if storage is already limited? Maybe add little help icons next to the points that clarify the limits.

michaelaubry(10000) 2 days ago [-]

Great feedback. I think you're right it could use some more context.

I think FAQs would also be nice.

seishan(10000) 3 days ago [-]

This is impressive! You've sold me on the product, and I'm excited to see how this product develops over the coming months and years.

michaelaubry(10000) 3 days ago [-]

Reach out to me on Twitter @michaelaubry

jasonshapiro(10000) 3 days ago [-]

Kudos! You have to admire anyone for executing on a launch this well as a solo dev. SaaS products targeting the more casual professional have a huge potential to grow right now.

IMO the killer feature will be the seamless import/export of content from phones to Web UI to social media account. Any platform allowing a frictionless experience in this regard will open the market to the much broader audience, which I think you're well branded for.

I'd love to hear more about your experience building the rendering stack with ffmpeg - and whether or not you recommend using it.

michaelaubry(10000) 3 days ago [-]

Instagram makes it super difficult to post on the platform. If you're a partner then I think it's possible.

It's something I would like to do. If anyone knows folks inside Instagram connect me Ill ship it.

I am trying to have an agnostic approach and focused on leaving things in the cloud as much as possible without too much extra upload/download time.

saadshamim(10000) 3 days ago [-]

out of technical curiosity, how do you render the video? is the whole thing like a canvas element?

michaelaubry(10000) 3 days ago [-]

Combination of several rendering techniques.

The idea is as long as you have all the information the user wants in in each layer. Then you can do a lot of things with this understanding.

The major pieces of information you need to collect while working on a 2D plane are these.

- The x,y coords - The width and height - A path to the asset - additional properties like colors and opacity

From having this critical information stored nicely means you can email it to a graphic designer and have them decode it and follow the 'map'.

You can send it to a client side process and have it interpret the information.

You can send it to a server and have it interpret the information.

You can do a one shot kill and build a system that pieces them together using one approach.

You can send each layer to a different process. You can create a specialized technique for each layer type.

The key is collecting the information. The actual rendering can be done in many ways. FFMPEG, canvas, screen shots, send it to a human lol. Using OS commands, etc.

bitcoinmoney(10000) 3 days ago [-]

Awesome product. Seems like I could use this.

michaelaubry(10000) 3 days ago [-]

Let me know if I can help. Hit me up on Twitter or send me an email. Ill work with you and get you set up :)

radley(10000) 3 days ago [-]

So, Adobe Spark?


Torn(10000) 2 days ago [-]

OP's app looks more like Canva (https://www.canva.com, which Spark is also a copycat of), especially the icons in the left nav. The timeline view is super neat though.

igorstellar(10000) 3 days ago [-]

To be fair, as someone who doesn't know what it is, I clicked 'Watch Video' on Spark website and that video did not show me what it is and how to use the product but rather some nonsense-story. Story creator website showed me exactly how the app looks like and how to use it right away without having me to click anything. Awesome work, bookmarking it for future use!

earthnail(10000) 2 days ago [-]

This looks amazing, congrats!!

One quick note: on your features page, the trim section has the same text as resize gifs. You pasted the wrong copy there ;).

Again, congrats on your great work!

[EDIT] found one more: when I use your demo, the onboarding tutorial stops after step three. Regardless of whether I click next, click 'skip', or drag the playhead marker, I never get shown step 4 out of 6.

michaelaubry(10000) 2 days ago [-]

Great catch. Yeah I think for Step 3 or 4 it's looking for a CSS class selector I removed. Need to buff that one out. Its going in the backlog. Thanks for letting me know. You rock!

boothead(10000) 3 days ago [-]

Whoa, very cool!

michaelaubry(10000) 3 days ago [-]

Thank you!

4theBroken(10000) 2 days ago [-]

Isn't this the same as Adobe Premiere?

michaelaubry(10000) 2 days ago [-]

It has aspects of premiere. Sort of a merger between the two. It's tailor made for social media content. Which means adding some motion graphics with the ability to trim audio and video.

jacob_rezi(10000) 3 days ago [-]

Looks awesome, Michael! Looks like we did the exact same thing using the Ouch Scribbles. Take a look - https://www.rezi.io/

michaelaubry(10000) 2 days ago [-]

HAHA good choice. Rezi looks very cool.

kylejtorres(10000) 3 days ago [-]

So cool to see this on HN. I've used the product to make a lot of my social media posts. It's so easy to use and makes me look like way more of a pro than I actually am. Keep up the great work!

michaelaubry(10000) 2 days ago [-]

Kyle the man. The first tester and had some UX input that was super valuable. Thanks for sharing the love.

I want to see you back on the product making more videos soon!

harryf(10000) 3 days ago [-]

Looks great but trying the tour on my iPhone was troublesome - feels like it wasn't really designed to be used on a phone. Which is a shame if true, because that's where all the social media influencers etc live

michaelaubry(10000) 3 days ago [-]

Fair point.

There is definitely a wide audience on desktop. According to my analytics. Most of the traffic came from desktop. At best it got to 50/50.

That said. I am focused on nailing this on mobile as well, once I get some resources.

I have used all responsive techniques. So the bones are there I just need to give it a little love :)

I agree itll be killer to produce on the go.

davjhan(10000) 3 days ago [-]

This is insane. The home page is super clear and the product itself is very slick to use. Good job on defining the right vision on the product and prioritizing the exact set of features to launch with to solve the specific niche problem space (podcast/video promotors on instagram). You seem to have a good handle on feature creep.

michaelaubry(10000) 3 days ago [-]

I love it. It's funny because I think I spent way too much time on this relative to the advice I was getting.

If I didn't have great people in my life then I would be stuck on a never ending feature rabbit hole.

When I start my next company I plan to have more balance between MVP, user interviews, pre sales, business metrics, market research, etc.

This venture was purely intuition and a major desire to build this :)

The great thing is I love building this product.

ricahe2559(10000) 3 days ago [-]

This looks great, Michael! Do you take individual screenshot on a headless browser for all layers and stitch them using FFMPEG?

I played around with FFMPEG to when my company needed dynamic videos (ie. They wanted to create videos like the ones Facebook automatically generates). And that's at least what I did.

michaelaubry(10000) 2 days ago [-]

Yeah thats a great approach.

miguelmota(10000) 3 days ago [-]

Awesome demo! It's really intuitive to use.

michaelaubry(10000) 3 days ago [-]


I have plans to make it even easier. I think for some users an interactive setup would be useful.

lefrenchy(10000) 3 days ago [-]

This is an awesome looking product, I think it will really take off for small shops that want to be able to quickly put out content. Nice work, wishing you the very best!

michaelaubry(10000) 3 days ago [-]

Yeah thats the goal. Let me know if you know anyone who can find value. Always happy to have a conversation.

Sreyanth(10000) 2 days ago [-]

Two things that drive product value & growth insanely:

    1. Dogfooding your own product
    2. A free-tier pricing that actually helps one-time users
You nailed both of them. Congrats on the launch. Using it right away. :)
gamerfreakish(10000) 1 day ago [-]

What is dogfooding?

Kye(10000) 3 days ago [-]

Mowgli is a very good dog.

michaelaubry(10000) 3 days ago [-]

He's the best.

tiffanyh(10000) 3 days ago [-]

I've been a happy paying https://clipchamp.com/ User for sometime but will have to give this a try.

Congrats Michael.

People underestimate how hard it is to build and ship. So massive kudos to you on your new product.

michaelaubry(10000) 2 days ago [-]

Not a rocket ship but not a baloney sandwich either haha.

I have vaguely heard of them. Cool name. Id love to learn your use case, pain points, and what I need to do as a product creator to make a significantly better product.

If you're interested I would like to get inside your brain. Hit me up on Twitter @michaelaubry or email [email protected] . com

grativo(10000) 3 days ago [-]

This is a really great product! You did this with a lot of heart! I saw your most recent YouTube tutorial and the ending proved that you are very dedicated and driven to this. Only big things from here on out! Great job once again!!

michaelaubry(10000) 3 days ago [-]

Ah man. Thanks for checking out the videos haha. Trying to educate folks and explaining things always helps you deepen your knowledge on a thing. So a major win-win. I think it's also been helping my SEO.

What about the end stood out? Which video?

bgdam(10000) 3 days ago [-]

Dude this is fantastic. Also you really need to charge more. Start by doubling the prices for all the plans. It's easily worth it and people will pay.

caseyf7(10000) 3 days ago [-]

Also consider lowering expectations for support. The current pricing won't absorb many high maintenance customers abusing your support line.

timdorr(10000) 3 days ago [-]

I completely agree. I'd love to pay for this on a per-seat basis. $50-100/mo/seat seems about right here.

I also don't know if including storage in the pricing is helpful. It might make more sense to represent that in time. If this is being sold to users too inexperienced to use AE, then the idea of bitrates and pixel resolutions probably will be foreign to them. But time is something everyone knows and can easily reason about.

You might also experiment with completely removing that restriction. Storage is cheap and hosting video isn't really a huge value add. The features and functionality are the real selling point, so price on those things and the real value you offer to users.

hanspeter(10000) 2 days ago [-]

This product will most likely also sell with a higher price, but for an initial launch I would recommend keeping prices low.

What you want is to learn if your product is something people are willing to pay for. Once that is established, you can tune the pricing to increase the revenue.

ChicagoBoy11(10000) 3 days ago [-]

Bro I tried the demo and on the surface this looks insanely well executed. If my job goes remote next year (I'm a school admin/support person, so if that's the case I'll have a lot of downtime), you wanna take on an unpaid intern just in exchange for learning from this?

michaelaubry(10000) 2 days ago [-]

Im from near Chicago. I love the excitement. Hit me up on Twitter DMs @michaelaubry

looperhacks(10000) 2 days ago [-]

This looks like a really great product, but scrolling the landing page is laggy AF (Firefox). Doesn't make me want to try it out.

michaelaubry(10000) 2 days ago [-]

Got it. Sorry about that. Hopefully youll check it out in the future.

alecfreud(10000) 3 days ago [-]

great, here come more Gary V style videos...

kidding aside, this is awesome. excellent work.

michaelaubry(10000) 3 days ago [-]

Flood the internet with Gary Vee - as if it weren't already.

mmckelvy(10000) 3 days ago [-]

Looks great! Question regarding the features. Do I need to record my videos using something else or can I do the actual recording using your product?

michaelaubry(10000) 3 days ago [-]

HAHA no one has asked me this.

I have this built out and hooked up to S3 I disabled it as I thought users wouldnt want it.

In the early days. I would use the webcam, record me talking. Then caption it on the spot, add some letterbox text and publish to my Instagram story.

I might have to reveal it and bring it back. This is of course if people want it and it solves a pain point worth paying for.

BHSPitMonkey(10000) 3 days ago [-]

Nice work! FYI, on https://storycreatorapp.com/features you've accidentally used the same caption/copy for the last two features.

michaelaubry(10000) 2 days ago [-]

Hahaha completely overlooked that. Thanks for letting me know. Fixing it RN :)

ivanvanderbyl(10000) 3 days ago [-]

This is really cool, love the niche this fits into — and for other future founders, there's a lot of low hanging Adobe apps which could follow this path: Audition? Acrobat?

One thing I would suggest: increase your prices, at least at the top end. $83/month for the Business plan is way too cheap. At least put an extra zero on there.

michaelaubry(10000) 3 days ago [-]

Good points.

Im always on the side of charging more where it makes sense. Still learning the market.

I am doing B2B integrations where they get access to the tool embedded in their product and an S3 link sent to their system for $20k annual.

This breaks down to $1.6 monthly.

trilinearnz(10000) 3 days ago [-]

This is really impressive, and a great example of the modern web put to productive use. I especially liked how you could start messing around with the demo without having a registration prompt (you only need that when exporting, which I think is perfectly reasonable).

I wish you the best of luck with your venture!

michaelaubry(10000) 2 days ago [-]

Thanks I just added that. I think it's good to give users the ability to see what's its all about before committing.

berkayozturk(10000) 2 days ago [-]

Incredible work! Can I ask which technologies you used to build the editor? Wappalyzer detects Next.js, React and Vercel.

tessela(10000) 2 days ago [-]

The timeline looks nice, I wonder if it is opensource.

jnfr(10000) 3 days ago [-]

Congrats Michael! Your hard work over the last few months is evident. I wish you only the best from here onwards!

michaelaubry(10000) 3 days ago [-]

Thanks for the love and support. Just getting started!

pier25(10000) 3 days ago [-]

This is super impressive. Congrats!

Would love to know more on the frameworks you used, infrastructure, etc.

michaelaubry(10000) 3 days ago [-]

Yeah nothing unordinary.

React.js on the FE with a lot of custom components. Used styled components. Used Apollo and zustand.js as global state/cache management.

Node.js with FFMPEG binary as a way to process videos

Prisma2, GraphQL and postgresql for data

Vercel for hosting

maximevoisin11(10000) 3 days ago [-]

Amazing ! Can you make it collaborative ? If yes, then you have a 'Figma for video editing' : you made After Effects online and collaborative. Huge ! (and then I'd want to invest ;) )

michaelaubry(10000) 3 days ago [-]

Yes and I know exactly how to do it.

- is it a part of my vision: yes - does it get me excited: HELL YES - do I understand if there is a need in the market: no - do I currently have the resources to build it: no

I'd like to first secure some runway. I've been bleeding out for awhile.

Second, I would like to talk to users and if this is a serious problem worth solving and not just something that is cool. Then I am all for it ready to hit it hard.

Some things that get me pumped for the future.

- realtime collaboration - simple keyframe animations - vast library of templates - guided/interactive video creation - API for other developers and organizations to create video

techsin101(10000) 3 days ago [-]

as a developer i am curious about how it works? do you use ffmpeg? do you upload all videos on server?

michaelaubry(10000) 2 days ago [-]

FFMPEG is a huge component. There isnt a silver bullet there are many moving parts.

Essentially FE handles collecting information. BE handles parsing information and using various rendering techniques to stitch together the pieces.

nishanth_v(10000) 3 days ago [-]

In the pricing section I see there's a limit on videos. Is this a monthly limit or total limit. For example, free tier says 5 videos, is that 5 per month?

michaelaubry(10000) 3 days ago [-]

This is per month.

I am still experimenting with the pricing and do have paying customers.

I find the average user needs anywhere between 5 videos a month to 10. These are serious users who are consistent with their posting and are making a return.

Those who are learning and new to the content game can get a few videos rendered per month to see if it's for them.

Agencies are a different persona though and they would most likely need more than 10.

pdxandi(10000) 2 days ago [-]

This looks amazing, well done! Really intuitive and easy to use.

A minor suggestion: the phrase 'Free Demo' on the landing page sticks out to me and feels awkward. I think maybe changing it to 'Try' or 'Demo' would be more fitting. Adding 'Free' seems unnecessary and distracting.

I also think it should be highlighted, maybe changing it to blue text. Or maybe green.

Just my opinion.

michaelaubry(10000) 2 days ago [-]

Great feedback. I had demo before some people said it made it feel like an enterprise product where demo was booking a sales demo. So I tried to make it clear it wasnt that.

Maybe try sample or something could be more fitting.

What do you think?

jychang(10000) 2 days ago [-]

I showed this to a youtuber friend with 4m subscribers. His first response was 'looks easier than our current workflow, but we upload in 4K currently and the demo maxes out at 720p'.

To be fair, he probably uses After Effects way more than the average expected subscriber, so this isn't a representative sample.

gen220(10000) 2 days ago [-]

This is legit feedback if you're targeting this segment. I have relatives in the YT content "industry", and they're (inordinately?) obsessed with uploading content at the highest quality YT will allow. If something isn't pumping out 4K, it doesn't pass their smell test, for better or worse. So, if you allow 4K exports, I'd advertise it loudly! :)

txu(10000) 3 days ago [-]

Super impressive work since I first saw it on IndieHackers. Congrats!

michaelaubry(10000) 3 days ago [-]

Hell yeah. Lets gooo

michaelaubry(10000) 3 days ago [-]

I quit my job over a year ago. Been bootstrapped on savings. I picked up a gig last year and only made $30K in 2019.

Hoping my efforts pay off.

rememberlenny(10000) 3 days ago [-]

I signed up. This is immediately useful and totally worth the cost. Loving the easy video captioning.

Copenjin(10000) 2 days ago [-]

I don't even need to make videos but this is so polished I'd like to try it out anyway, great work.

xenospn(10000) 3 days ago [-]

Looks awesome! BTW, the 'About' page has a missing image (I'm assuming it's you).

ActsJuvenile(10000) 3 days ago [-]

How many people do you have on your team?

tiffanyh(10000) 3 days ago [-]

Unsolicited suggestion.

Hit up Courtland at Indie Hackers. He has a podcast and your story would be a perfect fit.

Should help drive some traffic your way as well

thecupisblue(10000) 3 days ago [-]

Amazing work man! This looks and feels great!

Dropped it on producthunt, hope you get some boost there!

29athrowaway(10000) 3 days ago [-]

This looks very polished. Good work, man.

statictype(10000) 3 days ago [-]

I'm sure it will. How long did you spend working in this before the first release?

(Is this the first release)

dceddia(10000) 3 days ago [-]

Nice job! It looks like a ton of effort. What's the tech stack for something like this? All homegrown? I've wanted to mess around with screencast editing but programmatic video stuff in general seems daunting.

rswail(10000) 2 days ago [-]

This looks awesome! One prospective customer group, I'm helping a teacher friend of mine (she teaches Y1-4) as she produces basically an episode of Playschool/Sesame Street per day to her students.

She's working in iMovie and looking to move to Final Cut Pro, specifically for better editing and effects. So this looks like an awesome halfway house.

I can also see a marketplace for templates around your editing product that would be another great sideline for creators.

Questions though about storage and video streaming. Are you offering the streaming service along with the editing?

Or is this purely the editing stage and then there's a take the raw footage and upload to somewhere like mux.com for the encoding and streaming requirements?

My contact details in my profile here, really interested in this :)

michaelaubry(10000) 2 days ago [-]

Thats really cool. I will definitely follow up.

So the videos are hosted for you and a part of the plan. I encode the video so it's supported on phones and all social media platforms.

To post on Instagram you have to download it. You can easily embed the video using the hosted link onto blogs and you can tweet the link, use the link in SMS, etc.

I'd love to help more :)

michaelaubry(10000) 2 days ago [-]

I don't see any contact info though. You can email me if you are interested michael @ storycreatorapp.com or @michaelaubry

Looking forward to it. See you on the inside!

tomc1985(10000) 3 days ago [-]

This obsession with 'stories' is getting out of hand. It's like that horrible 'Made with Love' trend from a few years ago.

michaelaubry(10000) 3 days ago [-]

You sound like a real fun person

swframe2(10000) 3 days ago [-]

See also: veed.io, slide.ly, kamua.com

antoineMoPa(10000) 3 days ago [-]

For sure there is a lot of competition in online video tools.

jamalx31(10000) 3 days ago [-]

Honestly, one of the best products I've seen this year.

Wistar(10000) 3 days ago [-]

And probably the most complimentary HN comment section I have ever seen.

viraj_shah(10000) 3 days ago [-]

This is awesome.

michaelaubry(10000) 2 days ago [-]


kanobo(10000) 3 days ago [-]

I was excited to see at least some AE features in the browser or more-than-beginner type editing -- but it's more akin to iMovie or a basic templated editor? Impressive nevertheless for one person to bootstrap, congrats.

michaelaubry(10000) 3 days ago [-]

Fair enough.

I needed to get an MVP out and start figuring out ways to make money before I completely go into debt.

My goals for the future are to give more customization to the motion graphics and to feature some of the core elements a professional in hurry will need. Nothing too complex but just right.

You can also think of it being a mix with other types of video editing tools. I do plan to roll out keyframe animations when I get some cashflow. I am excited for that.

dylan604(10000) 3 days ago [-]

I came to say the same thing. If you're going to compare yourself to AE, then you better bring some heavy hitting features. The iMovie comparison is exactly what I was thinking. It's a FisherPrice 'baby's first NLE'. That sounds a lot harsher than I mean for it to be, but it is apropos.

People that have never used an editor and wants to get into it, then this very well may be an amazing option for them. For $32/mo though, that's a big ask. For $50/mo you can have access to the full Adobe suite, and this is but a fraction of that.

I wish you well, and it definitely looks like you've spent a lot of hard work on this. Just don't try to compare yourself to something you're clearly not trying to be.

PS: that's the royal you, not directing at the parent

langitbiru(10000) 3 days ago [-]

Nice. The only thing that makes me hesitant to ditch Adobe subscription is After Effects. There are replacements for Photoshop, Illustrator, but not for AE.

michaelaubry(10000) 2 days ago [-]

If it meets your needs then hop aboard.

I'd love to learn more about what you're working on and how you plan to use video/motion graphics.

Hit me up via email or Twitter @michaelaubry. Id love to chat.

uxcolumbo(10000) 2 days ago [-]

For pro users there is a replacement for AE - check out

Davinci Resolve Fusion


And it's a perpetual license as well - useful if you prefer the older licensing model. Remember what happened to Adobe users in Venezuela.

But storycreatorapp looks great for content creators who don't have the time to learn video editing & FX and just want to get things done quickly and focus on their content. Definitely going to check it out more...

matlin(10000) 3 days ago [-]

This is awesome. Hats off to you.

But... 'After Effects for dummies' is selling it short. This is a web based video editor! After seeing Figma do it's thing, I was waiting for someone to do it for videos. We're nearing the point where you can do 100% of your work in the web. Keep up the good work!

michaelaubry(10000) 2 days ago [-]


Thanks for for the support :)

AlexDanger(10000) 3 days ago [-]

The site is fantastic. I think you have massive potential upside if this is marketed appropriately.

Do you need any help with sales and marketing? I'd be interested in having a discussion if you'd prefer to focus on building things.

michaelaubry(10000) 2 days ago [-]

Definitely. Id much rather be engineering.

Reach out to me on Twitter or email @michaelaubry michael @ storycreatorapp . com

nojvek(10000) 3 days ago [-]

The home page took over a minute to load for me. How is this being served? seems like you could optimize this.

michaelaubry(10000) 3 days ago [-]

Using Vercel.com

It's all serverless. Could be assets taking a second or internet connection.

Lightspeed in incognito is saying first contentful paint was 0.5 s.

Could be extensions or just a rare exception. If you could provide some more insights I am happy to see how it can be faster. Always fascinated by speed.

tyscorp(10000) 3 days ago [-]

It might be getting the ol' classic hackernews hug-of-death special.

epberry(10000) 3 days ago [-]

I think the focus on speed and bundled content is important. These are the two biggest weaknesses of After Effects today imo.

michaelaubry(10000) 3 days ago [-]

Yes bundled content is a big pain point of mine. My biggest pain points with After Effects.

1. When I want to create a creative video I typically look on Envato for inspiration. Then I pay $30 to Envato on top of the $270 a year for After Effects. Then I have to learn how to install the template or asset. I wanted a marketplace and library integrated with the tool.

2. I also want the After Effects experience to be like Figma , Canva, or Sketch for making videos quickly. I don't want to fiddle with too many knobs and counter intuitive tools. While they are powerful and After Effects will always be GOAT. Its too much sometimes, especially in a world of speed and iteration.

kuldeep_kap(10000) 2 days ago [-]

Congrats! This looks amazing!

If you don't mind me asking, how did you decide to commit to complex app like this? It must've taken a long time to build the MVP. Did you interview potential customers before hand or just went I with your gut or your own pain points? How did you know people will be willing to pay for this, considering there are a lot of similar tools out there?

michaelaubry(10000) 2 days ago [-]

I made a lot of rookie mistakes and I am ok with that. It was purely a gut feeling.

I am genuinely interested in video. I have been fascinated by it since a kid. So I knew I wanted to solve this problem.

I also love design tools and a good challenge. So I went against the grain and followed my gut. It brought a lot of pain and useful learnings. I wouldn't have it any other way.

Next time I will know what not to do. Always talk to the market and implement the MOM test. Do follow your gut but do mix that with conversations.

batt4good(10000) 3 days ago [-]

What kind of stack did you use for this awesome project?! I've never really known how to get started with an app like this outside of wrapping a bunch of functions around ffmpeg.

michaelaubry(10000) 3 days ago [-]

Combination of things. It's like a cake haha.

React.js FE Node.js BE

Vercel FTW

arvidkahl(10000) 3 days ago [-]

This is an extremely impressive product. Signed up a couple weeks ago, it's spectacular.

michaelaubry(10000) 3 days ago [-]

Hell yes. You're a smart man.

sak5sk(10000) 3 days ago [-]

Hey, nice job! The tool looks really cool and seems like it would save a lot of time for people who are not into video editing but want to add cool effects.

All the best to you!

michaelaubry(10000) 2 days ago [-]

Yup working towards that everyday. Makes me happy. Cheers :)

splatzone(10000) 3 days ago [-]

This is so cool! Some specific thoughts

- My finger slipped and it wheecht me out of the editor before I'd finished editing the video - maybe it could prompt me to confirm if I want to leave the editor/save my work if I've made changes?

- Is it possible to disable the browser right click/context menu on the timeline? I expected to get some additional options but got the browser context menu instead

So cool! Well done

michaelaubry(10000) 3 days ago [-]

Great point. I do an auto save every 3 seconds, but yes if you go back before then it's game over haha.

I will check to see if it's possible to disable that I remember this being a hard thing to do on another project I was working on.

Ill look into it again :) worst case I'll see if I can trigger a save function on a back event.

I'd love to have a custom context menu. What options would you expect to find?

dvt(10000) 3 days ago [-]

First of all, hats off to an amazing product: very nicely polished, marketed, and packaged.

However, and I'm sure you know this much better than I do, this seems like a niche-of-a-niche type of product, no? YouTube already has some rudimentary video editing capabilities, Vimeo has a pretty nice video editor, and there's (quite literally) dozens of others. Can you even compete in this landscape? I feel that why After Effects/Vegas Video still exist is exactly because they're pretty hardcore tools.

michaelaubry(10000) 3 days ago [-]

Great points.

I believe so. I don't think it's a zero sum game and competition is great.

I have a vision for this product and my main goal is to overload the user with value and an amazing product.

I want to have a marketplace and be the go to spot for the best templates (not overly cookie cutter, all beautiful designs), everything needs to be well designed and have a premium feel.

I also plan to roll out features other players don't have - they have been copying me for a minute.

So I am not going to reveal them all but just know my goal is to bridge the gap between power and ease of use. Mix that with premium assets, a market place, and the user always in mind as a priority. I think I'll have a fighting chance.

leemac(10000) 3 days ago [-]

Ahh! This is so cool!! Really neat idea, very slick.

Played around with the demo, very well done! I love the walk-through.

Small issue: I noticed you can click on multiple music/sounds and they start to stack with each playing over the other (using Windows/FireFox). That was the only issue I encountered during a quick play-through.

Good luck!

michaelaubry(10000) 3 days ago [-]

Awesome. Ill put this in the backlog. Maybe letting the user know they can only have one audio at a time and asking if they want to swap.

ape4(10000) 3 days ago [-]

Very nice but I want desktop (Windows) app. Partly because I am often in low bandwidth situations.

michaelaubry(10000) 3 days ago [-]

Got it. This is something to consider.

It is possible to use blobs and write to localStorage. I'll take offline mode into serious consideration when I get some cashflow and a team.

jacobwsmith(10000) 3 days ago [-]

Fantastic landing page - I feel like I got a quick understanding of what the offering was and how it could help me.

michaelaubry(10000) 3 days ago [-]

Nice. That's good. I feel like I can continue to improve.

Things I am focused on are making it easy to find the type of content the user wants to create and sending the user down the path to success in the least amount of steps possible.

I think the messaging is always a work in progress. I am glad it makes sense and is clear.

Who do you think this tool is for?

mwizzle(10000) 3 days ago [-]

This looks great! What stack is it built on? How long did it take you to build?

I'd love to learn about how you decided to build it, scoped out what features to build and cut, the whole behind-the-scenes. Pretty please! :D

michaelaubry(10000) 3 days ago [-]


React.js for the UI and state management. Vercel for serverless hosting and easy deployments FFMPEG for stitching Node.js on the backend handling the rendering Prisma 2 as a ORM for database interaction GraphQL to perform operations on the data

This took me longer than anticipated about 11 months.

Let me know if you have any more questions happy to answer. You can DM me on twitter @michaelaubry

chadlavi(10000) 3 days ago [-]

as someone who doesn't need this product in order to make money, that's surprisingly expensive! people are really paying 30 bucks a month to edit instagram story videos?

tiffanyh(10000) 3 days ago [-]

I can't believe people are paying $x per month to just put files in a folder to let them sync.

Dropbox is a multi-billion $ business.

Never underestimate what people are willing to pay for.

michaelaubry(10000) 2 days ago [-]

Pricing is extremely weird and there are interesting Twitter discussions around this.

It's crazy to me people are willing to pay $5 a day for frothy milk with a shot of espresso but Starbucks isn't doing too bad neither is Blue Bottle, etc.

As I expand the asset library, each template, and animated text I make takes a lot of time and skill that is one click for users.

Lastly just like Starbucks and other great companies it's all about crafting a pleasurable experience that people enjoy engaging with.

I want people to come to the product and have fun. Personally I LOVE using Figma, Canva, and Sketch. They are delightful for me.

Just my 2 cents.

aldanor(10000) 3 days ago [-]

Looks pretty awesome.

Minor nit: hitting backspace to delete a clip (pretty common in editing apps) forces the browser to go back.

You might also want to catch other types 'leave page' events and ask for confirmation so that the user doesn't lose their work accidentally.

kreelman(10000) 3 days ago [-]

Also, trim music gave an error for me too. Do you have a place to drop bug reports?

michaelaubry(10000) 3 days ago [-]

Thanks for catching that. Hot keys have been a pain. I also found an issue with copy pasting inside text boxes.

I promised myself and my startup friends I wouldn't touch code until I handled doing marketing and sales.

It's on the backlog now, thanks for letting me know. This feedback goes along way :)

thrownaway954(10000) 3 days ago [-]


dude you really out did yourself. usually i criticize people because their product homepage doesn't tell nor show anything about their product so i'm confused about what it does and how to use it.


your homepage not only tells me exactly what this does, but shows me to the point where i'm excited to use it. good on you. others should learn from you.

my one thing that i noticed (and this could be a revenue generator for you) is the lack of templates currently. the more templates you have for us noob video editors, the better. not only that, but allowing the community to contribute and/or sell their template would be a HUGE win for you.

jamalx31(10000) 3 days ago [-]

100% his landing page feels like a breath of fresh air comparing to other landing pages mess

kreelman(10000) 3 days ago [-]

Have to agree with above.

Had an issue with sign up via Google, but I got past that and really quickly I was able to build a video!

It's really enjoyable to use. Well done!

...You may have also picked the absolute perfect time to put a product out like this. We're all mostly stuck at home, coming up with great ideas that we want to communicate.

I hope it takes off.

Watch this space!

michaelaubry(10000) 3 days ago [-]

This means a lot. I have spent some time getting it down. A lot of blood, sweat, and tears haha.

I agree with the templates. I am trying to allocate my time wisely and I think expanding that will be major.

I am drawing up a way to make it even easier for people to create amazing videos.

When you click 'add new video' I plan to make it interactive and guide you along every step.

Your options will be

Do you want to create a video

- from scratch - from a template - for your podcast - for your product - for your brand

Based on the selection I will walk the user through an simple experience. Have them upload their assets, select a vibe, enter some information, etc. Then generate the video data and put it on the timeline.

I am doing sales, customer support, engineering, and design. So once I can get some runway I will be able to roll it out.

To be honest I hate sales and marketing. I just want to build, and if I can get to ramen profit. Then I can pay people to do what I hate.

Moghammed(10000) 2 days ago [-]

Yeah, I can't think of anything i'd need a video for, but still made one just to see if it works as well as the intro makes it look. And it really does work that well!

One small issue and a small feature request, tho:

- Undo of resizing images should restore it to the size that it was before you started dragging, currently it restores it to a lot of sizes during dragging.

- It would be great if you could switch between different kinds of text effects without having to remove the entire effect and re-enter your text.

[EDIT: found some more small bugs]

- Copy & paste hex colors for text doesn't work b/c it will paste the text element instead

- One time the download link for the video didn't work, it just closed the modal and i had to render it again. Worked fine the next time

All in all very well done! Will definitely be spreading the word about this one!

marktolson(10000) 3 days ago [-]

Well done, looks amazing. Uploading videos (webm) doesn't seem to be working right now. Interested to know what kind of infrastructure you're running on.

michaelaubry(10000) 3 days ago [-]

Ahh thanks for bringing that to my attention. I have a strict check on these formats. If its not in the array it shall not pass.

const supportedVideos = [ 'video/ogg', 'video/mp4', 'video/webm', 'video/mov', 'video/quicktime', 'video/x-matroska' ];

No particular reason why I dont support webm never really thought about it. I'll update it and run some tests.

Thanks for letting me know.

Historical Discussions: Latest Firefox rolls out Enhanced Tracking Protection 2.0 (August 05, 2020: 767 points)

(781) Latest Firefox rolls out Enhanced Tracking Protection 2.0

781 points 2 days ago by LinuxBender in 10000th position

blog.mozilla.org | Estimated reading time – 3 minutes | comments | anchor

Today, Firefox is introducing Enhanced Tracking Protection (ETP) 2.0, our next step in continuing to provide a safe and private experience for our users. ETP 2.0 protects you from an advanced tracking technique called redirect tracking, also known as bounce tracking. We will be rolling out ETP 2.0 over the next couple of weeks.

Last year we enabled ETP by default in Firefox because we believe that understanding the complexities and sophistication of the ad tracking industry should not be required to be safe online. ETP 1.0 was our first major step in fulfilling that commitment to users. Since we enabled ETP by default, we've blocked 3.4 trillion tracking cookies. With ETP 2.0, Firefox brings an additional level of privacy protection to the browser.

Since the introduction of ETP, ad industry technology has found other ways to track users: creating workarounds and new ways to collect your data in order to identify you as you browse the web. Redirect tracking goes around Firefox's built-in third-party cookie-blocking policy by passing you through the tracker's site before landing on your desired website. This enables them to see where you came from and where you are going.

Firefox deletes tracking cookies every day

With ETP 2.0, Firefox users will now be protected against these methods as it checks to see if cookies and site data from those trackers need to be deleted every day. ETP 2.0 stops known trackers from having access to your information, even those with which you may have inadvertently visited. ETP 2.0 clears cookies and site data from tracking sites every 24 hours.

Sometimes trackers do more than just track. They may also offer services you engage with, such as a search engine or social network. If Firefox cleared cookies for these services we'd end up logging you out of your email or social network every day, so we don't clear cookies from sites you have interacted with in the past 45 days, even if they are trackers. This way you don't lose the benefits of the cookies that keep you logged in on sites you frequent, and you don't open yourself up to being tracked indefinitely based on a site you've visited once. To read the technical details about how this works, visit our Security Blog post.

What does this all mean for you? You can simply continue to browse the web with Firefox. We are doing more to protect your privacy, automatically. Without needing to change a setting or preference, this new protection deletes cookies that use workarounds to track you so you can rest easy.

Check out and download the latest version of Firefox available here.

All Comments: [-] | anchor

Razengan(10000) 1 day ago [-]

I love Firefox as an alternative to Chrome but I hate how it goes of its way to be as cumbersome as possible.

Witness this example of the most terrible UI ever; how frustratingly convoluted it is to delete specific cookies:

1: Open Preferences

2: Search 'cookies'

3: See a bunch of cluttered stuff

4: Scroll all the way down to 'Cookies and Site Data'

5: Click 'Clear Data'

6: Oops, that's not it. Cancel and try 'Manage Data'

7: Search 'google' for example (to avoid their crappy tactic of signing you into Search when you sign into YouTube or Gmail, but that's another story)

8: Click 'Remove All Shown'

9: Click 'Save Changes'

10: Get hit with a modal alert in your face showing you the list of changes to confirm.

11: Click 'Remove'


corford(10000) 1 day ago [-]

It isn't great but imho it's better than Chrome which confuses me each time and feels super limited:

1. Open settings

2. Privacy and security

3. Scroll down to find the tiny 'See all cookies and site data'

4. Use the search box for the site you want

5. Then you only have the option of removing everything from that site?? There's a little X icon to the right of each cookie but clicking it does nothing

Unless there's a better way I haven't seen (I use chrome infrequently)

symlinkk(10000) 2 days ago [-]

What if cookies were opt in? E.g. blocked by default everywhere unless you decide to allow them, same as access to the webcam for example.

xook(10000) 2 days ago [-]

Side benefit: It helps a ton with testing changes instead of clearing everything and starting over!

dgb23(10000) 2 days ago [-]

I want this rather than the weird cookie messages that you get everywhere now.

eklavya(10000) 2 days ago [-]

I wish Apple realised that more competition in browsers on iOS is better for the user and they shouldn't be greedy about it. How wonderful would it be to have proper Firefox in iOS.

parsimo2010(10000) 2 days ago [-]

I use Firefox on iOS. It has some tracking protection. Probably not the full-fat version of desktop Firefox, but I find it to be more tolerable than Safari on iOS.

nabaraz(10000) 2 days ago [-]

If only I can get Chromecast working on Firefox. I would never touch Chrome ever again.

rmorey(10000) 2 days ago [-]

I mean, you would still be touching chrome, as Chromecast devices are well, running Chrome... but your point is taken

typon(10000) 2 days ago [-]

My friend who works in an adtech company:

'Protip: Use Firefox instead of Chrome. We get very little data from Firefox users'

luxuryballs(10000) 2 days ago [-]

I assume the same applies for Edge so if I want my ads to be a tailored as possible but don't want to use Google apps then use the most recent Edge browser since it's Chromium under the hood?

stiray(10000) 2 days ago [-]

Protip #2, for what cant be traced using conventional methods, they will use fingerprinting and those add-ons take care about most common methods of fingerprinting - canvas, webgl, fonts and audio:





I would really love to have more addins like this, doing one thing and doing it good. They will kill fingerprinting and as a proof, I was downvoted the next moment i posted the links in another post but I want you to know there is a way out.

addicted44(10000) 2 days ago [-]

Couldn't that also be because Firefox has a higher percentage of expert users than Chrome, which has pretty much become the default browser, would.

suby(10000) 2 days ago [-]

There's a tacit understanding there that their collection and use of data is wrong, or at least that it is against one's interest to have their data collected.

winter_blue(10000) 2 days ago [-]

I've gone back and forth between Firefox and Chrome.

It's just that Firefox is noticeably slower than Chrome (or at least was about a year ago). Even simply switching between open tabs around what feels like 0.5 seconds, whereas on Chrome, tab switching is instantaneous. I've run Firefox in multi-process mode (since that was an option), but even with it, in general it seems like Firefox gets noticeably slower than Chrome when you have a lot of open tabs. I've always preferred and wanted to use Firefox over Chrome, but the performance degradation was too much to bear.

formerly_proven(10000) 2 days ago [-]

Even less data if you use uBlock Origin and possibly uMatrix (which is very high maintenance, but also reveals the utter insanity of the web).

Without an adblocker the internet is such a slow heap of trash that I'd never go back to not using one. This is also one of the main reasons I use my iPhone so little, since it doesn't really have any way to adblock.

rladd(10000) 2 days ago [-]

Latest Firefox still doesn't protect against browser fingerprinting. This from the EFF Panopticlick:

  Your browser fingerprint appears to be unique among the 303,579 tested in the past 45 days.
  Currently, we estimate that your browser has a fingerprint that conveys at least 18.21 bits of identifying information.
daffy(10000) 2 days ago [-]

Qutebrowser with a Firefox backend would be nice.

The-Compiler(10000) 2 days ago [-]

Still waiting for Servo or https://mozilla.github.io/geckoview/ to be usable as a library on desktop platforms. Unfortunately, that hasn't happened so far...

tgb(10000) 2 days ago [-]

I'm trying to get myself to finally switch to Firefox thanks to this post. But I'm having problems with ctrl +/- changing of font sizes. In Chrome, that setting affects one domain only. Here I'm getting confused because I open a new tab and the font is tiny or huge, even sometimes on sites I've never visited before. Ctrl 0 fixes it but then I seem to reset sites that I do want enlarged (like HN). What's the logic here? I haven't figured out the steps to reproduce this, it seems random.

irontinkerer(10000) 2 days ago [-]

I've had the same experience! Well sort of, I think it's related to the experimental zoom I enabled. Trying to track down how to reproduce so I can open a bug, but it's sporadic for me. Other than that, I'm pretty happy with the browser

tarkin2(10000) 2 days ago [-]

I understand: cookies are deleted every 24 hours unless you've visited the site before in the last 45 days.

Does this mean if I visit a site twice the cookie stays but otherwise it's gone in 24 hours?

ysavir(10000) 2 days ago [-]

I think it means that 3rd party cookies will be deleted in 24 hours unless you visited the site of origin in the last 45 days.

So if you get a 3rd party cookie from www.marketingsite.com, but never visited that site, it will be deleted in 24 hours. But if you get a 3rd party cookie from facebook, and you're a semi-active facebook user, the cookie will be kept.

xref(10000) 2 days ago [-]

Hopefully this will make affiliate-link based sites like Wirecutter change their ways. I dont mind them earning affiliate money but I can't even click a link at Wirecutter anymore since their bounce redirect to Amazon or wherever is totally blocked by my DNS settings originally via AdGuard now via nextdns.io

untog(10000) 2 days ago [-]

Can you add an exception for the site? It seems like a slightly inconsistent position that you don't mind them earning affiliate money yet you're actively blocking them from doing so.

vikbytes(10000) 2 days ago [-]

This is fantastic. I never liked the cookies sticking around forever, and managing them manually was a massive pain if you wanted to keep some of them.

Not to mention Firefox is usually brought to its knees when trying to delete large segments of History/Cookies at once.

jimmaswell(10000) 2 days ago [-]

What cookies are you trying to get rid of? Why are they a big deal?

Yoric(10000) 2 days ago [-]

> Not to mention Firefox is usually brought to its knees when trying to delete large segments of History/Cookies at once.

Is that still the case? I remember that one of the developers was seriously working on improving that.

ainar-g(10000) 2 days ago [-]

> This is fantastic. I never liked the cookies sticking around forever, and managing them manually was a massive pain if you wanted to keep some of them.

You might like the CookieAutoDelete plugin[1]. It's a recommended plugin which allows you to set a list of domains and domain patterns which retain their cookies while others are deleted. I've been using it for a couple of months now, and I love it.

[1] https://addons.mozilla.org/en-GB/firefox/addon/cookie-autode...

dependenttypes(10000) 2 days ago [-]

You might want to try uMatrix. It lets you choose which sites are allowed to use cookies.

bzb3(10000) 2 days ago [-]

I'm waiting for it to get so good that it becomes effective at removing the ads Mozilla added to the new tab page.

ziddoap(10000) 2 days ago [-]

As another user pointed out, this is anti-tracking not anti-advertisement.

I can't help but wonder why you don't just go to the front page of your settings, and select 'Blank page' on the 'New tab' field though. Unless I am misunderstanding where you are talking about.

neilsimp1(10000) 2 days ago [-]

You can remove everything from the New Tab page to make it completely blank now, although I do agree I'd prefer if it was blank by default with no ads or anything.

compscistd(10000) 2 days ago [-]

Firefox > Preferences > Home > New Windows and Tabs (very first option!) > Homepage and New Windows, New Tabs > Blank Page

I like that Firefox Home is the default option because you as the user are offered something. If you don't like it, you say no, go to the settings page, and are never offered it again. If you do like it or don't mind it, you keep it. Win-win.

If Firefox Home wasn't on by default, those that might have preferred it wouldn't know what they're missing. It seems difficult to think someone would actually prefer ads but if there's some engagement, then that speaks for itself.

dkdk8283(10000) 2 days ago [-]

How else do they pay developers?

Wowfunhappy(10000) 2 days ago [-]

Tracking Protection is quite explicitly not an ad blocker. It blocks tracking, and that's not the same thing.

markosaric(10000) 2 days ago [-]

Nice improvements! Firefox keeps delivering! But Chrome browser market share is now up on 71% while Firefox is down to 7.3% :(

aembleton(10000) 2 days ago [-]

How is that market share determined? Is it by looking at data from tracking?

Those tracking numbers will reduce, the more that Firefox cracks down on it.

User agents hitting server logs are probably more accurate now.

UI_at_80x24(10000) 2 days ago [-]

I've been a user since they first split from Netscape. Very happy to see continued efforts in this project. Congrats folks! I'm glad you exist.

Related: I don't know if this applies to other platforms, but the newest version of FireFox on FreeBSD (79.0,1) generates errors on every website you visit stating insufficient security. (including google & mozilla.org) This is somehow related to not having a virus scanner installed or something.

This is the about:config setting to disable that:


drewg123(10000) 2 days ago [-]

I'm running the same version of Firefox on FreeBSD (firefox-79.0,1) and I don't see this (with http2=true). I'm running 13-current (r363668) and using the 12-stable quarterly packages (FreeBSD:12:amd64/quarterly)

tdhz77(10000) 2 days ago [-]

I keep encouraging people to use Firefox over the others. My efforts keep failing. Mozilla's focus on privacy is important and should be to others. People have been burned in the past and are willing to give up privacy for something that just works. The irony.

m463(10000) 2 days ago [-]

I use umatrix to block stuff and a side effect is that EVERYTHING is much faster.

Some sites routinely load hundreds of items from dozens of third-party servers, so this shouldn't be surprising.

noahtallen(10000) 2 days ago [-]

FF's UX around things not related to browsing is slightly worse on all platforms. It's most obvious on iOS where some of the styles for your "library" just don't work correctly on iPhone 10's. I also think their model for managing non-browsing stuff is too complex comparatively.

To access many things like history and bookmarks, you need to use the dropdown menu on the right, which has like 30 different options in it, some of which seem similar but are actually different. Those options are sometimes also available from different locations, so it can be confusing how to access what you're looking for.

Other browsers are a lot more polished in this area, like Brave, Chrome, and even the new MS Edge. None are perfect, but I find their UX and UK slightly easier to use and generally less clunky. If FF fixed that, I'd be completely sold on it. But as is, I keep going back to the drawing board.

twblalock(10000) 2 days ago [-]

I have several reasons for not using Firefox:

Font rendering looks non-native on most platforms.

Scrolling behavior seems non-native on most platforms.

The combined location/search bar seems slower, and gives less useful results, than the same feature in other browsers.

Inferior developer tools.

Mozilla's marketing portraying itself as the white knight of the open web is tiresome and contradicted by past behavior. Pocket is still installed by default, and remember the Mr. Robot scandal?

Tracking is easily defeated in other browsers. Besides, if you use Gmail and Google search, and watch videos on Youtube, you've already decided to give your data to Google and you might as well use Chrome too.

bishalb(10000) 2 days ago [-]

I would use it regularly if it weren't noticably slower than Chrome and in many cases unresponsive (I get the yellow bar at the top that says site isn't responding or something like that, when the same site opens fine on chrome). I have the developer edition installed and am logged in via alternate profiles on some sites there. Often times, it displays a blank for a few seconds on heavy sites(with just 2 tabs open) before loading the content.

The developer console is also slower than Chrome's.

mitchdoogle(10000) 2 days ago [-]

Do people think Firefox doesn't work? I've noticed zero difference between using Firefox and Chrome and I've used both daily for five years or so.

jm4(10000) 1 day ago [-]

I've been using Firefox since it was called Phoenix and I used Mozilla before that. I'm pretty annoyed with Firefox lately. There's the pocket junk, the significantly worse battery life, the DOH stuff really pisses me off. How can they pitch themselves as the privacy browser when they siphon off my DNS traffic by default? Why does anyone give them a free pass on that? About the only thing they have going for them is they're not Chrome and that's not saying much. Lately, I find myself spending much more time in other browsers to see if one of them can become my daily driver. I fear that Firefox is becoming irrelevant.

chrisjc(10000) 2 days ago [-]

Maybe it's just a coincidence, but since we're doing more and more things over screenshare these days, I've noticed that all my peers are starting to use Firefox. I like to think it's because they've seen me use it when I'm sharing my screen.

muststopmyths(10000) 2 days ago [-]

I recently gave up on firefox after trying for years to use it for my more sensitive browsing (email, bank, investments, etc.) where I could not run javascript blockers.

3 big problems (and many minor little things that i hit every day but haven't bothered to record)

- composing email in gmail is horrible. unexplained bursts of lag where it hangs for several seconds and may or may not lose anything I typed in that interval. This one is recent as of the past couple of months.

- outlook email just not updating or refreshing until I restart the browser.

- lagginess in most if not all input boxes (could be related to the first problem above).

- every few updates it will lose all my containers and I have to make them from scratch.

It may be something these sites are doing wrong, but I don't have the patience any more. Chrome works, Edgemium works, so I switched.

still use firefox for facebook container, but that's about it.

alleyshack(10000) 2 days ago [-]

I want to switch to Firefox, but of all things, its tab management is keeping me away. On my last try, I gave up after five minutes because I couldn't see all my open tabs at once (had to scroll). Does anyone know of an extension or setting I can use to force Chrome-like tab behavior, where all tabs are shown at once regardless of how small they become?

ric2b(10000) 2 days ago [-]

I use tree style tabs, which is amazing for tab hoarders like me: https://addons.mozilla.org/en-US/firefox/addon/tree-style-ta...

LocalPCGuy(10000) 2 days ago [-]

For people with a lot of tabs, I recommend trying a Vertical Tabs add-on. I am currently using Vertical Tabs Reloaded, but there are a couple different ones. Usually people have plenty of horizontal space and you have a scrollable sidebar with all your tabs in it.

cpeterso(10000) 2 days ago [-]

I don't think you can prevent tabs from scrolling off screen, but you can set the 'browser.tabs.tabMinWidth' about:config pref to a tiny number so you can fit more tabs on screen.

You can also see the full list of tabs (with titles) in the tab overflow dropdown menu. It's the down arrow button to the right of the tab strip. The dropdown menu only appears after you open at least ~20 tabs. I set the browser.tabs.tabmanager.enabled about:config pref = true to always show it (because I like see the full tab titles).

atombender(10000) 2 days ago [-]

I'm a fan of ETP, except it breaks a lot of sites that use embeds.

For example, sites that use Twitter or Instagram [1] embeds won't show the embeds. And there's no way, as far as I can tell, to whitelist those.

The only solution is to whitelist the sites that have embeds, but that ends up enabling all the tracking and stuff you don't want.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1446243

ilikehurdles(10000) 2 days ago [-]

Sounds like a good idea would be to somehow run the embeds in their own isolated container? Not sure how feasible that feature would be to implement.

duhi88(10000) 2 days ago [-]

Pretty sure that embeds enable all the tracking stuff you don't want, too.

sequoia(10000) 2 days ago [-]

Note to website/blog authors: please don't include twitter embeds in articles as the only way to read/view/hear the content!!

- I use an ebook reader (Kobo) with Pocket to read saved articles and it can't load the embeds

- I listen to articles with the screen reading feature on iOS and when it goes over embedded tweets it's a nightmare

Many articles are nonsensical when you cut out the twitter embed that added key context or information to the article. Consider copy/pasting the text into the article & then linking to the tweet.

floatingatoll(10000) 2 days ago [-]

If you haven't already, please consider opening up a Webcompat issue [1] about this. They'll definitely want to know since that's the sort of scenario that could end up breaking in multiple browsers — not just Firefox! — as tracking protection spreads across the ecosystem.

[1] https://webcompat.com/issues/new

Ijumfs(10000) 2 days ago [-]

It all seems pretty pointless when every tracker tracks you by IP address and the only real way to defeat this is to use Tor, which BTW is blocked on an ever-growing list of sites.

anonymousDan(10000) 2 days ago [-]

Hmm, is there anyway to configure tor to reduce the number of hops in order to improve performance? It seems like multihop Tor is overkill for this use case. Wouldn't solve the problem with sites blocking Tor I guess.

M2Ys4U(10000) 2 days ago [-]

IP is a much coarser level of tracking, for most people.

GCNAT, roaming (WiFi <-> mobile), shared networks (NAT/IPv6 Privacy Addresses). All of these things mean you can't be sure you're tracking the _same_ person, unlike with cookies.

mancerayder(10000) 2 days ago [-]

I'm happy to see stuff like this.

I'd be even happier if the tricks to get video to play were somehow canceled.

I have adblocker and video blockers, but somehow, news sites have a video that plays. If I scroll off the page, the video pops out into the lower right hand of the page and resumes playing, even if the big version of the video at the top of the page was stopped /paused (which it is by default), and it needs to be stopped again. On mobile (Android) this is a double nightmare, even in Firefox, because the little video has a tiny little X, and somehow my finger doesn't ever hit X the first time. I can plug in a USB-A connector in the right way faster than I can press that little X.

Is there an explanation for this, and am I the only one?

Shacklz(10000) 2 days ago [-]

> but somehow, news sites have a video that plays

This is so annoying, engaging in such obviously obnoxious UX patterns should be regulated and punished with fines when user-intent tries to be circumvented.

The 'just dont visit that website lol'-trope really doesn't cut it anymore; operating a news paper organization nowadays is such a cutthroat business that you simply can't not do it if your competition does it, there really needs to be regulation to level the playing field to stop this kind of bullshit

cactus2093(10000) 1 day ago [-]

The much bigger problem for me is I can't really go to any news site without the full page being hijacked blocking the content until I turn off my adblocker.

I feel like adblockers are becoming less and less useful overall, even as they become more widespread and more advanced, because I have to turn them off to see anything on the internet.

Am I the only one with that issue?

danShumway(10000) 2 days ago [-]

Short version of a long story! A few years ago Google implemented autoplay blocking in Chrome. It was designed to fix problems like this, but was riddled with issues.

I wrote about it at https://danshumway.com/blog/chrome-autoplay/. The spec evolved a little bit since then, so not everything in that post is up to date, but most of the core problems still remain (or did the last time I checked).

Firefox was forced to follow suit, and to their credit their spec was a lot more sensible, but it was really only papering over the problems in the Chrome spec. It wasn't at a fundamental level thinking about video/audio differently than Chrome was, it was just trying to do the same thing minus the egregiously bad decisions.

I feel like a lot of the problems with hijacked interactions on the web can be traced back to spec histories like this.

A good implementation of video/audio blocking:

- wouldn't reveal to the page that audio was blocked, it would either silently mute the audio or refuse to render the video without reporting an error.

- wouldn't have exceptions based around trying to interpret user intent or (in Chrome's case) exceptions based on how you navigated to the page.

- wouldn't try to distinguish between things like GIFs, animated backgrounds, and videos (they're all moving pictures that use data and distract motion-sensitive users, you don't need to treat them differently)

My (subjective) opinion is that video autoplaying never really got better because we never really tackled the problem correctly from the start, and since then we've just been continuing to apply band-aides on top of a fundamentally broken strategy.

> am I the only one?

Depending on how much you hate this and how much effort you're willing to put into getting rid of it, disabling 3rd-party Javascript will fix the problem on most news sites.

Nearly all of them that I run into load the Javascript to run the player from a separate script being served from a separate subdomain or CDN. It's usually possible (even for news sites that require Javascript) to block that script in specific, or load just enough Javascript to get the page rendering and nothing else.

Santosh83(10000) 2 days ago [-]

Indeed am seeing more and more sites that somehow get their videos to autoplay even though I have that explicitly disabled in Firefox settings. I thought user agents had ultimate control over 'mere bytes' sent over the wire but somehow these sites manage to trick the browser and override user preferences.

calmworm(10000) 2 days ago [-]

... then when you do manage to tap/click the X, on some sites it says 'closing video...' for 3 more seconds before it closes. Also very annoying.

mschuster91(10000) 2 days ago [-]

You're not the only one, and the worst examples are news sites. Pretty darn annoying if you're in a boring videocall, want to read some news and then suddenly your speaker blaring ads reminds everyone you're not paying attention...

tome(10000) 2 days ago [-]

Isn't the small video in the corner a feature actually implemented in Firefox itself (picture-in-picture) and nothing to do with the web page? If so then that seems doubly bad!

floatingatoll(10000) 2 days ago [-]

If you temporarily set about:config 'image.animation_mode' to the value 'none', and then restart Firefox, does it stop that video from autoplaying? If so, it's because they're using an animated GIF, which isn't a video.

(You can right-click the value you modified in about:config to Reset it back to the default, so that animated GIFs work again, after you're done testing that.)

grishka(10000) 2 days ago [-]

Video players, or at least media files themselves, are often hosted on a separate domain. You have to find it in uBO and block it.

dependenttypes(10000) 2 days ago [-]

I never understood this feature. Why not use an addon instead? Ublock origin for example is much better compared to the firefox tracking protection.

ocdtrekkie(10000) 2 days ago [-]

Because the default should be to block tracking. We just shouldn't allow it. Imagine someone telling you a popup blocker (something even IE8 had standard) should be acquire separately.

ceejayoz(10000) 2 days ago [-]

> Why not use an addon instead?

The general population has no idea such an addon is necessary. I applaud Mozilla including at least basic privacy blocking by default now.

nightski(10000) 2 days ago [-]

I'm on the latest Firefox and it's allowing a Set-Cookie with the Secure flag over http (for localhost). This seems wrong to me, but maybe it's allowed for localhost specifically? Anyone else experience this?

bugmen0t(10000) 2 days ago [-]

Yes. localhost is considered a Secure Context via https://w3c.github.io/webappsec-secure-contexts/#potentially... - This is intentional to testing APIs which require a Secure Context easier to use during development. E.g., ServiceWorkers.

onyva(10000) 2 days ago [-]

The only browser we can trust today. Glad to see they're making privacy online a priority, on top of an already amazing browser I use regularly and exclusively both on Linux and iOS.

subhro(10000) 2 days ago [-]

Honest question, what's wrong with Safari, that you can not trust it?

colordrops(10000) 2 days ago [-]

Off topic, but every time I see an article like this I load up Firefox and try it out to see how it has progressed. I inevitably stop using it, and this time I decided to introspect and see why. It turns out that it's mouse wheel scrolling doesn't feel as snappy as other browsers, and for some reason it bothers me a lot. I'm going to try changing the scroll wheel settings and see if I can stick with it.

Shorel(10000) 2 days ago [-]

I am more sensitive to scrolling than most.

Wheel scrolling is awful in all browsers. The way to scroll is to use middle mouse button and move the pointer.

Only doing this you can accurately judge scrolling performance. I don't want 'snappy'. I want smooth.

mrandish(10000) 2 days ago [-]

I am super-sensitive to smooth scrolling performance and am kind of obsessive about it. I use FF 100% of the time (unless testing something cross-browser) with an extension called 'Yet Another Smooth Scrolling WE' https://addons.mozilla.org/en-US/firefox/addon/yass-we/

If you spend a bit of time exploring the extension's options you should be able to find a combination that hits your personal 'sweet spot'. I've also found that really dialing in consistent smooth scrolling performance can require optimizing other factors including: OS settings, mouse software driver settings (I use Logitech Options), video card options (VSync especially) and even monitor options (often disabling motion 'enhancement' modes) because the end result is only as good as the whole stack.

It can help to ensure your hardware has sufficient performance to maintain your desired scrolling performance even when multi-tasking and the browser loading website with a bunch of JS and assets, especially on laptops and wireless connections. UMatrix helps block a lot of these loads.

If you have intermittent variable results, another thing to check is other FF extensions. I run quite a few extensions to customize my experience and have run across a couple (unrelated to scrolling, screen or visual appearance) that make scrolling performance on some sites inconsistent during background page load, perhaps some rare interaction of threads. Fortunately, none of them were any of the popular ones or ones I find essential.

I don't think it's possible to achieve truly 'perfect' scrolling behavior 100% of the time on all sites yet, at least I haven't been able to on any desktop browser or hardware combo I've tried. Currently, I'm quite happy with my FF configuration as all the sites I visit regularly perform very well and >95% of one-off sites do also.

m463(10000) 2 days ago [-]

I've noticed some sites affect scrolling somehow, either because they intercept it somehow, or you scroll over a text box or some other widget and the scrolling targets the box instead of the page.

I think if you can keep the mouse off the page and on the scrollbar at the side (I'm on mac) it scrolls more predicably.

weaksauce(10000) 2 days ago [-]

that's funny because in chrome i hate the scrolling because it doesn't have the middle mouse click autoscroll feature and all the 'solutions' to fix that are janky webextensions.

murrayn(10000) 1 day ago [-]

I'm so glad these FF config options exist:

mousewheel.acceleration.start mousewheel.acceleration.factor

I have them set to 1 and 15.

Works with Thunderbird as well!

robbyking(10000) 2 days ago [-]

For a while now I've gone and forth between Firefox and Chrome (Chrome always felt more consistent to me, but I liked Firefox's privacy and security features), but recently I went all in on Firefox it and feels a lot nicer than it did even 6 months ago.

I don't know what changed, but I haven't looked back.

gruez(10000) 2 days ago [-]

> It turns out that it's mouse wheel scrolling doesn't feel as snappy as other browsers

are you sure it's not because you have smooth-scroll enabled?

sharps1(10000) 2 days ago [-]

You can make the behavior more like other browsers by setting (in about:config)

to true (by double-clicking).

EDIT: Formatting.

psteinweber(10000) 1 day ago [-]

Exactly my story. Had this with my retina MBP from 2015 and couldn't understand how this is supposed to be usable.

Tried everything, even resetting the OS.

Have a new MPB since Monday, tried Firefox again, no issues at all.

philliphaydon(10000) 2 days ago [-]

Honestly I don't notice any difference between FF and Chrome for scrolling. What do you mean it doesn't feel as snappy?

stiray(10000) 2 days ago [-]

Add those add-ons:





I would really love to have more addins like this, doing one thing and doing it good. They will kill fingerprinting and as a proof, I was downvoted the next moment i posted the links.

simonebrunozzi(10000) 2 days ago [-]

Not sure that's the reason why you were downvoted.

Perhaps (I am trying to guess) it would have been more helpful to explain why these four add-ons, individually, are so necessary. Or perhaps a more expanded comment on why you picked these, and what effect they provide.

DavideNL(10000) 2 days ago [-]

So which version of Firefox will have this feature?

floatingatoll(10000) 2 days ago [-]

The post says that it's being rolled out gradually on Release channel, so as long as you haven't blocked automatic updates in various ways, you'll get it soon (assuming that the rollout continues without any significant issues being uncovered that halt it).

Presumably if you want to run ahead of that, there's ways to get it early via Beta or Nightly channels, but I don't know what those are.

svnpenn(10000) 2 days ago [-]

> Latest Firefox rolls out Enhanced Tracking Protection 2.0; blocking redirect trackers by default

Isnt that what is used with Google Search? For example if you go here:


the first result appears to be:


but its really:


martin-adams(10000) 2 days ago [-]

If that's true, surely that's going to play havoc for the Search Console

tamwahba(10000) 2 days ago [-]

It seems like the feature here is 'blocking' the trackers by deleting cookies from sites you haven't interacted with. It's not preventing the redirect as far as I understand.

input_sh(10000) 2 days ago [-]

More details here[0], but in short, they'll 'block them' by deleting cookies and site data of redirect trackers every 24h, preventing long term profile building, while not breaking the redirects.

There's also a 'Google search link fix' recommended extension that fixes those URLs (replaces https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&c... to https://en.wikipedia.org/wiki/Sunday in your example). Also available for Chrome and Opera.

[0] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Pri...

ffpip(10000) 2 days ago [-]

Go to uBlock Origin and add these to My Filters


sleavey(10000) 2 days ago [-]

I find this annoying and noticed this change a year or so ago. Google used to just give you the (as far as I could tell) unadulterated link. In fact, at the time I wondered how/if Google were actually able to track how many people clicked the top link in the results, because it seemed to be the real URL and stayed real even when clicked. I figured they either had so much faith in PageRank that they didn't need to monitor clicks, or they were hiding their tracking in a less obvious way.

Anyway, does anyone know a way in Firefox to stop sites from changing a URL's target when it gets clicked? This seems like it should be an about:config option.

shock(10000) 2 days ago [-]

Unfortunately, even with Enhanced Tracking Protection 2.0 you can still be tracked by using ETags. There are extensions for Firefox that block ETag tracking, but I haven't reviewed a specific extension sufficiently to recommend one.

ffpip(10000) 2 days ago [-]

Remove Etags with the CleanURL addons. Just turn off everything else it does, if you only want to remove ETag.

Clean URL cleans many links. No utm_source, no amazon trackers, no google/yandex changing links just right before you click it. It has adblock built in, so turn it off if you already have an adblocker.

You can also do it with https://addons.mozilla.org/en-US/firefox/addon/netflix-party...

Historical Discussions: In spite of an increase in Internet speed, webpage speeds have not improved (August 04, 2020: 684 points)
The Need for Speed, 23 Years Later (June 01, 2020: 4 points)
The Need for Speed, 23 Years Later (June 09, 2020: 2 points)
The Need for Speed, 23 Years Later (June 03, 2020: 2 points)
The Need for Speed (1997) (June 20, 2020: 1 points)

(694) In spite of an increase in Internet speed, webpage speeds have not improved

694 points 3 days ago by kaonwarb in 10000th position

www.nngroup.com | Estimated reading time – 8 minutes | comments | anchor

23 years ago, the internet was quite different from the one we use today. Google didn't exist yet, fewer than 20% of U.S. households had internet access, and those who did were using a dial-up connection.

It's no wonder that people complained about slow speeds on every website we tested back then, because the internet and the computers used to access it were painfully slow.

What is surprising is that, despite today's much faster network speeds and computer processors, people using the internet today are still plagued by the exact same frustration: slow websites.

The Internet Is Faster, but Websites Aren't

You might be wondering whether people simply don't notice how much faster today's sites are because their expectations have increased over time. While it's true that people's estimates of wait times are sometimes exaggerated, in this case it's not just a matter of distorted perceptions.

For the past 10 years, Httparchive.org has recorded page load times for 6 million popular websites. (Httparchive.org is a part of the InternetArchive.org, whom you may know as the folks behind the WayBack Machine). The results are not encouraging: for webpages visited from a desktop computer, the median load time hasn't improved. Today's websites aren't that much faster than they were 10 years ago.

Median page Onload time tracked by the Httparchive.org has remained about the same over the past 10 years, while, over the same period, the average internet speed for users in the U.S. (tracked by Akamai and Cable.co.uk) has steadily increased.

As you might guess, the story on mobile is even worse — connection speeds have improved for sure, but, over the past 10 years, the mobile page load times tracked by Httparchive have actually increased.

The average connection speed for U.S. mobile users has increased steadily in the past 10 years; meanwhile, the load times for mobile web pages over the same period have more than doubled. Connection speed data through 2017 is from Akamai (which in 2014 adjusted its methodology to record connection speeds from a larger sample of devices); data from 2018 and 2019 is from Opensignal.com, a mobile analytics company. Page OnLoad times for mobile webpages are as recorded by Httparchive.org.

Increases in internet speed clearly haven't solved the problem of slow websites. Of course, network speed is not the only factor that affects performance, so it's not reasonable to expect speeds to have completely kept pace with network connectivity. But it seems like huge increases in network speed should make it at least somewhat faster to browse the web.

You may be wondering if this data is really accurate, which is a fair question, as there are many different ways to measure performance and speed, such as by sampling different selections of websites or using different milestones to identify when a page is loaded. For example, in 2018, Google reported an average time of 7 seconds for mobile pages to load content above the fold. But, since above-the-fold loading times from 10 years ago aren't available, we can't draw conclusions about trends in that particular metric. In any case, even this more favorable number is still 7 times slower than the recommended response time for navigating web pages.) The Httparchive.org data is unique in that it's been collected using the same approach for the entire decade, allowing longitudinal comparison. This data strongly suggests that the websites people encounter today aren't that much faster than they were a decade ago.

How Slow Is Too Slow?

The basic rules of human perception of time provide a framework for understanding the effects of webpage delays: people can detect delays as short as 1/10th of a second, so anything that takes longer doesn't feel 'instant.' Delays of just 1 second are enough to interrupt a person's conscious thought process, changing the experience into one of waiting for the system to catch up, rather than feeling as though you are directly controlling the interface. This delay reduces conversion.

We've described how these delays of just a few seconds seriously hurt the experience of those trying to use a website. But you don't have to take our word for it — in the past 10 years, the effects of slow-loading web pages on site abandonment and conversion has been proven time and again:

  • 2009: Google and Bing both reported that even half-second delays in load time resulted in measurably lower conversion metrics (number of searches and revenue per user).
  • 2010: A Mozilla experiment which found that reducing the page load time by 2 seconds led to 15% higher conversion.
  • 2011: Gomez reported on 150M pageviews across 150 sites and found that pages that took 6 seconds to load were 25% more likely to be abandoned than pages which loaded in 2 seconds.
  • 2016: Google found that increasing the load time of its SERPs by half a second resulted in a 20% higher bounce rate.
  • 2016: Google found 53% of mobile visits ended if a page took longer than 3 seconds to load.
  • 2017: Akamai aggregated data from 17 retailers (7 billion pageviews) and found that conversion rates were highest for pages that loaded in less than 2 seconds; longer load times correlated with 50% drops in conversion rates and increased bounce rates, especially for mobile visitors.
  • 2018: BBC found that for every extra second of page load time, 10% of users will leave.
  • 2024: just kidding, but we're sure there will be many more compelling data points to add to this list in the future, because this problem won't go away.

If you're still not convinced that response time matters in your specific industry or for smaller organizations, check out this collection of case studies by WPOStats.com, which documents how performance optimization dramatically improves traffic, sales, and usage metrics across many different types of websites and industries.


All this data still does not precisely define a magic number at which your site is 'fast enough' or 'too slow.'

If you're in a group of people being chased by a bear, you only need to be faster than the slowest person in the group. But that's not how websites work: being faster than at least one other website, or even faster than the 'average' website, is not a great achievement when the average website speed is frustratingly slow.

Subsecond load times would be ideal; realistically, this target is incredibly difficult to achieve while also delivering rich and engaging content. But, if there's one thing that's been amply proven in all the experiments listed above, it's that each incremental improvement in speed pays off. Instead of settling for being 'fast enough,' investment in performance optimization should be driven by asking, 'how much more successful would we be if we were 1 second faster?' Reducing page load times by even a second, will improve your users' experience, and increase your conversion rates. The slower your website is, the more you have to gain from making it faster.


  1. Httparchive.org "Loading Speed" Report, 2010-2019, https://httparchive.org/reports/loading-speed#ol
  2. Akamai State of the Internet Connectivity Reports, 2010-2017 https://www.akamai.com/us/en/resources/our-thinking/state-of-the-internet-report/global-state-of-the-internet-connectivity-reports.jsp
  3. Cable.co.uk, "Worldwide Broadband Speed League" reports. (This data is download speed rather than the connectivity speed reported by Akamai for 2010-2017, so is slightly inflated relative to the Akamai data) https://www.cable.co.uk/broadband/speed/worldwide-speed-league/
  4. Opensignal.com, a mobile analytics company, "The State of Mobile Network Experience" 2019 report, https://www.opensignal.com/reports/2019/05/global-state-of-the-mobile-network
  5. Opensignal.com, a mobile analytics company, "The State of Wifi vs. Mobile Network Experience as 5G Arrives", https://www.opensignal.com/reports/2018/11/global-state-of-the-mobile-network

All Comments: [-] | anchor

rayiner(10000) 3 days ago [-]

It's comical. I've got 2 gbps fiber on a 10 gbps internal network hooked up to a Linux machine with a 5 GHz Core i7 10700k. Web browsing is just okay. It's not instant like my 256k SDSL was on a 300 MHz PII running NT4 or BeOS. Really, there isn't much point having over 100 mbps for browsing. Typical web pages make so many small requests that don't even keep a TCP connection open long enough to use the full bandwidth (due to TCP's automatic window sizing it takes some time for the packet rate to ramp up).

iso8859-1(10000) 3 days ago [-]

How does a web page decide whether a TCP connection is kept open?

Surely the webpage can only use HTTP, and there is no concept of a connection in JavaScript or HTML/CSS.

So it must be the responsibility of the browser to reuse connections, or to response multiplex efficiently in HTTP/2.

johnwalkr(10000) 2 days ago [-]

I have 10Gbps fiber and have the same experience, although downloading from steam or whatever is nice. It's still worth a small premium for being a new uncongested fibre network that doesn't slow down at night but the overall speed doesn't make a difference for the reasons you mentioned.

thereisnospork(10000) 3 days ago [-]

As someone else with gratuitously fast internet I almost wish I could preemptively load/render all links off of whatever page I'm on in case I decide to click on one. (I imagine this would be fairly wasteful).

pdimitar(10000) 3 days ago [-]

iMac Pro (10-core, 64GB ECC RAM, 2TB NVMe SSD) with 1GbE connection here.

Ever since I bought this machine (and swapped the ISP) I came to understand that it was not my machine at fault; it's most websites that are turtle-slow.

minerjoe(10000) 3 days ago [-]

Funny. I'm using a thinkpad T60 from over 10 years ago, clocked down to 1000 Mhz, because why not, and running a modified version of the links browser that I've hacked together with guile for expandability and every web page takes less than a second to load and render. Truly flying around the web. For many tasks I can run circles around your setup.

When I visit a web page, I load 1 file. 1 file. Show the page and then start loading the images, if any.

You can gain much by not loading any javascript, not caring a bit about css and having a very fast HTML parser and renderer.

Konohamaru(10000) 3 days ago [-]

Niklaus's Law strikes again.

schmudde(10000) 3 days ago [-]

I think it's a good law, for what it's Wirth.

zukzuk(10000) 3 days ago [-]

... which is just a special case of https://en.wikipedia.org/wiki/Jevons_paradox

joncrane(10000) 3 days ago [-]

I think this is a general problem with technology as a whole.

Remember when channel changes on TVs were instantaneous? Somehow along the way the cable/TV companies introduced latency in the channel changes and people just accepted it as the new normal.

Phones and computers were at one point very fast to respond; but now we tolerate odd latencies at some points. Apps for your phone have gotten much much bigger and more bloated. Ever noticed how long it takes to kill an app and restart it? Ever notice how much more often you have to do that, even on a 5-month old flagship phone? It's not just web pages, it's everything. The rampant consumption of resources (memory, CPU, bandwidth, whatever) has outpaced the provisioning of new resources. I think it might just be the nature of progress, but I hate it.

notyourwork(10000) 3 days ago [-]

> Ever noticed how long it takes to kill an app and restart it? Ever notice how much more often you have to do that, even on a 5-month old flagship phone?

Is this an android problem? I don't really ever have to close apps unless the app itself gets stuck into a broken state and forcing close to restart can correct the issue.

lumost(10000) 3 days ago [-]

Alternately, the value of low-latency experiences is not as high as we believed - or it's poorly measured.

Particularly in Enterprise software, the time to complete a workflow or view specific data matters a lot - the time to load a page is a component of that, but customers will gladly trade latency for a faster e2e experience with less manual error checking.

In consumer the big limiting factor is engagement, a low-latency experience will enhance engagement. However it's possible to hide latency in ways that weren't possible before such as infinite scrolls and early loading of assets. The engagement on the 50th social feed result has less to do with the latency to load the result, and more to do with how engaging the content is.

dylan604(10000) 3 days ago [-]

It used to be software would get written, and then iterated over to optimize and refine it to make it smaller/faster. That got too expensive, so the devs just depended on CPU speeds increasing and drives getting larger.

daxfohl(10000) 3 days ago [-]

Microwaves are another example. Used to just turn a knob to the number of minutes you want, and done. Now it's five button presses. (Maybe not exactly a latency thing, but a UX one that makes it slower).

massysett(10000) 3 days ago [-]

It is the nature of progress: progress is doing more with less. That's increased productivity.

Of course software now uses more computing resources, so that's not doing more with less. But the computer is cheap. What's expensive is the humans who program the computer. Their time is expensive, and getting experienced, expert humans is even more expensive.

So we now have websites that have rich features bolted together using frameworks. Same for desktop software, embedded systems, and whatever else. They're optimized for developer time and features, not for load time because that's not expensive, at least not in comparison.

As a user the only solution I see to this is to use old fashioned dumb products rather than cheaply developed "smart" ones. For instance I'm not going near a smart light switch, or a smart lawn sprinkler controller. Old dumb ones are cheap and easy and fast and predictable.

cuddlybacon(10000) 3 days ago [-]

> Remember when channel changes on TVs were instantaneous?

Remember when turning a TV on was <0.5s?

My current dumb TV takes a good while to turn on. When I press the power button, it takes about 1s for the indicator light to change than another 2 or 3 to begin displaying anything.

Aaronstotle(10000) 3 days ago [-]

I once had a glitch on my iphone back in iOS 9.x. The glitch made it that all transitions/animations were disabled and it was a fantastic experience to click an app and have it open instantly. Turning off animations in IOS settings doesn't make it as fast as that glitch did unfortunately.

wnevets(10000) 3 days ago [-]

>Somehow along the way the cable/TV companies introduced latency in the channel changes and people just accepted it as the new

One of the worst parts of the over the air digital switch over was how much harder it was to channel surf quickly.

dghughes(10000) 3 days ago [-]

>Remember when channel changes on TVs were instantaneous? Somehow along the way the cable/TV companies introduced latency in the channel changes and people just accepted it as the new normal

Phones used to be rotary dial but then touch tone phones with number buttons were introduced. I was reading an article about human brains and its expectations. Going from a touch tone phone to an old rotary style phone seems excruciatingly slow to our brains. Depending on the number a 1 on each it's very close in duration but a 9 or a 0 on a rotary compared to a touch tone 9 or 0 seems glacial in speed.

fractal618(10000) 3 days ago [-]

You had me at:

> Remember when channel changes on TVs were instantaneous?

There's nothing less satisfying than smushing down those rubbery remote control buttons for an extra 2000 to 4000 milliseconds to change the channel.

Don't get me started on entering the wrong channel numbers. That gives me PTSD.

leadingthenet(10000) 3 days ago [-]

You call it progress, I call it regress.

It seems to me that we have lower performance, exponentially higher resource consumption, and often no more functionality in a "modern" web app / Electron app compared to some desktop apps from the 90's. And for what? To have even worse UI's and UX's that never conform to platform guidelines? Where's the progress?

wanderoff42(10000) 3 days ago [-]

Your last paragraph is so true. I feel that's an issue we are facing a lot in every domain. I see people writing highly inefficient back-ends Because they believe anyways we are going to scale it horizontally, so it doesn't matter. The similar applies to frontends too I guess, people don't care to optimize apps either. They're like anyways the minimum configuration on which our apps run are getting better day by day, so why not build a resource hogger!

I personally believe we should start having 'bootcamps' which talk about optimizations, and the cost of stupid designs. I'm also looking forward to compile time and run-time optimizations so that we don't even have to rely on the developers for it.

skohan(10000) 3 days ago [-]

I remember thinking this when OS's started butting transparent blur effects on various UI layers. As someone who's worked with computer graphics, I understood that high quality blur effects are relatively expensive. Most computers these days can handle it without breaking a sweat, but It's like why are we using these resources on the desktop environment, who's job is basically to fade into the background and facilitate other tasks?

I don't think it's the nature of progress so much as it is laziness. Most developers (myself included) don't worry much about optimization until the UX performance is unacceptable.

I sometimes wonder what the world could be like if we just froze hardware for 5 years and put all of our focus on performance optimization.

DoreenMichele(10000) 3 days ago [-]

The rampant consumption of resources ...has outpaced the provisioning of new resources. I think it might just be the nature of progress, but I hate it.

Someone had to more or less decide to handle it that way for some reason. So I am skeptical that 'it's just the nature of progress.'

formerly_proven(10000) 3 days ago [-]

> Somehow along the way the cable/TV companies introduced latency in the channel changes and people just accepted it as the new normal.

The technical reason is that digital TV is a heavily compressed signal [1] (used to be MPEG2, perhaps they have moved on to h.264) with a GOP (group of pictures) length that is usually around 0.5-2 seconds. When you switch channels, the MPEG-2 decoder in your receiver needs to wait until a new GOP starts, because there is no meaningful way to decode a GOP that's 'in progress'.

[1] And the technical reason for the compression is that analog HD needs a lot more bandwidth than analog NTSC/PAL/SECAM, while raw HD transmission would need an absurd amount of bandwidth per channel (about a gigabit/s for 1080p30). So HD television pretty much requires the use of digital compression. Efficient digital video compression requires GOP structures.

iforgotpassword(10000) 3 days ago [-]

While I can see the technical reason why channel changes take a while with digital TV, I've always wondered why switching the digital input of your TV, or just changing the resolution of the connected devices takes so long. On most TVs it's over two seconds. The signal is in most cases just a stream of standalone, uncompressed frames. Switching from HDMI1 to HDMI2 should take a few milliseconds.

commandlinefan(10000) 3 days ago [-]

> I think it might just be the nature of progress, but I hate it.

I suspect that the root cause is that nobody understands what's going on from the UI down to the hardware, and nobody is incentivized (or even allowed) to spend the time it would take to actually do so.

kiba(10000) 3 days ago [-]

I think it might just be the nature of progress, but I hate it.

Not really. It's just not a priority.

sandworm101(10000) 3 days ago [-]

Remember when you could look at a book, a guide, to see what was on and when? It never took more than a minute or two. Then came the TV channel with the slowly-scrolling list of channels (early 90s). Try figuring out today which shows will be available tomorrow at a particular time... and whether you have subscribed to that channel. Good luck. I don't think it is even possible anymore.

jberryman(10000) 3 days ago [-]

The history of the automobile is another such example. Brief summary: it turns out the invention of the car didn't really save anyone time, it just enabled sprawl.

The purpose of technology (in the POSIWID sense) is to concentrate wealth.


lazyjones(10000) 3 days ago [-]

> a general problem with technology as a whole.

No, not with technology as a whole. With software ...

minimuffins(10000) 3 days ago [-]

I don't think it's the nature of progress. There's no question we have the _capacity_ to engineer our way out of these problems. They aren't unsolvable. But a lot would have to change before the necessary resources are mobilized against these kind of problems instead of churning out yet more bloat, which, let's face it, is what most of us are doing with our time every day.

Related: https://www.youtube.com/watch?v=kZRE7HIO3vk

I'm not expert enough to say if his technical solutions are correct, but it's a pretty good explication of the problem.

noja(10000) 3 days ago [-]

I swear cell phones have a slight latency compared to a landline too.

Unklejoe(10000) 3 days ago [-]

> Remember when channel changes on TVs were instantaneous?

I think about this every time I use the Hulu 'Guide' on my Fire TV. It's extremely slow and cumbersome.

I remember using the early digital cable boxes in like 2005 which were much more responsive, and honestly, the UI was much better too.

TheOtherHobbes(10000) 3 days ago [-]

Having used 14.4k dialup on a 486DX2-66MHz at 640 x 480 VGA, I really don't think 2020s technology is outstandingly slow.

I don't even think most phone apps are bloated, because most phone apps - and web sites - are just electronic forms decorated with a bit of eye candy.

Security and reliability worry me far more. Many sites have obvious bugs in $favourite_browser, and some just don't work at all. Some of this is down to ad blocking, but that shouldn't be a problem - and the flip side is that blocking ads, trackers, and unwanted cookies seems to do wonders for page load speeds.

0xdeadbeefbabe(10000) 3 days ago [-]

> Ever noticed how long it takes to kill an app and restart it? Ever notice how much more often you have to do that, even on a 5-month old flagship phone

At least car crashes are still low latency.

partiallypro(10000) 3 days ago [-]

The main culprit, imo is javascript. People/clients want more and more complex things, but javascript and its libraries are the main culprit. Image compression, minification...it helps, but if the page needs a lot of JS, it's going to be slower.

Slightly off topic, but I have a site that fully loads in ~2 seconds but Google's new 'Page Speed Insights' (which is tied to webmaster tools now) give it a lower score than a page that takes literally 45 seconds to fully load. Please someone at Google explain this to me. At least GTMetrix/Pingdom actually makes sense.

s1t5(10000) 3 days ago [-]

I'm not a web developer so I really have no idea about this - is WebAssembly a viable solution or have I just absorbed some hype without understanding the problems with JS?

bearjaws(10000) 3 days ago [-]

Users expect load times proportional to the content expected, if I am loading photoshop I don't expect it to be quick.

However if I am loading Reddit, I expect it to be fast, and it seems the websites that should load the fastest are now loading the most 'non-essential' JS, leading to worse performance than peoples expectations.

In my experience JS adds at most hundreds of miliseconds, and thats because people add dozens of marketing/tracking code that bog the site down.

If you run ghostery / ublock the javascript eval time shrinks dramatically. Our web app has a very large Angular app and it still renders in under 200ms, but we don't have any 'plugins' due to working with PHI.

EE84M3i(10000) 3 days ago [-]

Something I've noticed is that Dead Code Elimination for Javascript sucks. Yes, there are the (relatively) single 'tree shaking' algorithms, but there because of various language issues, it's very hard to mark code as completely unused. Because of this it's extremely common to pull in a library and have a bunch of code that's never used in production!

xenospn(10000) 3 days ago [-]

Giant CSS bundles as well.

csark11(10000) 3 days ago [-]

What's your site's URL?

jefftk(10000) 3 days ago [-]

If you post the links for the two sites I would enjoy looking at them to figure out why PSI is giving such strange scores

xchip(10000) 3 days ago [-]

The question is, why does your site needs so much javascript? Most prob is for tracking :/

neurostimulant(10000) 3 days ago [-]

Page Speed Insight mainly measure how fast your above-the-fold content loads. It doesn't matter if your page loads a bunch of heavy js and images as long as it's deferred/lazy-loaded and doesn't block initial render. For example, amp pages actually load a lot of js for its component, but it doesn't block above-the-fold render and thus scored really great on page speed insight. Personally, I think page speed insight is one of google's strategy to encourage people to use AMP more.

Edit: Also, I find it comical that when you include recaptcha v3 on your website, your page speed insight score can drop almost 20 points. It is as if google don't want you to use recaptcha at all.

cel1ne(10000) 3 days ago [-]

Yes, and the main culprit of JavaScript is excessive GC, because of excessive object-creation.

Developers are careless because JS is so fast, but they forget the hundreds of milliseconds of GC-pauses that occur because they never think about the memory they are allocating and throwing away each second.

heisenbit(10000) 3 days ago [-]

Angular, React, Vue, etc. are all getting better reducing the bloat. Javascript packaging and tree shaking is also much better. A lot of old compatibility stuff for IE etc. also can be dumped now. JS compilation got faster. The reason we are not seeing any speedup is the code that is not needed: Tracking and advertisements - this stuff seems to fill any size and speed gains.

dgb23(10000) 3 days ago [-]

That statement seems to general.

JS in of itself is not a performance issue in many cases. It can even improve performance in terms of speed/responsiveness.

rozenmd(10000) 3 days ago [-]

From what I've seen building PerfBeacon.com, it seems more like images in e-commerce sites are the worst offenders.

Slowest site I've tested so far had a page size of 8.5MB, 80% of that was images.

austincheney(10000) 3 days ago [-]

More specifically the culprit is generally unnecessary string parsing. Every CSS selector, such as querySelectors and jQuery operations, requires parsing a string. Doing away with that nonsense and learning the DOM API could make a JavaScript application anywhere from 1200x-10000x faster (not an exaggeration).

Most JavaScript developers will give up a lung before giving up accessing the page via selector strings. Suggestions to the effect are generally taken as personal injuries and immediately met with dire hostility.

vlunkr(10000) 3 days ago [-]

I think JS does play a big role, but my guess is that the 3rd party stuff is lots heavier than the scripts that actually make the page work. You have to write tons of code before the load time even matters, but when you have a bunch of 'analytics', advertising, and social media integration scripts it adds up, especially when each ad is essentially it's own page with images and scripts.

If you use Privacy Badger or similar plugins, you see that it's not uncommon for websites to have an obscene amount of these.

TLDR: I think ads are slowing the internet down way more than React apps.

nomel(10000) 3 days ago [-]

Perhaps they test with simulated reduced bandwidth?

johannes1813(10000) 3 days ago [-]

This reminds me a lot of Freakonomics podcast episode (https://freakonomics.com/2010/02/06/the-dangers-of-safety-fu...) where they discuss different cases where increased safety measures just encouraged people to take more risk, resulting in the same or even increased numbers of accidents happening. A good example is that as football helmets have gotten more protective, players have started hitting harder and leading with their head more.

Devs have been given better baseline performance for free based on internet speeds, and adjust their thinking around writing software quickly vs. performantly accordingly, so we stay in one place from an overall performance standpoint.

Taek(10000) 3 days ago [-]

In the case of devs it's not just staying in the same place though. You get more complete frameworks, more analytics, better ad engines, and faster development pace.

It might not be what we wanted, but it is a benefit

pravus(10000) 3 days ago [-]

This is known as Jevons paradox in economics and the classic example in modern times is rates of total electricity usage going up while devices have become ever more energy efficient.


kbuchanan(10000) 3 days ago [-]

To me it's more evidence that increased speed and reduced latency is not where our real preferences lie: we may be more interested in the capabilities of the technology, which have undoubtedly improved.

ClumsyPilot(10000) 3 days ago [-]

To me increasing tendency of Boeing to crash is evidence that safety is not where our real preferences lie.

To me increasing tendency of junk stocks getting AAA rating is evidence that profitable investment is not where our real preferences lie.

To me increased prevalence of obesity and heart disease is evidence that staying healthy and alive is not where our real preferences lie.

HumblyTossed(10000) 3 days ago [-]

I'm unclear who you mean when you say 'our' and 'we'. Developers? Producers? Consumers?

alkonaut(10000) 3 days ago [-]

Despite an increase in computer speed, software isn't faster. It does more (the good case) or it's simply sloppy, but that's not necessarily a bad thing because it means it was cheaper/faster to develop.

Same with web pages. You deliver more and/or you can be sloppier in development to save dev time and money. Shaking a dependency tree for a web app, or improving the startup time for a client side app costs time. That's time that can be spent either adding features or building another product entirely, both of which often have better ROI than making a snappier product.

tuatoru(10000) 3 days ago [-]

Why are you valuing the time of a developer so much more highly than the time of all the users of the web page?

Page load time affects every user; additional features only improve life for a few of them.

jbob2000(10000) 3 days ago [-]

It's the marketing team's fault. I proposed a common, standardized solution for showing promotions on our website, but no... they wanted iframes so their designers could use a WYSIWYG editor to generate HTML for the promotions. This editor mostly generates SVGs, which are then loaded in to the iframes on my page. Most of our pages have 5-10 of these iframes.

Can someone from Flexitive.com please call up my marketing coworkers and tell them that they aren't supposed to use that tool for actual production code?

Can someone also call up my VP and tell them they are causing huge performance issues by implementing some brief text and an image with iframes?

Can someone fire all of the project managers involved in this for pushing me towards this solution because of the looming deadline?

oblio(10000) 3 days ago [-]

Is your company making money? If so, they're doing their job :-)

rutherblood(10000) 3 days ago [-]

there's a really funny thing that happens with websites on phones. i try clicking something sometime after the page loads and just when i click it, bam! something new loads up on the page, all these elements shift up or down and what I just clicked on isn't at its place anymore and i end up clicking on something completely different. All this happens in split of a second just between the moments when I look and decide to click on something and before my finger actually does the tap. This is so so common across the mobile web. Such a stupid little thing but also highly annoying. Even if we could collectively solve this one little thing, i would consider our UI sensibility to have improved over the years.

cel1ne(10000) 3 days ago [-]

I know this from Android, but not from iOS.

ffggvv(10000) 3 days ago [-]

this isn't that surprising when you see they mean "throughput" and not "latency" when they talk about speed.

webpages aren't super large files so it would depend more on the latency of the request not Mbps

em-bee(10000) 3 days ago [-]

exactly. internet speed never was the issue. i am in a place where international network speed is a fraction of the domestic speeds. (often less than one mbit) yet websites are still just as fast. it all depends on how fast the server responds to the request, and almost never on how much the site has to load, unless there is a larger amount of images involved.

bilater(10000) 3 days ago [-]

But could the argument be made that we are loading a shit load more content so even though it feels slower you're getting a richer UX to work with?

Dahoon(10000) 3 days ago [-]

Richer is often worse though. So the worst of both sides.

CM30(10000) 3 days ago [-]

You could probably say the exact thing about video game consoles and loading times/download speeds/whatever. The consoles got more and more powerful, but the games still load in about the same amount of time (or more) than they used to, and take longer to download.

And the reasoning there is the same as for this issue for webpage speed or congestion on roads; the more resources/power is available for use, the more society will take advantage of it.

The faster internet connections get, the more web developers will take advantage of that speed to deliver types of sites/apps that weren't possible before (or even more tracking scripts than ever). The more powerful video game systems get, the more game developers will take advantage of that power for fancier graphics and larger worlds and more complex systems. The more road capacity we get, the more people will end up driving as their main means of transport.

There's a fancy term for that somewhere (and it's mentioned on this site all the time), but I can't think of it right now.

avani(10000) 3 days ago [-]

I think you may be referring to Jevon's Paradox: https://en.wikipedia.org/wiki/Jevons_paradox

m00dy(10000) 3 days ago [-]

new Playstation console has the fastest ssd in the market to address that issue.

setr(10000) 3 days ago [-]

One of my heuristics for video game quality is the main menu transition speed -- you only care about the menu animations once, on first view; after that, you want get something done (eg fiddle with settings, start the game, items, etc). So it should be fast, or whatever animation skippable with rapid tapping. Any game designer that doesn't realize this likely either doesn't have any taste, or is not all that interested in anyone actually playing the game.

This heuristic has served me stupidly well, and repeatedly gets triggered on a significant proportion of games -- and comes out correct

The actual level loading times of games doesn't matter all that much. Games go out of their way to be(feel) slow/sluggish/soft/etc

oblio(10000) 3 days ago [-]

While I agree with the idea and I am not happy about slow apps, the truth is, it's focused on technical details.

People don't care about speed or beauty or anything else than the application helping them achieve their goals. If they can do more with current tech than they could with tech 10-20 years ago, they're happy.

Dahoon(10000) 3 days ago [-]

>People don't care about speed

Every statically backed research on customer behaviour I have ever seen says otherwise. The more you slow down the page or app the less customers like and use it or buy the product being sold. As someone with a homemade site for our business I can say that it is extremely easy to be faster than 95% of sites out there and it makes a huge difference, also on Google. Tiny business with homemade website in top 1-3 on Google was mindbaffling easy because everyone use too many external sources and preschool level code. Especially the so-called experts. Most are experts in bloat.

pea(10000) 3 days ago [-]

So this has gotten to the point for me where it is a big enough burning painpoint where I would pay for a service which provided passably fast versions of the web-based tools which I frequently have to use.

In my day-to-day as a startup founder I use these tools where the latency of every operation makes them considerably less productive for me (this is on a 2016 i5 16GB MBP):

- Hubspot

- Gmail (with Apollo, Boomerang Calendar, and HubSpot extensions)

- Intercom (probably the worst culprit)

- Notion (love the app - but it really seems 10x slower than a desktop text editor should be imo)

- Apollo

- LinkedIn

- GA

- Slack

The following tools I use (or have used) seem fast to me to the point where I'd choose them over others:

- Basecamp

- GitHub (especially vs. BitBucket)

- Amplitude

- my CLI - not being facetious, but using something like https://github.com/go-jira/jira over actual jira makes checking or creating an issue so quick that you don't need to context switch from whatever else you were doing

I know it sounds spoiled, but when you're spending 10+ hours a day in these tools, latency for every action _really_ adds up - and it also wears you down. You dread having to sign in to something you know is sluggish. Realistically I cannot use any of these tools with JS disabled, best option is basically to use a fresh Firefox (which you can't for a lot of Gmail extensions) with uBlock. I tried using Station/Stack but they seemed just as sluggish as using your browser.

It's probably got a bunch of impossible technical hurdles, but I really want someone to build a tool which turns all of these into something like old.reddit.com or hacker news style experience, where things happen under 100ms. Maybe a stepping stone is a way to boot electron in Gecko/Firefox (not sure what happened to positron).

The nice things about tools like Basecamp is that because loading a new page is so fucking fast, you can just move around different pages like you'd move around the different parts of one page on an SPA. Browsing to a new page seems to have this fixed cost in people's minds, but realistically it's often quicker than waiting for a super interactive component to pull in a bunch of data and render it. Their website is super fast, and I think their app is just a wrapper around the website, but is still super snappy. It's exactly the experience I wish every tool I used had.

IMO there are different types of latency - I use some tools which aren't 'fast' for everything, but seem extremely quick and productive to use for some reason. For instance, IntelliJ/PyCharm/WebStorm is slow to boot - fine. But once you're in it, it's pretty quick to move around.

Can somebody please build something to solve this problem!

midrus(10000) 2 days ago [-]

Ohh God, can't upvote this enough. I feel the same. I work as a frontend dev and I just can't believe the amount of stuff we do in the user's browsers just because it's convenient for us, developers or because asking backend devs to do it will take longer or because it's the way to do it in React. SPAs can be faster of course, but most of the times they are not, and they are a lot worse than their equivalent rails or django app because your company just doesn't have the resources Facebook has. And even Facebook is terribly slow, so not sure what the benefits are at the end of the day.

Talking of reddit, I just cannot use it. I rely on old.reddit.com for now and the day it goes away I will only use it from a native client on my phone, or just not use it anymore.

I feel like repeating myself in every single comment I do on this topic but I really believe that tools such as Turbolinks, Stimulus or my favourite: unpoly are highly underrated. If we put 20% the effort we put on SPAs on building clean, well organized and tested traditional web applications we would be in a much better place, and faster (in the sense of shipping and of performance).

We should focus more on the end user and the business and a bit less on what's cool for the developers.

temporama1(10000) 3 days ago [-]

JavaScript is not the problem.

Computers are AMAZINGLY fast, EVEN running JavaScript. Most of us have forgotten how fast computers actually are.

The problem is classes calling functions calling functions calling libraries calling libraries.....etc etc

Just look at the depth of a typical stack trace when an error is thrown. It's crazy. This problem is not specific to JavaScript. Just look at your average Spring Boot webapp - hundreds of thousands of lines of code, often to do very simple things.

It's possible to program sanely, and have your program run very fast. Even in JavaScript.

xmprt(10000) 3 days ago [-]

I don't think people are claiming javascript execution speed is the culprit. Javascript can be slow but computers are also fast. However, loading all that javascript takes a long time especially if the website isn't optimized properly and blocks on non-critical code.

easterncalculus(10000) 3 days ago [-]

I think the problem is that languages like Javascript and object oriented languages in general actually incentivize this kind of design. Most of the champions of OOP rarely ever look at stack traces or anything relating to lower-level stuff (in my experience, in general). Then you take that overhead to the browser and expect it to scale to millions of users. It just doesn't make sense. No amount of TCO is going to fix the problem either.

APIs are going to be used as they're written, and as documented. So as much as there is a problem with people choosing to do things wrong, I think the course correction of those people is a strong enough force. At least in comparison to when the design incentivizes bad performance. There's basically nothing but complaining to the sky when the 'right' way is actually terrible in practice.

bob1029(10000) 3 days ago [-]

It doesn't have to be this way. I am not sure when there was a new rule passed in software engineering that said that you shall never use server rendering again and that the client is the only device permitted to render any final views.

With server-side (or just static HTML if possible), there is so much potential to amaze your users with performance. I would argue you could even do something as big as Netflix with pure server-side if you were very careful and methodical about it. Just throwing your hands up and proclaiming 'but it wont scale!' is how you wind up in a miasma of client rendering, distributed state, et. al., which is ultimately 10x worse than the original scaling problem you were faced with.

There is a certain performance envelope you will never be able to enter if you have made the unfortunate decision to lean on client resources for storage or compute. Distributed anything is almost always a bad idea if you can avoid it, especially when you involve your users in that picture.

pjmlp(10000) 2 days ago [-]

I am not aware of any such rule, given that I keep coding server rendering using Java and .NET stacks since ever.

The rule to pay attention to is not to follow the fashion industry of people wanting to sell books, conference talks, trainings, while adopting an wait-and-see attitude.

If you wait long enough then you are back at the beginning of the circle, e.g CORBA/DCOM => gRPC.

coding123(10000) 3 days ago [-]

I think that when the UI (in general) was starting to go server side rendered (again) people started to find ways to make it client rendered for speed (again). In fact I can imagine the guys at google building the first gmail said 'It doesn't have to be this way.'

FpUser(10000) 3 days ago [-]

'With server-side (or just static HTML if possible), there is so much potential to amaze your users with performance.'

Actually I am amazing my users with C++ data servers and all rendering done by JS in the browsers. What I do not do is hooking up those monstrous framework. My client side is pure JS. It is small and response feels instant.

tomc1985(10000) 3 days ago [-]

There is no rule, just tons and tons of mindless cargo-culting.

austincheney(10000) 3 days ago [-]

> It doesn't have to be this way.

Your could hire competent developers who know how these technologies actually work. Server side rendering is better but still not ideal, because the incompetence is reduced from the load event to merely just later user interactions. The performance penalty associated with JavaScript could be removed almost entirely by suppling more performant JavaScript regardless of where the page is rendered.

dave_sid(10000) 3 days ago [-]

You know what? I don't think the trend for client side rendering is the problem. It seems logical. The problem is they hijacking of client side development by frameworks like React that produce a 1 Gig bundle of JS soup and dependencies rolled into a memory hogging ball, when all you needs is 50 kilobytes of basic vanilla JavaScript that would download into the client before you can say "chrome has run out of memory"

cactus2093(10000) 3 days ago [-]

This type of anti-big-js comment does great on Hacker News and sounds good, but my personal experience has always been very different. Every large server-rendered app I've worked on ends up devolving to a mess of quickly thrown together js/jquery animations, validations, XHR requests, etc. that is a big pain to work on. You're often doing things like adding the same functionality twice, once on the server view and once for the manipulated resulting page in js. Every bit of interactivity/ reactivity that product wants to add to the page feels like a weird hack that doesn't quite belong there, polluting the simple, declarative model that your views started off as. None of your JS is unit tested, sometimes not even linted properly because it's mixed into the templates all over the place. The performance still isn't a given either, your rendering times can still get out of hand and you end up having to do things like caching partially rendered page fragments.

The more modern style of heavier client-side js apps lets you use software development best practices to structure, reuse, and test your code in ways that are more readable and intuitive. You're still of course free to mangle it into confusing spaghetti code, but the basic structure often just feels like a better fit for the domain if you have even a moderate amount of interactivity on the page(s). As the team and codebase grows the structure starts to pay off even more in the extensibility it gives you.

There can be more overhead as a trade-off, but for the majority of users these pages can still be quite usable even if they are burning more cycles on the users' CPUs, so the trade-offs are often deemed to be worth it. But over time the overhead is also lessening as e.g. the default behavior of bundlers is getting smarter and tooling is improving generally. You can even write your app as js components and then server-side render it if needed, so there's no need to go back to rails or php even if a blazing fast time to render the page is a priority.

EGreg(10000) 3 days ago [-]

This is coming from someone who built an entire server-side rendering framework with PHP and then added Node.js for sockets and other realtime stuff...

Client first apps are the future.

Look, this is what I was able to get with a mix of clientside and serverside... does it load fast?


The document is loaded from the server and then the client comes and fills in the rest. That first request can preload a bunch of data, to be sure. But then it can't be cached.

Please read THIS as to the myriad reasons why client-first is better:


xenocyon(10000) 3 days ago [-]

Is that really the central problem though? Or is it that there is so much cruft?

Most every time I try to load a web page my sole aim is accessing a little plain text and perhaps a picture or two if clearly relevant and illustrative. But (if it weren't for ublock or the like), for the few KB of the content I actually want I have to wade through irrelevant stock photos, autoplay videos, innumerable placements serving promotions/ads/clickbait, overlays, demands for entering my email address or logging in, social media icons and banners - and that's to say nothing of the stuff I don't see, the trackers and the scripts. Surfing the web like this is frankly a strain, one that we've accepted as normal because everyone does it.

If we serve cruft faster, certainly that will improve speeds, but those gains might simply motivate the powers that be to add more cruft so - just as the case with network speeds - we'll end where we started. We need to be radical and tear web pages down rather than merely focus on serving them faster through technical means.

jameslk(10000) 3 days ago [-]

Do you have any data to prove that server-side rendering will lead to 'faster' websites? The article definitely doesn't provide any.

Server-side rendering means the web page will be blank until the server responds. If a majority of the heavy lifting is done on the server, you increase the opportunity of slower server response times. That's a worse UX than a web page gradually loading on the client-side.

Things like CSS cannot be rendered on the server, yet CSS is often a bottleneck to rendering. Same goes for images and fonts. Where's the data showing 'client storage' and 'client compute' are the culprits of slow websites?

ZainRiz(10000) 3 days ago [-]

This sounds a lot like the old argument of developers not being careful about how much memory/cpu they use. Engineers have been complaining about this since the 70s!

As hardware improves, developers realize that computing time is way cheaper than developer time.

Users have a certain latency that they accept. As long as the developer doesn't exceed that threshold, optimizing for dev time usually pays off.

strken(10000) 3 days ago [-]

While SPAs are somewhat inefficient, I'm convinced that the unnecessary bloat is mostly related to

A) advertising and/or tracking

B) improperly compressed or sized assets

C) unnecessary embedded widgets like tweets or videos

D) insane things like embedding files into CSS using a data URI and therefore blocking rendering

E) nobody using prefetching properly

These are very loose figures for my computer and internet connection, but a small server-side rendered site is in the 10s of KB, and loads within 2 or 3 seconds. A small client-side rendered site is in the 100s of KB, maybe a little more or a little less, and takes 4 or 5 seconds. The sites that I really hate are in the 5 MB+ range and don't load for anywhere up to 10 or even 20 seconds, which goes above and beyond the bloat caused by client-side rendering.

wnevets(10000) 3 days ago [-]

It seems like a lot of websites could replace react/angular/framework with some simple jquery and html but that's not cool and not good on your resume. So now we have ui frameworks, custom css engines with a server side build pipeline deploying to docker just for a photo gallery.

Semiapies(10000) 3 days ago [-]

Server-side is no panacea. I starting paying attention recently, and WordPress-based sites frequently take well over a second to return the HTML for pages that are essentially static—and that's considered acceptably fast by many people running WP-based sites. Slow WP sites are even worse.

joe_the_user(10000) 3 days ago [-]

"Why should I care about future generations – what have they ever done for me?" Groucho Marx

Sure, off-loading the work onto the client doesn't help speed.

But Groucho would say now, 'Why should I care about the client? What has the client ever done for me?'

Sure, the web pages don't load any faster 'cause they're now running a cr*p load of javascript. And that javascript is running more and more annoying ads. And that's because ads support most websites and there's a finite ad budget in the world and that budget is naturally attracted to the most invasive ads available.

I often consult d20pfsrd.com, a site that hosts the open gaming license rules to the Pathfinder rule system (D&D fork/spin-off). Information itself is just static text and once was, apparently, supported by text adds. But now, naturally, it serves awful video ads as well. I would strongly suspect the site isn't getting more money for this, it's just that now that advertisers can run this stream of garbage, advertisers must run this stream of garbage.

ihuk(10000) 2 days ago [-]

admin.google.com is a great example of unnecessary, over-engineered and almost comically bad client side rendering.

First off it's painfully slow. Then you go to manage users. There's a list of users; so far so good. Then you try to add a user. First there's a loading(?!) indicator. Then add user dialog shows up. You fill in the form and add a user. Dialog closes and the list of users does not refresh. You don't see the user you just created. It shows up only after you reload the page. How does something like that event happen?

Diesel555(10000) 3 days ago [-]

From Dan Abramov from the React team yesterday https://twitter.com/dan_abramov/status/1290289129255624706

> We've reached peak complexity with SPA. The pendulum will swing a bit back to things on the server. But it will be a new take — a hybrid approach with different tradeoffs than before. Obviously I'm thinking React will be a part of that wave.

Combine that with NextJS's new features in server side rendering, I think we are going back to that. My React site is server side rendered.

1vuio0pswjnm7(10000) 3 days ago [-]

If the speeds have increased, are we (users) paying an increase in price? If yes, what are we subsidising?

reaperducer(10000) 3 days ago [-]

I am not sure when there was a new rule passed in software engineering that said that you shall never use server rendering again and that the client is the only device permitted to render any final views.

Maybe it's coming from the schools.

I worked with a pair of fresh-outta-U devs who argued vehemently that all computation and bandwidth should be offloaded onto the client whenever possible, because it's the only way to scale.

When I asked about people on mobile with older devices, they preached that anyone who isn't on the latest model, or the one just preceding it, isn't worth targeting.

The ferocity of their views on this was concerning. They acted like I was trying to get people to telnet into our product, and simply couldn't wrap their brains around the idea of performance.

I left, and the company went out of business a couple of months later. Good.

amw-zero(10000) 3 days ago [-]

There's no arguing against server rendering being simpler - it objectively consists of fewer components. But to say that it is more performant by nature? You can't actually argue that. There are plenty of performance downsides to rendering a full page in response to every user action. Don't forget, an SPA can fetch a very small amount of data and re-render a very small part of the page in response to a user action. There is a noticeably performance benefit to doing that.

Performance doesn't only boil down to the first page load. Hopefully your application sessions are long, and the longer page load time gets amortized across the session if interactions are performant after that.

Note, I primarily work on enterprise apps where the sessions are long, and the workflows are complicated. Of course page load time matters much more for a static site or a blog / content site.

But to claim that SPAs are all cost with no benefit is just disingenuous. Of course they have their own set of trade offs. But there is a reason people use them, and it's not some conspiracy fueled by uneducated people. Server rendering isn't some objective moral higher ground.

stiray(10000) 3 days ago [-]

I think this is comming from the same direction as desktop computing.

Try to install windows 7 on brand new machine (hopefully you will get the drivers). Regardless of all the new 'improvements' in windows 10, it will fly.

What we did to hw performance increase is just staggering. Instead of having software that works much faster, we ate the performance for the sake of cheaper development - filling software with lasignas of huge libraries that in most cases are not needed, employing incompetent developers that know how to code but are clueless about the computer/os/browser they are running their code on, not optimizing anything,...

Suboptimal technologies, suboptimal languages (to make the developers less expensive for the companies), lack of knowlidge. It stacks up, todays webpage is easly a few megabytes, for a kb of text. Due to huge list of third party dependancies that are not really needed, but are there for minor details on web page. It is just crazy and far worse than when server side rendering was 'THE thing'.

The result is here, not only on web.

neycoda(10000) 2 days ago [-]

Static things should be generated server-side (even if those static things are dynamically-gwnerated), and things that change on the page after load, interactively or by timers, should be rendered browser (client) side.

Client side rendering has become popular because it reduces server load... but unfortunately increases processing time in the browser, which can slow things down for users.

There's solutions (like Gatsby, which is its own layer of complexity) and cheats and workarounds, but the standard should be that if a page will contain the same information for a certain state on an initial load for that page, that content should be generated server-side. Anything that can't be or is dependant on browser spec or user interaction should be client-side.

I just don't believe in making the user process a bunch of repetitive static stuff that can be cached on the browser from the server or compressed before sending. There's gotta be more consideration of user experience over server minimization.

Pxtl(10000) 3 days ago [-]

Users say they want this, but then people get upset when list filters don't automatically update the listing when you click them, or when you have a set of drilldown cascaded dropdown and the first doesn't automatically filter the second, or when inapplicable widgets are visible that should be hidden when inapplicable.

reissbaker(10000) 3 days ago [-]

I don't think the problem is client rendering, especially when you consider latency. Sending views over a network, especially a high-latency, low-reliability one like a cell network, isn't going to beat the performance of doing UI rendering on-device.

Same goes with storage. What's faster: a readdir(3) call that hits an SSD to see how many mp3s you have downloaded, or traversing a morass of cell towers and backbone links in order to iterate over a list you fetch from a distributed data store running in AWS? It's the readdir(3) call.

Giant bundles of unnecessary JS are also bad for performance, but there's a reason why when we had more limited computing resources, we didn't try to make every screen an HTML document that needed a roundtrip to some distant server to do anything. Computing happened on your computer. That's also why native apps on smartphones exist: Apple tried to make everything websites with the first iPhone, it was unbearably slow, and so they pivoted to native apps.

Plain old documents are best as HTML and CSS. Highly-interactive UI isn't.

sunseb(10000) 3 days ago [-]

Laravel LiveWire + Alpine.js can do 80% of SPA needs from your PHP backend code.


glouwbug(10000) 3 days ago [-]

You're not the only one who thinks this way: https://twitter.com/ID_AA_Carmack/status/1210997702152069120

tonymet(10000) 3 days ago [-]

I'm developing a tiny-site search engine. upvote if you think this product would interest you. The catalog would be sites that load < 2s with minimal JS

agumonkey(10000) 3 days ago [-]

Totally, also websites of low size simply, and websites with simple structure (something that renders nice in a terminal)

slx26(10000) 3 days ago [-]

I'm making a site that's very light, minimal everything... except for a section that will contain some games. How do you handle those cases?

MaxBarraclough(10000) 3 days ago [-]

Wirth's Law: Software is getting slower more rapidly than hardware is becoming faster.


joncrane(10000) 3 days ago [-]

Yes! There's a 'law' or 'rule' for this! My favorite other ones are Hanlon's Razor and the Robustness Principle.

mnm1(10000) 3 days ago [-]

This is often intentional. Take a look at any OS or software with animations. Slowness for slowness sake. The macOS spaces change has such a slow animation, it's completely useless. Actually, macOS has a ton of animations to slow things down, but luckily most can be turned off. Not the spaces thing. Android animations are unbearable and slow things down majorly. Luckily they can be turned off, but only by unlocking developer mode and going in there. It's clear whoever designed these things has never heard of UX in their lives. And since these products are coming from companies like Google and Apple, which have UX teams, it leads me to think that most UX people are complete idiots. Or UX is simply not a priority at all and these companies are too stupid to assemble a UX team for their products. Hard to say which is the case.

sgloutnikov(10000) 3 days ago [-]

This post reminded me of this quote:

'The hope is that the progress in hardware will cure all software ills. However, a critical observer may observe that software manages to outgrow hardware in size and sluggishness. Other observers had noted this for some time before, indeed the trend was becoming obvious as early as 1987.' - Niklaus Wirth

ehsankia(10000) 2 days ago [-]

Yes and no. I think they grow hand in hand. It's similar to how games nowadays still run at ~30-60fps just like games 10 years ago and games 20 years ago. It might sound like a silly example but it showcases what I mean. As long as you're above a certain threshold, most people would rather have better graphics than more framerate. Similarly, both users and developers would rather have access to more stuff than have a website that loads slightly faster.

I could try to write a game in pure assembly, and that may run super freaking fast, but that would take me orders of magnitude more time than writing the same game in Unity. Similarly, I could write an app from scratch, or I could make a web app with dozens of powerful frameworks like React, and ship it with Electron, in probably a fraction of the time. As long as the app is usable, then this is the smarter move. I can develop 10 features in the time it takes to develop 1 for my competitor.

The industry moves fast, users want features, not a site that loads 100ms faster.

cyberlurker(10000) 3 days ago [-]

(Somewhat related) Although I've been out of the scene a while, it always felt like PC gaming was best when the GPU manufactures hadn't just introduced a significantly improved architecture AND the consoles were at the end of their life cycle. The worst times were when the newest, super expensive GPU monster would come out and the newest consoles were released.

I started to not feel excited for more powerful hardware. The performance ceiling was higher but I felt the quality (gameplay, performance, art style, little details) of games temporarily dropped even though the graphics marginally improved on the highest end.

the-pigeon(10000) 3 days ago [-]

Yeah this is the way it always will be.

Speed is determined by business requirements, not capabilities.

I have hundreds of opportunities for optimizations in my apps. I could make them fly. But the business side says the current speed is good enough and to focus on new functionality. So that's what I do.

Tomis02(10000) 3 days ago [-]

The problem of course isn't the internet 'speed' but latency. ISPs advertise hundreds of Mbps but conveniently forget to mention latency, average packet loss and other connection quality metrics.

opportune(10000) 3 days ago [-]

Correct me if I'm wrong but I believe ISPs use a combination of hardware and software to "throttle down" network connections as they attempt to download more data. For example if I try to download a 10GB file on my personal computer I'll start off at something like 40mbps and it will take 15 seconds before they start allowing me to scale up to 300mbps. I assume that when downloading things like websites, which should only take tens or hundreds of ms, this unthrottling could also be a significant factor in addition to latency, depending on what the throttling curve looks like.

Also ISPs oversell capacity, which they've probably always done, so even if you're paying for a large bandwidth that doesn't mean you'll ever get it.

sarego(10000) 3 days ago [-]

As someone who just recently worked on reducing page load times these were found to be the main issues

1- Loading large Images(below the fold/hidden) on first load 2- Marketing tags- innumerable and out of control 3- Executing non critical JS before page load 4- Loading noncritical CSS before page load

Overall we managed to get page load times down by 50% on average by taking care of these.

Wowfunhappy(10000) 3 days ago [-]

Can someone who understands more about web tech than me please explain why images aren't loaded in progressive order? Assets should be downloaded in the order they appear on the page, so that an image at the bottom of the page never causes an image at the top to load more slowly. I assume there's a reason.

I understand the desire to parallelize resources, but if my download speed is maxed out, it's clear what should get priority. I'm also aware that lazy loading exists, but as a user I find this causes content to load too late. I do want the page to preload content, I just wish stuff in my viewport got priority.

At minimum, it seems to me there ought to be a way for developers to specify loading order.

Moru(10000) 3 days ago [-]

I wish 50% would be enough but some local news pages take 15 seconds after the last modernisation. And that is with ad-blockers...

Rapzid(10000) 3 days ago [-]

Add to that a ton of un-cached images will run up against the browser concurrent connection limit(non-HTTP2) and cause queuing.. Now you have latency back in the mix between batches :/

bdickason(10000) 3 days ago [-]

The article doesn't dig into the real meaty topic - why are modern websites slow. My guess would be 3rd party advertising as the primary culprit. I worked at an ad network for years and the number of js files embedded which then loaded other js files was insane. Sometimes you'd get bounced between 10-15 systems before your ad was loaded. And even then, it usually wasn't optimized well and was a memory hog. I still notice that some mobile websites (e.g. cnn) crash when loading 3p ads.

On the contrary, sites like google/Facebook (and apps like Instagram or Snapchat) are incredibly well optimized as they stay within their own 1st party ad tech.

davidcsally(10000) 3 days ago [-]

As someone working on improving we vitals metrics, ad networks are 100% the biggest issue we face. Endless redirects, enormous payloads, and non optimized images. And on top of all that, endless console logs. I wish ad providers had higher standards.

kevincox(10000) 3 days ago [-]

Do you know why modern sites are slow? Because time isn't invested in making them faster. It isn't a technical problem, people take shortcuts that affect the speed. They will continue to do so unless the website operators decide that it is unacceptable. If some news website decided that their page needed to load in 1s on a 256Mbps+10ms connection they would achieve that, with external ads and more. However they haven't decided that it is a priority, so they keep adding more junk to achieve other goals.

It's simply Parkinson's Law

jakub_g(10000) 3 days ago [-]

Facebook desktop web is far from being optimized. On my fiber connection & high-end non-Mac laptop, loading 'Messenger' tab takes about 15-20 seconds.

Having said that, FB and Google properties (like YouTube) have insane edge over the rivals by having full control over the advertising stack.

TwoBit(10000) 2 days ago [-]

I think the reason why web sites are as slow as they are is because they can be. Double the speed of the Internet and all computers and soon enough the content will expand to be just as slow as before.

netflixandkill(10000) 3 days ago [-]

JS whitelisting and ad blocks anecdotally confirm this.

The number of sites that will be loading images and js from three or four or more different ad/tracking and CDNs is nuts, plus the various login and media links, and I feel zero guilt for not participating in this advertising insanity.

Tightly put together pages with only a handful of JS loads are damn near instant over gigabit fiber.

SilasX(10000) 3 days ago [-]

You had me until the end. I still see significant lag and mismatches on Facebook's page from simple actions like pulling up my notifications or loading a comment from one of them. For the later, I see it grind its gears while it loads the entire FB group and then scrolls down a long list to get to the comment I want to see.

(This is on a Macbook Air from 2015, but these are really simple requests.)

erickhill(10000) 3 days ago [-]

I agree on the Ad Networks causing massive site bloat.

However, in terms of Facebook, I'd say it was well optimized considering its complexity prior to the recent redesign. But ever since the new design, my Macbook Pro can barely type on that site anymore. My machine has a 2.5 GHz Quad-Core Intel Core i7 and 16 GB of RAM.

That's pretty sad. Responsive design is a great idea, but in terms of how it is sometimes implemented you're getting X numbers of styles to load a page.

austincheney(10000) 3 days ago [-]

Not a surprise. Most people writing commercial front end code have absolutely no idea how to do their jobs without 300mb of framework code. That alone, able to write to the basic standards and understand simple data structures, qualifies my higher than average salary for a front end developer without having to do any real work at work.

phendrenad2(10000) 3 days ago [-]

You found someone willing to pay for developers who can get better performance out of a site? Must be FAANG?

jerf(10000) 3 days ago [-]

An uncompressed RGB 1920x1080 bitmap is 6,220,800 bytes. When your webpage is heavier than a straight-up, uncompressed bitmap of it would be... something's gone wrong.

We're not quite there, since web pages are generally more than one screen, but we're getting close. Motivated searchers could probably find a concrete example of such a page somewhere.

propelol(10000) 3 days ago [-]

They don't have any idea because very few are willing to pay to have a website that's faster, so there is no incentive to learn it.

sumtechguy(10000) 3 days ago [-]

That mentality is not 'new'.

No, you can not add the whole c++ std lib into our code. Yes, I know it is useful. Yes, that will save you 2 hours of work. However, the code no longer even fits into the 1MB of flash we have. Yes we can ask for a new design the management would love to spend 500k on spinning a new design and getting all of the paperwork for it done, and our customers would love replacing everything they have for functionally equivalent hardware that now costs 50 dollars more each. But at least you saved 2 hours writing some code.

aequitas(10000) 3 days ago [-]

There is also a big 'runs-on-my-machine' factor. Where the machine is the developers high-tier laptop hooked up to Gigabit LAN and Fiber WAN with <5ms ping and >100Mbit symmetric.

Luckely there are tools like Lighthouse[0] but with all the abstractions and frameworks inbetween it is often impossible to introduce the required changes without messing up the quality or complexity of the code/deployment.

[0] https://developers.google.com/web/tools/lighthouse

paradite(10000) 3 days ago [-]

In other news:

In spite of an increase in mobile CPU speed, mobile phone startup time have not improved (in fact they became slower).

In spite of an increase in desktop CPU speed, time taken to open AAA games have not improved.

In spite of an increase in elevator speed, time taken to reach the median floor of an building have not improved.

My point is, 'webpage' has evolved the same way as mobile phones, AAA games and buildings - it has more content and features compared to 10 years ago. And there is really no reason or need to making it faster than it is right now (2-3 seconds is a comfortable waiting time for most people).

To put things in perspective:

Time taken to do a bank transfer is now 2-3 seconds of bank website load and a few clicks (still much to improve on) instead of physically visiting a branch / ATM.

Time taken to start editing a word document is now 2-3 seconds of Google Drive load instead of hours of MS Office Word installation.

Time taken to start a video conference is now 2-3 seconds of Zoom/Teams load instead of minutes of Skype installation.

ClumsyPilot(10000) 3 days ago [-]

This hipster atttiude of replacing proper software with barely functional webapps really gets on my nerves.

People use and will continue using Skype and especially MS Office. It is much more functional that gSuite alternatives and moving people to castrated and slow webapps is not progress.

SilasX(10000) 3 days ago [-]

>My point is, 'webpage' has evolved the same way as mobile phones, AAA games and buildings - it has more content and features compared to 10 years ago. And there is really no reason or need to making it faster than it is right now (2-3 seconds is a comfortable waiting time for most people).

What features? I don't know anything substantive a site can deliver to me today that it was not capable of 10 years ago. The last major advance in functionality was probably AJAX, but that doesn't inherently require huge slowdowns and was around more than 10 years ago.

The rest of your comparisons are dubious:

>Time taken to do a bank transfer is now 2-3 seconds of bank website load and a few clicks (still much to improve on) instead of physically visiting a branch / ATM.

This is the same class of argument as saying that (per Scott Adams), 'yeah 40 mph may seems like a bad top speed for a sports car, but you have to compare it hopping'. (Or the sports cars of 1910). Yes, bank sites are faster than going to ATM. Are they faster than bank sites 20 years ago? Not in my experience.

>Time taken to start editing a word document is now 2-3 seconds of Google Drive load instead of hours of MS Office Word installation.

Also not comparable: you pay the installation MS Word time-cost once, and then all future ones are near instant. (Also applies to your Skype installation example.)

mc32(10000) 3 days ago [-]

Network latency. Bandwidth is the new MHz.

superkuh(10000) 3 days ago [-]

This could be part of it. The shift to mobile computers which are necessarily wireless which means a random round trip time due to physics, which means TCP back-off. That, combined with the tendency to require innumberable external JS/CDN/etc resources that require setting up new TCP connections work together to make mobile computers extra slow at loading webpages.

cozzyd(10000) 3 days ago [-]

Bandwidth and latency aren't the same thing! High-latency networks sometimes don't ever load some websites, even if there is reasonable bandwidth. I remember one time when I was in Greenland having to VNC to a computer in the US to do something on some internal HR website that just wouldn't load with the satellite latency.

emptyparadise(10000) 3 days ago [-]

I wonder how much worse things will get when 5G becomes widespread.

sumoboy(10000) 3 days ago [-]

You know for sure your phone bill will increase.

zoomablemind(10000) 3 days ago [-]

Another factor is a wider use of all sorts of CMS (WordPress etc) for content presentation, combined with often slower/underpowered shared hosting and script heavy themes.

On some cheap hosters it may take a second just to startup the server instance, that's before any of the outgoing requests are done!

ClumsyPilot(10000) 3 days ago [-]

Wordpress is not actually that bad if you use it responsibly

commandlinefan(10000) 3 days ago [-]

Yep - to the executives, saving a couple of (theoretical) hours of development work is worth paying a few extra seconds per page load. Of course, the customers hate it, but the customers can't go anywhere else, because the executives everywhere else are looking for ways to trade product quality for (imaginary) time to market.

superkuh(10000) 3 days ago [-]

The last 5 years there has been a dramatic shift away from HTML web pages to javascript web applications on sites that have absolutely no need to be an application. They are the cause of increased load times. And of them, there's a growing proportion of URLs that simply never load at all unless you execute javascript.

This makes them large in MB but that's not the true cause of the problem. The true cause is all the external calls for loading JS from other sites and then the time to attempt to execute that and build the actual webpage.

Eric_WVGG(10000) 3 days ago [-]

The last two years have had a dramatic shift away from SPA and toward JAMstack, which is more or less pre-rendered SPA. The result is not only faster than SPA, but faster than the 2008-2016 status quo of server-scripted Wordpress and Drupal sites.

I blew off JAMstack as another dumb catchphrase (well, it is a dumb catchphrase), but then inherited a Gatsby site this past spring. It absolutely knocked my socks off. The future is bright.

tomjen3(10000) 3 days ago [-]

Yes. And while I will continue to defend it in case of interactive dashboards, webapps and status screens if you are not writing on of those, you shouldn't be using javascript. If we want to fix the internet, browsers should ask the user for permission for each cookie and javascript should be disabled by default.

thomasfortes(10000) 3 days ago [-]

Yeah, it is perfectly possible to build javascript heavy websites without making them unusable until all is downloaded and processed...

I'm rebuilding a site for a client and using a bunch of dynamic imports, if you don't touch a video route, you'll not download the videojs bundle at all, I set a performance baseline for the site to be interactive and anything that makes it go over the baseline needs a good reason to be there.

wnevets(10000) 3 days ago [-]

That is what happens when you hire react devs to build your static website.

Alex3917(10000) 3 days ago [-]

> The last 5 years there has been a dramatic shift away from HTML web pages to javascript web applications on sites that have absolutely no need to be an application.

SPAs are a lot easier to keep secure though, so if you don't want your private data leaked then they're a much better option.

Softcadbury(10000) 3 days ago [-]

Yes you're right, but keep in mind that with SPA, only the first loading is slower, then you don't need to reload again when changing pages.

elbelcho(10000) 3 days ago [-]

If left unoptimized, JavaScript can delay your pages when they try to load in users' browsers. When a browser tries to display a webpage, it has to stop and fully load any JavaScript files it encounters first.

throwaway0a5e(10000) 3 days ago [-]

To quote an exec at a major CDN:

'wider pipe fit more shit'

(yes he actually said that, to an entire department, the context was that people will fill the pipe up with junk if they're not careful and it made more room to deliver value by not sucking)

thrownaway954(10000) 3 days ago [-]

exactly this. i remember when programmers took the time to make sure their programs didn't take up alot of memory. as we got more ram, many became lazy about memory optimization cause well... the computer has plenty. same thing here with webpages. there was a time where you needed to optimize your site cause of the modem that everyone used. now everyone has dsl or higher so there isn't an incentive to optimize your site.

joncrane(10000) 3 days ago [-]

I've recently started using a Firefox extension called uMatrix and all I can say is, install that and start using your normal web pages and you'll very quickly see exactly why web pages take so long to load. The number and size of external assets that get loaded on many websites is literally insane.

WrtCdEvrydy(10000) 3 days ago [-]

It's the cascade of doom... we have an internal benchmark without ads, and it's funny to see without ads we load something like 6Mb of JS but if you load the ads, the analytics cascade of hell will load 350Mb of JS to display one picture.

colmvp(10000) 3 days ago [-]

I've been using uMatrix for ages and it was baffling to me how some websites that are literally just nice looking blogs have an unreal number (i.e. 500+) of external dependencies.

ipnon(10000) 3 days ago [-]

It's easy to convince atechnical people to adopt uMatrix when you show them that they can watch shows without ads and load pages in less than a second.

anonyfox(10000) 3 days ago [-]

I just rewrote my personal website ( https://anonyfox.com ) to become statically generated (zola, runs via github Action) so the result is just plain and speedy HTML. I even used a minimal classless „css framework" and ontop I am hosting everything via cloudflare workers sites, so visitors should get served right from CDN edge locations. No JS or tracking included.

As snappy as I could imagine, and I hope that this will make a perceived difference for visitors.

While average internet speed might increase, I still saw plenty of people browsing websites primarily on their phone, with bad cellular connections indoor or via a shared WiFi spot, and it was painful to watch. Hence, my rewrite (still ongoing).

Do fellow HNers also feel the „need for speed" nowadays?

pdimitar(10000) 3 days ago [-]

Very interested on how you used Zola. The moment I wanted to customize title bars and side bars and I was basically on my own. Back then I didn't have the desire (or expertise) to reverse-engineer it.

Have you found it easy to customize, or you went with the flow without getting too fancy?

40four(10000) 3 days ago [-]

I have a similar setup for my personal site, although it's still a work in progress. I've really been interested in JAMstack methods lately. I build the the static site with Eleventy, and have a script to pull in blog posts from my Ghost site. To bad I haven't really written any blog posts though, maybe one day :) Anhhow, I really like Cloudflare workers, would recommend!

astura(10000) 2 days ago [-]

Looks good and loads instantly.

onemiketwelve(10000) 3 days ago [-]

Thank you sensei

ww520(10000) 3 days ago [-]

That's very cool. Nice little project to speed the site. One data point. A cold loading takes about 2.2 seconds; subsequent loads take about 500ms, from a cafe in the Bay Area using a shared wifi.

The cold loading stats:

     Load Time 2.20 s
     Domain Lookup 2 ms
     Connect 1.13 s
     Wait for Response 68 ms
     DOM Processing 743 ms
     Parse 493 ms
     DOMContentLoaded Event 11 ms
     Wait for Sub Resources 239 ms
     Load Event 1 ms
Edit: BTW, the speed is very good. I've tried similar simple websites and got similar result. Facebook login page takes 13.5 seconds.
otter-in-a-suit(10000) 3 days ago [-]

I have done the same with Hugo on my blog[0], but actually had to fork an existing theme to remove what I would call bloat.[1]

The interesting thing for me is, while I personally certainly feel the 'need for speed' and appreciate pages like yours (nothing blocked, only ~300kb), most people do not. Long loading times, invasive trackers, jumping pages (lazily loading scripts and images), loading fonts from spyware-CDNs - are only things 'nerds' like us care about.

The nicest comment on my design I heard was 'Well, looks like a developer came up with that' :)

[0] https://chollinger.com/blog/ [1] https://github.com/chollinger93/ink-free

ntumlin(10000) 3 days ago [-]

Very impressive. One cool thing you can further do to improve perceived speed potentially at the expense of some bandwidth is to begin to preload pages when a link is hovered. There are a couple of libraries that will do this for you.

It can shave 100 - 200 ms off the perceived load time, and since your site is already near or below that threshold it might end up feeling like you showed the page before anyone even asked for it.

user5994461(10000) 3 days ago [-]

There is no support for comments in the blog and no pictures at all. No images, no thumbnails, no banner, no logo, no favicon.

Also, no share button. No top/recommended articles. No view counter.

Once you start adding medias it will be quite a bit slower. Once you start implementing basic features expected by users (comments and related articles for a blog) it's gonna be yet again slower.

I remember when my first article went viral out of the blue, I think have to thank the (useless) share buttons for that. Then it did 1TB of network traffic over the next days, largely due to a pair of GIF. That's how bad pictures can be.

djaychela(10000) 3 days ago [-]

That's fantastic - as near to instantaneous as you need, and it's actually slightly odd having a page load as quickly as yours does; we've become programmed to wait, despite all the progress that's happened in hardware and connectivity. The only slightly slow thing was the screenshots on the portfolio page as the images aren't the native resolution they're being displayed at.

Does the minification of the css make a big difference? I just took a look at it using an unminifier, and it was a nice change to see CSS that I feel I actually understand straight away, rather than thousands of lines of impenetrable sub-sub-subclasses.

marvinblum(10000) 3 days ago [-]

That's perfect! Most pages should load instantaneous, at least those serving text for the most part.

I did the same for my website [1], and I hope this becomes more of a standard for 'boring old' personal pages and blogs.

[1] https://marvinblum.de/

nicbou(10000) 1 day ago [-]

Yes, a lot of people are browsing in less-than-ideal conditions. Many apps fall on their face when you try to use them on a German train with spotty reception.

jesterson(10000) 2 days ago [-]

That's fantastic, all _static_ site need to have this rendering speed, but unfortunately static content applicable to very narrow niche. Most sites have to provide dynamic content to certain range and this is where it becomes incredibly slow

1vuio0pswjnm7(10000) 3 days ago [-]

'Do fellow HNers also feel the need for speed nowadays?'

I stopped using graphical browsers many years ago. I use a text-only browser and a variety of non-browser, open source software as user-agents. Some programs I had to write myself because AFAIK they did not exist.

The only speed variations I can detect with human senses are associated with the server's response, not the browser/user-agent or the contents of the page. Most websites use the same server software and more or less the same 'default' configurations so noticeable speed variations are rare in my UX.

randomguy3344(10000) 3 days ago [-]

Hey brother, I made an account just to reply to your comment, I enjoyed your website and grew my knowledge reading it.

Just wanted to let you know there's a typo @ https://anonyfox.com/tools/savings-calculator/

```Aside from raw luck this ist still the best```

umyemri(10000) 3 days ago [-]

Alignment of list on https://anonyfox.com/grimoire/elixir/ seems a bit off.

Love the style though. Very crisp, very snappy.

NorwegianDude(10000) 3 days ago [-]

Seems mostly good to me after cloudflare caches it, but you have made one annoying mistake: you forgot to set the height of the image, so it results in content shift. Other than that, it's great! :)

rafaelturk(10000) 3 days ago [-]

zola ?

mbar84(10000) 3 days ago [-]

If you would specify the width/height of the image, you could avoid the page reflow that makes the quicklinks jump down.

Historical Discussions: The Truth Is Paywalled but the Lies Are Free (August 03, 2020: 677 points)

(678) The Truth Is Paywalled but the Lies Are Free

678 points 4 days ago by praptak in 10000th position

www.currentaffairs.org | Estimated reading time – 24 minutes | comments | anchor

Paywalls are justified, even though they are annoying. It costs money to produce good writing, to run a website, to license photographs. A lot of money, if you want quality. Asking people for a fee to access content is therefore very reasonable. You don't expect to get a print subscription to the newspaper gratis, why would a website be different? I try not to grumble about having to pay for online content, because I run a magazine and I know how difficult it is to pay writers what they deserve.

But let us also notice something: the New York Times, the New Yorker, the Washington Post, the New Republic, New York, Harper's, the New York Review of Books, the Financial Times, and the London Times all have paywalls. Breitbart, Fox News, the Daily Wire, the Federalist, the Washington Examiner, InfoWars: free! You want "Portland Protesters Burn Bibles, American Flags In The Streets," "The Moral Case Against Mask Mandates And Other COVID Restrictions," or an article suggesting the National Institutes of Health has admitted 5G phones cause coronavirus—they're yours. You want the detailed Times reports on neo-Nazis infiltrating German institutions, the reasons contact tracing is failing in U.S. states, or the Trump administration's undercutting of the USPS's effectiveness—well, if you've clicked around the website a bit you'll run straight into the paywall. This doesn't mean the paywall shouldn't be there. But it does mean that it costs time and money to access a lot of true and important information, while a lot of bullshit is completely free.

Now, crucially, I do not mean to imply here that reading the New York Times gives you a sound grasp of reality. I have documented many times how the Times misleads people, for instance by repeating the dubious idea that we have a "border crisis" of migrants "pouring into" the country or that Russia is trying to "steal" life-saving vaccine research that should be free anyway. But it's important to understand the problem with the Times: it is not that the facts it reports tend to be inaccurate—though sometimes they are—but that the facts are presented in a way that misleads. There is no single "fact" in the migrant story or the Russia story that I take issue with, what I take issue with is the conclusions that are being drawn from the facts. (Likewise, the headline "U.S. Says Hussein Intensifies Quest For A-Bomb Parts" is technically accurate: the U.S. government did, in fact, say that. It was just not true.) The New York Times is, in fact, extremely valuable, if you read it critically and look past the headlines. Usually the truth is in there somewhere, as there is a great deal of excellent reporting, and one could almost construct a serious newspaper purely from material culled from the New York Times. I've written before about the Times' reporting on Hitler and the Holocaust: it wasn't that the grim facts of the situation were left out of the paper, but that they were buried at the back and treated as unimportant. It was changes in emphasis that were needed, because the facts were there in black and white.

This means that a lot of the most vital information will end up locked behind the paywall. And while I am not much of a New Yorker fan either, it's concerning that the Hoover Institute will freely give you Richard Epstein's infamous article downplaying the threat of coronavirus, but Isaac Chotiner's interview demolishing Epstein requires a monthly subscription, meaning that the lie is more accessible than its refutation. Eric Levitz of New York is one of the best and most prolific left political commentators we have. But unless you're a subscriber of New York, you won't get to hear much of what he has to say each month.

Possibly even worse is the fact that so much academic writing is kept behind vastly more costly paywalls. A white supremacist on YouTube will tell you all about race and IQ but if you want to read a careful scholarly refutation, obtaining a legal PDF from the journal publisher would cost you $14.95, a price nobody in their right mind would pay for one article if they can't get institutional access. (I recently gave up on trying to access a scholarly article because I could not find a way to get it for less than $39.95, though in that case the article was garbage rather than gold.) Academic publishing is a nightmarish patchwork, with lots of articles advertised at exorbitant fees on one site, and then for free on another, or accessible only through certain databases, which your university or public library may or may not have access to. (Libraries have to budget carefully because subscription prices are often nuts. A library subscription to the Journal of Coordination Chemistry, for instance, costs $11,367 annually.)

Of course, people can find their ways around paywalls. SciHub is a completely illegal but extremely convenient means of obtaining academic research for free. (I am purely describing it, not advocating it.) You can find a free version of the article debunking race and IQ myths on ResearchGate, a site that has engaged in mass copyright infringement in order to make research accessible. Often, because journal publishers tightly control access to their copyrighted work in order to charge those exorbitant fees for PDFs, the versions of articles that you can get for free are drafts that have not yet gone through peer review, and have thus been subjected to less scrutiny. This means that the more reliable an article is, the less accessible it is. On the other hand, pseudo-scholarhip is easy to find. Right-wing think tanks like the Cato Institute, the Foundation for Economic Education, the Hoover Institution, the Mackinac Center, the American Enterprise Institute, and the Heritage Foundation pump out slickly-produced policy documents on every subject under the sun. They are utterly untrustworthy—the conclusion is always going to be "let the free market handle the problem," no matter what the problem or what the facts of the case. But it is often dressed up to look sober-minded and non-ideological.

It's not easy or cheap to be an "independent researcher." When I was writing my first book, Superpredator, I wanted to look through newspaper, magazine, and journal archives to find everything I could about Bill Clinton's record on race. I was lucky I had a university affiliation, because this gave me access to databases like LexisNexis. If I hadn't, the cost of finding out what I wanted to find out would likely have run into the thousands of dollars.

A problem beyond cost, though, is convenience. I find that even when I am doing research through databases and my university library, it is often an absolute mess: the sites are clunky and constantly demanding login credentials. The amount of time wasted in figuring out how to obtain a piece of research material is a massive cost on top of the actual pricing. The federal court document database, PACER, for instance, charges 10 cents a page for access to records, which adds up quickly since legal research often involves looking through thousands of pages. They offer an exemption if you are a researcher or can't afford it, but to get the exemption you have to fill out a three page form and provide an explanation of both why you need each document and why you deserve the exemption. This is a waste of time that inhibits people's productivity and limits their access to knowledge.

In fact, to see just how much human potential is being squandered by having knowledge dispensed by the "free market," let us briefly picture what "totally democratic and accessible knowledge" would look like. Let's imagine that instead of having to use privatized research services like Google Scholar and EBSCO, there was a single public search database containing every newspaper article, every magazine article, every academic journal article, every court record, every government document, every website, every piece of software, every film, song, photograph, television show, and video clip, and every book in existence. The content of the Wayback Machine, all of the newspaper archives, Google Books, Getty Images, Project Gutenberg, Spotify, the Library of Congress, everything in WestLaw and Lexis, all of it, every piece of it accessible instantly in full, and with a search function designed to be as simple as possible and allow you to quickly narrow down what you are looking for. (e.g. "Give me: all Massachusetts newspaper articles, books published in Boston, and government documents that mention William Lloyd Garrison and were published from 1860 to 1865.") The true universal search, uncorrupted by paid advertising. Within a second, you could bring up an entire PDF of any book. Within two seconds, you could search the full contents of that book.

Let us imagine just how much time would be saved in this informational utopia. Do I want minute 15 of the 1962 Czechoslovak film Man In Outer Space? Four seconds from my thought until it begins. Do I want page 17 of the Daily Mirror from 1985? Even less time. Every public Defense Department document concerning Vietnam from the Eisenhower administration? Page 150 of Frank Capra's autobiography? Page 400 of an economics textbook from 1995? All in front of me, in full, in less than the length of time it takes to type this sentence. How much faster would research be in such a situation? How much more could be accomplished if knowledge were not fragmented and in the possession of a thousand private gatekeepers?

What's amazing is that the difficulty of creating this situation of "fully democratized information" is entirely economic rather than technological. What I describe with books is close to what Google Books and Amazon already have. But of course, universal free access to full content horrifies publishers, so we are prohibited from using these systems to their full potential. The problem is ownership: nobody is allowed to build a giant free database of everything human beings have ever produced. Getty Images will sue the shit out of you if you take a historical picture from their archives and violate your licensing agreement with them. Same with the Walt Disney Company if you create a free rival to Disney+ with all of their movies. Sci-Hub was founded in Kazakhstan because if you founded it here they would swiftly put you in federal prison. (When you really think about what it means, copyright law is an unbelievably intensive restriction on freedom of speech, sharply delineating the boundaries of what information can and cannot be shared with other people.)

But it's not just profiteering companies that will fight to the death to keep content safely locked up. The creators of content are horrified by piracy, too. As my colleagues Lyta Gold and Brianna Rennix write, writers, artists, and filmmakers can be justifiably concerned that unless ideas and writings and images can be regarded as "property," they will starve to death:

Is there a justifiable rationale for treating ideas—and particularly stories—as a form of "property"? One obvious reason for doing so is to ensure that writers and other creators don't starve to death: In our present-day capitalist utopia, if a writer's output can be brazenly copied and profited upon by others, they won't have any meaningful ability to make a living off their work, especially if they're an independent creator without any kind of institutional affiliation or preexisting wealth.

Lyta and Brianna point out that in the real world, this justification is often bullshit, because copyrights last well beyond the death of the person who actually made the thing. But it's a genuine worry, because there is no "universal basic income" for a writer to fall back on in this country if their works are simply passed around from hand to hand without anybody paying for them. I admit I bristle when I see people share PDFs of full issues of Current Affairs, because if this happened a lot, we could sell exactly 1 subscription and then the issue could just be copied indefinitely. Current Affairs would collapse completely if everyone tried to get our content for free rather than paying for it. (This is why you should subscribe! Or donate! Independent media needs your support!)

At the end of last year, I published a book on socialism, and at first some conservatives thought it funny to ask me "if you're a socialist, can I have it for free?" They were quieted, though, when I pointed out that yes, they could indeed have it for free. All they needed to do was go to the local socialized information repository known as a public library, where they would be handed a copy of the book without having to fork over a nickel. Anyone who wants to read my book but cannot or does not want to pay for it has an easy solution.

I realized, though, as I was recommending everyone get my book from the library rather than buying it in a bookstore, that my publisher probably didn't appreciate my handing out this advice. And frankly, it made me a little nervous: I depend for my living on my writing, so if everyone got my book from the library, it wouldn't sell any copies, and then my publisher wouldn't pay me to write any more books. We can't have too many people using the socialized information repository when authors are reliant on a capitalist publishing industry! In fact, a strange thing about the library is that we intentionally preserve an unnecessary inefficiency in order to keep the current content financing model afloat. Your library could just give you DRM-free PDFs of my book and every issue of Current Affairs for free, but instead they make you go to the magazine room or check out one of a limited number of copies of the book, because while we want books and magazines to be free, we cannot have them be as free as it is possible to make them, or it would hurt the publishing industry too much. (Libraries preserve the fiction that there are a select number of "copies" available of a digital book, even though this is ludicrous, because abandoning the fiction would hurt publishers. They could offer every book ever written to anyone at any time. They just can't do it legally.)

I also realized, however, that I wouldn't care how many people got my book for free if my compensation operated on a different structure, where I was paid by the number of people who read it rather than the number of people who bought it. "Impossible!" you say. "Where would the money come from?" We can imagine such a set-up quite easily, though. We have our universal public knowledge database, and anyone who wants to can type in the title of any of my books and read them for free.** But the number of people who read the book is tracked, and I am compensated two dollars for every person who reads it (a pittance, but that's about what authors get for their sales). Current Affairs, likewise, is granted a budget proportional to its readership. Compensated from where? Budget from where? Why, from the universal public knowledge database of course. But from where do they get their money? Why, from taxes.* Free at point of use services are not some alien concept. The NHS compensates doctors while charging patients nothing. (Of course, compensation for producers wouldn't even be that much of an issue in a society with a Universal Basic Income and where the basics of life were guaranteed. I wouldn't care about making any money on my books if I could live decently regardless.)

Now, I am sure there will be those who argue that any universal knowledge access system of this kind will inhibit the creation of new work by reducing the rewards people get. But let us note a few facts: first, dead people cannot be incentivized to be creative, thus at least everything ever created by a person who is now dead should be made freely available to all. The gatekeepers to intellectual products made by the dead are parasites the equivalent of a private individual who sets up a gate and a tollbooth in the middle of a road somebody else has already built and starts charging people if they want to pass. Actually, since parasites latch onto the living, they are better compared with corpse-eating worms.

Second, creators are already exploited: Spotify is very much like the universal searchable information database for music, it just operates for profit rather than for artists, and rights-holders get a fraction of a cent per Spotify play, an amount that must itself split between the label, the producer, the artist, and the songwriter. The CEO of Spotify has said that if artists want more money, they should make more music. (He is worth $4 billion.) And if you ever want to make a professor laugh, ask how much they make from royalties on their published academic articles. As Adam Habib, Vice-Chancellor for research at the University of Johannesburg explains, academic publishing is a "completely feudal system":

"The costs of the research production are borne by the universities, and as a result, by public monies, in most cases. Then, private companies publish the research, and charge the universities and public institutions for the very research outputs that they paid for. This is effectively the subsidy of the private sector by public money. There is a myth that this is an example of entrepreneurialism. In my view, all it does is facilitate enrichment at public cost with huge consequences for those most disadvantaged."

This problem has not been fixed by the rise of "open-access" scholarship, because it hasn't removed the profit motive, so poor countries are still getting screwed by the existing publishing model.

Third, when considering the free information repository's effects on content creation, you cannot look only at one side of the equation. The question of how much productivity would be inhibited by the state declining to enforce the copyrights of academic journal publishers and Getty Images must be weighed against the phenomenal unleashing of human productive power that universal free access to all human knowledge would create. You must add up how much researchers could do with the time that they now have to spend trying to track down and access things. No more would a certain thing only be in a certain library and accessible through an inter-library loan request. No more would librarians have to spend any time managing subscriptions rather than helping with searches. Researchers in the developing world would no longer be utterly unable to compete with American libraries that can afford vast fees. (I can tell you, personally, that as someone who is constantly having to find obscure used books for research and then order them and wait sometimes weeks for them to arrive, I could produce far more quickly if I could see the full content of the book in ten seconds, and I am constantly exasperated by Google Books' "snippet view.")

Furthermore, we would have to consider what would happen in a society where the relative accessibility and cost of truth versus lies was adjusted. What if every online course was free? What if textbooks cost nothing instead of $200? What if we made it as easy and cheap as possible to find things out and were guided by the desire to create the greatest possible access to knowledge rather than by economic considerations? I do not know what would happen, but I hope some rogue state (or microstate or seastead) that doesn't mind pissing off the world's most powerful corporations and governments tries to storm the Bastille of information and free every bit and byte from its artificial prison. The only thing stopping them is law, and what is law but a threat?

The good news about our times is that the possibilities for democratizing knowledge are greater than ever. We could not have started Current Affairs in 1990 unless we had about ten times more money than what we actually had. Sharp left YouTubers are fighting hard to combat propaganda and debunk bad arguments, there are tons of great podcasts, and even Twitter has its uses. (Where else do you get to yell at powerful and influential people and personally tick them off?) But it is still true that Fox News and PragerU and the American Enterprise Institute have a hell of a lot of money to blast out their message as widely as possible. There is nothing on the left of remotely comparable size and influence.

But we are working on it. We are a long way from the world in which all knowledge is equally accessible. Hopefully someday our patchwork of intentionally-inefficient libraries will turn into a free storehouse of humanity's recorded knowledge and creativity. In the meantime, however, we need to focus on getting good and thoughtful material in as many hands as possible and breaking down the barriers we can. At Current Affairs we have no paywall, even though this might cost us some money, because we are trying to make it as easy as possible to hear what we have to say. This is what the right does. They tell people what to think, offer them books and pamphlets and handy five-minute YouTube videos. On the left we are not nearly as slick.

We can't afford to keep our reach to those who like us so much that they are willing to pay money to listen, because then the free bullshit wins. It's hard for small media institutions to figure out the right balance of depending on ads, paywalls, and donations. The money has to come from somewhere, after all. A lot of the times, that means a heavy dependence on ads—the traditional model of magazines has been ad-revenue based, not subscription-based—so that paywalls are actually the less corrupted model; a podcaster who sells their product on Patreon rather than giving it away but filling it with mattress and "box-of-shit-a-month" ads has an important kind of freedom: they only have to please the audience, not the sponsors. At Current Affairs, we sell subscriptions to keep the lights on, but even one person who could have read an article and doesn't is a loss. (I wish I could give my book to everyone too but my publisher won't let me. I did make another one free, though.) The Guardian and the Intercept provide a lot of valuable material to the public for free because they don't have paywalls, but the Guardian is funded by a trust and the Intercept by a benevolent billionaire. (Such funding sources make things much easier. Attention benevolent billionaires: we have a donate page.) Perhaps paywalls can help publications like the New York Times and the New Statesman from having to partner with "branded content" suppliers like Shell Oil and Cigna, but at the expense of limiting reach. More reason to have publications funded by the centralized free-information library rather than through subscriptions or corporate sponsorship.

Creators must be compensated well. But at the same time we have to try to keep things that are important and profound from getting locked away where few people will see them. The truth needs to be free and universal.

* I will not be reading angry emails from Modern Monetary Theorists. I am assuming a country without a sovereign currency, so there.

** You could, of course—and I am sure many people would want to—offer the universal database with a paywall and something like an "allowance" each month (e.g. 5 books, 100 newspaper/magazine articles, 50 scholarly journal articles, 20 films, and as many court records as they like) above which there would be a cost, and waive that cost for anyone below a certain income level. But this would involve means-testing, which makes everything needlessly complicated and inevitably means that some people will not access things who otherwise would. We do not means-test the public library and we should not means-test the universal public knowledge database.

All Comments: [-] | anchor

thundergolfer(10000) 4 days ago [-]

This is a long-ish article, as per usual for N Robinson, and I think people ITT are focusing only on the start. Yes free vs. subscription news media is an interesting and important issue, but this article goes on to make a more wide-reaching and incisive criticism of how Capitalism structures the production and distribution of information or knowledge (Current Affairs is a socialist magazine).

Information production is dominated by the super-rich and the state, and access to knowledge (not merely information) is carefully controlled for the benefit of and use by the highly educated, wealthy elite.

An aspect that I often think about is how the vast majority of truly top-tier multimedia (books, articles, movies) has been created decades ago. A person could, for example, spend multiple lifetimes reading the amazing academic and artistic literature in the public domain. But such an existence is terribly unprofitable, so it's heavily discouraged in favour of a content-diet full of pop-culture ephemera. Alan Kay said about Pop Culture, that it 'is all about identity and feeling like you're participating, It has nothing to do with cooperation, the past or the future—it's living in the present.' Such over-stimulating, disorienting, and fashionable multimedia, perhaps perfected by Buzzfeed, is making us energetic consumers, and idiots. Amused to death drinking from a firehose of 'free'.

fortran77(10000) 4 days ago [-]

No, they're focusing only on the title. People on HN are no better than the people who get their news from Facebook memes. They just _think_ they're better.


hinkley(10000) 4 days ago [-]

I wonder sometimes if all of those people who sat around dreaming about what the Web would do for the world ever get together and ask what the hell happened.

What's worst is that a lot of these conversations happened when memory of Eternal September were still relatively fresh. For my own part, I was maybe 22, new to the industry, and had not yet begun to appreciate Carlin's, 'Scratch any cynic and you will find a disappointed idealist.' Boy howdy.

The infrastructure that grants us the Enlightenment or Democracy also, when taken ad absurbum, brings us to Idiocracy.

The price of freedom is eternal vigilance. Turns out Reason extracts the same bargain.

mkolodny(10000) 4 days ago [-]

The fact that I can read your comment suggests that the Web has done something pretty incredible - it has made it possible for anyone with internet access to share information with billions of people.

I think that this article is written from an idealist perspective. It's asking: How can we take this accessible Web of journalistic information, and make access to it free, while still providing professional journalists with an income?

The author's answer to that question is: have the government pay journalists rather than citizens. One way to make that happen could be to centralize journalism in a non-profit, Spotify-esque app, and have donors and government sponsor the app.

Another similar, less ideal option is to create this centralized, Spotify-esque app for journalism, and then paywall the app. Since paying a monthly fee for dozens of individually paywalled newspapers/websites seems unreasonable to me, I don't pay for any of them. Yet, I do pay for Spotify. And if I could pay a monthly fee to read all journalism on the web, I probably would.

wing-_-nuts(10000) 4 days ago [-]

When I first saw google books and google scholar I thought 'Oh my god, this is amazing, this will change everything!'. I imagined that a decade or two in the future, being able to read pretty much any book or paper I wanted for a reasonable monthly fee. The search functionality was an absolute godsend!

Now? Google books has been pretty much sidelined and abandoned. I feel like I've watched the second library of Alexandria burn, and I am heartbroken.

My only consolation, is sites like sci-hub and libgen have made textbooks and papers downloadable, albeit illegally. I still really wish I could search within every book.

fossuser(10000) 4 days ago [-]

I think this is one case where the end of the product is not Google's fault.

Books got enormous push back from legacy publishers. They also had trouble with copyright law around orphaned works, but copyright protection increasing over time from 14yrs to 90yrs after the death of the author (and moving from requiring registration to being default) really made it hard for them to do what they wanted.

It's a shame that something initially intended to 'promote the progress of science and useful arts' is now used to inhibit it.

divbzero(10000) 4 days ago [-]

Google Books in 2004 [1], Google Scholar in 2004 [2], and Google Patents in 2006 [3] felt like the golden age of Google's mission to "organize the world's information and and make it universally accessible and useful" [4]. If you think back to the state of the web at the time, those services were expansive in their scope and ambition when they were launched. I give Google credit for continuing to run and maintain them to this day.

[1]: https://en.wikipedia.org/wiki/Google_Books

[2]: https://en.wikipedia.org/wiki/Google_Scholar

[3]: https://en.wikipedia.org/wiki/Google_Patents

[4]: https://www.google.com/search/howsearchworks/mission/

GordonS(10000) 4 days ago [-]

I still find Google Scholar really useful for searching for papers. It only has access to the full text for some percentage of papers, but you can usually find ones of interest in scihub.

sunstone(10000) 4 days ago [-]

And the Current Affairs site is free so...what is that telling us?

rmorey(10000) 4 days ago [-]

Current Affairs is primarily a print magazine (i am a delighted subscriber). They publish some (though not all) of their articles for free on the web

makomk(10000) 4 days ago [-]

I think a more accurate description is that the elite narrative is paywalled but the one for the proles is free - and the elite are absolutely convinced their narrative is the truth. The contents of, say, New York Times or Washington Post reporting on the state of US Covid-19 testing and contact tracing and how it compares to the rest of the world, the evidence on mask wearing, or anything to do with Trump or protests in no way resembles the truth.

Reading them leaves people absolutely convinced that - for example - the US is behind South Korea in per-capita coronavirus testing, despite this not having been true for a long time. Seriously, the number of people I've seen on here and in the Times comment section pointing to this one specific, untrue thing as clear proof Trump must've sabotaged Covid testing because there was no way that the US could have remained behind South Korea for so long otherwise was astounding - and even more astounding was the cleverness of the articles worded to trick people into believing this, or the boldness of the Times in literally printing a news headline accusing Trump of being the liar for pointing out the exact opposite was true.

hootbootscoot(10000) 4 days ago [-]

I highly disagree. Rupert Murdoch is surely one of the most elite of the elites and his vast media empire has managed to brainwash certain folks into calling his vision 'populism' due strictly to his outreach efforts.

It would also appear that you didn't read the article, as your point is not the topic of the article.

The article is about copyright law running counter to information accessibility, running counter to an interpretation of 'freedom of speech'.

The article is about economic rather than technological barriers to information sharing.

That's the problem with knee-jerk first-paragraph-read comments.

smolder(10000) 4 days ago [-]

How does it matter at all whether we're testing more than South Korea now? They did test far more at the onset of the crisis and have effectively contained the spread. We have dramatically more cases and a greater need for testing now than they do. Why would you get hung up on current testing rates when comparing the US to S. Korea? It is beside the point of who responded better. (Which they almost certainly did.)

loriverkutya(10000) 4 days ago [-]

Lies are not free, they are paid by ads and a complete industry is built on keeping this going. The price of the ads are built into the price of the product/service you buy, so we are basically paying for the lies too, it's just not as obvious as paying to get through the paywall.

jaggirs(10000) 4 days ago [-]

No, you are getting paid for the lies and advertisements you see. You get paid with free apps.

shultays(10000) 4 days ago [-]

They are free for consumers

andyjohnson0(10000) 4 days ago [-]

I think the meaning is pretty clear. Dishonest/misleading sources of information are intentionally designed to have a vary low barrier to entry, while trustworthy sources are forced to use off-putting paywalls due to the inherent higher cost of what they do. Hence the asymmetry.

dilandau(10000) 4 days ago [-]

Great point. You could say that the 'lies' are propaganda, as much as the advertisement that they are the vehicle for. Inundated with this much propaganda, which points to the impotence of the individual, no wonder we're seeing so much mass unrest.

deltron3030(10000) 4 days ago [-]

Conspiracies are a never ending source for new ownable keywords and terminology, which combined with the 'research it yourself mantra' only drives people deeper into the ecosystem of alternative book stores and prepper online shops etc. If people would know how the web works they'd be more sceptic about the media they encounter and their calls to action.

andrewtbham(10000) 4 days ago [-]

> it is not that the facts it reports tend to be inaccurate

yes it is.

I really lost faith in nyt when they ran article about running out of running out of charge in a Tesla. When they were caught red handed lying they still wouldn't admit to it. Their response was: 'Problems With Precision and Judgment, but Not Integrity, in Tesla Test'

They just don't get it. People are sick of their BS and they need to just comp to it and change. The days of BSing people are over.




SnotJockey(10000) 3 days ago [-]

Yeah, That pissed me off too, It stank of petrochemical company bribery or willful ignorance.

That said they still produce some quality journalism and I wouldn't write off an entire news organization for one shitty decision (to back a bad review).

Just like I don't judge all of Tesla for the fact they don't separate their freeway and street driving data so we can properly analyze their self driving crash statistics misleading potential customers.

neycoda(10000) 3 days ago [-]

If you think the New York Times is bad, you should try looking up all of the crazy assertions made by Fox News show hosts.

arp242(10000) 4 days ago [-]

That's 7 years ago? I don't know anything about that entire story so I can't comment specifically, but the NYT publishes about 150 articles a day, so there are bound to be some stinkers over the years. It's not great, but dismissing an entire publication over it doesn't strike me as fair.

javajosh(10000) 4 days ago [-]

It's a mistake to judge an organization on one, or even a handful of incidents. There should be a pattern of inaccuracy and bias before you 'cancel' a news source.

This is one of the unsung problems of the modern world, the growing impossibility of keeping an accurate 'reputation' in your brain. Virtually everyone and everything has been accused of being evil, in general, or specifically. It's no longer good enough to just 'have good feelings' about a name, because I guarantee you've read something about every name that is negative. Mere accusation is trivial to make, and society has somehow gotten away from swift and harsh punishment against baseless accusation. Without that (very useful) habit, the heuristic of equating reputation with number of accusations is meaningless, and yet we all still do it.

mancerayder(10000) 4 days ago [-]

Tesla is a scientific error, but what about their 1619 Project stuff?

For me it was the wild Op Eds literally calling for a revolution, and a two day focus on Trump on the front page and the side op eds while there were riots within view of my window. I remember specifically seeing 5 helicopters within sight, after a second day of rioting, seeing nothing mentioned in the home town NYT that we had sacked Soho and Midtown in consecutive days, hundreds of businesses attacked, and deciding then and there that it was a dangerous, misleading newspaper. After several years of subscribing, I unsubscribed.

jaccarmac(10000) 4 days ago [-]

'But it's important to understand the problem with the Times: it is not that the facts it reports tend to be inaccurate—though sometimes they are—but that the facts are presented in a way that misleads.'

Shall we make sure that the quote at the top of the thread includes at least one full sentence as context?

blueyes(10000) 4 days ago [-]

What does the author of this piece want?

The 'truth', or even a plausible interpretation of the facts, is expensive to produce. It is even more expensive when you do it every day covering a wide array of topics.

If you value facts and good analysis, pay for them. It's that simple. Sure, the New York Times gets a lot of things wrong. So does everyone else, including many commenters on HN.

* But a lot of what the New York Times reports, that people disagree with, are things that governments and companies say and do. Those governments and companies have reasons to lie, but the enunciation of the lie is a fact. Judgment is left to the reader, if he has it.

* Publications are also reflections of the societies and political systems in which they are embedded. They speak based on the assumptions and contest of forces in that milieu.

* Most publications are firmly rooted in the attention seeking dynamics of their industry. Many of their errors can be categorized as: amplifying the causes of fear or outrage. Again, if you keep that in mind, their stories are easier to parse.

* Finally, a publication is not a monolith. It is a human organization with good reporters and bad reporters, good days and bad days. The Tesla article written by Stross was ludicrous. The reports by Judy Miller that led us into the Iraq War were abhorrent.

The New York Times should be ashamed of those lapses. But it is putting out a large number of factual reports everyday. Much larger than any equivalent number of human beings in America.

stjohnswarts(10000) 4 days ago [-]

Meh that was years ago. Things ebb and flow, but what got me recently was their uptake of cancel culture. They're even throwing out editors because their reporters and others there have glommed on to the idea that non-progressive -opinions- are not welcome. I am a progressive but I do not buy into this the shame-shame-shame bell attitude that has taken the internet by storm. Vanguarded by twitter.

ksec(10000) 4 days ago [-]

I am extremely naive on the topic, so be easy on me.

1. Does that mean Tesla gets to know every single thing about my car? Where I have driven to, and how I use it, what time I use it?

2. Both Nytimes and Top Gear act on the same thing ( note. I barely read the NYT article ), If Clarkson were not driving in a normal manner, and flying around a test track burning tires, it will quickly run out of battery. And due to the way filming and production, they didn't actually ran out of battery.

I would have thought a normal thing to do would be to disagree with the results, and suggest driving in normal condition and battery conserve manner would results in better milage. Instead Tesla went all out and start suing.

I would call this spinning the truth in some sense. But I wouldn't all this BS.

skrtskrt(10000) 4 days ago [-]

NYT has beat the drums of war for Afghanistan and Iraq as well as the recent Bolivian coup and pretty much every other imperialist, fascist action backed by the US the world over.

Its 'liberalism' is a thin sheen over another mass media corporation continually enabling the military-industrial complex at the cost of tens of millions of lives.


afpx(10000) 4 days ago [-]

I've been looking for an alternative to the NYT that has as good or better writing and reporting. Do you have any suggestions?

thatwasunusual(10000) 4 days ago [-]

> The days of BSing people are over.


cs702(10000) 4 days ago [-]

Focusing on a small part of one sentence which is only tangentially related to the OP's main topic seems... unfair to me.

I found the OP to be a well-written, well-reasoned, thought-provoking article and would highly recommend reading it in its entirety before passing judgment on it.

fabbari(10000) 4 days ago [-]

  Just out of completeness you could have added the actual response to Tesla's comments: 

  To say the least: it's not as clear cut as you make it to be. My personal opinion - for what is worth it - is that between Musk and a journalist of the NYT I would still trust the journalist more. But I am naive that way.
andreilys(10000) 4 days ago [-]

I lost faith when they tried to dox Scott Alexander for no reason other than to generate a few clicks over the controversy.

Dotnaught(10000) 4 days ago [-]

The linked article concludes:

'Creators must be compensated well. But at the same time we have to try to keep things that are important and profound from getting locked away where few people will see them. The truth needs to be free and universal.'

Let's also consider whether lies should be made more expensive. Free costs more than it's worth.

m463(10000) 4 days ago [-]

That's an interesting thought.

Coal plants offer cheap power, mostly because they ignore the externalities of pollution.

If you could identify the externalities of lies and tax or fine them in some way, maybe they would be minimized.

(tax what you want less of)

joe_the_user(10000) 4 days ago [-]

Let's also consider whether lies should be made more expensive.

Whether lies should be more expensive or not, lies cannot be made expensive. Even in a totalitarian state, false rumors spread easily. The cost of production of lies is close to zero, the cost of distribution of lies also close to zero.

It's like all of HN is eager to apply an argument that makes sense for cheap, knock-off auto parts ('Cheap is too expensive', etc) to decent public journalism. It doesn't work.

sandoooo(10000) 4 days ago [-]

I am in wholehearted agreement on a tax on lies so long as I get to decide which articles are lies.

marcus_holmes(10000) 4 days ago [-]

As always, who judges whether something is true?

reportgunner(10000) 4 days ago [-]

> You want the detailed Times reports on neo-Nazis infiltrating German institutions, the reasons contact tracing is failing in U.S. states, or the Trump administration's undercutting of the USPS's effectiveness—well, if you've clicked around the website a bit you'll run straight into the paywall.

Excuse me but why would I want to pay for that ? I don't earn by reading articles online and the news websites are already littered with advertisements and autoplay videos. I get tracked around the web by these ads and I'm supposed to even fund that ?

Thank you but I will pass.

> The New York Times is, in fact, extremely valuable, if you read it critically and look past the headlines. Usually the truth is in there somewhere, as there is a great deal of excellent reporting, and one could almost construct a serious newspaper purely from material culled from the New York Times.

... and can't you do the same with Infowars ? Critically read it and look past the headlines ? Perhaps the truth will be there somewhere too. (albeit as a total negation of the Infowars article)

blaser-waffle(10000) 4 days ago [-]

Don't even use Infowars and NYT in the same sentence. The NYT has some notable shortcomings, but Infowars is straight-up propaganda used by its host to shill for questionable food supplements and fake corona-virus cures.


HPsquared(10000) 4 days ago [-]

I'd like more microtransactions and would gladly pay a few cents or even a dollar to read a good article.

The business model is to bundle everything together, for a long subscription - most want at least a month, perhaps a year, full subscription.

freddie_mercury(10000) 4 days ago [-]

All that would do is massively incentivize clickbait headlines, if they could actually get paid $1 for each person who clicked on it.

Do you really want a world with even more clickbait headlines and articles artificially spread over 20 pages? Because that's what microtransactions would lead to.

teraku(10000) 4 days ago [-]

It doesn't work.

Because what if you pay 0.5$ for an article, and then you find out it's not what you expected/badly written/...

You basically are not happy and less likely to do it again. If you put payment to the end, then people are less willing to pay, because they already consumed the article. You can't take the information out of the head again if he's not willing to pay.

It's a tricky thing

raziel2p(10000) 4 days ago [-]

How do you know it's a good article before you read it, though? Articles aren't (and shouldn't) be produced like movies, video games or books, with marketing budgets and hype.

alexashka(10000) 4 days ago [-]

The author would've been better served by pointing out the paywalls of research done in universities, funded by people's taxes.

That'd be somewhat close to 'the truth is paywalled'.

Newspapers and truth are a contradiction - truth is boring, newspapers cannot afford to be boring because they compete on entertainment value or illusion of 'being informed', not truth value, therefore every newspaper panders and skews 'the truth' in exchange for entertainment or 'being informed' value, that's their job.

Paywalled or not, newspapers are drug dealers for the junkies addicted to a 'need to be informed'. Twitter is beating them to the punch. On twitter you can be infinitely 'informed' and be an active participant in 'informing' others, which is why journalists and 'intellectuals' can't get enough of it.

Herrin(10000) 4 days ago [-]

He does, starting at the section that begins:

> Possibly even worse is the fact that so much academic writing is kept behind vastly more costly paywalls.

irontinkerer(10000) 4 days ago [-]

The first link in the article is not a 'lie'.

'Portland Protesters Burn Bibles, American Flags In The Streets' has a RT video that shows exactly that, and links to several other confirmed Twitter accounts all stating the same thing.

How is that a lie?

devman0(10000) 4 days ago [-]

Not saying it's a lie, but having problems corroborating the original footage. No major news agencies are carrying it, I can't even find an AP or Reuters on it and they generally would report on this kind of thing. For instance, AP has a report of flag burning on Aug 1st, but nothing about bibles.

I'm skeptical because there have been times footage from other events have been reused out of context to push an agenda. Just a few days ago Trump ran a campaign ad that used Ukrainian protest imagery to demonstrate chaos in the US, for instance.

a_bonobo(10000) 4 days ago [-]

Where does OP's article say that the bible-burning incident was a 'lie'?

New York Post says 'a bible', and your video a single book being burned, can't see whether that's a bible. The original video has been deleted (link is dead here: https://twitter.com/stillgray/status/1289512762733785088 ) but quickly picked up by the usual far-right BS twitterati.

>The fire was later put out by members of Moms United for Black Lives Matter, who doused the flames with bottles of water and stomped on the embers,

so that article OP cites is outrage bait, it's not an informed source.

mundo(10000) 4 days ago [-]

Through selective omission, by presenting the non-central as central. Did the events in that article happen? Yes, probably. Would someone who gets their information from Breitbart have a roughly accurate idea of what happened at the Portland protests? No.

awinder(10000) 4 days ago [-]

There's been a lot of coverage around false-flag sorts of behavior in Portland, but the fact that this thing is being pumped through the lie pipelines that originate with places like RT (Russian state-controlled media) is kinda a dead giveaway.

We can get into a nuanced discussion of 'is it really lying' since there's a video, but I have not been a party to any conversations in my lifetime that start with a nuanced discussion of 'is something technically a lie' that end with a good feeling in my gut. So that's kinda a clue that despite the technicalities of language something uncool is happening.

oxymoran(10000) 4 days ago [-]

Because it doesn't agree with the narrative that Mr. Robinson is trying to push.

arp242(10000) 4 days ago [-]

That may perhaps not be the best example – although the linked Breibart article is hardly a stellar journalism and some of the commentary is wildly off-base – but the general pattern described in the article seems to be mostly right to me.

IMHO dismissing/criticising a rather lengthy article just on one less-than-ideal example is not really good form.

cousin_it(10000) 4 days ago [-]

> Now, I am sure there will be those who argue that any universal knowledge access system of this kind will inhibit the creation of new work by reducing the rewards people get. But let us note a few facts: first, dead people cannot be incentivized to be creative, thus at least everything ever created by a person who is now dead should be made freely available to all. The gatekeepers to intellectual products made by the dead are parasites the equivalent of a private individual who sets up a gate and a tollbooth in the middle of a road somebody else has already built and starts charging people if they want to pass.

This seems wrong. Longer-lasting copyright does incentivize authors more, because it can be sold for more.

hypersoar(10000) 4 days ago [-]

The typical copyright term in the US is life of the creator + 70 years. Are potential revenues a hundred years from now really figuring all that much into the price of copyrighted work sold today?

Super_Jambo(10000) 4 days ago [-]

But we are positively drowning in music, video and written words. We don't need more incentives for people to create copyrighted work. We need more effective markets for consumers to judge it.

mlthoughts2018(10000) 4 days ago [-]

Also, an author could write books that are given to some type of trust or foundation with explicit orders not to publish them until certain favorable copyright or distribution laws are available.

So it could be possible to "be incentivized" to publish new works even after you're dead. For the estate of an author that has developed a reputation, say like JD Salinger, this would be a very smart thing to do if you wanted to influence policy beyond your death.

JMTQp8lwXL(10000) 4 days ago [-]

Please get a library card if you wish to read quality (often paywalled) news sources for free. Your tax dollars (going to your city/county) are already paying for access. Best of all, it's digital. Your library card number will grant you huge access to all of this digitally and in near real time (e.g., the databases are updated daily).

nitwit005(10000) 4 days ago [-]

It's a lot easier now that there are websites, and even mobile apps, tied to library systems. My local library system offers some impressive stuff like current and past newspapers, magazines, Associated Press videos, etc.

symlinkk(10000) 4 days ago [-]

Are you referring to digital scans or what? My library card doesn't allow me to access nytimes.com.

dredmorbius(10000) 4 days ago [-]

Few of the sources I find most useful (technical books, academic journals, historical magazine and newspaper articles) are available even with a library affiliation.

LibGen and SciHub are vastly more convenient and comprehensive.

ferros(10000) 4 days ago [-]

I would be willing to pay an amount for better quality news than what can be found for free, however news outlets always seem to lean one way. I want somebody to just report and not tilt it in one direction.

marmaduke(10000) 4 days ago [-]

You can aggregate across news sources, eg factr.com

goto11(10000) 4 days ago [-]

> I want somebody to just report and not tilt it in one direction.

This is unfortunately not possible. Even if an outlet only providing objective, verifiable facts there is some angle or bias in what facts you chose to report and what issues you chose to investigate.

op03(10000) 4 days ago [-]

I use twitter for that. Don't follow journos/news orgs. Just follow topics and saved search terms.

throwanem(10000) 4 days ago [-]

You want something you can trust absolutely and not have to engage with critically at all. It's a reasonable desire, but one much more likely to be fulfilled in the realm of religious faith than that of news or any other human activity.

lambdadmitry(10000) 4 days ago [-]

The FT should serve you well.

lvs(10000) 4 days ago [-]

This is called 'false balance.' Reality doesn't have sides.

specialist(10000) 4 days ago [-]

'Objectivity' in journalism does not mean unbiased, or The Truth.

Simple recipe to follow:

- Show your work.

- Cite your sources.

- Corroborate.

- Sign your name.

- Admit your mistakes.

Voila, you're a journalist.

lm28469(10000) 4 days ago [-]

> I want somebody to just report and not tilt it in one direction.

No one on this planet is 100% unbiased. It doesn't matter if it's free or $100 per article, it's still written by a human being with personal opinions, experiences, prejudices, &c.

The easiest way to get close to the 'truth', which is subjective in many cases anyways, is to read from many sources and cross check the facts between them.

actuator(10000) 4 days ago [-]

I think it is incredibly hard to not tilt one way. After all, organisations are run by humans and we all have our own leanings. It is very much possible when we hire people those leanings bias us towards a certain ideology and we may hire like minded individuals. Also, people who don't fit the ideological tilt might feel left out and may leave on their own. So, over time a media organisation will come to represent one side more.

This all in addition to the biases that can come from the owners/company bankrolling the media house.

FranzFerdiNaN(10000) 4 days ago [-]

What is unbiased? If you were a late 18th century french reporter writing a report about the death of king Louis XVI, would you write 'King Louis was murdered', 'King Louis was executed', 'former king Louis executed by means of guillotine for high treason', 'citizen Louis Capet beheaded on the order of the National Convention' or something else? All of those descriptions can be considered true and yet none are unbiased.

AnimalMuppet(10000) 4 days ago [-]

Reuters is pretty good (at least for US and world news; I don't know how they are for UK).

AP seems to me to be fairly reasonable (not dramatically slanted one way or the other).

dvfjsdhgfv(10000) 4 days ago [-]

This could be possible some 20 years ago, maybe. But these day even if you choose to just describe bare fact, you will still be criticized for the choice of words, such as 'riot'. Moreover, if you describe facts in full, including some background details, you will be crticized by people shouting 'How is this relevant?!'

sul_tasto(10000) 4 days ago [-]

The FT is good, as is stratfor.com

ivanjovanovic(10000) 4 days ago [-]

One reasonable way to deal with that would be to subscribe to several sources which lean in different directions and then explore the situation from various angles.

fredley(10000) 4 days ago [-]

Pay for quality, but understand the tilt. If you really want to, pay for two services with opposing tilts, but equal (high) quality.

bitwize(10000) 4 days ago [-]

What can I say, mate? Reality has a well-known left-wing bias. So do 'the better angels of our nature'.

quietbritishjim(10000) 4 days ago [-]

One compromise is to find a news source that is at least open about its bias. The Economist is a famous example of this: it will openly declare support (and opposition) for particular political candidates and policies.

luis2527(10000) 4 days ago [-]

Did anyone realize that this article is free?

ngcazz(10000) 4 days ago [-]

The amount of free content that Current Affairs has published on the website is a minuscule fraction of their print (subscriber) output.

rezeroed(10000) 4 days ago [-]

The paywalled content is not, in my experience, the truth, just someone else's lies. I use 'lies' as freely as the title. 'Leaning' or 'spin' might be more accurate.

rezeroed(10000) 4 days ago [-]

Don't be shy; feel free to comment. This site claims to encourage conversation.

SamWhited(10000) 4 days ago [-]

This is specifically why I choose to support sources that don't have paywalls, but are still trustworthy news sources like Vox, The Guardian, etc. In these particular cases they do have an editorial stance that I happen to agree with but others may not, but the facts are still there and they're largely trustworthy. They're also open about their editorial stance and biases, so if you don't agree you can still get the facts and be skeptical about the editorial stance. But either way, personally I like both these examples so I try to support them instead of organizations that put up a paywall. I'm sure if your worldview is different you can find organizations that are legit but also have different funding models to support, or just learn to separate facts from editorializing or op-eds.

starik36(10000) 4 days ago [-]

I don't read the Guardian, but Vox has a significant bias. I wouldn't call them trustworthy.

rawfan(10000) 4 days ago [-]

I just want Netflix for journalism. I don't want a super expensive sub to a dozen papers. I want one subscription, they all should be included (maybe offer different tiers if you are an arse) and it shouldn't be more than $20/month.


5etho(10000) 4 days ago [-]

and how much you think journalists should earn? 10k/yr?

freddie_mercury(10000) 4 days ago [-]

Why do you assume that $20 is enough to support multiple newspapers?

That's nearly how much the New York Times ($17/month) alone costs and they lost money every year in recent history until 2018 and are only now turning the corner as a business.

On the surface I'd assume that actual journalism -- not just clickbait opinion pieces -- is going to cost you way more than $20 a month if you want access to multiple sources.

mythz(10000) 4 days ago [-]

You likely want Apple News+ [1] at just under $10 USD /mo - a single cost for a la carte access to a large pool of publications, basically the only premium subscription model that's going to be able to gain mainstream adoption.

[1] https://www.apple.com/apple-news/

specialist(10000) 4 days ago [-]

Hard pass on algorithmic recommenders.

I have always wanted human curation, filters, opinion.

listenallyall(10000) 4 days ago [-]

The absolute worst posts on HN start with 'I just want...' and then selfishly lowball the cost of whatever it is they want.

Anyway, you've described a cable bundle model, not Netflix's model.

hugh-avherald(10000) 4 days ago [-]

The problem with a model like Netflix for journalism is that it will take the same path Netflix did.

rvz(10000) 4 days ago [-]

Apple News+?

alpineidyll3(10000) 4 days ago [-]

In raw dollar terms this would undoubtedly >2x-3x the capital available to journalism as an industry. People who say (truthfully): 'journalism needs more money' need to confront the reality that the current biz model doesn't work. Either we coerce Google news to tithe like Australia just did, or something like this sounds like a better solution than just letting journalism die.

chadcmulligan(10000) 4 days ago [-]

Not sure how available this is elsewhere but I get press reader access for free through the state library in my browser. It has a lot of papers and magazines (though the interface isn't the best)

schnable(10000) 4 days ago [-]

This is Apple News+ I think?

dukoid(10000) 4 days ago [-]

In Germany, part of the problem is that old media lobbying has successfully prevented a reasonable (text based) internet presence of public broadcasting...

Some Google-translated coverage: https://translate.google.com/translate?sl=auto&tl=en&u=https...

antibuddy(10000) 4 days ago [-]

Public broadcasting in Germany is no better than private media. In fact most topics are reported the same and even the opinion is the same. But even ignoring this, it is a big problem that publicly funded media is in direct economic competition with privately funded media. This is exacerbated by the official entertainment mission (Unterhaltungsauftrag[1]) and the usage of advertisement.

[1]: https://www.die-medienanstalten.de/fileadmin/user_upload/Rec... in §11

mips_avatar(10000) 4 days ago [-]

Reminds me a lot of google's effort to create high-quality scans of every book ever written. They got pretty far before the publishers stopped them. They now sit on a sizeable percentage of all books published, locked away in their servers where nobody can access it.

JMTQp8lwXL(10000) 4 days ago [-]

It's not all for naught. Eventually the books will end up in the public domain, and if Google exists then, can choose to make scans public.

Valgrim(10000) 4 days ago [-]

Could they still use this to train an AI model, legally speaking?

kyleblarson(10000) 4 days ago [-]

Of course they left the Wall Street Journal (paywalled and quite expensive) out of their list because it doesn't fit their narrative.

brendoelfrendo(10000) 4 days ago [-]

As in... the WSJ publishes lies but is paywalled? That's generally not my experience; WSJ is nowhere near where it used to be, but isn't among the trash tier of journalism, yet.

The 'media bias' chart from Ad Fontes media puts them at a slight right slant, with reporting accuracy more-or-less on par with publications like Newsweek, The Economist, and WaPo.[0][1]

[0] Chart: https://www.adfontesmedia.com/gallery/ [1] Methodology: https://www.adfontesmedia.com/how-ad-fontes-ranks-news-sourc...

dsign(10000) 4 days ago [-]

That's why I buy my 'news' at Amazon, as e-books on the topics I'm interested, since for 99.9999999999% of the things happening in the world, nobody really cares about how many months they will have to wait for my opinion.

na85(10000) 4 days ago [-]

If you choose to abstain from current events then you assume responsibility for the outcome. Functioning democratic societies require active, engaged citizenry.

Passive, disengaged voters are what creates the conditions for Donald Trump to arise.

Havoc(10000) 4 days ago [-]

Are there some sort of bundle subscriptions available?

Buy all of that stuff separately won't be cheap

thewhitetulip(10000) 4 days ago [-]

even if there were a bundle of that, people'll still find a way to complain about it and they'd still not want to pay for news.

Yaa101(10000) 4 days ago [-]

This is why I sponsor the Guardian, they stay open and give good news.

vixen99(10000) 4 days ago [-]

The Guardian as with other newspapers has to selectively report the news which it does using a vocabulary which implicitly or explicitly reflects their opinions about the news. Naturally the overall output confirms the prejudices of their readership. It's 'good news', they say.

jamisteven(10000) 4 days ago [-]

"History is always written by the winners. When two cultures clash, the loser is obliterated, and the winner writes the history books—books which glorify their own cause and disparage the conquered foe. As Napoleon once said, 'What is history, but a fable agreed upon? '"

kerkeslager(10000) 4 days ago [-]

I struggle to see how this is relevant.

mattacular(10000) 4 days ago [-]

This isn't history, it's current affairs.

blaser-waffle(10000) 4 days ago [-]

Napoleon got a citation, but who is the original quote from?

rbecker(10000) 4 days ago [-]

> Breitbart, Fox News, the Daily Wire, the Federalist, the Washington Examiner, InfoWars: free!

As is salon.com, theatlantic.com, theroot.com, vox.com, huffingtonpost.com, slate.com, and also somehow missing from that list, CNN!

I appreciate the point the author is trying to make, and realize this is tangential, but I would be grateful if he didn't try to deceive me with the choice of examples.

fock(10000) 4 days ago [-]

is CNN really putting up anything except news - of which reuters/AP/UPI already give a pretty good overview for free? And while all the left/neoliberal (and clickbaity) things you noted are free, they are less concerned with news, but more opinion – to my knowledge they don't try to make the impression of a news outlet as much as the listed ones (which carry typical news-speak even in their name!)

Historical Discussions: Math Overflow users resolve PhD thesis crisis (August 04, 2020: 637 points)

(646) Math Overflow users resolve PhD thesis crisis

646 points 3 days ago by DarkContinent in 10000th position

mathoverflow.net | Estimated reading time – 3 minutes | comments | anchor

@user161819 I wanted to make a comment but it got too long, so putting it as an answer. But please take it just as a comment for later, once everything is finished:

If I understand your comment to my answer correctly, you are aiming to change your algorithm for the torus so it works with cr(G). I think the whole MO community is keeping their fingers crossed, wishing you that you can successfully complete everything in time!

Looking at the far horizon, I wanted to make a suggestion to you. Once you have changed your torus algorithm and completed your thesis, you will have effectively two algorithms in your hands for the torus: The old one based on pcr(G) and the new one based on cr(G). I am saying the obvious here, keep both of them, they can really be fruitful for future research.

(1) Obviously, your two algorithms could support research on the big open question whether pcr(G)?=cr(G) or not. They could produce experimental evidence, ideas, and insights for a future proof of equality, or an actual counterexample. (Again, I am saying the obvious here.)

(2) To really pressure-test pcr(G)?=cr(G) on the torus, it would be interesting to also try the best known to date lower bound for cr(G) 129e3n2

for graphs with e>7n. This lower bound is from Eyal Ackerman (2019): 'On topological graphs with at most four crossings per edge', Computational Geometry, 85: 101574, 31, doi:10.1016/j.comgeo.2019.101574 (probably you are aware of it from the Wikipedia article that you quoted).

I think your question and this whole topic are really important. László Székely calls it one of the 'foundational problems' and devotes a whole section to it in his article Turán's Brick Factory Problem: The Status of the Conjectures of Zarankiewicz and Hill. In: R. Gera et al. (eds.)(2016): Graph Theory—favorite conjectures and open problems. 1.)

For now, fingers crossed that you can complete your thesis in time!

All Comments: [-] | anchor

fizixer(10000) 3 days ago [-]

OP is incredibly fortunate. Or maybe mathoverflow is that active/supportive.

As a STEM grad student (not in math), I had more than a couple such moments of crises, when I posted my questions on various stackexchange websites. I got either useless replies, or no replies.

iflp(10000) 2 days ago [-]

Mathoverflow is different from most of the other SE sites in that it's only for research level questions. There is a separate site, math.stackexchange, for other math-related questions.

lifeisstillgood(10000) 3 days ago [-]

The walled garden sweet spot

Stack overflow and it's cousin sites have many serendipities like this - and I happily conjecture this happens more here than facebook or twitter.

I think the reason is that despite being a walled garden (ie proprietary) it still has a promise to open up the content and makes effort to moderate and grow the community - in other words what they are really selling is not the SEO but the sweet spot between 'anyone posts anything' of an 'ideal' internet where no rentiers exist but no one can find anyone else, and the much more corporate hand of Facebook.

I am not sure reddit exists in this sweet spot either - mostly because there is just sooo much reddit.

mcintyre1994(10000) 2 days ago [-]

StackExchange sites feel like sites that you can browse without them trying to get you addicted and trying to stop you leaving. They do have the sidebar which sometimes shows interesting content but it doesn't feel optimised for addiction or stickiness - more genuine discovery.

I'm not sure what sort of advertising they do on other stack exchange sites if any, but sticking to a job board on SO makes it so much more pleasant to read than a social feed throwing random junk products at you every few minutes until you go away.

Dirlewanger(10000) 3 days ago [-]

Both reddit/SO and say, classic forums, each have their own method of content discovery (reddit/SO always prioritize new items, forums push you to long-running threads), but both have their blind spots. With reddit you can end up with a lot of duplicates because a subreddit's dashboard decays stuff pretty fast based on the frequency of posts. It makes long-running discussion impossible. With forums you can have your long-running discussions, but you sometimes have to wade through page after page to find those specials nuggets of info.

gowld(10000) 3 days ago [-]

SO is a garden, not a walled garden.

neves(10000) 3 days ago [-]

I always say that the wonder of the Internet is the collaborative Wikipedia, not Facebook walled garden. Stack Overflow network of sites is another of the great Internet wonders. Non technical people can not grasp how fantastic they are. Younger developers do not imagine a world without SO.

As a private company, probably someday they will lost their techno-utopian magic (as Google already lost). It will be a very sad day.

lukeplato(10000) 3 days ago [-]

PG recently tweeted something relevant to this thought [1]

'Twitter is a few people saying interesting things amidst a much larger number saying mean or mistaken things. So are books. But you don't suddenly get sentences from bad books in the middle of reading one of the good ones. Maybe this gets fixed in version 2 of social media. Maybe version 2 is halfway between the randomness of Twitter and the predictability of Substack.'

[1] https://twitter.com/paulg/status/1290738446059950080

Shog9(10000) 3 days ago [-]

A couple relevant bits of info about MathOverflow:

- The site is operated by Stack Overflow/Exchange, but is owned by MathOverflow, Inc a non-profit corporation[0]. As such, it retains the right to exist independently of the Stack Overflow company - to my knowledge, it is the only public Stack Exchange site for which this is true.

- Like all public Stack Exchange sites, authors retain ownership of their work, which is published under a CC-BY-SA license. Regular archives are uploaded to Archive.org and can be obtained there or via Bittorrent[1]

In short, not a walled garden, and not Stack Overflow's garden.

[0]: https://meta.mathoverflow.net/questions/969/who-owns-mathove...

[1]: https://archive.org/details/stackexchange

supernova87a(10000) 3 days ago [-]

I have to say also that this type of crisis is not surprising (unfortunately) for math, or similar highly theoretical, loner fields. I can guess that the student asking the question is not being very open with his/her advisor, has worked and struggled for long hours alone, thinking they have to solve it on their own, and is not super communicative and checking in about important aspects of the thesis. Because he/she thinks it has to be a surprise 'breakthrough' result -- a heavy obligation of the field's expectations.

No responsible advisor would let the work get to such a state, so late in the game. Major fault of the advisor too, here.

jfkebwjsbx(10000) 3 days ago [-]

Advisors are also very much at fault, not just students.

The last year of my PhD I ended up being pretty much alone because my advisor had changed research topics a year before and therefore was not interested nor up to date, so any of her inputs were not very useful.

A couple friends of mine also struggled with their advisor because he actively avoided communication for some reason. I guess he had a personal or health issue.

So even on good faith, advisors can end up making students life quite stressful for one reason or another

generationP(10000) 3 days ago [-]

As far as notational clusterfucks go, crossing numbers (along with the three standard definitions of a ring) are one of the best-known ones to still be biting people on a regular basis. ('Positive' and 'natural number' are sufficiently well-known that people are careful.) But imagine how it felt to do group theory back when 'group' could mean any of 'abstract group', 'subgroup of GL(n)', 'finite group', 'monoid', 'semigroup' and combinations thereof.

tgb(10000) 3 days ago [-]

The simplest gotcha I know is: is f(x) = 1/x piece-wise continuous?. This is calculus 1 level material and yet author's disagree significantly on this point, sometimes without specifying it! Some say yes, others would require f to have finite left and right limits at every point. This mattered for a point of my thesis and my advisor was very unhappy with me calling these function piece-wise continuous.

OskarS(10000) 3 days ago [-]

The way I was taught was that back in the olden days, 'group' always referred to groups of permutations (and the operation was always composition), and it was Cayley that introduced the much more general and abstract notion of groups that we use now. He could do that because it's relatively trivial to prove that the the two definitions are basically the same: every group is isomorphic to some group of permutations according to Cayley's theorem: https://en.wikipedia.org/wiki/Cayley%27s_theorem

wenc(10000) 3 days ago [-]

This reminds me of my Ph.D. crisis. (which I'm sure many former grad students can relate to)

I was in my 6th year. All my friends had graduated, and my stipend had run out. I was 2 weeks away from submission and discovered that one of my assumptions was wrong, which potentially distorted/invalidated all my studies -- to fix these studies would have potentially delayed submission for months. It was a very subtle assumption violation (and it wasn't even that wrong) and my committee probably wouldn't even have noticed. I was tempted to sweep it under the carpet and not let it keep me from graduating.

But I knew it was wrong. I felt that if I sacrificed my integrity then, the moral failure would mark me for life. No one would know -- but I would know. So I decided to fix the issue, re-do the studies and live with the reality that I would have to delay my defense.

Turns out when you're desperate -- and many grad students can attest to this -- a resourcefulness that you never thought you had kicks in ('where were you during all my years of grad school?'). I don't remember how, but I somehow managed to wrangle new studies out in 3 days (which would have previously taken me months). I made the deadline in the end.

The lesson I learned was that committing to doing the right thing has its costs, but in some cases it also forces one to explore attacks never previously considered. Asking on MathOverflow is one such attack.

jvvw(10000) 2 days ago [-]

I discovered that the proof of a fundamental proposition in my maths PhD thesis had a mistake in it two weeks before my viva, after I had submitted. Not sure if that was a better or worse time to discover it! After a rather stressful few days, I managed to reprove it and took the correct proof along to the viva. The examiners were 'well, the result was obviously correct so we weren't worried about it'!

sadfev(10000) 3 days ago [-]

Your comment is a lifeline to me in this difficult time.

Why are we not resourceful in normal time?

foepys(10000) 3 days ago [-]

> Turns out when you're desperate -- and many grad students can attest to this -- a resourcefulness that you never thought you'd had kicks in

This is getting millions of students through high school, college, and university every year.

Everybody has at least one thing they pushed back for way too long and then developed almost super-human-like abilities to do it. Doesn't guarantee a good grade, though.

mjklin(10000) 3 days ago [-]

The book "Gödel, Escher, Bach" was printed with a special process only available at one printer far away from where Hofstadter was living, and was extremely time consuming. At the last minute the whole thing had to be redone, so he had to get on a plane, print a few pages, then fly back to his job. This took several months. (He talks about it in the 20th anniversary edition.)

dmch-1(10000) 3 days ago [-]

My laptop broke down just hours before I had to submit the thesis. It was the day of the deadline. I did have printed copies of the thesis (I had to submit printed copies), however I discovered bad typos on the front page. But, my laptop was down with the only digital version in it. In the end, I corrected the typos using a photocopier. I overlayed the typos with corrections on cut out pieces of paper, and photocopied. It was quite a struggle, but I made it!

jjeaff(10000) 3 days ago [-]

Good on you for doing the right thing. We have far too little thinking like this in the world.

It reminds me of a story I heard from a businessman that was walking with an associate down the street. He stopped at one of those old metal newspaper dispenser machines to buy a paper. For those that don't know, the way they work is once you have put in your quarter the machine opens up and all the papers are just stacked in there.

His associate asked him to grab a newspaper for him. So he pulled out one paper, closed the machine and put in another quarter and then pulled out another. The associate asked, 'Why didn't you just grab a second paper for free? It's just a quarter.' The businessman's response was, 'If I'm willing to sacrifice my integrity for a quarter, how much easier will it be to sacrifice it when something serious is on the line?'

supernova87a(10000) 3 days ago [-]

There's another (slightly kinder) version of this:

When people go to grad school, then during their studentship get married and have a kid -- suddenly their productivity goes up dramatically and output per time increases by many factors. What happened?

It turns out when you have self-motivating reasons to get out (and a target on your head from your spouse to earn some money for godssake), you find ways to focus on what's important and drop the rest.

No more idling away for hours on silly ideas that don't get you closer to handing in your thesis. No more trying random libraries that get your code to run 2% faster. No more goofing around after 6pm with other students just because you have the time -- you have to get home and be a breadwinner for your family. You have to get shit done.

You start to ask, 'even if I don't know exactly what the thesis will say, how should it be organized and what kinds of conclusions will make up the writing? And what experiments do I need to fill in those charts/paragraphs, and no more?' What's the minimum I need to do to get out of here? Not, 'What amazing interesting thing could I explore?'

Limits and constraints sometimes free the mind dramatically. The side effect is maybe you don't get to explore ideas that go nowhere, but that's a discussion about the purpose of the PhD and for another topic I guess.

(And sometimes, if you think, well I don't have a kid, so what's the rush? Well, someday you might have a spouse, a kid, and every day of time you left in grad school is a day for your future self -- and family -- and $$ -- left behind in time. Work to free your future self... now, while you have the time.)

no_identd(10000) 3 days ago [-]

>'the faculty will not accept asymptotics'

the hell does that mean?

impendia(10000) 3 days ago [-]

(Math professor here).

Ambiguous and poorly explained. (Note the question immediately afterwards asking for clarification.) But probably something along the general lines of 'My advisor said that, if my main theorem is an asymptotic estimate instead of an exact formula, then this would not be judged to be novel/strong enough to earn a Ph.D.'

thom(10000) 3 days ago [-]

This seems like a deeply flawed qualification, that smart people, noticing interesting properties of deep problems, are forced to panic in such ways.

omginternets(10000) 2 days ago [-]

On the surface, I agree: there's an interesting problem that's worth solving, and a purely artificial limit is forcing people to do a bang-up job at solving it.

But if you dig a bit deeper, I can see two counter-arguments:

1. The real risk -- by which I mean 'the risk I have most often observed in the wild' -- is that a Ph.D expands to fill the time it's given, without ever wrapping up and producing a publishable result. This happens so often that it's practically expected in some places.

2. Having a deadline, oddly enough, also serves as a catalyst for birthing an idea... for 'pinching it off' as the expression goes. At some point you have to stop planning and start executing. You can see the deadline as a forcing function.

Ph.Ds are needlessly traumatic and procedural in many ways, but I'm no longer sure that hard deadlines are a net negative.

oconnor663(10000) 3 days ago [-]

A neat comment on the accepted answer:

> From OP's point of view this could be viewed as glass half-full rather than glass half-empty. Their dissertation results hold unequivocally on the sphere and might hold on the torus, though it is an open problem if they do. It is certainly legitimate to study what follows from a given conjecture being true. It could even be spun as a feature rather than a bug of the dissertation. If the results in fact fail on the torus then you know that the conjecture must be false. Potentially, it could open up a fruitful avenue of attack.

Kind of reminds me of Terence Tao's post on what solving big problems looks like: https://terrytao.wordpress.com/career-advice/be-sceptical-of...

finolex1(10000) 3 days ago [-]

As an aside, looks like one of the comments in the Math Overflow link above was made by Terence Tao himself

thechao(10000) 3 days ago [-]

When Terence Tao writes stuff like this, I'm always very happy that I got to experience the Moore-method for learning math (at UT Austin). A group of us would be dumped into a class with a common topic and we'd just have to prove things (topology, algebra, analysis) ... on the blackboard, in front of everyone. The best work we did was when something started going wrong and then we'd all start arguing about the proof, building count-conjectures on the fly and riffing on the math. The worst work was when someone went and found a proof ahead of time and just showed the answer. There's so much to learning where the sharp bits of math are; proofs are the razor-thin path through the briar patch.

It was only later that I found out that history, the study of art & literature, and philosophy can all teach you the same thing. The important part is that you're interested in the topic.

theossuary(10000) 3 days ago [-]

It reminds me of this math overflow answer: https://mathoverflow.net/a/338620

I suspect the reason OP's thesis worked out okay is because his intuition wrt the problem is correct, even if his formulation was a bit off. Very cool, sounds like a good mathematician to me

(644) US to ban transactions with ByteDance and WeChat in 45 days

644 points about 15 hours ago by baylearn in 10000th position

asia.nikkei.com | Estimated reading time – 4 minutes | comments | anchor

PALO ALTO, U.S./HONG KONG -- U.S. President Donald Trump has issued executive orders to ban any U.S. transactions with WeChat, the messaging app owned by Tencent Holdings, and ByteDance, owner of TikTok, within 45 days, describing the Chinese-owned companies as threats to national security.

'The spread in the United States of mobile applications developed and owned by companies in the People's Republic of China . . . continues to threaten the national security, foreign policy, and economy of the United States,' said Trump in the two executive orders signed on Thursday.

'Additional steps must be taken to deal with the national emergency,' he added.

The orders intensify U.S. pressure on Chinese tech companies, after the U.S. published details this week of a 'clean' network initiative to act against Beijing on a number of fronts, from app use to cloud storage.

They also appear to confirm that Microsoft, which is in talks with ByteDance to buy TikTok's operations in the U.S. and several other countries, in effect has a 45-day window to reach a deal.

One executive order issued by Trump will prohibit any transaction between U.S.-based individuals or companies and ByteDance, whose TikTok video messaging app is extensively used in the US. The second prohibits any transaction with Tencent that relates to WeChat.

The definition of 'transaction' will be clarified by the Secretary of Commerce, also in 45 days, according to the White House announcement.

Shares in Tencent, which is listed in Hong Kong and also traded in New York, fell almost 7%.

Trump accused TikTok and WeChat in the executive order of being espionage tools for Beijing to spy on the U.S. as they capture huge amounts of user data.

In addition, Trump said both apps are used for propaganda purposes as they reportedly censor 'content that the Chinese Communist Party deems politically sensitive and may also be used for disinformation campaigns that benefit the Chinese Communist Party.'

The order will likely force U.S. app stores including Google's and Apple's, to remove TikTok and WeChat. It is unclear whether using or downloading the apps will be prohibited in the U.S. after 45 days, according to legal experts.

'The specific impacts are not yet known and are subject to regulations to be issued by the Commerce Department . . .The restrictions may affect US person's ability to use these apps or result in other, more tailored restrictions,' said Nicholas Turner, a Hong Kong-based counsel at law firm Steptoe & Johnson.

'Downloading the app is more likely to be banned because it involves signing a user agreement with the companies, which is a transaction by definition,' said Ye Jun, a partner at Chicago-based law firm Getech Law specializing in corporate and patent law.

'It is more difficult to ban the use of the apps. If users already have them on their phones, it is near impossible to ask them to delete or stop using, unless the U.S. can build a 'great firewall' to block them once for all,' Ye added.

The latest move by the administration will also put additional pressure on ByteDance to reach a deal with Microsoft. Trump said last week that if TikTok's U.S. operation is not taken over by an American company by September 15, the app will be banned in the country.

ByteDance's founder and CEO Zhang Yiming said in an internal company letter on Tuesday that the U.S. government's real goal has always been banning the app instead of 'forcing a sale of TikTok'.

TikTok has been highly popular among young Generation Z social media users in the U.S., where the app had been installed 180 million times as of early July, according to data from market research firm Sensor Tower, in a country of 328 million people.

While WeChat does not have as broad a user base in the U.S. as TikTok -- of the app's 279 million overseas downloads in the past six years, the U.S. contributed less than 7%, or 19 million -- it is used by many Chinese living overseas as a tool for both personal and business communications.

However, Trump said the Chinese Communist party was using the app to 'keep tabs on Chinese citizens who may be enjoying the benefits of a free society for the first time in their lives.

Tencent declined to comment. ByteDance did not respond to requests for comment.

All Comments: [-] | anchor

supernova87a(10000) about 12 hours ago [-]

Regardless of how I feel about this particular company or transaction, to me this is a bad overreach of presidential power. I guess it's merely a reflection of the incompetence/inaction of Congress to study the matter and do something about it, as is their responsibility.

Why do I say so?

1. The justification for this is that it's a 'national emergency with respect to the information and communications technology and services supply chain'. Supply chain? Are you kidding me? The permissions given to the executive to declare emergencies for critical goods and services such as related to war time -- these extend to a voluntary communications app? Strains belief, and however you feel, this is not a good precedent to allow.

2. CCP is censoring / monitoring / scraping users' data, so this is a national emergency.... but not for 45 days and then also ok if we can buy the company on our terms.

This is yet another thing I guess time to throw up your hands and say, this is how we live now. One throw-it-against-the-wall proclamation after another.

Even if you're somewhat ok with it, are you really ok with this principle being applied, when someday it may not go how you want, for something you care about?

rapsey(10000) about 12 hours ago [-]

Justifications are meaningless. The US wants to curtail Chinese influence and power, so it finds some mumbo-jumbo to justify it.

president(10000) about 1 hour ago [-]

Please do some research before you downplay one of the most significant geo political events of our time. China poses an existential threat to democracy and freedom of the western world and we are (and have been) in a cold war with China. While you bask in your normalcy bias, realize that your ability to complain about your government is due to the fact that others before you have fought for your right to do so. If you care about the progress we have made and the future for your and others' children, you would not want the next century to be a Chinese century.

techntoke(10000) about 10 hours ago [-]

Does anyone remember when Trump bombed Syria while hosting Xi, and then bragged about how beautiful the chocolate cake was?

euix(10000) about 14 hours ago [-]

Didn't Pompeo say something this week along the lines of banning China Mobile servicing telephone calls to the U.S.? That means if you carry a cell phone from China to the U.S. you can't get roaming service anymore?

I am surprised Lenovo hasn't been singled out yet. They supply enterprises throughout the U.S. a lot of corporations that hand out laptops to their employees all use Lenovo.

rtx(10000) about 13 hours ago [-]

They might be forced to sell too.

systematical(10000) about 13 hours ago [-]

They ruined the ThinkPad line, please sell it to someone who can restore the brand.

ken47(10000) about 1 hour ago [-]

No one forced American corporations to do business with China. America's current lack of 'greatness' is America's own fault. There is a nonzero chance that America is going to face serious economic distress in the coming decade. Those who made poor leadership decisions for America's economy are desperate to find a scapegoat. No one forced American corporations to do business with China.

president(10000) about 1 hour ago [-]

Your scapegoat narrative is an unproductive one and apologist at worst. As usual, people are getting wrapped up in the 'America bad' trope while ignoring the fact that there are real people being targeted and having their lives tracked and terrorized by the Chinese government. I don't know what country you're from but if someone was behaving badly in your house, I would be kicking them out too. You'd have to be stupid to let a burglar in your house and offer them a room.

geokon(10000) about 14 hours ago [-]

The ongoing anti China circus through executive orders is arbitrary and draconian to say the least... and while I've been mostly ignoring it so far, a lot of Americans live in China and have money on Wechat. Are they now violating this week's 'law'?

The actual executive order: https://www.whitehouse.gov/presidential-actions/executive-or...

It's quite short and just adds a lot of FUD

jhart99(10000) about 13 hours ago [-]

I just went to my Wechat to try to transfer my money left in there to a friend. Transfer isn't going through.

trident1000(10000) about 14 hours ago [-]

Theres an incredible amount of foreign propaganda in western social media anyway. Twitter took down like a million CCP accounts the other month (they made an announcement about it) and thats just the ones they know about. This is a step thats at least in spirit probably makes sense but there are more larger issues. Reddit literally has an API to comment...you think all those policial default subs are real comments? Not a chance. Reddit doesnt care because it drives engagement.

qeternity(10000) about 14 hours ago [-]

A comment API makes sense as reddit originally had vibrant third-party app developers. It's not like it can only be used for bots to post.

0xy(10000) about 14 hours ago [-]

Political PACs were openly bragging about manipulating reddit's political subs in 2016 with thousands of accounts and reddit did absolutely nothing. [1] [2]

[1] https://www.motherjones.com/politics/2014/09/david-brock-hil...

[2] https://www.thedailybeast.com/hillary-pac-spends-dollar1-mil...

Deimorz(10000) about 14 hours ago [-]

Every website effectively has an API to comment (or do any other action that's available through the site). You can just use HTTP requests.

suyash(10000) about 14 hours ago [-]

yup they are all over popular websites with fake accounts, YT, FB etc

realusername(10000) about 14 hours ago [-]

Almost every CCP-owned channel on Facebook has bought fake subscribers, it's so visible they don't even try to hide it. As an example, CGTN French has 20M subscribers, more than any local French newspapers despite being virtually unknown in the country. If you go by subscribers count, it would be the most popular news platform of the country...

Shank(10000) about 13 hours ago [-]

According to Sam Dean (LA Times): 'Video game companies owned by Tencent will NOT be affected by this executive order! White House official confirmed to the LA Times that the EO only blocks transactions related to WeChat.' [0]

So that clears up at least a little of the ambiguity.

[0]: https://twitter.com/SamAugustDean/status/1291576813685108736

supergirl(10000) about 9 hours ago [-]

not yet, but this just shows how unpredictable US is. that will surely have an effect on many businesses.

nfc_(10000) about 12 hours ago [-]

So does this mean Apple will have to remove WeChat from the App Store everywhere in the world including China?

If Apple does this, what will it mean for their huge business in China?

bigpumpkin(10000) about 12 hours ago [-]

Trump just neutered Iphone sales in China. Apple can't even get revenue from WeChat now since those are transactions.

ulfw(10000) about 14 hours ago [-]

Forget I said anything. HN is infiltrated by propaganda without people even realising.

forsaken(10000) about 14 hours ago [-]


woutr_be(10000) about 13 hours ago [-]

Can you point out some obvious propaganda? And not just some argument you disagree with?

bmitc(10000) about 13 hours ago [-]

Does anyone know what the definition of a transaction is? The executive action is not clear at all and provides no definition of transaction. Big surprise.

Furthermore, what is the government going to do if I refuse to stop using WeChat? Is the U.S. going to embark on blocking the app at the network level? If not, how will they enforce this? What if a person as defined in the order travels to China? Are they banned from using the app there, where it's one of two apps basically required to get around efficiently?

This is just pure authoritarianism on the part of Trump. I am sick of him hijacking this country. And all of this is in a bid to get re-elected. He has been holding this country hostage, trying everything he can to ensure his re-election.

For me personally, I use WeChat on a daily basis as it is the primary method of communication between myself and my fiancee, a Chinese citizen, who is still banned from re-entering the U.S. despite living and working here. So now it's the U.S. government's policy to implement it's own authoritarian controls on what citizens can and can't use.

Honest to god. This country is a lost cause. It's filled with people who think their freedom allows them to demand access to military weapons and that their liberties are at risk due to requiring masks, but these same people will applaud this.

perennate(10000) about 12 hours ago [-]

I think most likely Apple and Google will have to remove TikTok and WeChat from the app store in U.S. If you install manually on Android you would probably be able to continue to use it. The executive order only bans transactions (since that is the limit of the power granted to president by IEEPA), so the executive has no authority to 'block the app at the network level' -- the IEEPA explicitly restricts the prohibition of 'any postal, telegraphic, telephonic, or other personal communication, which does not involve a transfer of anything of value'.

AFAIK the term 'transaction' comes directly from IEEPA since that is the law that lets the president make this executive order, so if you want more details on what is included in 'transaction' you can see the text at https://www.law.cornell.edu/uscode/text/50/chapter-35

TMWNN(10000) about 14 hours ago [-]

Explanation of CFIUS and why and how Trump can use it: https://www.lawfareblog.com/tiktok-and-law-primer-case-you-n...

JumpCrisscross(10000) about 14 hours ago [-]

Our 70s-era national security regime needs desperate Congressional overhaul. I take particular peeve with the prohibitions on judicial review of CFIUS actions.

danhak(10000) about 12 hours ago [-]

What a bizarre and interesting turn. I'd give a lot to be able to sit in on the meetings between TikTok and Microsoft right now.

I wonder if FB will come over the top with a bid.

tomerico(10000) about 10 hours ago [-]

FB would have too much scrutiny and chances for the deals to fail to pass regulators. If they are really looking for a quick and guaranteed deal, they'd look elsewhere.

almost_usual(10000) about 12 hours ago [-]

> I wonder if FB will come over the top with a bid.

Doubt it, they already have a new competing product. As soon as TT engagement drops due to friction the user will switch.

rollschild(10000) about 2 hours ago [-]

Let's face it: it has nothing to do with 'treat people the way they treat you.' It's just Trump trying to win/secure the votes from his loyal supporters.

Does TikTok grab user data and analyze them for ads, recommendations, etc? Sure, so do Instagram. But does TikTok send those data back to Chinese government so that it would potentially threaten U.S. national security? We need proof, but so far none. Here's an article of French hacker Elliot Alderson (@fs0c131y) analyzing TikTok code: https://medium.com/@fs0c131y/tiktok-logs-logs-logs-e93e81626...

So IMHO the two real reasons Trumps banned TikTok and WeChat are: 1. His supporters are mad that everything is made in China. It violates some basic economics principles but that's fine. 2. What happened to that Oklahoma rally, Trump really took it personal.

This ban put hundreds of thousands of Chinese people (including those who are already U.S. citizens or green card holders) in the U.S. to a difficult position. They don't have other ways to communicate with there friends/families in China other than WeChat. But I guess, those people (including myself), are the least important factors Trump worries about, if he even cares at all.

MrStonedOne(10000) 38 minutes ago [-]

> But does TikTok send those data back to Chinese government so that it would potentially threaten U.S. national security? We need proof, but so far none.

How about the No place to hide laws that mandate this? Proof enough for you?

sreejithr(10000) about 8 hours ago [-]

I agree 100% with this. I mean, there are other developing economies with serious tech sectors like India which operate according to the rules based open market system we have in place.

If the US doesn't take action against Chinese protectionism, why should other countries abide by the rules and give US access to their markets? The US provides China it's market even though China closes it down for everyone.

I think US being soft on China sets a bad precedent.

thewileyone(10000) about 7 hours ago [-]

India has a lot of protectionism in place for their core industries as well.

Example is call centers. Foreign call centers, like in the Philippines, are not allowed to service any company that does has operations in India.

notsureaboutpg(10000) about 6 hours ago [-]

Hahahahaha, India operates according to open market system?

India was well known for silly electronics tariffs, government overregulation and favoritism, and tit for tat retaliation and mercantilism well before the US and China started their economic war.

iandanforth(10000) about 13 hours ago [-]

Under what authority can he issue this order?

perennate(10000) about 13 hours ago [-]

The order says:

> By the authority vested in me as President by the Constitution and the laws of the United States of America, including the International Emergency Economic Powers Act (50 U.S.C. 1701 et seq.) (IEEPA), the National Emergencies Act (50 U.S.C. 1601 et seq.), and section 301 of title 3, United States Code,

sneak(10000) about 14 hours ago [-]

An important note that nobody seems to be reporting on: this is only possible because Apple and Google have control over which apps are permitted on your device.

The US government has no legal authority to ban publishing (including apps) in the US. They do, however, have the ability to regulate trade between the operators of the App Store (Apple) and the Play Store (Google), and app publishers, which is what they're doing here.

Ban the business relationship, and they've effectively banned the publishing. It's an end run around the US constitution, and everyone should be up in arms about it: not just at the US government, but also at Apple and Google for creating a legal chokepoint for mass censorship by the US federal government.

This would not be possible if end users actually exercised control over their own devices. This is how the web works, and it's how everything else on your device should work, too.

ac29(10000) about 12 hours ago [-]

> Apple and Google have control over which apps are permitted on your device

Has Google ever unilaterally removed sideloaded apps from Android?

kiba(10000) about 14 hours ago [-]

And how would we prevent the Chinese from simply subverting our democracy?

JCharante(10000) about 13 hours ago [-]

On the bright side banning Chinese apps doesn't have an impact on the intentional users. By that I mean is that there seems to be a lot more app stores than just Play Store, and it's even common for Chinese apps to offer an apk download off their website. People who are unknowingly using Chinese developed apps are probably going to be the most affected because they're not familiar with side-loading applications and are just going to switch to their nation's version of the application which is probably the intention. It is kinda fair because as soon as a major country starts to practice protectionism then the rest have to or else they'll get unfairly exploited.

voisin(10000) about 14 hours ago [-]

> everyone should be up in arms about it

I feel the same way! I can't understand why there aren't bipartisan calls for Trump to knock it off. Is it that most people aren't fully comprehending what's going on, or are those of us upset about this being dramatic?

billfruit(10000) about 14 hours ago [-]

What if the app publishers stop taking payments and ads from the US, in that case there may not be no longer cash transactions, so will the restrictions apply?.

ENOTTY(10000) about 13 hours ago [-]

That isn't my impression of how the web works. The second your bits leave your LAN, they begin traveling over infrastructure that is owned and operated by another private entity. They could rescind your access at any time, especially if the government orders it so.

Phaedor(10000) about 14 hours ago [-]

And this is why as a European I think that we should take a long and hard look at our dependency of US technology. Its becoming more and more clear that US is not a stable actor.

I think it would be prudent for us to force alternatives to apple and google app stores (and other software) that are housed and owned in Europe. I have no confidence that US intelligence is not accessing our data and setting up profiles on Europeans that could be used for nefarious purposes.

sidibe(10000) about 13 hours ago [-]

'Threat' aside, it does seem fair for the West to reciprocate bans on Google/Facebook etc, but not sure the best way to communicate with family in China now if the western messaging apps are banned there and vice versa

mdorazio(10000) about 13 hours ago [-]

Email? I know that seems dismissive, but people got along just fine before Facebook, texting, WeChat, etc. I suspect in reality there will be an endless whack-a-mole game with messaging services where people use a different one fairly frequently.

dicomdan(10000) about 13 hours ago [-]

The family in China should contact their representative and ask them to unblock Signal, Telegram, or any other method of communication. If they care about their citizens abroad they will surely do so.

mandeepj(10000) about 13 hours ago [-]

Have u tried connecting via a vpn ?

afrojack123(10000) about 12 hours ago [-]

Only took a couple decades to do this. A messaging app that allows China to read everything but not other countries should be illegal to begin with. Entirely secure or not at all is the way.

nicbou(10000) about 12 hours ago [-]

Isn't this exactly what US companies do, and what the US government wants to do?

paxys(10000) about 13 hours ago [-]

None of these Chinese companies would have grown to what they are today without intense government-enforced protectionism at home, and until China agrees to compete on a fair playing field I'm perfectly fine with them all being banned outside of their firewall.

Would China ever agree to let Instagram, Facebook, YouTube, Snapchat, WhatsApp, Messenger etc. or any new startup operate freely and make money there? It should never have had this one-way economic benefit to begin with.

est(10000) about 7 hours ago [-]

> None of these Chinese companies would have grown to what they are today without intense government-enforced protectionism at home

As others have mentioned, Alibaba, Taobao, Alipay grew the same time as mazon, Paypal and eBay was available in China. Also newegg had a much better headstart than jd.com but still failed.

At one time even Yahoo Messenger, MSN Messenger was a serious threat to Tencent QQ. Wordpress was much more popular than QZone.

logicchains(10000) about 13 hours ago [-]

>None of these Chinese companies would have grown to what they are today without intense government-enforced protectionism at home

Ah yes, I remember when the Chinese government killed off Vine to give TikTok a head start.

fqye(10000) about 9 hours ago [-]

You couldn't be wronger.

Alibaba wasn't protected by the government at all. Ebay's then CEO bought a local e-biz site eachnet and relocated to Shanghai but still failed at competing with Alibaba.

Amazon bought some local e-biz site too and failed at competing with Alibaba too.

MSN was a huge player in China with far more users than QQ when it was taking off. ICQ was in China too. At the time IM wans't a big deal for big players. Tencent went IPO as very small startup, far less than Sina, Sohu etc., China's portals.

Chinese internet giants could win because they adapted fast to local market.

bleepblorp(10000) about 13 hours ago [-]

It's pretty much impossible for lower-income countries to develop economically without protecting their emerging businesses from being slaughtered by established foreign competition.

The priority of the Chinese government is the benefit of its citizens (leaving aside that the CCP doesn't consider ethnic minorities in China to be full citizens, as this is a separate issue) and its domestic economy, not the benefit of the American tech sector's senior executives and shareholders.

It's not reasonable to expect, much less demand, that the Chinese turn their tech economy over to silicon valley by allowing unrestricted US entry. It's not in their economic interests, nor is it in their security interests, and no amount of US bullying will change this.

China is not Europe; it's not going to hand over the keys to its economic future to the United States just because the US asks for them.

baconandeggs(10000) about 3 hours ago [-]

What are you talking about? All countries protect their industries. Are we forgetting about Bombardier? All military contractors are almost part of the US government at this point. And Apple, and now Microsoft apparently are leveraging Trump to gain every bit of advantage they can. The US forces entire countries to give favorable deals to american companies and has for decades.

echevil(10000) about 9 hours ago [-]

How so? Google operated in China for more than a decade and never got a major market share, and they already did fairly well compared to others. Facebook was largely unheard of after being available in China for many years before finally got banned in 2010. Amazon has been operating in China forever but is basically irrelevant to most Chinese. Most American companies did terrible job in localization and I see no chance of them winning the competition with their local competitors.

If they face strong local competition in any other country, they'd fail too

alexmingoia(10000) about 13 hours ago [-]

Two wrongs don't make a right. US companies don't have to trade with China.

But the US banning trade with China is nothing but an attack on Americans and my freedom to trade and communicate.

xvector(10000) about 13 hours ago [-]

Those companies are free to operate in China if the comply with Chinese law.

The US hasn't issued any legal barriers for TikTok to jump over, they just straight out banned it.

ejanus(10000) about 9 hours ago [-]

The best argument I have read here.

dirtyid(10000) about 13 hours ago [-]

Since TenCent is huge:

>Video game companies owned by Tencent will NOT be affected by this executive order!

>White House official confirmed to the LA Times that the EO only blocks transactions related to WeChat

>So Riot Games (League of Legends), Epic Games (Fortnite), et al are safe


I guess Chinese users will resort to Chinese IP VPNs, there's already some to get around geofenced mainland content.

E: much downvote

Spartan-S63(10000) about 13 hours ago [-]

I had no idea that Tencent had full ownership of Riot games. Same thing with 40% ownership of Epic Games. Wow!

cwhiz(10000) about 3 hours ago [-]

The entire western world should treat China exactly the same way China treats them.

Western companies have a difficult time operating in China, but Chinese companies have zero problems operating in western countries. It's completely unfair and if we follow this to conclusion, the future will be one of only Chinese international corporations.

Would WeChat or TikTok even exist if the Chinese market were open to existing western chat and social media software?? Unlikely.

If China wants their companies to be able to access western markets then they must allow western companies to access their markets.

Quite frankly, the entire western world should be banding together to oppose this nonsense from China. We should outright ban any goods or technology that originates in China until China changes its behavior and opens its economy.

tfcata(10000) about 3 hours ago [-]

China has not forcefully seized foreign company's promising assets so good so it's impossible to consider for sale, by both a measly 'offer' of maybe 1/10, or less, of the market value and threatening from the leader, has it?

kjjw(10000) about 1 hour ago [-]

No thanks, while the US protects its digital economy from Chinese companies, it is also slapping tariffs (or is threatening) on European goods.

The US doesn't have any friends any more because it acting like a shit to them all. Meanwhile every day it looks more like a dictatorship with no regard for its populace. Honestly it's time the rest of the western world moved on. Relationship over.

bjo590(10000) about 3 hours ago [-]

> We should outright ban any goods or technology that originates in China

If we were to start that ban many of Walmart's non-grocery (and some groceries) shelves would be empty for considerable time.

I'm unsure there even are alternatives that can meet US demand for everything China provides.

svara(10000) about 3 hours ago [-]

It's one thing to call for a system of fair regulations that provide a level playing field; it's another thing altogether to essentially expropriate a company through executive action because it was doing too well.

Your comment makes sense and I mostly agree with it, but not on this submission.

yibg(10000) about 1 hour ago [-]

Regardless if it's justified or not I think this is a bad move by the US on the global stage.

The US just banned apps that didn't break any law, for vague "national security" reasons. And due to the global reach of American tech companies it becomes effectively a global ban.

Now every country that's not the US will need to 1) think twice about doing business in the US and 2) re-examine American control over the products and services their citizens can and cannot receive. Specifically Apple and google in this case.

Most countries and companies are already cautious when doing business in China because of the unpredictable and arbitrary nature of doing business in China. Now the US is headed towards the same direction.

So congratulations, the US is becoming more and more like China. Doesn't sound like a win to me.

turingbook(10000) 29 minutes ago [-]

>Western companies have a difficult time operating in China, but Chinese companies have zero problems operating in western countries.

Ask Elon Musk about the first part, and ask Huawei guy about the latter.

dontcarethrow2(10000) about 3 hours ago [-]

Quite frankly, the entire world can learn from China on how to protect its own economy from bullies. I'd hate to live in a world where the Microsofts buys all the Skypes and ruins all sorts of innovations you all claim comes from this freedom. You shareholders only have $ in your eyes, this means shit to the young people looking towards their future. I don't mind you all protecting your interests, you're human.. but it definitely adds too much taint to your 'genuine' perspectives.

jeswin(10000) about 3 hours ago [-]

> The entire western world should treat China exactly the same way China treats them.

You could use 'democratic nations' instead of western world. I agree with you otherwise.

What will a dictatorship do if it had hundreds of billions in trade surplus every year? Well, they are eventually going to spend it on weapons.

baron_harkonnen(10000) about 2 hours ago [-]

> the future will be one of only Chinese international corporations

A world ruled by corporations headquartered in a single country, with other countries having their politics and economy determined by the interests a single nation has been the rest of the worlds situation for 50+ years. It's somewhat ironic to see Americans suddenly getting concerned about this.

> Would WeChat or TikTok even exist if the Chinese market were open to existing western chat and social media software?? Unlikely.

The argument that if the Chinese market was fully open then their local companies would have been more quickly devoured by single, US based corporations doesn't do much to convince me that what the Chinese government is doing is wrong.

And don't fool yourself, this had nothing to do with what's 'fair' for America or Americans, all of this is so strong arm Chinese companies into surrendering to corporations that happen to be based in the US on favorable terms. This is the US government working as the goons to enforce Facebook, Google and in this case specifically Microsofts global dominance.

supergirl(10000) about 3 hours ago [-]

so china won is what you're saying? their system is superior if the west has to switch to it

cd_2001(10000) about 2 hours ago [-]

Please excuse me for my semi-rant. I don't mean to be too critical, but I think framing every news about China in this 'the West vs China' angle is really counterproductive.

First of all, 'the West' is a very vague concept, quite unlike talking about China. Does it include all of Europe? Does it include Moscow then? Does it include parts of Latinamerica? Maybe you mean NATO (but then you should include Turkey too). Or maybe you only mean Five Eyes (Australia, Canada, New Zealand, the United Kingdom, United States). This isn't just nitpicking: your point might make sense if you mean Five Eyes, but it might not make sense at all if you include, say, Greece , Turkey and Israel.

Second, talking only about 'the West' vs China marginalizes the rest of the world. Most of the people in this world live in neither the West nor in China and yet they never seem to matter according to most discussions I see here in Hacker News (I, for one, identify with neither).

This is specially bad when people implicitly say things like 'the West values freedom and democracy, whereas China...'. If democracy and human rights were something unique for 'the West', then you shouldn't be surprised that China doesn't like those concepts (in fact, they oppose liberal value precisely because of this: 'this only works in 西方, not here in China').

西方 = the West

Al-Khwarizmi(10000) about 2 hours ago [-]

Why 'the entire western world'? As an European, why should we align with the US in this?

The EU doesn't have any tech giant because, contrary to China, we have received US tech giants with open arms. So our economies bleed money and lose jobs to the likes of Amazon, because hey, 'the free market'. What do we stand to gain by siding with US companies in this US companies vs. China companies wars?

Perhaps it would be better for us to just imitate China, start being protectionistic and banning foreign tech companies so we can grow our own and prosper. It works. In the US you're pretty much telling us that it works - so well that you are doing the same.

Now that the US has shown it can ban foreign tech companies if it's in their own interest, and all the moralizing talk about the free market has been exposed as the self-serving propaganda it has always been, what reason do be exactly have to keep playing this game where we are the biggest losers?

AndyMcConachie(10000) about 2 hours ago [-]

Are folks unaware of the history of westerners and trade in China, and how this kind of attitude could be interpreted by Chinese people not ignorant of their country's history?

In the late 19th century Great Britain fought two Opium Wars to force China to open up its markets to British opium. The British were forcing Indian farmers to grow opium and then taking it to sell to China. They wanted access to Chinese silver, but China had little reason to give it to them. So Britain's answer was to get Chinese people addicted to opium and get the silver through trade of opium for silver.

The Qing dynasty, for obvious reasons, didn't like this idea and forbid the British from importing their opium. So the British fought two wars to force them to accept their imports.

The Economist magazine was established in the mid 19th century as a pro-slavery, pro-imperialist magazine. Their arguments during the lead up to both opium wars sound a lot like yours. I want to make people aware of this, because it's important we understand how that kind of argument sounds to many Chinese people less ignorant of Chinese history than most westerners.

cmurf(10000) about 2 hours ago [-]

There is no political or economic will to embargo Chinese made goods. It didn't get this way overnight, it wouldn't be possible to end it overnight. It would translate into wholesale asset destruction of many businesses who depend on supply chains that would take years to recreate. It would make hundreds of millions of poor even more poor: here and in China. It would take away cheaper versions that make people feel just a little bit more in touch with people who can afford every day things. It would be an economic nuclear bomb just to prove a point.

And for ~40 years the U.S. at least has considered attacks on its economy as a national security threat - it is extremely likely an embargo leads to an actual war. Making everyone worse off in the short term is a fast track to violence.

Besides, the vast majority of consumers have voted with their wallet.

Bhilai(10000) about 2 hours ago [-]

We are not China and we dont want our countries to behave or become like China.

krtkush(10000) about 3 hours ago [-]

All this is leading me to have a stronger belief in Nassim Taleb's 'The Most Intolerant Wins' theory[1]. Though, in this case there is no minority as such. China was intolerant to the US's ideology and exploited its openness for its own benefit. Now, unfortunately, it has come to a point where even US has to be intolerant like China to be able to get back at it. Thoughts?

[1] https://nassimtaleb.org/2016/08/intolerant-wins-dictatorship...

bigpumpkin(10000) about 2 hours ago [-]

Meanwhile, American brands find their refuge in China:


kgersen(10000) about 3 hours ago [-]

But seen from Europe, your 'western world' = mainly USA ...

which 'existing western chat and social media' aren't from the US ?

minusSeven(10000) about 3 hours ago [-]

I think China already promised that one or two years ago during the trade war with US but nothing major has come of it. I doubt China would follow through because it can have potential consequence in their censorships rules.

I guess eventually all other countries would resort to having laws specific to handling Chinese companies and all laws common for companies of all other countries.

as300(10000) about 2 hours ago [-]

Well, if there was truly a quid-pro-quo, then the United Kingdom should pay China reparations for its atrocious actions during the opium wars, similar to what Germany had to do to survivors of the holocaust.

If there were truly a quid-pro-quo, the West would be clamoring for statehood for Puerto Rico like they do for Taiwan.

If there were truly a quid-pro-quo, calls for the 'entire western world' to band against China would make as much sense as calls for 'the entire western world' to band against the United States, who did quite a bit to rig the global economy in its favor after WW2 and Bretton-Woods.

If China treated the Western world the way the Western world treated China, there would be hell to pay. And yet people like you just can't stand that they would act in their own self interest, and become a superpower without asking the permission of the West.

Countries will take whatever advantages are available to them to develop and benefit their people. China never tried to hide what they were doing. All of the world's economists were aware of their economic protectionist policies for the past 50 or so years. Let's call a spade a spade. This 'outrage' about China's policies in the year 2020 is nothing more than a frustration with the West's ruling classes for failing to respond to China's rise at all in the last 50 years. And there is clearly a sense of entitlement as well, as though those of use living in the West deserve to live in the hegemonic country despite not working as hard or valuing progress as much as those in other societies.

The Western world never cared about Chinese people or what was happening in China until it started to threaten their own global dominance. Now all of a sudden you have elite clamoring for civil rights and justice in China, while those same elites profited for decades off of cheap Chinese labor after the Great Leap Forward and Tiananmen Square.

Sure, you should advocate for your country to change its policies if its getting a bad deal from China. But turning this all into a 'West v. China' thing is how we got here in the first place.

nafizh(10000) about 3 hours ago [-]

Add to this the current ongoing Genocide against the Uighurs. Utterly insensible to let such a Government access western markets which eventually only bolsters its genocidal prerogatives.

dheera(10000) about 1 hour ago [-]

The real loser of this battle will be Apple, not China; they may just lose all their Chinese users in the US to Android in a couple months.

The US is and will always be a free speech country, and they cannot go to the extent of imposing a firewall like China does. The extent to which they can take action is to require WeChat and TikTok off the app stores of the US. Which would leave Android users the option to install the app directly from an APK file, bypassing the Google store (no rooting required) while Apple users basically have no recourse without jailbreaking.

markus_zhang(10000) about 2 hours ago [-]

Lots of western companies are operating in China. If they feel it's too difficult they will pull it out, but please don't act like a representative to all of them as if you know everything.

BTW please stop purchasing anything that originates from China immediately if you want to follow your own words.

throwaway64054(10000) about 14 hours ago [-]

The (US-HQ'd) company I work for has a small but significant office in China. We are not a household name, even within tech, so I doubt we'd be a target of any retaliation by China.

However, my worry is this: Whilst we have a lot of really good people in China, we don't do any business in China (and there's no realistic prospect of us doing so). So at this point operating in China seems like a huge exposure to risk for relatively little reward in the long term.

But I don't see an alternative for us. Pull out of China and business continuity would take a huge hit; stay in China and accept the risks & uncertainty.

Basically, I'm torn about the whole situation.

balola(10000) about 10 hours ago [-]

Your Chinese employees will be fine, but foreign personel coming in should indeed be very careful not to pique the party's attention as it will actively be seeking pawns.

komali2(10000) about 14 hours ago [-]

How bad of a hit would your business continuity take? Not to get dramatic, but, is it worth risking the lives of your employees over? Personally I won't step foot within PRC borders given my staunch pro-Taiwan sovereignty stance. It's a vicious state that has in the past indicated it will go to any lengths to protect its stature.

sudoaza(10000) about 6 hours ago [-]

This should be taken to the WTO if it wasn't coopted by the US and allies

xvector(10000) about 4 hours ago [-]

Won't make a difference, Trump would sooner pull out of the WTO than have his decisions dictated by it.

gumby(10000) about 14 hours ago [-]

Can someone explain the motivation?

bleepblorp(10000) about 14 hours ago [-]

It's one of those American policies that has a triple motivation: it provides an opportunity for the GOP to look 'tough' on foreigners, it provides the opportunity to protect the US tech sector's market position by outlawing a potential competitor, and it satisfies nationalist ('national security') factions in Washington that want to roll-back Chinese economic development.

Foreign policy initiatives that satisfy the needs of three or more power blocs in Washington DC tend to have extremely bad outcomes; the last major policy that had a triple motivation was the 2003 invasion of Iraq.

(final paragraph edited for clarity)

mienski(10000) about 14 hours ago [-]

The stated motivation is to reduce misinformation from China in lead up to election. Pretty wild that Facebook which was 100% proven to have been used for Russian misinformation in the last, and this election, somehow dodged a bullet on this...

My (totally personal opinion guess) is that this is just another way to poke the bear as part of the tit-for-tat with China and maybe a minor contributing factor that someone has got in his ear and blamed TikTok for his low campaign event turnout after there was a TikTok meme to RSVP to his events to make them sound like they were going to be huge, then embarrass him with low numbers.

Cynical perspective is that one of his friends with a stake in Facebook got in his ear about China using TikTok to make fun of him or something, and now Instagram Reels is right there to pick up the slack. Nothing would surprise me with this govt anymore. I mean the news is all talking about TikTok now instead of COVID-19... troop bounties... so on...

babesh(10000) about 13 hours ago [-]

This is obviously just a tactic by the Trump administration to distract people from the US economy tanking because of the mishandling of covid-19. If you haven't noticed, the US has much bigger problems. How about covid-19, the economy, lack of universal healthcare, weak education system, huge deficits, etc?

I am not blaming the mishandling of covid-19 solely on the Trump administration. I am just pointing out their Wizard of Oz distraction tactic and how the vast majority of people fall for it.

There is an economic competition between the US and China but this move is pure silly. WeChat isn't doing anything in the US. It's for Chinese people to talk to family in China.

em500(10000) about 8 hours ago [-]

Stoking animosity towards outsiders is a time honored tactic of all bad regimes (including that of the PRC) to distract citizens from their own failings. Time honored because it works almost every time.

babesh(10000) about 11 hours ago [-]

I think that negative votes on Hacker News are an indicator of people denying the truth. Bring it on.

neves(10000) about 14 hours ago [-]

Which workarounds you suggest around the Great Firewall of the United States of America?

stuqqq(10000) about 14 hours ago [-]

The GFW of China is to prevent its people to see outside. The GFW of America, at least for now, is to prevent China from peeking in. One is implemented as an allow list, the other one is a deny list.

_-___________-_(10000) about 14 hours ago [-]

Might be time to start a VPN service with exit points in China!

euix(10000) about 14 hours ago [-]

This. It was Cisco that originally helped build the GFW in China, this is like ironies to the power of ironies, exponential levels of irony.

voisin(10000) about 14 hours ago [-]

Is there any precedent for a president to do such a thing? I am completely astonished and dumbstruck that everyone is ok with this.

TearsInTheRain(10000) about 14 hours ago [-]

The government frequently bans foreign products that have national security implications.

casefields(10000) about 9 hours ago [-]

>That happened in 2012, the only legal challenge to CFIUS since it was established in 1975, when Chinese-based Sany Group's affiliate Ralls Corp. bought wind farms in Oregon. Then- President Barack Obama ordered Ralls to not only divest the wind farms, but also forced it to remove items added to the facilities, including concrete foundation, and barred employee access to the premises. Sany complied.


karterk(10000) about 13 hours ago [-]

Because many US companies have been at the receiving end of unfair trade practices, copyright theft and blatant IP violations from the mainland. It's good that we are finally doing something about that.

logicslave12(10000) about 14 hours ago [-]

Dumbstruck that we wouldn't allow an adversarial country collect data on all of our citizens locations? And potentially more?

suyash(10000) about 14 hours ago [-]

Dumbstruck to find that there is a leader with a spine in this country? That's quite something.

hiimtroymclure(10000) about 14 hours ago [-]

At least in my social circles, this has been one of the few things trump has done that both dems/conservatives agree with. Why would you be ok with CCP collecting info on US citizens and spreading misinformation to benefit a foreign regime?

badRNG(10000) about 14 hours ago [-]

The outcome of this ban has important implications for potential bans in the future that affect apps at odds with administrations. End-to-end apps like Signal come to mind.

pfisch(10000) about 13 hours ago [-]

Now we should intermittently just turn off websites and services of Chinese companies in the US, randomly lose and/or return shipments, etc.

They have been cheating us forever. We should do it too if that is the way they want to do business.

michaelyoshika(10000) about 12 hours ago [-]

Totally, I've just thrown out everything made in china from my house. You should do the same!

PopeDotNinja(10000) about 10 hours ago [-]

For WeChat, how much impact will this have on Americans? As an American, I tried signing up to communicate w/ someone in China, and I couldn't get through the verification process.

Obviously TikTok has a big USA presence.

thaumasiotes(10000) about 9 hours ago [-]

> As an American, I tried signing up to communicate w/ someone in China, and I couldn't get through the verification process.

I signed up before there was a verification process. But my understanding is that you get vouched for by an existing account. Since you were signing up to communicate with someone, wouldn't that person have been a natural choice to verify you?

tmpz22(10000) about 14 hours ago [-]

Doesn't this eviscerate Chinese investment in SV?

almost_usual(10000) about 13 hours ago [-]

No because wealthy people care about making more money not principles or nationalism.

A4ET8a8uTh0(10000) about 4 hours ago [-]

I am going to ignore value judgment over whether it is fair, or whether it even makes sense and am going to jump straight to enforcement framework and EO interpretation.

EO appears to single out two entities WeChat ( Tencent subsidiary ) and TikTok ( ByteDance subsidiary ). EO appears to indicate that the restrictions will be governed by sanctions framework.

Tencent owns a fair amount of gaming outfits so based on ownership, for example, Grinding Gears could be affected since Tencent owns 80% stake there. Gears seems to interpret the order in an optimistic way leaning heavily on phrase 'any transaction that is related to WeChat', but ignores 'with Tencent Holdings Ltd.' and how it is likely going to be interpreted by the banking. In short, Tencent interpretation right now is 'it applies only if it only blocks transactions related to WeChat.'

I personally have less generous read, but if a lawyer could actually weigh in, that would work:P

Original text:

Section 1. (a) The following actions shall be prohibited beginning 45 days after the date of this order, to the extent permitted under applicable law: any transaction that is related to WeChat by any person, or with respect to any property, subject to the jurisdiction of the United States, with Tencent Holdings Ltd. (a.k.a. Téngxùn Kònggǔ Yǒuxiàn Gōngsī), Shenzhen, China, or any subsidiary of that entity, as identified by the Secretary of Commerce (Secretary) under section 1(c) of this order.[1]



I am not a lawyer yo. Don't be an idiot.

pbhjpbhj(10000) about 3 hours ago [-]

This joke was good:

> In addition, the application captures the personal and proprietary information of Chinese nationals visiting the United States, thereby allowing the Chinese Communist Party a mechanism for keeping tabs on Chinese citizens who may be enjoying the benefits of a free society for the first time in their lives. //

Because only a fascissistic dictatorial government would be shooting on it's own citizens, eh.

Who is using TikTok for something that's got any business or political sensitivity?

I'm surprised Trump doesn't just admit 'China is beating us on internet platforms, we're not going to let them win financially'. He could present it as 'leveling the playing field'. That sort of overtly authoritarian dictatorial xenophobic fascism seems to be lapped up by the USA public at the moment?

Or does anyone want to defend that some video of me dancing to music is somehow critical to keep private??

I can see there's a side effect of the USA administration not wanting any apps they don't have a backdoor to being used in USA.

They don't say which companies will be subject to the order, really open and free (presumably to allow maximum market manipulation [fluctuation on the speculated actions, then controlled movement according to which companies the order covers] and a month and a half to get relevant bribes).

Tl;dr depends if Grinding Gears pays Trump enough campaign funding I'd guess.

c789a123(10000) about 6 hours ago [-]

Great action, I fully support!

wildrhythms(10000) about 4 hours ago [-]

What's the motivation?

khuey(10000) about 13 hours ago [-]

Seems pretty weak to be honest. Definitely not the 'American Great Firewall' that some were predicting.

joe_the_user(10000) about 12 hours ago [-]

After 45 days, would anyone logging into a Wechat server in China be guilty of circumventing an embargo? Would they have to actually be sending money?

danboarder(10000) about 14 hours ago [-]

This is sad, we in the US are losing the little high ground we had and no longer lead by example to bring the world closer together by encouraging free markets and free people. I'm against this protectionist nationalism, it only leads to future conflict.

tomp(10000) about 8 hours ago [-]

The only thing that matters is long-term consequences. This could be war, but literally nobody wants that, least of all China. It also could be beating China down on its knees, and forcing it to free its people and its markets. That would be a remarkable outcome (not sure Trump can get there, though - I expected something in this direction in his dealings with North Korea, but he dropped the ball... I don't know, maybe he just got bored?)

sreejithr(10000) about 8 hours ago [-]

Not really. There are other developing economies with serious tech sectors like India which operate according to the rules based open market system we have in place.

If the US doesn't take action against Chinese protectionism, why should other countries abide by the rules and give US access to their markets? The US provides China it's market even though China closes it down for everyone.

I think US being soft on China sets a bad precedent.

yumraj(10000) about 9 hours ago [-]

I'd rather that US lose the moral high ground than lose its bread and butter.

karterk(10000) about 13 hours ago [-]

US is merely reciprocating the policies of CCP: let them allow US companies to compete fairly in China as well.

toto444(10000) about 1 hour ago [-]

> I'm against this protectionist nationalism, it only leads to future conflict.

And unfortunately this is what the top comment preaches and I am very disappointed to see that the more-educated-than-the-average-person readers of HN fall into the trade war / retaliation fallacy.

Trade always benefits both parties. Otherwise they would not start trading in the first place.

Markoff(10000) about 12 hours ago [-]

it's not only about protectionism, WeChat is also used for tax evasion through wechat payments, let alone security risks with Chinese gov having access to all photos, documents and videos of residents of US who have this spyware on their phone

frequentnapper(10000) about 10 hours ago [-]

You need to study up on game theory. Being the only nice guy surrounded by jerks who take advantage of you benefits them and wipes you out.

baby(10000) about 14 hours ago [-]

I don't think the US has ever lead by example. Europe has though.

ekianjo(10000) about 13 hours ago [-]

> no longer lead by example to bring the world closer together by encouraging free markets and free people.

It does not work very well if you are the only nice guy within a group of a--holes. You just end up being beaten up. Free Markets work well if pretty much everyone adopts the same rules of commerce, but China is way beyond that point.

tannhaeuser(10000) about 10 hours ago [-]

I don't get it. It's US companies standing to loose their stronghold (near monopoly) on social, advertisement, and other forms of monetizing the web if the US creates a precedent for 'national security' in this way, as in 'we're welcoming social networks and free speech as long as it benefits the US and can be searched without warrant.' Quite predictably, governments all over the world will be pressurized to question why they should give US companies (bred by teethless US antitrust) a free pass to destroy their publishing industry. Publishers themselves will put this onto the agenda in their own best interest. The French are already on the fence to create new digital tax legislation after EU/US negotiation have been aborted by the US side. Maybe hurting Google, Facebook, Twitter & co is seen as desired collateral damage?

TheOtherHobbes(10000) about 8 hours ago [-]

Election. Team Trump believes it needs to take a firm stand on something or other.

There are much more obvious national security threats, but this is kindergarten tit for tat politics for ratings. So don't expect any action on those.

Or any understanding of second order consequences for the US. Or a coherent national IT security strategy, which is what the US and EU really need.

donw(10000) about 9 hours ago [-]

The US lost that near-monopoly years ago.

Broadly speaking, I think we're witnessing the birth of three distinct global powers.

One is centered on the US, and consists (broadly) of Canada, Mexico, the rest of Latin America, the Commonwealth nations, Japan, Korea, Israel, and Taiwan.

An American civil war would destabilize this to a large degree, so we'll have to see if that plays out.

Edit, since people seem confused: This would be an absolute nightmare scenario, but it is in the realm of possibility, and would massively change international political structures.

Another is centered on Germany and France, and controls Europe, parts of Africa, and the Middle East.

Finally, we've got China, which will likely control a big chunk of Africa and Southeast Asia.

Russia is a tough one, but I see them siding with China or Europe.

Contested territory will include Taiwan, bits of the Middle East, and an escalation of the border disputes with Japan. Likely bits of eastern Europe, plus conflicts on the China/India border...

What a time to be alive.

Edit: Just because somebody says that something is a possibility doesn't mean that they want that thing to happen.

Me coming down with a case of COVID is a possibility. In fact, I'm operating under the assumption that I will be infected at some point, regardless of precautions (masks, hand-washing, stepping back from my competitive doorknob-licking career, etc.)

That doesn't mean I want a case of Horribly Shitty Virus With Not-Yet Well-Documented Complications.

I'm just prepared for the worst (as best as I can), and hoping for the best (because why not?)

pelliphant(10000) about 6 hours ago [-]

Are you still trying to find logic in trumps actions?

The popular opinion on reddit seems to be that this is just trump being told that tiktok users where responsible for his failed tulsa rally, and personally, that explanation seems just as likely as any of the other explanations given here...

baybal2(10000) about 8 hours ago [-]

> and other forms of monetizing the web if the US creates a precedent for 'national security'

By far not a precedent. USA has been doing this even with its Western European 'allies' for as long as NATO existed.

As for Chinese cos. Forced sales at firesale prices are not unprecedented too: Sany

jb775(10000) about 3 hours ago [-]

You're overthinking this. It's one thing for a popular app to come from another country. It's another thing when that country has complete control over business decisions, is your biggest global competitor, and is known to play dirty.

gr2zr4(10000) about 8 hours ago [-]

What still amazes me after 5 (?) years since Snowden's revelations it's how EU hasn't banned US social platforms yet.

FooBarWidget(10000) about 9 hours ago [-]

The Trump administration is stirring up China hate as a means of getting reelected. Anything goes. The national security concerns may or may not be factual, but Trump doesn't care about that one bit.

It baffles me that people cannot see this very obvious fact. Just proof that his strategy is working.

cwhiz(10000) about 3 hours ago [-]

China doesn't allow many western companies to operate in China. Why should the US allow Chinese companies to operate in the US??

This is an outrage that we should not allow. They steal our intellectual property, create state funded companies that are given monopoly access to their markets, and then unleash their stolen products on the world market. And if American or other western companies try to compete on the ground in China, they can't.

Would WeChat or TikTok exist if western chat and social media apps were given total access to Chinese markets??? Extreme doubt.

The eventual end result of bending over to China is a world where all of the goods, services, and software are owned by China. This imbalance that China has created is unworkable and cannot be allowed to continue.

snarf21(10000) about 4 hours ago [-]

Doesn't Tencent own interests in all of these game companies? Does mean that even things like Fortnite are illegal now?

WhyNotHugo(10000) about 7 hours ago [-]

What I also don't get is why they'd be so stupid. Just tax the companies locally and make a profit.

As long as they contribute to the local economy, provide a service, and operate within the bounds of the law, that should suffice.

Banning them is just worse for everyone.

krn(10000) about 9 hours ago [-]

> if the US creates a precedent for 'national security' in this way

It can be a real threat to national security[1]:

> Thursday's order alleges that TikTok 'automatically captures vast swaths of information from its users,' such as location data and browsing and search histories, which 'threatens to allow the Chinese Communist Party access to Americans' personal and proprietary information -- potentially allowing China to track the locations of Federal employees and contractors, build dossiers of personal information for blackmail, and conduct corporate espionage.'

It has nothing to do with other Western nations. It's about protecting your citizens from aggressive foreign states that operate based on completely different values.

I would consider going even further, and banning all Chinese researchers from the top US schools[2]:

> Seventy-one institutions, including many of the most prestigious medical schools in the United States, are now investigating 180 individual cases involving potential theft of intellectual property.

> Almost all of the incidents they uncovered and that are under investigation involve scientists of Chinese descent, including naturalized American citizens, allegedly stealing for China.

[1] https://www.cnn.com/2020/08/06/politics/trump-executive-orde...

[2] https://www.nytimes.com/2019/11/04/health/china-nih-scientis...

foobarbecue(10000) about 4 hours ago [-]

Let me explain: xenophobia. Trump is playing to his base.

president(10000) about 1 hour ago [-]

As you stated, this is about national security, which should be of the utmost urgency and concern. So you think saving the likes of Google, Facebook, Twitter is more important than saving the liberty of citizens and the lives of the overseas dissidents that are being endangered?

jariel(10000) about 4 hours ago [-]

a) Publishing is altogether a different situation.

b) Small nations can't feasibly have their own social app networks to any degree of scale.

c) The US is not China. For the most part FB is not a national security risk, whereas the Chinese apps could become that. China uses it's apps to observer and control every aspect of behaviour in China - the wherewithal, mans, intent etc..

d) This is tit for tat: China does not allow foreign social media in it's house.

I'm not so sure I agree with this, but it's not so outlandish.

hintymad(10000) about 9 hours ago [-]

Isn't this just reciprocity? China banned a long list of media companies: FB, Google, YouTube, Twitter, Netflix, and every traditional media. Now the US is banning two apps made in China.

puranjay(10000) about 9 hours ago [-]

What's stopping Volkswagen from lobbying Germany to ban Teslas because all the cameras in it pose a 'national security risk'?

This is, by all account, a shortsighted move

mark2996(10000) about 1 hour ago [-]

I really think its as simple as Trump hates TikTok because 95% of TikTok users are not Trump supporters. Every thing else is just excuses/rationalization for the ban (although some of the reasons may actually be good, they aren't what Trump cares about)

chvid(10000) about 7 hours ago [-]

You have posted the reason why this and the TikTok ban is most likely a bluff.

This (ordering companies such as Apple and Google to sever all business with TikTok and WeChat's owners and by extension removing them from all app stores) is a cannon that only can be fired once and will be so loud that it will fundamentally damage the centralised app store model.

This is similar to the muslim travel ban and will be overturned by some court and the Trump administration probably expect this. But does this anyway for the purpose of political communication.

thewarrior(10000) about 14 hours ago [-]

Some people here are wondering about the implications of this. What this means IMO is that all Chinese investments in SV need to be liquidated at fire sale prices in the next 45 days. Regardless of how people here feel about China this is a huge escalation. The US is inflicting huge losses on Chinese companies for no clear violation of US laws on their part. The pandoras box is now open.

Do countries get to do this to each other whenever they feel like it now ? Can China force Tesla to sell its Chinese operations because Teslas data gathering poses a national security risk ? China has some pretty serious means available to it for escalation. China can ban Boeing from China forcing the US taxpayer to incur serious losses in keeping the company afloat.

The whole thing is pretty stupid overall. Most people don't realize that during the 2008 crisis it was Chinas 500 billion dollar stimulus that kick started demand and pulled the world out of a depression. China and the US are interdependent and hold up the global system upon which global growth depends. If China slows down as a result of all this that reduces global growth. Pushing China to the wall can make them take extreme steps like undercut the entire dollar based financial order leading to mass instability. The US might come out victorious anyway but its not worth the risks. Not to mention a war which if it breaks out could lead to WW3.

Previous attempts to contain China were much more tactful with things like the TPP and the Iran deal. Right now the world is hurtling towards the abyss and most people here don't even realise it.

A meta point I'd like to add is that currently 10 % of the earths population in the 'Westosphere' controls 60 % of the worlds wealth. This is untenable in the long term and all this flailing about will not stop a reversion to a more balanced world. Its better that this happen gracefully than in a violent fashion.

pldr1234(10000) about 10 hours ago [-]

Quite a shallow post that essentially boils down to:

'We should not be this hard with China, let's keep doing what we were doing in the past.'

Your post history also looks very interesting, for anyone investigating state-actor social nets.

ezVoodoo(10000) about 12 hours ago [-]

WAL-MART, Starbuck, Ford, General Motors, GE, Microsoft, Apple, Tesla, UnitedHealth, HP, IBM, JPMorgan, Pepsi, CocaCola... all these companies are easy targets of China. But the Chinese government is not likely to make any move unless really cornered.

mlindner(10000) about 10 hours ago [-]

China has been doing this for a very long time to US companies. I'm not sure what pandora's box is being opened here. They just didn't selectively to two companies, they do it to all of them (other than Tesla who was the exception to the rule).

AimForTheBushes(10000) about 13 hours ago [-]

Anytime you do business with China your essentially doing business with the CCP.

We would all like China to be a positive part of the global community but the fact of the matter is you can't ignore their overall policy. A possible genocide against the ethnic Uighurs, aggressive expansion in the South China Sea, normalization of censorship and suppression, treatment of Hong Kong & Taiwan: all of which are a complete contrast of Western values.

Could China force Tesla to sell Chinese operations? Absolutely. I don't think any one would be surprised. Companies that enter the Chinese market (if they can in the first place) are subjected to getting a smaller piece of the pie from the start.

Facebook has relentlessly tried to tap the Chinese market but are blocked by the CCP. So why does TikTok get a free pass to the American market? Why does __any__ Chinese company get a pass?

I think we need to continuously evaluate our relationship with China.

aey(10000) about 13 hours ago [-]

Despite all this, US tech firms still have a harder time operating in China.

> A meta point I'd like to add is that currently 10 % of the earths population in the 'Westosphere' controls 60 % of the worlds wealth.

True wealth is really hard to measure. Stocks and dollars and euros are not wealth. They are mediums of exchange no different then dogecoin.

When I am hungry, I can't eat them directly, I can only trade them for food. And the amount of food I will get varies based on supply and demand.

fastball(10000) about 10 hours ago [-]

Wasn't China already doing stuff like this?

curiousgal(10000) about 10 hours ago [-]

> forcing the US taxpayer to incur serious losses in keeping the company afloat.

That would be a first now would it.

peteretep(10000) about 13 hours ago [-]

> Do countries get to do this to each other whenever they feel like it now

No, but wake me up when the US does this to another free and open country, rather than one that has a history of aggressively doing this to Western countries.

onetimemanytime(10000) about 9 hours ago [-]

>>it was Chinas 500 billion dollar stimulus that kick started demand and pulled the world out of a depression.

unlikely. That much doesn't move the needle, world economy right now is $140+ TRILLION. USA gave out over $700 Billion for that matter in 2008-2009 https://en.wikipedia.org/wiki/American_Recovery_and_Reinvest...

EU gave $200+ billion, plus each country did their own thing.

Ask FB or Google about their Chinese websites. How are they doing? This should have been done long time ago, tit for tat.

As for the rest, I doubt China can resort to stupid stuff. US and EU are not happy with them for a lot of things, including Covid, and China desperately needs them. China allowed manufacturing not to help us, but to gain taxes, employment and to steel IP.

kstenerud(10000) about 9 hours ago [-]

This is nothing new; it's geopolitics. The big boys have been doing this since the dawn of time.

Think of it like a playground: Everyone generally gets along, but the big kids can and sometimes do monopolize things they think are important at the time. The smaller kids make way for the bigger kids, and every now and then there's a fight between the bigger kids when they disagree strongly enough. They exchange words, jostle and block each other, or maybe it even comes to blows, and then things return to the status-quo for awhile.

That's how geopolitics work, except that it's usually a LOT more subtle or at least hidden from the public eye. Trump isn't a member of the political elite and is completely lacking in tact, so it's a lot harder to keep an appearance of calmness when he's in the room. Expect a much cleaner appearance of normalcy once he's out of office.

euix(10000) about 13 hours ago [-]

I think we are already passed the point of any of this mattering. Increasingly the rhetoric seems to me to just be: what matters isn't who is in the right or what is just or fair. Increasingly the only thing that matters is which side you are on. Are you with us or are you against us? It's just dogma at this point.

In this kind of environment people that are rational either stick to their tribe or learn to keep their mouths shut because it's no longer safe to express a contrarian opinion (just like in China). You see already see this in comments sections throughout western publications and forums where questioning policy or taking the opposite side means you are a bot/stooge/unpatriotic/50 cent warrior, etc, etc, etc.

keiferski(10000) about 13 hours ago [-]

Can China force Tesla to sell its Chinese operations because Teslas data gathering poses a national security risk ?

China can force any company out of the country for any arbitrary reason whatsoever. As far as I know, they are still banning Houston Rockets games because Morey supported Hong Kong in a tweet.

nujabe(10000) about 10 hours ago [-]

'China can ban Boeing from China...'

Not if they don't want an airline industry. China does not have the capacity to build a jet engine yet alone a commercial plane in service.

exprath35(10000) about 13 hours ago [-]

> Do countries get to do this to each other whenever they feel like it now ? Can China force Tesla to sell its Chinese operations because Teslas data gathering poses a national security risk ?

Yes? Are you unaware that China has been doing exactly this for many, many years?

Just as two examples, AWS was forced by China to sell off its Chinese operations to a Chinese company because of arbitrary national security reasons. Blizzard Entertainment was forced by China to go through a Chinese intermediary who controls and publishes all Blizzard games in China.

loceng(10000) about 1 hour ago [-]

I'd prefer we reference CCP instead of China to differentiate the Chinese people from their tyrannical rulers.

The laws in America not outright making it illegal for tyrannical organizations who operate concentrate camps and partake in genocide (Uighurs) to have access to our democratic societies and benefit economically doesn't mean it's not something you should stand for - and where the laws likely should exist for that.

Likewise, the CCP would likely just takeover Tesla's operations - maybe paying them, maybe not. The difference in the US, currently at least, is TikTok has options: 1) they don't sell and the ban comes, they don't receive any $x billions compensation, or 2) they understand a ban is likely in the US (and growing other democratic nations) and they make the smarter economic decision of selling for $x billions - maybe at fire sale pricing, however there are many competitors who could still compete and be capable of taking over operations, so perhaps not as low as otherwise it could be.

We're not hurtling towards the abyss, that's fear mongering; you may be right that SV startups may have to divest their China based investors, however that may or may not hurt them - and that would be the result of countering external costs to supporting and allowing CCP that we haven't been adequately accounting for for decades.

This is a strategic political-economic pressure decision to put pressure onto the CCP - notice that we're paying attention and won't stand for the level of abuse and tyrannical behaviours of the that we're becoming more and more aware of.

Economic pressure and preventing access to democratic societies for economic gain IS the graceful way to go about this, rather than allow CCP to gain more reach, access for propaganda for intelligence gathering and manipulation, to then spoil politicians further and become more entrenched in our societies economically and via investments in the most popular apps, etc.

And finally, the CCP isn't suicidal - their $500 billion stimulus you reference was for their own survival as well; they also wouldn't want this system they've been creating getting its fingers strongly into the rest of the world economy (which they depend on to maintain) to collapse, the collapse of which would mean things would shift greatly and more likely away from their own benefit.

Nitramp(10000) about 9 hours ago [-]

> A meta point I'd like to add is that currently 10 % of the earths population in the 'Westosphere' controls 60 % of the worlds wealth. This is untenable in the long term and all this flailing about will not stop a reversion to a more balanced world. Its better that this happen gracefully than in a violent fashion.

That has been the case since at least the start of colonialism, so something like 200 years now. It depends on what you mean by long term, but that is already a long time. (And it has been sort of non-violent since the de-colonization wars in the 60s).

rtx(10000) about 14 hours ago [-]

What are your views on IP theft by some companies there. Should that be considered first strike by China.

airstrike(10000) about 13 hours ago [-]

> Do countries get to do this to each other whenever they feel like it now ?

No, countries get to do this to each other whenever they feel like since the dawn of mankind. 'Now' isn't any different.

zkid18(10000) about 14 hours ago [-]

Looks like Trump had been promoted to the external M&A consultant for SV tech companies.

deevolution(10000) about 14 hours ago [-]

Are we witnessing the end of free markets? Or the growth of decentralization? These sorts of moves are making a very strong case for decentralized platforms.

peteretep(10000) about 13 hours ago [-]

More likely the end of one-sided free markets. There's no suggestion that the US is about to pull the same shit on countries that don't try and hobble US tech companies.

sozy777(10000) about 13 hours ago [-]

OK This is huge. Trump isn't just banning TikTok he's banning Tencent and it's holdings too from operating in the US. To put this is perspective Tencent owns Riot Games (League of Legends) and has stakes in Epic Games (Fortnite), Ubisoft (Assassin's Creed) and Activision Blizzard (Call of Duty). Overall Tencent owns 108 companies across media, entertainment, fintech and education. This is going to be a mess. (universal music and Spotify also owned partially)

Also how will impact companies of which Tencent owns less than 50%?

dicomdan(10000) about 13 hours ago [-]

White House has clarified that this will not affect the mentioned gaming companies and other holdings. Only WeChat itself.

Rebelgecko(10000) about 13 hours ago [-]

It's only WeChat. Reddit, Riot, Epic etc are unimpacted

threatofrain(10000) about 14 hours ago [-]

Rather than every nation deploying their own internal version of X, surely the sane endgame is open source software.

bmitc(10000) about 12 hours ago [-]

Ironically, Tencent is a major contributor to open source software.


ars(10000) about 14 hours ago [-]

Someone has to store the videos though.

iask(10000) about 14 hours ago [-]

Is DJI on the pending list?

rootsudo(10000) about 13 hours ago [-]

Time to buy a drone before they're sold out.

gzu(10000) about 14 hours ago [-]

So does this ban transactions with all of Tencent or just those in relation to WeChat? https://www.whitehouse.gov/presidential-actions/executive-or...

  Section 1.  (a)  The following actions shall be prohibited beginning 45 days after the date of this order, to the extent permitted under applicable law: any transaction that is related to WeChat by any person, or with respect to any property, subject to the jurisdiction of the United States, with Tencent Holdings Ltd. (a.k.a. Téngxùn Kònggǔ Yǒuxiàn Gōngsī), Shenzhen, China, or any subsidiary of that entity, as identified by the Secretary of Commerce (Secretary) under section 1(c) of this order.
Markoff(10000) about 12 hours ago [-]

considering the 'OR' I would assume it includes any transaction with Tencent Holdings Ltd. including their games, QQ etc.

obmelvin(10000) about 14 hours ago [-]

Similarly, and I honestly feel a bit silly asking this, but does this ban US investors from purchasing Tencent stock? It doesn't seem so - are these literally just bans on app transactions? Or does this also mean that, for example, WeChat couldn't host infra on Azure in Africa?

cjbprime(10000) about 14 hours ago [-]

And what's a 'transaction'?

neves(10000) about 14 hours ago [-]

You won't be able to play overwatch anymore.

Deimorz(10000) about 14 hours ago [-]

Did you edit in the quote from the EO after? It specifically says 'any transaction that is related to WeChat'. The TikTok/ByteDance order doesn't include a 'related to' qualifier like that, the same section in it just says:

    any transaction by any person, or with respect to any property, [...] with ByteDance Ltd.
It's still not clear what exactly that will mean, but the intention definitely seems to be to restrict it to WeChat and not hit everything from Tencent.
hnick(10000) about 13 hours ago [-]

They have since clarified, only WeChat apparently


jianshen(10000) about 13 hours ago [-]

What is his motivation for this right now? Is it just the distraction of the week or was there a specific incident recently that triggered this?

dicomdan(10000) about 12 hours ago [-]

One factor is that China has had a Firewall blocking all foreign digital competitors and information sources for a long time.

demadog(10000) about 12 hours ago [-]

There's likely a lot that is not and will never be covered in the news that the NSA and CIA know. I'm sure they look at our commentary on here, with 1% of the geopolitical information, and chuckle.

whateveracct(10000) about 14 hours ago [-]

Instagram Reels is launching at a 'convenient' time.. . . ..

verdverm(10000) about 13 hours ago [-]

I've had a number of tik tok-ers tell me they don't care what app it is. They care about the community and creators, and they will find each other on whatever replaces TT.

I was intrigued by this sentiment, the idea that the conceptual social network is the thing of value, not the application one interacts with it through.

sschueller(10000) about 10 hours ago [-]

Especially right after Zuckerberg gets grilled in congress for being anticompetitive.

Meanwhile the president threatens to ban facebook's biggest competitor...

bmitc(10000) about 12 hours ago [-]

I do wonder if there's a deeper connection than Facebook simply having this feature ready to go and releasing it early. Was there cooperation of the sorts 'you ban TikTok and we'll release Instagram Reels early to help distract upset users'?

cblconfederate(10000) about 6 hours ago [-]

AFAIK , tiktok's demographic (teenagers) is no way going to do the same things on facebook . It's probably going to be a new company (Franken-Vine maybe)

Historical Discussions: 1Password for Linux development preview (August 04, 2020: 602 points)
1Password for Linux Development Preview (August 04, 2020: 18 points)

(604) 1Password for Linux development preview

604 points 3 days ago by terabytest in 10000th position

discussions.agilebits.com | Estimated reading time – 3 minutes | comments | anchor

A full-featured Linux desktop app has been our most requested feature by far and responsible for the longest forum post in our history. Today we're thrilled to announce that 1Password is coming to Linux!

Planned for official release later this year, we couldn't wait to share the news with you so today we're unveiling an alpha a development preview so you can join in on the fun.

Not exactly an LTS release

This isn't a long-term support release! You can expect many updates and changes over the next few months as many features are not complete yet. For example, the app is currently read only: there is no item editing, creation of vaults, or item organization.

As such this initial release should be used for testing and validation purposes only and is not suitable for business critical environments. For a stable experience on Linux you'll want to use 1Password X in your browser.

A true Linux app

Our new app is built to meet the security and performance expectations of Linux users. Its backend is written completely in Rust, a secure systems programming language that has made a lot of waves in the Linux community. We're especially proud to be using the incredible to power the end-to-end encryption that keeps your data safe.

In addition to a secure foundation we've tailored the app to integrate with Desktop Linux with features like:

And this is just what's available in the preview. There will be many more exciting features that will take advantage of the power of Linux as we get closer to release.

Free accounts for open source teams

Our new app is built on great open source projects like the Rust programming language for the underlying logic, and React for a responsive component-based UI.

Building an app for Linux wouldn't have been possible without these giant shoulders to stand upon so we want to give back to the free software community. If you work on an open source team that needs a password manager, open a PR against our repo and we'll give you and everybody on your team a free account. You and your team will be able to continue using this account even when the development preview ends.

Thank you so much for your contributions and making the world a better place.

apt-get install 1password

If you're excited to report issues, work with us to resolve them, and update to verify fixes, then you're welcome to join us on this journey. To get started read the guide Get to know 1Password for Linux to find installation and troubleshooting instructions.

We maintain signed apt and rpm package repositories for Debian, Ubuntu, CentOS, Fedora, and Red Hat Enterprise Linux. We also have an AppImage for as-of-yet unsupported distributions. Let us know what distribution you use and how well 1Password works there.

We hope you are as excited about a 1Password Linux app as we are. Our development team will be active in this forum so when you find issues please report them here.

All Comments: [-] | anchor

daffy(10000) 3 days ago [-]

Why would I want to trust some company with all my passwords?

1123581321(10000) 3 days ago [-]

Generally because the chance they serve a compromised client is lower than that your homegrown storage will be compromised, and because the UI affordances make it easier to use more unique passwords and multi-step authentication.

dastx(10000) 3 days ago [-]

Ooh boy. This is amazing. The horrible Linux support they had was the reason I left 1Password. This might make me go back to 1Password.

igravious(10000) 3 days ago [-]

Don't they have browser plugins? That's how use Lastpass on Ubuntu. I mean, besides Electron apps, who actually makes native Linux apps? (joking, not joking)

dteare(10000) 3 days ago [-]

So sorry we scared you away! Linux has been on our radar for a while but our biggest challenge was finding a way to share code between apps without writing everything in C++. Thankfully we found Rust, a systems language built around efficiency and safety. From there we were able to build the common core we've always dreamed of and were off to the races. Please give us another chance and let us know what you think. <3

m0zg(10000) 3 days ago [-]

BitKeeper is perfectly adequate across Windows, Linux and iOS. It's also free.

m0zg(10000) 2 days ago [-]

That is, BitWarden

randomstring(10000) 3 days ago [-]

I see support for Debian and Ubuntu (https://support.1password.com/cs/getting-started-linux/), but I don't see anything about Raspbian or running on the Rasberry Pi. Hopefully Pi support isn't far behind.

bwoodruff(10000) 3 days ago [-]

We don't currently support ARM but we'll keep an eye out for feedback on this. Thanks! - Ben, 1Password

a-ungurianu(10000) 2 days ago [-]

Unrelated to the heading of this post, but looking at the screenshot it seems that it supports OTP generation within the app.

Having one place that both stores your password and generates the OTP seems to defeat the purpose of 2FA no?

pseudalopex(10000) 2 days ago [-]

It's better than not using 2FA.

bwoodruff(10000) 2 days ago [-]

Yes and no. A much better explanation than I can give here is covered in our blog post: https://blog.1password.com/totp-for-1password-users/ - Ben, 1Password

jinx1975(10000) 3 days ago [-]
the_svd_doctor(10000) 3 days ago [-]

How does this work if you have to change your password for some reason ? Do you change the 'What for?' field ?

musicale(10000) 3 days ago [-]

I pulled the plug on 1password because I hate subscriptions.

There is also no logical reason to pay agile bits for single-purpose back-end infrastructure when we already have dropbox, etc.. An encrypted password file is tiny.

Subscription apps (and subscriptions in general) are simply not scalable in their current implementation.

nucleardog(10000) 3 days ago [-]

I don't even mind subscriptions but they've still lost me as a customer for bundling the subscription pricing along with an extremely hard push into storing all my most sensitive data "in the cloud".

Still rocking my 1Password 4 license on Windows and OSX from years ago with no plans on upgrading. When I'm finally forced to, I'll simply switch to another product.

laactech(10000) 3 days ago [-]

I switched from LastPass years ago, and 1Password is great. I use a family account, and it's been easy to get my wife to use randomly generated passwords.

The only missing piece for me is a native Linux app since I use Ubuntu for all my development environments. The web browser extension works, but it's a noticeable difference moving between it and the windows desktop app. I'm super excited to give this a try.

dteare(10000) 3 days ago [-]

Awesome! I'm so happy to hear this. We'd love to hear about your experiences so please share after you give it a go.

c0nsumer(10000) 3 days ago [-]

After putting it off for a while, I finally moved from LastPass to 1Password around the beginning of the year.

Inertia kept me with LastPass, but I really should have fought that. There's just no comparison between the pile of slow crap that is LastPass and it. I guess I just accepted that a password filler would be clunky, but... no. 1Password is much, much better.

jychang(10000) 2 days ago [-]

Which platform is Lastpass slow at?

Just curious, since I don't have issues with Lastpass, but I mostly use the chrome extension.

sickcodebruh(10000) 3 days ago [-]

I use 1PW professionally, with our whole company having shared vaults for different departments and security levels, as well as at home, so my wife and I can share some credentials. Absolutely love it. I'm spending more and more time working from Ubuntu so I'm very happy to see this, it should make things just a bit easier.

bwoodruff(10000) 3 days ago [-]

Awesome! We'd be happy to hear any feedback once you start using it. :) - Ben, 1Password

techntoke(10000) 3 days ago [-]

Enpass is a really nice password app that works on Linux, and allows you to sync with various cloud providers. I have since switched to Password Store though which is open source and uses GPG for encryption, so you can pick your own algorithms.

princevegeta89(10000) 3 days ago [-]

Yup, have been using Enpass exactly for the reason that there has never been a 1Password for Linux until now.

Enpass always worked seamlessly, and it was a one-time purchase for me back then. (They changed it to a subscription model a year ago)

pseudalopex(10000) 3 days ago [-]

Enpass isn't open source and the only security audit skipped Linux.[1]

[1] https://dl.enpass.io/docs/EnpassSecurityAssessmentReport.pdf

ceocoder(10000) 3 days ago [-]

Unfortunately this only works with hosted 1Password (as far as I can tell), there doesn't seem to be any support for self hosted vaults. Can Roustem or anyone else from 1Password team clarify this?

This was the precise reason I switched to BitWarden 6 months ago, needed a solution where my passwords didn't leave my network.

mdaniel(10000) 2 days ago [-]

Depending on your use case, KeePassXC supports reading local vaults, but currently just reading them because I didn't have the need to try and round-trip the vaults for my on-call laptop.

I don't believe it would be an overwhelming amount of work to implement the write portion (err, aside from getting a security review) but I do seriously doubt that KeePassXC would accept the PR to change the backing store, meaning it would have to be a fork :-(

dteare(10000) 3 days ago [-]

I'm no Roustem but I'm close. :)

You're right, 1Password for Linux integrates tightly with the 1password.com service and as such does not support local vaults.

dahfizz(10000) 3 days ago [-]

Seriously, this is great to see. I've been using LastPass for a while but have been unhappy with it. If 1Password is taking Linux support seriously, I'll definitely switch.

beyer(10000) 2 days ago [-]

I assure you, we are taking Linux support seriously. When I joined the team, we had almost no one using Linux in house. This has changed in a big way and is only part of why we are now investing in making the best darn app we can for Linux.

Proven(10000) 3 days ago [-]

LastPass for Linux works without special issues.

neilv(10000) 3 days ago [-]

Is this open source? That page makes no mention of it.

When my shop standardizes on a password manager program/browserplugin for Linux development workstations, I currently don't see us picking a closed source one.

daffy(10000) 3 days ago [-]


bwoodruff(10000) 2 days ago [-]

The app itself is not open source, though we have built on a number of open source technologies. Arguably the most critical component is open source, and that is the ring encryption library.

Thorentis(10000) 3 days ago [-]

KeepassXC is the perfect solution in my opinion. It is open source, has a huge number of features (that don't get in the way of basic usage), and has mobile apps and desktop apps that work well on all platforms. Right now I am using it on Windows, Mac, AND Linux, as well as my Android phone. I have it syncing over Dropbox, but you can sync it however you like. The Android app automatically fetches the latest version, and supports auto-complete etc. I see no reason to pay for a password manager or use something that isn't open source.

below43(10000) 3 days ago [-]

I second this. KeePass is awesome (across all platforms)

octorian(10000) 3 days ago [-]

I'm the same way. The most important parts of the KeePass ecosystem to me are:

1. It runs on every platform I currently use, as well as any platform I might care to use, whether or not that platform is sufficiently 'popular' for a company to justify caring about it.

2. It isn't dependent on the continued healthy existence of a company to remain usable, as I could simply self-maintain in a worst case scenario.

These are very important things about a password manager to me, personally, which is why any of these more polished/popular options would be an extremely tough sell.

noisy_boy(10000) 2 days ago [-]

Doing the same using Syncthing for syncing. For basic password management across devices without having to go to 'cloud', I don't see a better alternative. The new polished UI for KeepassXC on Linux is a bonus.

xoa(10000) 3 days ago [-]

Nice to see progress here, though 1P continuing to move away from local control to force subscriptions is regrettable. Even so, the UI hasn't been matched yet IMO, which is important for getting the less technical to use it. We're sadly also still a ways away from passwords being eliminated entirely, so it's still very important to get everyone using one.

One thing still missing I really hope to see though is the local application (on all platforms) supporting hardware tokens for unlock (with a backup master option). That'd be a nice extra security+convenience option which would work across platforms.

ghostpepper(10000) 3 days ago [-]

For many/most types of software I am in the same camp of people who would prefer to pay more upfront for a license as long as the software continues working as is - I bought it because it worked and if I chose to pay more in the future for a better version, I will make that decision based on the new features added and not the old features being held hostage.

However, for something as high-value as a password manager, I think having a subscription model makes a lot of sense. I can't think of any other class of product where timely updates from the developers are so critical to the utility of the product. You could even argue that an unpatched, out of date password manager is worse than no password manager.

shmerl(10000) 3 days ago [-]

How does it compare to KeePassXC?

asdajsdh(10000) 3 days ago [-]

It's proprietary, non-free and all your passwords get send out into the wild. So it has all the things for a modern app.

comice(10000) 3 days ago [-]

I just moved from (paid) 1p to bitwarden at the weekend due to lack of proper Linux support. I was just testing bitwarden and found I couldn't easily get a good export of my passwords from 1p on Linux, because only their desktop apps support that. It won't run under wine and I ended up installing a Windows VM specifically to do the export.

Was so frustrated at this it pushed me to move to bitwarden. Good for them for sorting it though.

ben0x539(10000) 3 days ago [-]

... I wish I'd thought of booting into my Windows partition and installing 1password there, instead of spending an evening writing up an extremely overwrought export script on top of the commandline client.

prgmatic(10000) 3 days ago [-]

I just did the same over the weekend. Really loving BitWarden, it works and it's fast. It did take me a bit of time to export out, scrub & format CSV, then import to BitWarden.

pandatigox(10000) 3 days ago [-]

If you used Firefox or Chrome, then you could use 1Password X for Linux systems. But I'm guessing from the (paid) part that you weren't using their sync?

iuguy(10000) 2 days ago [-]

I was a 1Password user from when it was fully self-hosted until they started pulling bait and switch tactics to move people to subscriptions and online vaults[1]. I also had Windows licences, and Windows was certainly a 2nd class citizen while I used 1Password. And of course, it doesn't look like the 1Password Linux client is open source. The back-end certainly isn't.

I switched to Keepass[2] initially, synchronised with NextCloud but it wasn't intuitive enough for everyone. We moved everyone to Bitwarden a few years ago using bitwarden_rs[3] and have never looked back.

[1] - https://medium.com/@kennwhite/who-moved-my-cheese-1password-...

[2] - https://keepassxc.org/

[3] - https://github.com/dani-garcia/bitwarden_rs

rb666(10000) 2 days ago [-]

I'm also a big fan of Bitwarden. Have tested 1Password, LastPass and a few of the other password managers over the last 5 years. This is the one that ticks all the boxes, has the least bloat, does correct matching. I support it with pleasure!

faebi(10000) 2 days ago [-]

It's why I switched to Keeweb, it's keepass compatible, works anywhere and has desktop apps too.

atoav(10000) 2 days ago [-]

Using keepass (synced via nextcloud) for ages now, what would be the arguments for bitwarden?

meerita(10000) 2 days ago [-]

I believe you can still self-host if you want. I don't understand how can you be disturbed by offering a cloud service to host your passwords could be bad to switch. I'm happy sub of 1Password and i self-hosted my passwords before for a long time.

dgellow(10000) 2 days ago [-]

FYI, the Windows application is now at the level of the macOS version. I was also annoyed by the move to a subscription model and waited a while before accepting to switch. But I'm now quite happy paying yearly given how well they improved their cross-platform support.

mapgrep(10000) 2 days ago [-]

This comment is like a canonical example of why a company would choose not to develop for Linux. 1Password puts real resources and risk into supporting a platform that may not pay off, but it's Not Good Enough for much of the community because:

-The client is not open source

-the backend is not open source

-it's not "a first class citizen" right away (the Windows port is by all accounts improving)

I'm not trying to put down your comment but to point out that when you have a fragmented platform that is difficult to develop for plus a community that is often hostile to closed source or less than perfect feature parity you are going to be relatively deprived of commercial offerings. This is why we see less Linux support broadly. Not that you personally should change your opinions.

tikiman163(10000) 2 days ago [-]

You should consider LastPass. They've had considerably fewer security breaches than KeePass and is accessible either using a local app or from any modern browser regardless of OS. It even has a smart phone app, although the phone app requires a subscription.

lmedinas(10000) 2 days ago [-]

Same issue for me, having online vaults was the killer for me. Also the subscription made me really think if the service was worth it after all, specially after buying the apps they are asking me more money. I moved to Keepass (and opensource apps) and im fairly happy with it.

I dont think i will go back to 1Password.

trevor-e(10000) 2 days ago [-]

Yep, super annoying for me as well. I paid $60 which I feel is quite a lot for a password manager. When I built my new computer the other weekend I found everything on 1Password's site was subscription-walled, I finally found a download via a hidden Google backlink. But, of course, since I don't have a subscription it's now in read-only mode... feels really scummy of them.

m12k(10000) 2 days ago [-]

I'm still using 1Password self-hosted - works fine.

panpanna(10000) 2 days ago [-]

I think keepass has the base functions covered, they just need a slightly better UI and a simpler way to sync passwords.

Thankfully, it's open source so I'm sure someone with fix that soon ;)

Valkhyr(10000) 3 days ago [-]

As many others have said, I'll be interested once they provide local vault support on Linux.

I dislike subscriptions - not for the financial cost as such, but because I like to evaluate whether or not I want to pay for a given version or stay on the current version. I'm happy to pay for software that provides value to me, which 1Password does (and I did pay for the existing clients). The same applies to major version updates when they add value for me - though the reality is that my usage is very basic, and I am often happy with an older version of the same software for years, so subscriptions to support continued feature development feels like an unjustified lock-in to me.

I do subscribe to some services that have a significant backend/cloud-based component, but in the case of 1Password, I sync the vault via Dropbox, so a subscription instead of licence/upgrade based pricing feels completely inappropriate.

Since I am trying to move more of my computing to Linux, it looks like at some point I'll have to look for other options than 1Password, which is a shame :-(

guiambros(10000) 3 days ago [-]

I was on the same boat until recently. Long time 1Password user under Linux + local sync (since v3, 10+ years ago). Always feeling neglected by AgileBits.

Last year I got tired of having to fidget with WineHQ config every time I updated something, and decided to pony up for the cloud-based subscription.

It was the best decision ever.

Not only solves the compatibility issues (obviously), but also gave me the ability of managing different vaults, selectively share passwords within within the family, and also having some nice additional features (e.g., wiping out devices before intl travel).

All things considered, more than worth the subscription price.

The only two things that I miss from the native version:

1) ability to attach files to an entry

2) the flexibility of doing bulk operations (e.g., selecting multiple entries).

I solved the latter running 1P under a Windows VM, but hoping this Linux native version will solve now. 1 down, 1 to go.

hanklazard(10000) 3 days ago [-]


I left 1PW a few years ago as I felt them pushing towards a subscription model. I've tried a bunch of other open-source options, this one is best. Gives you a nice self-hosted bitwarden install without the overhead (in particular .log bloat) of the main bitwarden repo. You also get 2FA which I feel is essential.

nodesocket(10000) 3 days ago [-]

1Password has probably been the single biggest productivity increase tool I've ever purchased. Worth every cent, and new Linux support news is fantastic.

beyer(10000) 2 days ago [-]

Thank you so much for your support! This is only the beginning for our Linux client, so stay tuned for a lot more excitement!

rawoke083600(10000) 2 days ago [-]

I have been looking into a good passwd-manager for linux and my android phone for the last week or so: Most of them look so 'blingy' I can't get myself to trust it ! I'm 51% convinced myself just to code my own (I know , I know)

I've recently switched over from Ubuntu-Genome to Ubuntu-i3 and now for the life of me, I can't get chrome to sync/autofill/remember my passwords :( - Same user just different desktop/wm, why is chrome so good and so bad ! Anyone have any ideas how to fix chrome or can recommended a 'unix-spirit(do one thing etc...)' password-manager for Linux+Android+Chrome ?

0XAFFE(10000) 2 days ago [-]

I would go for bitwarden[1]. It's free software, can be self hosted and is available on most platforms and has nice browser addons. If you want to go the self hosted route, you might want to look into bitwarden_rs[2] because it uses way less resources than the official server.

[1] https://bitwarden.com/

[2] https://github.com/dani-garcia/bitwarden_rs

maxhille(10000) 2 days ago [-]


Using it for years now and am happy with it. Although it start from this pretty basic setup, you can extend your usage with mobile apps and browser extensions.

ryanmccullagh(10000) 3 days ago [-]

How does 1Password compare to LastPass?

For one, LastPass _always_ freezes Chrome on me a few times a day.

rootusrootus(10000) 3 days ago [-]

1Password is pretty stable, I have not had any problems with applications freezing.

I was going to switch to LastPass since I have two younger children that I'd like to use a password manager, but LastPass has the same 17+ restriction so I'm looking for other options. But if you don't have the same issue, 1Password is a solid choice.

emdowling(10000) 3 days ago [-]

I've been using 1Password every day for over 11 years now. The oldest passwords I've got stored are for Twitter and Dropbox (yes, the passwords have been changed but the records were first created in 2009).

It's one of those apps which has been made with proper craftsmanship and care, so while I'm not a Linux user, I'd have no problem recommending based solely on Agilebit's reputation.

burnte(10000) 3 days ago [-]

I used KeePass, then LastPAss, then tried 1Password about 8 years ago. I haven't even considered changing. I joined when they were still mostly focused on MacOS and iOS, the Windows and Android apps were secondary. Since then they really shifted to a totally cross platform experience, and I'm incredibly happy with the app. I'm glad they're branching out to Linux.

bashinator(10000) 2 days ago [-]

Yup. Password management is one of those things where I want to pick the best possible solution, over the 80% good for 20% of the cost. The risks of losing credentials are real, and terrible. Making shit easy for non-technical people is a real-world risk reduction. Making shit easy for technical people is also a real-world risk reduction, and letting me put 1P into automated workflows is great. If there's minor encroachment on territory currently held by Hashicorp Vault, then 'Go 1P!' - I love competition between two genuinely good products.

foobiekr(10000) 3 days ago [-]

I am a 1password user, and have bene for about the same amount of time, but I've been slowly looking for an alternative.

Unless I'm mistaken, 1Password no longer ephemerally decrypts passwords as needed and only while used and then scrubs the memory. [1, old but still] The excuse, if I remember it, was that garbage collected languages made this challenging. Even so, there is some irony in them moving away from the temporary, one-at-a-time, scrubbed approach just before all of the side channel attacks that allowed leaking memory across processes became widespread.

[1] https://nakedsecurity.sophos.com/2019/02/21/password-manager...

Scarbutt(10000) 3 days ago [-]

But why a whole electron app just to store passwords?

numbers(10000) 3 days ago [-]

Used Dashlane for 2-3 years and then tried 1Password and I haven't looked back. Dashlane has too many bugs to be useful all the time.

pjmlp(10000) 2 days ago [-]

Apparently that craftsmanship went astray with the adoption of Electron.

disposedtrolley(10000) 2 days ago [-]

I just checked my vault out of curiosity, and my first entry from 2009 is the credit card I used to purchase a 1Password licence shortly after!

It's robust software that does was it says on the box. I was initially reluctant to move out of my local vault but the online service has been impeccable.

karlshea(10000) 3 days ago [-]

It's made with proper craftsmanship and care on the Mac (which is primarily where I've been using it for years).

The Windows client is much better after the last major release, but it's never been as slick as the Mac version (the biggest wart now is the system tray/browser extension popup).

1Password X looks nice until you try and use it, and all the company reps on the forums are very argumentative about any feature request (look for the pushback they give about resizing their super-cramped browser extension popup—and the issues with hires screens stemming from how they built it, which assumes a fixed size).

I've also got a chip on my shoulder about the 'feature' they added that showed the most recently used websites in the iOS app with no way to disable it (they finally allowed setting the number to zero months later). The reps on their forums all come off with this attitude of 'this is the best way, and you're wrong if you don't like it' for just about every issue that comes up.

I like the app and will continue to use it, but if my main platform wasn't macOS/iOS I would have bailed long ago.

mkskm(10000) 3 days ago [-]

> It's one of those apps which has been made with proper craftsmanship and care

Is it? I've been using it for sometime as well but it seems like there is a lot of room for improvement. E.g:

- Support for unlocking via Watch ID on the Mac.

- Currently on iOS when searching for a password within an app, if a site prefix is included that doesn't match what's in 1Password the list will just show no results, with no way to navigate manually to the login. Instead, you have to close the app, open 1Password, and copy/paste the credentials back in. Typically the master password will have to be re-entered as well, despite touch ID being adequate a moment prior. Since it's rare to sign up via the web now for mobile apps, this is the most common scenario for me when using 1Password for apps on my phone (and occasionally websites as well).

- Improved UI/UX on mobile. Dashlane is way better in this regard. 1Password overemphasizes features I don't need like tags and favorites and has a pretty cluttered look in general.

I like the native Mac app and open/local vault format. (Dashlane by contrast has a very buggy desktop app and requires storing everything on their servers.) But I would jump at the chance to use an alternative with a simpler UI and better experience on mobile.

bredren(10000) 3 days ago [-]

Also a longtime user. Did you kick over to their subscription model or have you stuck with the old installs attached to the grandfathered permanent license?

Polylactic_acid(10000) 3 days ago [-]

What are peoples thoughts about firefox lockwise? I have been using keepass for forever but I want something that easily syncs between devices and doesn't require copy/paste in to the browser. Lockwise seems perfect but I haven't used it much yet.

felbane(10000) 3 days ago [-]

I've found great success with KeePass and Dropbox. I use the Kee plugin in Firefox that allows autofill in the browser, and KeePass2Android on my phone which supports pluggable autofill for browsers and apps on Android.

It's not as trivial to set up as Lockwise or 1Password, but I prefer this setup because:

- 100% Open Source - I own the keyring and can sync it across literally all of my machines, plus the cloud storage provider(s) of my choice seamlessly - The keyring is protected by a key that only I know, no third party is handling the unlock on my behalf

lorenzhs(10000) 2 days ago [-]

I use Lockwise, and the thing that drove me towards it is that it works on Linux and iOS without having to run a sync service (Dropbox/...) on my laptop. The sync is excellent and just works. The iOS app could use some work, though. It doesn't support adding or editing passwords (but Firefox for iOS does, and it's planned for Lockwise according to the issue tracker), and it's REALLY slow to start (several seconds) when using it to fill a password in the browser. I don't know what's up with that, but it's annoying.

wincy(10000) 3 days ago [-]

Well one difference is you just login with a password and you have access to all your passwords. Whereas with 1Password you use the long key to set up a new device, so it requires either memorizing that key, writing it down, or having a different device with that key on it.

alexandrerond(10000) 3 days ago [-]

I switched to 'pass' from 1p and it is a breeze because it just works without all the bullshit and I don't have to place any trust on a company saying they do things right (they will never tell otherwise).

And 1password never cared about Linux. I had to custom-script data export, they pretty much held data hostage by making it difficult to migrate from the platform, not to speak of the undocumented data formats. But at least we did not have to install some closed source propietary thing to do something as critical as password management (browser sandboxing seems slightly better). If they cared they would open up their client's code for everyone to peek.

no_wizard(10000) 3 days ago [-]

I have to call this out as a bit of a hyperbole.

They already participate, quite openly, in security audits[0], and while yes, I'd love it if it was OSS too, but the reality of making money on these services is that (especially I believe at the time 1Password was founded), is it wouldn't have likely done them any good, really. In fact it could hurt their business. I believe 1Password was one of (but not the only!) pioneers of this being a successful consumer business.

Notably, I don't think its worth detracting from a fantastic product based solely on the license of its underlying software. I'm also not aware of any 1Password data breaches.

As far as exporting goes, you can simply generate a CSV file (or plain txt)[1] as well. Not sure what the issue there was, I'd be curious to know.

While I like OSS too, and prefer it when able, I think its a stretch to say they're holding their users hostage if they want to migrate away, not to mention being OSS isn't really a predicate as to whether the software & user experience is actually any good.

disclaimer: I don't work for 1Password, but I've used it for over a decade.



e8gy3(10000) 3 days ago [-]

I must be easily excitable because this makes me very excited.

beyer(10000) 2 days ago [-]

Me too! There will be a heck of a lot more excitement over the coming months now that our Rust client app team is firing on all cylinders.

Hnrobert42(10000) 2 days ago [-]

I don't understand all the subscription hate. Don't get me wrong. I, too, long for the days of DIY car repair and have a hard drive full of mp3s ripped from CDs (snagged from a Columbia House subscription, no less).

But this is just how ~the software~ business works now. Investors want predictable ROI. If your business can't/won't show *aaS revenue predictability, investors will take their money somewhere that can.

ecmascript(10000) 2 days ago [-]

Well just because that is how a lot of software gets deployed doesn't mean it sucks any less.

I hate subscriptions because the costs adds up very quickly. I try to save more than 50% of my income so having several subs is just not happening.

I'd rather not have any software at all or much crappier alternatives of it rather than paying every month.

29athrowaway(10000) 3 days ago [-]

Shower thought: An online password manager is equivalent to password reuse, because there's one password behind all your passwords.

doctoboggan(10000) 3 days ago [-]

Not really because the most common reason your password is leaked is some online service leaked it, not someone hacked your machine and found it.

In that scenario they would only have your single site password and not your master key.

eberkund(10000) 3 days ago [-]

Is it Electron based? I noticed the blog post said they used React to build the UI.

bwoodruff(10000) 3 days ago [-]

It is a hybrid. The UI is using Electron. The guts are Rust. - Ben, 1Password

colordrops(10000) 3 days ago [-]

I've been using Bitwarden across the browser, Linux, Mac, and android, and it works great, and is fully open source, unlike 1Password.

dexterdog(10000) 3 days ago [-]

And you can self-host it which I do

brandon272(10000) 2 days ago [-]

1Password is a great app but I can't justify spending $120 CAD per year on their family plan. (I know it's slightly less expensive to pay a year upfront, which I'd rather not do)

beyer(10000) 2 days ago [-]

I'm not sure where you're getting this price, is that your calculation with GST/HST?

Plans on https://1Password.ca are CA $7.95/month (CA ~$96 for the year) for monthly billing. We think this is a killer price for five family members without any restrictions like living at the same residence. You do save quite a bit with annual billing, but I understand that's not for everyone.

cycomanic(10000) 3 days ago [-]

I've been using keepassxc which is open source, extremely snappy (it's one of the fastest starting gui apps on my desktop) and offers all the functionality I think I need (ssh agent integration, secret service integration...). What does 1password offer that would make me switch? Why is everyone so excited about 1password and keepassxc is hardly ever mentioned?

benhurmarcel(10000) 2 days ago [-]

> What does 1password offer that would make me switch?

Mostly great sync across devices, and great apps on all devices (all mobile, all browsers, and all desktop OS except Linux for now).

If you mainly use one computer and don't mind tools which are a bit less polished, it's not that compelling.

xtat(10000) 2 days ago [-]

was onboard until i found its a gui

benhurmarcel(10000) 2 days ago [-]

There is already a command line tool available.

aborsy(10000) 3 days ago [-]

What would 1Password add to keepassxc+Dropbox for personal use? It only subtracts from the security.

I get that organizations with many users don't want to deal with database management. For personal use though, keepassxc is fine.

pseudalopex(10000) 3 days ago [-]

Better UI.

ngrilly(10000) 2 days ago [-]

Disaster recovery is simpler with 1P (if you lose your computer and your phone). You just need your 1P credentials instead of your Dropbox credentials and KeePass secret key.

dddw(10000) 3 days ago [-]

Congratulations! Is there als a CLI?

monadic2(10000) 3 days ago [-]

There is already a CLI client for 1password on linux!

bluk(10000) 3 days ago [-]

Yes, https://support.1password.com/command-line-getting-started/ for various platforms/architectures.

kanobo(10000) 3 days ago [-]

Is this better than just using KeepPassXC with a simple kbdx file synced to an online drive? It's an honest question, I've never used 1Password before.

the_snooze(10000) 3 days ago [-]

I switched from KeePass to 1Password mainly to get my family on it. They really appreciated the clean workflow once they got used to it after a couple days. And the auto-sync feature combined with the mobile app has been useful for when they have to enter the Netflix password on various devices.

sedatk(10000) 3 days ago [-]

Yes as it isn't subject to accidental overwrites of the data file It also has useful integrations like browser extensions so can autofill. Not sure if KeePass is capable of that. It's a paid app though.

abrowne(10000) 3 days ago [-]

I used older version of 1Password on Mac and use KeePassXC on Ubuntu now. 1P is definitely slicker, and presumably the subscription comes with support. For me, not a worthwhile trade-off, but could be worth if for some people, especially those with many devices to keep in sync and business users.

Historical Discussions: Scientists rename human genes to stop MS Excel from misreading them as dates (August 06, 2020: 583 points)

(585) Scientists rename human genes to stop MS Excel from misreading them as dates

585 points 1 day ago by virde in 10000th position

www.theverge.com | Estimated reading time – 7 minutes | comments | anchor

There are tens of thousands of genes in the human genome: minuscule twists of DNA and RNA that combine to express all of the traits and characteristics that make each of us unique. Each gene is given a name and alphanumeric code, known as a symbol, which scientists use to coordinate research. But over the past year or so, some 27 human genes have been renamed, all because Microsoft Excel kept misreading their symbols as dates.

The problem isn't as unexpected as it first sounds. Excel is a behemoth in the spreadsheet world and is regularly used by scientists to track their work and even conduct clinical trials. But its default settings were designed with more mundane applications in mind, so when a user inputs a gene's alphanumeric symbol into a spreadsheet, like MARCH1 — short for "Membrane Associated Ring-CH-Type Finger 1" — Excel converts that into a date: 1-Mar.

Studies found a fifth of genetic data in papers was affected by Excel errors

This is extremely frustrating, even dangerous, corrupting data that scientists have to sort through by hand to restore. It's also surprisingly widespread and affects even peer-reviewed scientific work. One study from 2016 examined genetic data shared alongside 3,597 published papers and found that roughly one-fifth had been affected by Excel errors.

"It's really, really annoying," Dezső Módos, a systems biologist at the Quadram Institute in the UK, told The Verge. Módos, whose job involves analyzing freshly sequenced genetic data, says Excel errors happen all the time, simply because the software is often the first thing to hand when scientists process numerical data. "It's a widespread tool and if you are a bit computationally illiterate you will use it," he says. "During my PhD studies I did as well!"

Examples of gene symbols being rendered as dates in Microsoft Excel.
GIF: The Verge

There's no easy fix, either. Excel doesn't offer the option to turn off this auto-formatting, and the only way to avoid it is to change the data type for individual columns. Even then, a scientist might fix their data but export it as a CSV file without saving the formatting. Or, another scientist might load the data without the correct formatting, changing gene symbols back into dates. The end result is that while knowledgeable Excel users can avoid this problem, it's easy for mistakes to be introduced.

Help has arrived, though, in the form of the scientific body in charge of standardizing the names of genes, the HUGO Gene Nomenclature Committee, or HGNC. This week, the HGNC published new guidelines for gene naming, including for "symbols that affect data handling and retrieval." From now on, they say, human genes and the proteins they expressed will be named with one eye on Excel's auto-formatting. That means the symbol MARCH1 has now become MARCHF1, while SEPT1 has become SEPTIN1, and so on. A record of old symbols and names will be stored by HGNC to avoid confusion in the future.

So far, the names of some 27 genes have been changed like this over the past year, Elspeth Bruford, the coordinator of HGNC, tells The Verge, but the guidelines themselves weren't formally announced until this week. "We consulted the respective research communities to discuss the proposed updates, and we also notified researchers who had published on these genes specifically when the changes were being put into effect," says Bruford.

As Bruford makes clear, the art of naming genes is very much driven by consensus. Like the lexicographers charged with updating dictionaries, the Gene Nomenclature Committee has to be sensitive to the needs of those individuals who will be most affected by their work.

This wasn't always the case, mind. In the early, frontier days of genetics, gene naming was often a playground for creative scientists, leading to notorious genes like "sonic hedgehog" (yes, named for that Sonic) and "Indy" (short for "I'm not dead yet"; a reference to the gene's function, which can double the life span of fruit flies when mutated).

Now, though, the HGNC has taken matters firmly in hand, and current guidelines don't cede much ground to whimsy or ego. The focus is on practical concerns: how do we minimize confusion? For that reason, gene symbols should be unique, and gene names should be brief and specific, says the committee. They cannot use subscript or superscript; can only contain Latin letters and Arabic numerals; and should not spell out names or words, particularly offensive ones (a rule that should hold true "ideally in any language").

Gene names should avoid offense "ideally in any language"

And while the decision to rename genes is not taken lightly, it's not unusual, says Bruford. Many gene symbols that can be read as nouns have been renamed to avoid false positives during searches, for example. In the past, CARS has become CARS1, WARS changed to WARS1, and MARS tweaked to MARS1. Other changes have been made to avoid insult.

"We always have to imagine a clinician having to explain to a parent that their child has a mutation in a particular gene," says Bruford. "For example, HECA used to have the gene name 'headcase homolog (Drosophila),' named after the equivalent gene in fruit fly, but we changed it to 'hdc homolog, cell cycle regulator' to avoid potential offense."

But Bruford says this is the first time that the guidelines have been rewritten specifically to counter the problems caused by software. So far, the reactions seem to be extremely positive — some would even say joyous.

After geneticist Janna Hutz shared the relevant section of HGNC's new guidelines on Twitter, the response from the community was jubilant. "THRILLED by this announcement by the Human Gene Nomenclature Committee," tweeted Hutz herself. "Finally!!!" responded Mudra Hegde, a computational biologist at the Broad Institute in Massachusetts. "Greatest news of the day!" said a pseudonymous Twitter user.

Why did Microsoft win in a fight against human genetics?

Bruford notes that there has been some dissent about the decision, but it mostly seems to be focused on a single question: why was it easier to rename human genes than it was to change how Excel works? Why, exactly, in a fight between Microsoft and the entire genetics community, was it the scientists who had to back down?

Microsoft did not respond to a request for comment, but Bruford's theory is that it's simply not worth the trouble to change. "This is quite a limited use case of the Excel software," she says. "There is very little incentive for Microsoft to make a significant change to features that are used extremely widely by the rest of the massive community of Excel users."

Bruford doesn't seem bitter about the situation, though. After all, she says, it wouldn't do to wait on a hypothetical Excel update to fix these problems when a long-term solution can be introduced by scientists themselves. Microsoft Excel may be fleeting, but human genes will be around for as long as we are. It's best to give them names that work.

Correction: The story has been corrected to clarify that Excel users can save spreadsheets that retain their formatting, avoiding the mistake where gene symbols are changed into dates. We regret the error.

All Comments: [-] | anchor

glofish(10000) 1 day ago [-]


the paper announcing the new guidelines of renaming genes, a work of fundamental importance to all scientists in the world, cannot be read without an expensive subscription to the journal.


Thanks science (sarcasm!)

rolph(10000) about 22 hours ago [-]

copy paste the DOI into scihub

ynodir(10000) 1 day ago [-]

Havent read the article, but come on. Tools should serve us, not the other way around. It'd be enough just to change the affected cells' type.

jmkjaer(10000) 1 day ago [-]

> Havent read the article

Please do. The article has a paragraph that addresses this.

curiousllama(10000) 1 day ago [-]

Excel datetime functions are garbage. you have to explicitly, manually tell it not to format things as dates, the most destructive data type, but it doesn't act that way for other formats (e.g., $ doesn't turn things into accounting format).

That said: every datetime function I've ever written is also garbage so... glass houses, I guess?

bronzeage(10000) 1 day ago [-]

Which is exactly why feeding all your input into a trash input parsing function by default is a horrible idea. If dates handling was just 1 extra button you click, the problem wouldn't exist. Overzealous default behavior, the #1 sin of Microsoft.

hermitcrab(10000) about 20 hours ago [-]

Excel is an amazing tool. But it also has some significant shortcomings:

* A well known tendency to mangle date and gene data under the guide of being 'helpful'. * Easy to making mistakes when cutting and pasting cells. * Difficult to see what is going on in a spreadsheet. * Poor handling of CSV files.

Some of these shortcoming are inherent to spreadsheets. Others are specific to Excel, but hard to overcome due to the weight of backward compatibility.

I have written a product for transforming and analysing tabular data (https://www.easydatatransform.com) that tries to overcome these issues:

* Doesn't change your input data file. * Doesn't re-interpret your data, unless you ask it to. * See changes as a visual data flow. * Operations happen on a whole table or column. * Good handling of CSV files.

Also it doesn't try to do everything Excel does.

It is a fairly new tool. Would appreciate some feedback.

tucaz(10000) about 19 hours ago [-]

Website looks great. Will try shortly and I'm more than willing to give you my money if it delivers. We need a product like this.

Edit: great call out to 7 days of non consecutive use

gverrilla(10000) about 17 hours ago [-]

I'm a business guy, so I don't know much about it, but looks very useful and easy! I have been off the game for 5 years now, but I used to run a magento ecommerce, and I think your software might have helped managing products listings and stuff like that at the time. You might wanna take a look at this costumer segment, particularly small and medium businesses. They probably have a process for this already, but your software might be a good replacement, even though they won't be actively looking for it because they already have something that works. Good luck!

zuno(10000) about 19 hours ago [-]

Thanks. I just checked the video. Looks neat! Excited to try it out.

mcv(10000) 1 day ago [-]

> 'Why, exactly, in a fight between Microsoft and the entire genetics community, was it the scientists who had to back down?'

Back down? Or pick a better tool. If Excel proves to be an unreliable tool for your job, use a better one. Alternatives exist, ranging from Google Docs, and LibreOffice, to simpler light-weight spreadsheets. Or possibly more specialist tools.

Why does everything always have to be put in Excel if Excel is such a poor tool for so many things?

tqi(10000) about 23 hours ago [-]

Characterizing this as a 'fight between Microsoft and the entire genetics community' is... a choice. But I guess Tech is this season's 'big bad', so every subplot has to tie back.

otherme123(10000) 1 day ago [-]

LibreOffice AND Google Docs fall in the same exact spot than Excel here. Try it. Type 'SEPT-2' or 'SEPT2' in any of them and see what happens.

asdff(10000) about 18 hours ago [-]

The vast majority of people who use tabular data a lot in biology are already just using python or R primarily, and most do this work on a cluster. This article is what happens when you send your .csv to a PI who hasn't bothered to brush up on any computer literacy after somehow mastering punch card programming, or the undergrad whose breadth of understanding beyond the web browser starts with MS office and ends with g suite.

HiFaraz(10000) about 8 hours ago [-]

But they did pick a better tool: they changed the name. Nomenclature is also technology.

toast0(10000) about 23 hours ago [-]

Better tools for managing genomic information are probably out there, but if everyone picks Excel (or workalikes that all do automatic date formating too), then hoping people switch is tilting at windmills.

Unilaterally declaring you will not use names that confuse Excel will fix the problem (until we get new month names, anyway), and not require anyone outside their sphere of influence to comply.

I had the same situation at work, as an aquired company we had servers named devX, the new parent had servers named devX and pushed ssh config files to our laptops that made it impossible to connect to our servers; we asked them to fix it the first couple times, but eventually renamed our servers to vedX to avoid the issue. It wasn't the right solution, but it was a stable solution and let us get on with our lives instead of fighting with IT.

acid__(10000) 1 day ago [-]

Excel may have failed in this specific task, but let's not pretend like its functionality doesn't run circles around Google Docs and LibreOffice.

Excel is a "pretty darn good" tool for 95% of tasks. If your work has highly varied workflows, then that flexibility more than makes up for its failures on the last 5%.

If you have very specific workflows on the other hand, you may find value in replacing Excel with a specialist tool. But let's not pretend that specialist tools don't also have their own shortcomings; at best they'll achieve 99.9% coverage of tasks.

alphanumeric0(10000) about 24 hours ago [-]

Research software dev here. This came as a huge shock to me when I started at my job. I work with very smart, dedicated people performing cancer research, why would they put up with this affecting their productivity? Humans really are adaptable creatures.

After a few months of working there my boss handed me 3 or 4 Excel spreadsheets to compare to ensure a recent change I made hadn't affected our data (we don't have much in the way of automated tests either). As a software developer, this was a deeply troubling request.

One option was to load them in to database tables so that I could perform SQL queries against the data (Postgres has COPY that works with CSVs), which isn't hard and probably the path most people should take, but I didn't want to write table definitions.

I ended up using https://github.com/BurntSushi/xsv (I am not affiliated with the project in any way). It's a command-line tool written in Rust that performs queries/joins/manipulation/basic analysis against CSV/TSV files. While not as analytically powerful as Excel or Postgres, I was able to verify the data was good and pipe out results into another file without writing any custom code, and without opening a single file.

MaxBarraclough(10000) about 23 hours ago [-]

Were the files expected to be identical? If so, diff would have done the job.

Perhaps not directly relevant, but the lesser known GNU Recutils looks neat. Perhaps some day I'll find an opportunity to try it out.


teruakohatu(10000) about 23 hours ago [-]

Have you checked out the tidyverse suite of packages on R? The tidyverse philosophy is don't make assumptions about data so you get detailed error messages when one row probably was not parsed correctly.

komali2(10000) 1 day ago [-]

In excel, can you set the a column to not format to dates? I think you can do that in google docs for example. Why don't they do that?

LatteLazy(10000) about 22 hours ago [-]

It's actually worse. If you expressly set the format of a column to text, then put 01/01 in it, you still get Jan-01 and a field value of 46000ish.

vikramkr(10000) 1 day ago [-]

A lot of people are attacking excel in this thread, just remember to give a fair share of the blame to people naming genes as well. These are meaningless names and changing them is frankly easier than changing a feature in excel (that the finance folk probably don't want changed). And its a good excuse to clear up some of the egregiously silly disease related names as well so we don't tell parents their kid is suffering from a debilitating mutation in luke-Skywalker-like upside-down cantaloupe 12b or whatever.

rolph(10000) about 23 hours ago [-]

no these are not meaningless names, there is undue confusication in the case of the more contemporary names such as [Sonichedgehog]

these are not meaningless lables https://en.wikipedia.org/wiki/Sonic_hedgehog

mentioned eslewhere in this thread is a nomenclature that allows one to easily find notes referring to the gene from your research library. It was a new cadre of young upcoming scientists that decided to break with tradition and use something familiar to lable genes according to game characters or pop icons.

dekhn(10000) about 22 hours ago [-]

In grad school I studied a gene which at the time was called Oct1 ('octamer binding protein 1'). My main problem was literature searches, which often found 'OCT-1' (organic cation transporter-1). Genomic naming is a total mess, I found it easier to just mentally compute the md5sum of a name, then memorize the first few digits (only need about 8-10 hex digits).

DMLoeffe(10000) about 21 hours ago [-]

> just mentally compute the md5sum of a name

What are you taking about lol

Brett_S(10000) 1 day ago [-]

If the scientists had asked someone who knew Excel well, then they would have been told to prevent autocorrect from running enter 'MARCH1 with the apostrophe at the start.

mrunkel(10000) about 4 hours ago [-]

This. Why does nobody know this?

lunchladydoris(10000) 1 day ago [-]

Or perhaps stop using Excel?

treeman79(10000) 1 day ago [-]

Excel is amazing at what it does.

I had a Manager that spent an entire week compiling a report on how much times people spent on tasks.

He was very proud as he was showing me this report. I asked him why he didn't use a pivot table.

While he was asking what that was, I spent two minutes on the same data that he had and reproduced his report. I had a friend and patron for life after that. I was involved in many major decisions for a fortune 50. All because I could work excel.

mobilio(10000) 1 day ago [-]

This happens before, and will happen again: https://qz.com/119578/damn-you-excel-spreadsheets-jp-morgan-...

chapium(10000) 1 day ago [-]

Excel is pretty efficient at ad-hoc text processing. You get real time feedback as you change rules and there are lots of handy functions to help you along. Its like a tabbed portable jupyter notebook.

DethNinja(10000) 1 day ago [-]

What's the alternative?

leecarraher(10000) 1 day ago [-]

Final Jeopardy : Trebeck: You know what, how about you just write down a number, any number at all. Could be a 1 or a 2, perhaps 3... and excel you answered; A smiley face emoji, simply stunning.

Excel is no longer motivated by the original intention of a spreadsheet, and now caters to the lowest common denominator, a piece of graph paper. As such MS has shifted focus from doing calculation to text and graphics layout tool. white text copied from a terminal : white text white background you got it! comma separated numbers : default a long string with commas in it want a plot : it is in the insert menu for some reason, since plots and numbers are no longer excels raison d'être

ubermonkey(10000) 1 day ago [-]

MSFT did the same thing with Project when they introduced something called 'Manually Scheduled Tasks.'

Project is fundamentally a critical path scheduling tool, or at least was. Task A must finish before Task B, which must complete before Task C. If A is delayed, then that delay pushes B and C out, too.

This is what it's FOR, more or less.

Manually scheduled tasks don't move. They're set with whatever dates you give them, and do not move in response to delays or whatnot from predecessor tasks.

People wanted this because some (dumb) people insisted that 'well, that task CANT move because it has to be done by then!'

This is akin to asking for the arithmetic engine to be turned off in Excel, because by golly you really need 2 and 2 to sum to 17.5.

jkaptur(10000) 1 day ago [-]

The date parsing being discussed has been the behavior for at least 20 years, probably more like 30.

Your comment about graph paper echoes a comment from a former Excel PM: 'The gridlines are the most important feature of Excel, not recalc.'


lvturner(10000) 1 day ago [-]

I think another question is that why isn't there a viable alternative to excel for this particular use case?

Is there, perhaps, a market for niche spreadsheet applications that serve one particular market?

hairofadog(10000) 1 day ago [-]

I've been wishing for something like this lately in the apple ecosystem: sort of a "plain text" tabular editor that can handle large data sets. Numbers feels like it's designed for presentations more than data crunching, and Excel (or at least the price of excel) would be overkill for my needs (which largely consists of data cleanup, spot-checking, and quick calculations). I'd happily pay for an IA-Writerish spreadsheet app.

Edit: or maybe Sublime Text is more what I mean. In any case I'm hoping someone will pop in and say, "well why aren't you Snappets!?" and then I'll go buy a copy of Snappets.

jpindar(10000) 1 day ago [-]

I'm sure there is, which is great for one's own use but you can't make everyone else use it.

takluyver(10000) 1 day ago [-]

Educated guesswork: this isn't one particular use case, but a whole bunch of use cases with one thing in common: gene names are put in a table. Excel is the familiar, ubiquitous tool for tabular data.

yummypaint(10000) 1 day ago [-]

Why in the world is excel the application of choice? Last time I tried to use it (for much less complicated things than genetics), it choked and became unusable when the filesize exceeded about 6MB. I have yet to encounter a spreadsheet oriented task that isn't better implemented in gnumeric. Maybe we should also shorten all publications so adobe reader can display them without crashing?

airstrike(10000) 1 day ago [-]

I regularly use 10MB+ files with circular calculations turned on and it's no biggie most of the time

zapdrive(10000) 1 day ago [-]

Maybe you should ditch your Pentium 1 and buy a newer computer?

catalogia(10000) 1 day ago [-]

> Why in the world is excel the application of choice?

Because if it weren't for Excel, most Excel users would have to hire programmers. (And Gnumeric is very obscure, how many non-programmers have heard of it?)

NikolaeVarius(10000) 1 day ago [-]

Excel is the application of choice because the world runs on Excel. Get with the times.

curiousllama(10000) 1 day ago [-]

I work in excel every day and, if this is true, you needed to spend like 30 seconds on google to find a half dozen solutions to this. Turn off auto-calculations, delete pivots, write fewer vlookups, etc...

It's roughly the equivalent of saying 'whenever my C program gets too big the 'core dumps,' whatever that means - a totally useless language'

tomp(10000) 1 day ago [-]

> For example, HECA used to have the gene name 'headcase homolog (Drosophila),' named after the equivalent gene in fruit fly, but we changed it to 'hdc homolog, cell cycle regulator' to avoid potential offense."

Ugh... everything is politics now.

_petronius(10000) 1 day ago [-]

As Thomas Mann said, everything is politics. I would add that has always been the case.

You can get frustrated about that fact, or you can use it as an opportunity for learning about how and why that is so, and gain a deeper understanding of the tightly linked systems of power relationships all around you.

krastanov(10000) 1 day ago [-]

As the sentence before the one you quoted explains, this is not done for politics. It is done because clinicians have to be taken seriously by the parents of the child having a mutation in that gene. Saying 'your child has a headcase mutation' ends up causing defensive reactions instead of discussing treatment. Maybe that is an irrational reaction on the parents' part, but not everyone is super rational the first moment they learn their child has an illness.

LatteLazy(10000) 1 day ago [-]

It's honestly amazing that Excel hasn't fixed this issue. It's pisses off an enormous number of users especially in basically any non-US country (even if 01/02 is a date, it isn't the second of January in most of the world...)

javier123454321(10000) 1 day ago [-]

Honestly though, dates make more sense being month first, that way if you are trying to discover which one comes first, 10th of february or 10th of january, you only have to go to the first parameter in format 01/10 vs 02/10 instead of 10/01, 10/02. Logically, the most broad data item should go first.

nikk1(10000) 1 day ago [-]

There are dozens of auto-formatting 'features' that piss me off each and every day across all of Microsoft's products.

airstrike(10000) 1 day ago [-]

Just type '01/02 or ='01/02' and get on with your life

mattnewton(10000) 1 day ago [-]

Why hasn't the field standardized around another tool? Is the list of excel plugins that something libreoffice couldn't replace it, where they could just make add the options they need? My guess is the field is mostly lacking people interested in working on tooling like this?

stjohnswarts(10000) about 21 hours ago [-]

But you can select from dozens of formats in the cell setup? Not sure I understand the issue?

danso(10000) 1 day ago [-]

I personally dislike this behavior in Excel, as I'm accustomed to working with plaintext data and using pandas/strftime to get datetimes as I need them. But I also figure that my situation as a technical user is likely in the minority, compared to people who are using spreadsheets every day to do manual data entry.

That said, I completely agree with the tangential issue of U.S. dates being misleadingly different in format compared to non-U.S. Always an issue when teaching data/spreadsheets to a class with at least one non-American – but also a good reason to teach them the value of ISO8601 :)

owl57(10000) 1 day ago [-]

I don't even know why anyone still uses this notation in today's connected world. 2020-08-06 is ideal. Russian standard (06.08.2020) is uglier (digits are not sorted by significance) but still unambiguous. But that slash horror? Why would anyone continue inflicting this on themselves and their peers!?

qayxc(10000) 1 day ago [-]

That's because it's not an issue at all.

It's people using a tool without knowing said tool. You can disable auto-formatting (or even better yet - set the column data type) with a simple click.

pintxo(10000) 1 day ago [-]

For Germany this converts to the first of Feb, and this way of entering dates is just so convenient, as you can type the whole date just using the keys from the num block. We do actually support this format in our app for date entry, simply because it's so convenient.

T-hawk(10000) about 20 hours ago [-]

Excel is stuck. It can't change this behavior if it cares about backwards compatibility. There are uncountably many recorded macros and all sorts of scripts and data tools everywhere, that implicitly and inadvertently depend on behavior like this. Like Javascript's warts, you have to declare these as features and live with them.

Of course Microsoft has thought about this, it's silly to think they never have and we're smarter. Yes, this behavior is annoying and US-centric, they know that. But they also know that breaking compatibility with all these scripts and macros would be the worse problem in the larger picture. That picture is huge, it would be on the order of the scope of the Y2K effort to modify every code everywhere that's ever touched Excel dates.

Would you want Excel to introduce a 'quirks mode' to handle this sort of thing?

vsareto(10000) 1 day ago [-]

>It's honestly amazing that Excel hasn't fixed this issue.

I'm surprised there hasn't been a dotfile option added yet

numpad0(10000) 1 day ago [-]

Excel's problem is it destroys the original keystrokes.

Anyone know why? It makes little sense to me.

dkarl(10000) 1 day ago [-]

Because usability is measured by the naive expectations of the least sophisticated users.

airstrike(10000) 1 day ago [-]

Because most of the time you actually want it to be a date and those are stored in the spreadsheet as integers

You can add a single quote before any value to prevent Excel from autoformatting

Or just turn off autoformatting entirely

proactivesvcs(10000) 1 day ago [-]

In this case, because it's much faster to enter data such as dates by entering the shorthand - e.g. 6/8 - than have to type 06/08/2020. The same reason it's very helpful to type '500' into a currency cell and have it convert to '£500.00', or the many other sorts of autotype.

takluyver(10000) 1 day ago [-]

Programmers love to complain about Excel, but we happily use YAML, which has essentially the same footgun: certain strings (like 'on') need quoting if you don't want it to interpret them as something else.

the_imp(10000) 1 day ago [-]

This was fixed in YAML 1.2, which was released in 2009. Don't blame the spec, blame the applications that use ancient libraries.

recursive(10000) about 23 hours ago [-]

> we happily use YAML

Don't include me in that group.

DarkWiiPlayer(10000) 1 day ago [-]

YAML offers a simple solution though: just quote all one-word text and, more importantly, quote all text you don't manually write into the YAML file.

Also, the main problem is that excel changes your input instead of just misinterpreting it. The difference being, the original will be gone.

elwell(10000) about 22 hours ago [-]

> Programmers love to complain about Excel

In Apache POI HSSF/XSSF (Java Excel API), the HSSF actually stands for 'Horrible Spreadsheet Format'.

As a fun example, note that we need isSheetHidden() and isSheetVeryHidden() functions. (http://poi.apache.org/apidocs/4.1/org/apache/poi/ss/usermode...)

mattnewton(10000) 1 day ago [-]

If I use those words a lot I will consider changing from YAML since it might be the wrong tool.

fabian2k(10000) 1 day ago [-]

We programmers also complain a lot about YAML, though maybe not enough and not as much as about Excel. But some YAML footguns like the country code for Norway being interpreted as a boolean are reasonably famous, and I think widely regarded as a bad idea.

rkachowski(10000) 1 day ago [-]

My mind is blown that Excel's usability is so bad that the representation of the human genome itself has to adapt around it's undesired behaviour.

As in, the history of genetics research is now irreversibly linked with the shortcomings of this one software product, which just happens to be incapable of describing the genetics of the organisms that created it.

mmcgaha(10000) 1 day ago [-]

I hate to sound like a salty old IT guy, but here we go. It is not the fault of Excel that people are using it wrong. They have the ability to import the data as text but they skip that step all together. If the user does not say up front what the column is, Excel has to guess. If Excel didn't try to guess, someone would be making a comment on how bad usability is when an obvious date field was getting interpreted as text.

throwaway936482(10000) 1 day ago [-]

Excels usability isn't bad though. Which is why everyone uses it. It falls down in this (and lots of other cases) but if you want to see what good usability looks like, look at what people use.

ep103(10000) 1 day ago [-]

My favorite is still that Excel can't handle dates before 1900

unnouinceput(10000) 1 day ago [-]

Quote: 'There's no easy fix, either. Excel doesn't offer the option to turn off this auto-formatting...'

What kind of loopy Excel variant the article's author is using? It's right there in settings, you can easy stop auto-formatting. Yes, default installation has it on, but it can be turned off.

epistasis(10000) 1 day ago [-]

I do not believe it can be turned off, or at least I've never encountered a version where it could be turned off by toggling options.

Create a simple CSV file like

Open it in Excel, and watch SEPT1 be silently converted to 1-Sep, with no way to return.

Exce is truly worse than most people can imagine.

sseagull(10000) 1 day ago [-]

Can you explain where? I just booted up a relatively clean copy of Excel 2019 and can't find it after a couple minutes.

I see a few autoformatting options, but nothing corresponding to dates. The only options I see are similar to this:


mplanchard(10000) 1 day ago [-]

From the article:

> Even then, a scientist might fix their own data, but as soon as someone else opens the same spreadsheet in Excel without thinking, errors will be introduced all over again.

conductr(10000) 1 day ago [-]

Also you can single quote 'MARCH1 to keep it as string. This is safer as when you share the file you don't know if the next person has the same settings change you described. Either way, it's only 27 genes they had to rename. Seems like a good choice on their part to just rename them. Hopefully they're doing this check during the naming stage for new genes.

jordigh(10000) 1 day ago [-]

To people asking, 'why do they use Excel?' that's like asking 'why must we be subjected to gravity?'

The whole world's data ultimately comes from or ends up in an Excel spreadsheet. Sure, we might use other intermediate data storage methods, but in the end it's going to go into some scientist's or some politician's computer, and by golly it's gonna be in Excel. Trying to rally against Excel is like trying to rally against fundamental forces of nature.

This is just an example of that fundamental force winning.

guenthert(10000) about 19 hours ago [-]

Ah, just make the data file larger. No one is going to use a spreadsheet tool when there are millions of rows (I hope).

agumonkey(10000) 1 day ago [-]

And other fields are probably stuck with bad word processing papercuts.

wittyreference(10000) 1 day ago [-]

This accords with my experience.

When I worked in clinical trial design/analysis, no one did their analyses in excel, but for initial data formatting and clean-up? /everything/ went through excel, even if the final cleanup was a python script.

You're not going to have the same ease of eyeballing your data in SPSS or SAS or R.

pratio(10000) 1 day ago [-]

True in every sense. We shifted from google sheets to O365, there's just no way to go around excel. Its efficient and every one from a novice to expert can find their way around it.

gsich(10000) 1 day ago [-]

Excel is quick and easy. From startup to import to calculation. You haven't imported a single line with Python or R.

TallGuyShort(10000) 1 day ago [-]

Also, on the list of dumb things that make me roll my eyes at Microsoft, interpreting the string 'MARCH1' as a date and trying to normalize date formats isn't exactly the dumbest.

bromuro(10000) about 23 hours ago [-]

> to people asking, 'why do they use Excel?'

I often read comments on HN starting with "to people asking" or "to the people wondering"... and introducing a new discussion, usually on another similar topic, basically hijacking the whole discussion.

donatj(10000) 1 day ago [-]

So much this. I work in educational software, and have a little write up about the troubles we've had with schools "fixing up" CSVs from their student information systems before sending them to us. The simple act of opening the CSV and hitting save loses data (Microsoft actually fixed some of the problem very recently). That said, they will use Excel, it's not in my control.

I received dozens of comments "Just use Open Office / Libre Office". I am but a man, I cannot change the world. I forgive that which I cannot control. We work with tens of thousands of schools. Getting all of their administrators to install Open Office, and thousands of administrative assistants to remember to use it rather than Excel is simply an impossibility.

- https://donatstudios.com/CSV-An-Encoding-Nightmare

azalemeth(10000) 1 day ago [-]

> ' To people calling, 'why do they use Excel?' that's like asking 'why must we be subjected to gravity?''

I respectfully disagree with this. Excel is fundamentally not suited to analysing *omics data. It's often the default program affiliated with a .csv filetype on people's computers, but trying to get an entire field of scientific research to rewrite itself based on its glorified bugs is...wrong, in my opinion.

If you see wrong things in the world, do you accept them as they are, or try -- however ineffectually -- to force change for the better? I for one bang the drum into the wind and try to get biochemists off it. I teach people to be very sceptical of excel in my stats courses, for example (aside from some showstopping bugs and downright dangerous defaults, its RNG is particularly crap).

anameforauser(10000) about 11 hours ago [-]

"Newton had a great time for a long time with his description [of gravity], and then at some point it was clear that that description was fraying at the edges, and then Einstein offered a more complete version," As for flat spreadsheets, they serve their purpose just like any other tool. I don't think I heard people complain about spreadsheets for dealing with flat data, the same way no one complains about hammers to deal with nails. If we're trying to deal with interconnected and enterprise resources we don't maintain those resources in Excel. We store them in Enterprise Resource Planning software or build relational databases (which are as old as gravity). Nothing is wrong with spreadsheets, they're the sweetest way to deal with quick data until we use them for things they can't do.

lifeisstillgood(10000) 1 day ago [-]

Nah, I plan on having HDF5 or Apache Arrow be the portable table of data and then build WASM based viewers - it's the excel killer but the war of attrition will be long !

BurningFrog(10000) 1 day ago [-]

If the gravity metaphor doesn't reach you, maybe this will:

The world speaks English. It's an illogical and absurdly spelled language. Esperanto is miles ahead on all logical measures. Yet we will never switch, both because the cost of switching is enormous, and because there is no way to coordinate/organize such a move.

throwawaylabor(10000) 1 day ago [-]

The hyperbole here is amazing. You're saying that a 40 year old computer program that runs exclusively on windows is as un-changable as a _law of nature_.

The english comparison made in a peer comment is more apt, but wow.

The tech industry is so self-centered we think that Scientists should change their fields because of our bugs.

rorykoehler(10000) 1 day ago [-]

Excel should belong to the commons at this stage

importantbrian(10000) about 22 hours ago [-]

Whenever people ask me about getting into the field and if they should learn Python or R or Scala or whatever else, I always respond with you need to get really good with excel/Google Sheets and SQL. Then worry about everything else. I can't tell you how many times I've done an analysis I'm really proud of in something like R Markdown only to have the end user say that's great now can I get it in an excel? I just deliver everything in excel as a default now.

rat_1234(10000) 1 day ago [-]

This calls to mind a quote from Cormac McCarthy's Blood Meridian:

  It makes no difference what men think of war, said the judge. 
  War endures. As well ask men what they think of stone. War 
  was always here. Before man was, war waited for him. The 
  ultimate trade awaiting its ultimate practitioner. That is 
  the way it was and will be. That way and not some other way.
Just swap our war for Excel! 'It makes no difference what men think of Excel, said the judge. Excel endures. As well ask men what they think of stone.'
pmarreck(10000) about 24 hours ago [-]

OK then. Use Excel. But at least know the basics of Excel. Such as simply right clicking the column header and picking a different format so that it will stop auto-interpreting the input.

meow1032(10000) 1 day ago [-]

I don't disagree completely with this, but just want to point out that it's kind of a bad smell to have computational biologists who are - as someone in the article puts it - computationally illiterate. I have met lots of these types over the years, and usually their methods are kind of a gong show. If you can't properly sanitize your data inputs on your column headers, why should I trust that you've treated the rest of your data properly?

Duller-Finite(10000) 1 day