Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

November 19, 2019 17:34

Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: Court: Suspicionless Searches of Travelers' Phones and Laptops Unconstitutional (November 12, 2019: 2061 points)

(2061) Court: Suspicionless Searches of Travelers' Phones and Laptops Unconstitutional

2061 points 7 days ago by coloneltcb in 16th position

www.aclu.org | Estimated reading time – 2 minutes | comments | anchor

BOSTON — In a major victory for privacy rights, a federal court in Boston today ruled that the government's suspicionless searches of international travelers' smartphones and laptops at airports and other U.S. ports of entry violate the Fourth Amendment. The ruling came in a lawsuit, Alasaad v. McAleenan, filed by the American Civil Liberties Union, Electronic Frontier Foundation, and ACLU of Massachusetts, on behalf of 11 travelers whose smartphones and laptops were searched without individualized suspicion at U.S. ports of entry.

"This ruling significantly advances Fourth Amendment protections for the millions of international travelers who enter the United States every year," said Esha Bhandari, staff attorney with the ACLU's Speech, Privacy, and Technology Project. "By putting an end to the government's ability to conduct suspicionless fishing expeditions, the court reaffirms that the border is not a lawless place and that we don't lose our privacy rights when we travel."

"This is a great day for travelers who now can cross the international border without fear that the government will, in the absence of any suspicion, ransack the extraordinarily sensitive information we all carry in our electronic devices,' said Sophia Cope, EFF senior staff attorney.

The district court order puts an end to CBP and ICE's asserted authority to search and seize travelers' devices for purposes far afield from the enforcement of immigration and customs laws. Border officers must now demonstrate individualized suspicion of contraband before they can search a traveler's device.

The number of electronic device searches at U.S. ports of entry has increased significantly. Last year, CBP conducted more than 33,000 searches, almost four times the number from just three years prior.

International travelers returning to the United States have reported numerous cases of improper searches in recent months. A border officer searched plaintiff Zainab Merchant's phone, despite her informing the officer that it contained privileged attorney-client communications. An immigration officer at Boston Logan Airport reportedly searched an incoming Harvard freshman's cell phone and laptop, reprimanded the student for friends' social media postings expressing views critical of the U.S. government, and denied the student entry into the country following the search.

The opinion is here: https://www.aclu.org/legal-document/alasaad-v-mcaleenan-opinion-summary-judgment.

All Comments: [-] | anchor

aerovistae(2940) 7 days ago [-]

Yeah, too bad the border control agents basically do what they want and this case means nothing.

They'll do the same thing, and someone will say 'But what about Alasaad v. McAleenan?' and in the small windowless holding room the agent will say 'What about it? You want to leave, unlock your phone.'

zackmorris(3469) 7 days ago [-]

I've been seeing this fallacy a lot lately in these times. If you're ever in this situation: the longer you're held without a lawyer, the higher the liability on your captor becomes. Period. This is where million dollar lawsuits by the ACLU and jail time for corruption begins.

The US was founded on habeas corpus, and it doesn't matter what laws have been written since, or what national threats have been used as an excuse to undermine our inalienable rights. Your duty is to the future people of the US so that their rights to life, liberty and the pursuit of happiness aren't infringed.

My feeling is that this house of cards of government officials skirting the law and pleading ignorance is going to come crashing down in the near future. They know that they are breaking the law. They're just assuming that you will be afraid and let them.

I had hoped that we were past all this and the 21st century would to be free of injustice, but unfortunately we're all going to have to fight just like our ancestors did. But I think that better times really are just around the corner, as long as we stay informed and unified.

Related: congresswoman Nanette Barragan questions Kirstjen Nielsen on asylum seekers being turned away at US border crossings:


landryraccoon(4165) 7 days ago [-]

Evidence from an illegal search cannot be used in court.


cmurf(1645) 6 days ago [-]

They can image my mobile device in, what, 5 minutes? Why hold it for days or weeks except as a form of coercion? You land and need your phone to retrieve the phone number for friends or family, to order rides, create and manage accommodations, and find directions. Taking someone's mobile device leaves them stranded.

neilalexander(4077) 6 days ago [-]

They can only image your device if they can connect to it. I don't know what Android is like in this regard, but an iPhone won't open up the USB interface to a foreign device until it has been unlocked and the device has been trusted by entering the passcode, which keeps some power in your hands.

If they just take the device off you, your power is taken away in a different way, but it's still taken away from you.

MaupitiBlue(4049) 7 days ago [-]

Haven't read the opinion, but I'd anticipate that this will be reversed on appeal. My guess is that this was either an Obama or Clinton judge.

Both my Crim Pro and Con Law professors skipped border search cases and summed them up with 'you have no rights at the border.' That's been the common law since King Tut.

pjc50(1404) 7 days ago [-]

> That's been the common law since King Tut.

Much immigration law is pure statute and dates from the 20th century. Border enforcement has varied hugely, and often included internal border and migration enforcement.

(The British Empire had almost what we would call EU-style 'free movement' exactly up to the point where nonwhite people started taking advantage of it in significant numbers, with the Ugandan Asians fleeing Idi Amin in the 60s.)

ars(3058) 7 days ago [-]

https://en.wikipedia.org/wiki/Denise_J._Casper Obama Judge.

But you should realize you are basically saying Judges are not impartial arbiters, but rather biased based on who appointed them.

dragonwriter(4218) 7 days ago [-]

> That's been the common law since King Tut.

Common law started long after King Tut, and in any case the Constitution isn't overridden by common law where they conflict. While this particular issue has been controversial, there idea that you have no rights at the border is unsupportable either in the text of the Constitution or the case law, and you should feel cheated that you had two professors that did such an inexcusably bad job of explaining the law, if your description of their lessons is correct.

akersten(10000) 7 days ago [-]

Does this apply to the 'special Constitution zone' that intrudes 100 miles inland from the US border? Doesn't matter if it's constitutional or not if you're in a no-constitution zone.

AnimalMuppet(3739) 7 days ago [-]

Hmm. Has the Supreme Court ever ruled that the 'special Constitution zone' was constitutional? Because, on its face, it looks like it's not...

dragonwriter(4218) 7 days ago [-]

> Does this apply to the 'special Constitution zone' that intrudes 100 miles inland from the US border?

This is specifically about border searches, so, insofar as such a zone exists (the description is hyperbolic and more useful for motivating activism that any kind of analytical application), yes.

> Doesn't matter if it's constitutional or not if you're in a no-constitution zone.

There is no such thing. There is an executive-asserted 100 mile zone (statute only says a reasonable distance, and AFAIK courts have not ruled on actual boundaries) from the border in which federal officials can conduct warrantless searches of vehicles to identify people without legal permission to be in the country.

dontbenebby(4040) 7 days ago [-]

IANAL, so can someone help me understand: How is this the final say if it's out of a court in Boston? Isn't that only one circuit?

kickopotomus(10000) 7 days ago [-]

It is not necessarily the final say. DHS may turn around and challenge the ruling which may then be picked up by the Court of Appeals for further review. However, the Court of Appeals does not have to review the case and even if they did, they very well may agree with the ruling from the district court.

mark-r(4210) 7 days ago [-]

Aren't rulings in one circuit often used as arguments in another circuit?

Obviously it's not the final word unless it comes from the Supreme Court.

colinmhayes(10000) 7 days ago [-]

The US has common law, so I believe this ruling becomes precedent unless it is overturned.

vilhelm_s(3607) 7 days ago [-]

It's not even a circuit yet, this is a district court judgement. It is not a final say.

Fellshard(10000) 7 days ago [-]

Open question, here: Doesn't this backfire and make TSA more likely to operate on the basis of profiling alone? What makes a traveler suspect?

This reminds me of Sowell's distinction between 'Type I' and 'Type II' discrimination: when all other information is forbidden from you, you'll end up using the only information you have as your heuristic, however noisy and biased that information might be compared to a more comprehensive source. This leads to incidental systemic discrimination.

That said, if 'suspicionless' is tied to concrete external evidence rather than mere profiling, this should be a good move.

ddenisen(10000) 7 days ago [-]

This ruling has nothing to do with TSA. TSA does not control US borders.

umvi(4218) 7 days ago [-]

Unpopular opinion: why is profiling bad in context of airplane security?

If 100% of airplane terrorist attacks are carried out by people of X country (Xians), and 0% of airplane terrorist attacks are carried out by people from Y country (Yians), it seems like a pointless charade of political correctness to check Xians and Yians with equal frequency.

I say profile the heck out of Xians. Stop and frisk 100% of Xians and 0% of Yians.

AdmiralAsshat(1438) 7 days ago [-]

So will the CBP be ordered to destroy all the images they made of devices searched at the border?

vilhelm_s(3607) 7 days ago [-]


> As part of the relief sought, Plaintiffs seek expungement of all information gathered from, or copies made of, the contents of Plaintiffs' electronic devices including social media information and device passwords. [...] Although this is not a criminal case, considering the remedy for the unconstitutional search in the criminal context is illustrative of the extraordinary nature of the remedy sought here. Even where law enforcement officers have conducted a search in violation of the Constitution, the "fruits of [the] search need not be suppressed if the agents acted with the objectively reasonable belief that their actions did not violate the Fourth Amendment." [...] even where criminal proceedings followed a border search that exceeded the bounds of the Fourth Amendment and the fruits of same were suppressed, expungement of the border agents' files would not necessarily follow. Nor should it where other deterrents to border agents' unconstitutional searches remain in place. [...] In light of this other relief, including declaratory relief, the Court DENIES the request for expungement of information


annoyingnoob(10000) 7 days ago [-]

There is hope for the US, hard to believe with the current state of affairs.

jakeogh(3264) 7 days ago [-]

You mean pulling out of the various wars? Or the awesome economey? Or just the BS media?

sneak(3056) 7 days ago [-]

They let the border pigs keep all of the illegally and coercively stolen private information from the thousands or millions of phones they've already imaged (under the basis of 'it was believed to be legal at the time'), so not really.

Third-party doctrine lets them silently access your emails and IMs and texts at rest, and Section 702 as well as creative definitions of 'access' let them sniff them all in transit. Privacy from the fed snoops is 100% an illusion; the entire population is under continuous realtime monitoring.

This border search thing is an important but ultimately minor victory; total surveillance is the name of the game in the United States now, and likely will remain so until the country no longer exists.

Strong encryption is the only defense we have against mass surveillance, but sadly most people don't care enough about their own privacy (much less that of others - the classic 'What do you have to hide?') to demand it in communications products and services, so the majority of personal networked communications happens in ways that are extremely simple for the corrupt federal government to collect, analyze, and permanently store.

rwmj(735) 7 days ago [-]

Does 'travellers' include non-US citizens?

kova12(10000) 7 days ago [-]

I'm not a lawyer, but afaik the Constitution protects equally citizens and not citizens

kpU8efre7r(4158) 7 days ago [-]

And that, ladies and gentleman, is the awesome side of the USA.

dang(179) 6 days ago [-]

Please keep nationalistic flamebait off HN. Also, could you please stop posting unsubstantive comments? You've done it repeatedly lately and it's not what this site is for.


corford(4181) 7 days ago [-]

What... legal theater?

tus88(10000) 7 days ago [-]

But are travelers (foreigners) part of 'the people'?

papito(10000) 7 days ago [-]

There are cases of doing this to U.S. Citizens, in some cases, quite disturbingly, to journalists.

jdeisenberg(4097) 7 days ago [-]

Yes. The Constitution is very careful in its use of 'people' (or 'person') and 'citizens'.

abathur(10000) 7 days ago [-]

While this is very relevant in a broad sense, it isn't at matter in this case.

The plaintiffs are 10 U.S. citizens and one lawful permanent resident.

hangonhn(4021) 7 days ago [-]

That's an interesting question. It's not restricted to only citizens (so applies to immigrants, etc.) but may not extend as travelers who are passing through or has no real connection to the country. Ref: US vs. Verdugo-Urquidez

babypuncher(10000) 7 days ago [-]

I would argue that any individual deserves most or all rights outlined in the Bill of Rights, regardless of citizenship or geographic location. These are fundamental human rights. I'm aware this is not how the constitution is written, it's just my opinion.

kpU8efre7r(4158) 7 days ago [-]


Finnucane(10000) 7 days ago [-]

If you're on US soil you're subject to US law, including all provisions of the Constitution. And the Constitution makes a distinction between 'citizens' and 'people' (i.e., everybody, including noncitizens)

baddox(4166) 7 days ago [-]

It's a bummer there is still so much serious political discussion about which people count as people.

thinkloop(3571) 7 days ago [-]

It's unfortunate that this is a reasonable question. I've been surprised to learn how conditional being treated like a human being really is (location, citizenship, etc.). It would be my preference for my country to treat all people kindly, fairly, equally.

dreamcompiler(4034) 7 days ago [-]

Good question. I was going to say 'yes' but the Heller decision has made that less clear.

dragonwriter(4218) 7 days ago [-]

> But are travelers (foreigners)

International travelers entering the US (who were subjected to the policy) are not all foreigners.

> part of 'the people'?

Yes, 'the people' in the Fourth Amendment (and most of the rest of the Constitution) are the people subject to the authority of the United States government, which is basically everyone within US territory, however transitorily, and not exempt from US authority by treaty (e.g., diplomats, etc., are excluded.)

luckydata(4207) 7 days ago [-]

It was always CLEARLY unconstitutional. It's worrying it took so long to have it declared as such.

ngngngng(3875) 7 days ago [-]

As an American I'm very confused why clear constitutional violations don't warrant any repercussions on the perpetrating individuals.

So I work for a company that deals with medical data, if I violate HIPAA, I don't get to just keep doing it until a federal court says stop and then walk away as if nothing happened. I'm personally held responsible for those violations. Not just my company, me.

Why aren't each of these agents that obviously and egregiously violated the constitution subject to imprisonment or fines for each violation?

colinmhayes(10000) 7 days ago [-]

The level that the bill of rights applies to non-residents isn't very clear. The fourth amendment specifies the right of 'the people', but it doesn't expand on who this includes. Some interpretations have it as us residents, some have it as all people.

bsimpson(3484) 6 days ago [-]

It makes me really sad that it took a court case to decide this.

waterheater(10000) 6 days ago [-]

Drafting a law would be worse. Smartphones have only been around for 12 years, so no good law could have been passed because we did not fully understand how the smartphone would shake out.

It took time because society needed to reckon with the newfound implications brought by such profound technological shifts.

Freedom doesn't come from a drive-thru window. You have to wait at the table until the issue has cooked.

mlang23(4206) 6 days ago [-]

Best thing is to just not have anything to do with the US of A. Death penalty, loaded guns everywhere, a general attitude of being the masters of the world which leads to things like suspicionless searches. No, thanks. I think the best would be if more people would actually stop dealing with the USA. Its becoming a trend. I once turned down an invitation to a conference in Hawai, and guess what, I was not alone.

bluGill(10000) 6 days ago [-]

Death penalty exists, but it is rare.

Loaded guns exist, but they are not common. Most of them are in the hands of well trained people who are harmless unless (in the unlikely event) that deadly force is the only way they can get out of the situations.

The US has its problems, but you have been listening to propaganda by anti US forces (you might want to figure out who they are and what their agenda is!) if you think either of the above issues are worth a worry.

umvi(4218) 6 days ago [-]

You are far more likely to be killed by a drunk driver in the US than by a gun. Your chances of being killed by a gun are basically zero (unless you are travelling to the US specifically to commit suicide), so that's a somewhat bogus concern

thedaemon(10000) 6 days ago [-]

Why are you on hackernews, a USA website / company?

umvi(4218) 6 days ago [-]

> a general attitude of being the masters of the world

Sounds like you don't appreciate the regional and resource stability that the US military brings to your country. You probably just take it for granted that cargo ships can travel freely and trade without worry of being attacked. Global trade wasn't this risk-free before the US navy became 'masters of the world'.

Maybe we need to dissolve NATO then because clearly the EU does not see defense as a priority and puts little to no effort protecting its interests since the US takes care of it. I vote the US pull out of NATO and stop being 'masters of the world' in the middle east, protecting shipping lanes with the navy, etc. EU can protect its own interests with its own defense budget. Good luck.

TheCuckedKing(10000) 6 days ago [-]

They are the world masters, though..

WilTimSon(10000) 6 days ago [-]

That whole attitude of being 'the greatest country in the world' always seemed so peculiar to me. I get that all countries tend to be patriotic and embellish themselves but that just sounds a bit ridiculous and antagonistic. For a country established by immigrants and populated by immigrants, you'd think the national identity would be a bit more versatile.

Instead, it's been homogenized to the point where people have to fight their own perceptions to reclaim their culture and identity. It's a shame, really.

currymj(3841) 6 days ago [-]

ironically, Hawaii has no death penalty and extremely restrictive gun laws. it's a big country!

abstractbarista(10000) 6 days ago [-]

As an American, I'm lovin' it! Guns are everywhere, yet shootings are very rare. I feel very safe knowing I and others around me all want and enjoy safety, and we have strong law enforcement to keep it that way.

(Although I agree that our 4th amendment has been horrifically raped in modern times.)

The vast majority of gun deaths are consentual (suicide), and of the homicidal remainder, the vast majority are within organized crime. Stay away from gang activity and don't flagrantly disobey police officers, and I guarantee you'll be completely fine.

If you disagree with all this, it's probably just not the country for you. You make a good decision staying out.

jopsen(10000) 6 days ago [-]

There are 16 countries rated higher on the 'human freedom index': https://object.cato.org/sites/cato.org/files/human-freedom-i...

I assume you won't travel to Sweden, they have the same score as the US :)

Don't get me wrong, I also frustrated the US doesn't so better -- not because the US is that bad, but because it would be so easy for the US to do better.

And for some reason I emphasize with the Americans :) (I live in Denmark, but I'm not scared of traveling to neither Sweden or the US -- but I've never been to a non-democratic country, no plans to either).

sojournerc(10000) 6 days ago [-]

He says while commenting on web site based in the USA

aledalgrande(3820) 6 days ago [-]

What surprises me the most is the huge divide between these horrific things that happen at government level and the fact that 99% of Americans I met were instead very lovely people.

jslove(10000) 6 days ago [-]

And worse - no proper support for Metric!

psv1(10000) 6 days ago [-]

Yes - on a personal level, I tend to avoid travelling to the US. For the same reasons that I avoid travelling to most of the Middle East or China.

jason_slack(2419) 6 days ago [-]

But you realize that ordinary people don't have a large choice in this right? We can vote and vote and vote, but until the majority wins, and helps facilitate change, we are stuck with how people perceive our country as well as some of the harsh realities of living here.

I feel like you insulted me for the actions of my country. Isn't this how racism and hate is spread? I'm now not worth knowing and should be shyed away from because of my country...

Edit: I think the downvotes are a bit uncalled for. The OP said people should avoid the US because of its practices. The OP didn't say ANY country with these practices, China, Middle East, North Korea, Russia. It was a very general statement that has is heavily opinion driven.

bsenftner(4198) 6 days ago [-]

As an American citizen, I've been trying to talk my family into leaving for years. Comments here reference the problem being the State and not the citizens. I beg to differ, the citizens in 'general America' are dangerous and stupid to themselves and everyone around them. The entitled attitude of middle and upper middle class anyone is so unreasonable, I sometimes walk around the gang-tagged zones in my own neighborhood and think 'we're a nation of trapped rats, we deserve this for the lies we keep repeating, we keep eating, we keep electing, and the most empty lie how we're the greatest nation on earth, what a hoot!'

ramblerman(3215) 6 days ago [-]

As a european who is not always very fond of the US, this is ridiculous, and lazy thinking.

The amount of generalization and black and white simplifications to come up with such a thing is impressive though.

DannyB2(10000) 6 days ago [-]

> Best thing is to just not have anything to do with the US of A.

The US is about 4% of the world population. The rest of the world about 96%.

(Google for 'US population' and 'world population', use mathematical process known as division.)

g105b(10000) 6 days ago [-]

This comment is as mindless as the generalisation it's applying to an entire population.

tonfreed(10000) 6 days ago [-]

Oh come on now. In Australia you can be detained without charge and all they have to do is mention the 't' word. I also had someone turn up to our tabletop session on the weekend saying trains had stopped because someone got shot outside the station, while I struggle to get a license for a muzzle loading antique flintlock rifle. Not to mention I can get fined for all matter of stupid stuff because of the nanny state.

The US isn't perfect, but of all the places in the world you could live it's undeniably one of the best. Probably second only to Germany in my favourite places to travel, tbh

cgb223(4209) 7 days ago [-]

What constitutes "reasonable suspicion" in this case?

gorgoiler(10000) 7 days ago [-]

I'm not sure to whom the reasonable suspicion has to be announced? It makes it hard for me, a lay person, to understand the practical ramifications of this decision.

Would the suspicion have to be announced to me, when I'm asked to unlock my phone, or would it have to be presented to a court, when they are asked to prosecute me based on the contents of my phone? If they find illegal content on my phone, wouldn't the prosecutor just tell the court "she matched a profile for the sort of person that carries illegal documents"?

The latter is only meaningful if the case goes to court, which feels like a stretch. There are many forms of punishment that can be meted out without having to go to court. Vexatious detention, confiscation of property, etc.

The former seems silly — surely I can't impede law enforcement because they haven't justified their authority to me?

JohnJEsq(10000) 7 days ago [-]

The search of a phone is like a search of any other thing in US law. Generally, searches require a warrant which implicitly requires probable cause, but there are some well-defined exceptions to the warrant requirement, and the probable cause requirement. One of these exceptions is the border search exception. US Courts have previously said this exception is reasonable for various reasons, and is an exception to both the warrant and probable cause requirements. The court in this case says it is an exception only for routine searches, and called the digital searches non-routine which don't fall entirely within the existing exception. Rather the court says non-routine searches don't require a warrant but do require a reasonable suspicion, a standard still below probable cause. A reasonable suspicion requires only the officer to have a suspicion they can state a reason for. For example, a phone's lock screen with CP would satisfy reasonable suspicion.

> Would the suspicion have to be announced to me, when I'm asked to unlock my phone[?]

No. Officers don't have to disclose to you their reasons. It's helpful, but not constitutionally required.

>[W]ould it have to be presented to a court, when they are asked to prosecute me based on the contents of my phone?

Not necessarily. In a criminal prosecution violations of the Fourth Amendment can be excluded by the court. The theory underlying the exclusion is that without the exclusion there would be nothing to stop cops from violating your Fourth Amendment rights. A criminal defendant must challenge the use of the evidence found, and the officer must give their reason when the judge is trying to determine whether to allow the testimony.

In the US, you can also sue the officers when they violate your rights and win money damages. That's what this case is. The officer as part of their defense must give their reasons to defend they didn't violate any constitutional rights.

>If they find illegal content on my phone, wouldn't the prosecutor just tell the court "she matched a profile for the sort of person that carries illegal documents"?

Probably. Some courts may additionally require the officer to give a basis of why you fit that profile, or why someone who fits that profile reasonably is likely to carry illegal things. Typically just saying it matches a profile is sufficient.

calibas(10000) 7 days ago [-]

> surely I can't impede law enforcement because they haven't justified their authority to me

Unless you've broken the law (and certain special cases), they have no authority over you. Because of the nature of their jobs, which requires them to sometimes take away a person's freedom or life, they must be held to a much higher level of accountability than your average person.

You cite fear of harassment as a reason for not standing up to police, and to me that says something is deeply wrong with our current system.

landryraccoon(4165) 7 days ago [-]

The point of the law is that even if law enforcement finds evidence of guilt via illegal search, they cannot use that evidence in court. This is a crucial protection for citizens. The point of the ruling isn't that it prevents the search, it prevents the result of that search from being used against you. If law enforcement wishes to establish the guilt of a suspect in court, it has to use only evidence it obtained through legal means.


Tomminn(4197) 7 days ago [-]

Next stop, how about letting a foreigner visit the US for a week without having their fingerprints taken.

forgotmypass9(10000) 6 days ago [-]

How about letting foreigners change airplanes without treating them like they plan to visit or be illegal immigrants? Canada, UK, I'm looking at you guys too.

Last time I changed airplanes in Canada: 'How long do you plan to stay in Canada?' 'Well my flight leaves in three hours.' 'Are you bringing any nuts or fruit in to Canada?'

compiler-guy(10000) 7 days ago [-]

I support this ruling very strongly, and am super-glad it went the way it did.

But at the same time, if it were as obvious as just quoting the constituion, it wouldn't have made it to the Supreme Court in the first place. The SC decides cases that are difficult, not cases that are easy.

Again, I believe this is the right decision, but it was no layup.

michaelmrose(10000) 7 days ago [-]

Your argument is poorly formed this was an extremely obvious case of reading the plain language of the law as already quoted. Which court it arrived in isn't proof positive of the complexity of the law. Its proof of how irrational and diseased our system of government is.

Perhaps you would like to advance an actual argument insofar as abusing access to portable computing devices to virtually riffle through an individuals entire life without benefit of court or even articulable suspicion? This is an intrusion into an individuals privacy worse than anything the framers could have imagined justified only a by a right to search that exists to discover contraband THINGS not contraband thoughts.

Lest we forgot Texas had to have the supreme court tell them that they couldn't outlaw gay sex not that long ago.

gabelevi(10000) 7 days ago [-]

This was just a federal court, not the Supreme Court.

toby-(10000) 7 days ago [-]

This didn't go to the Supreme Court. A district court in Boston heard this case.

abtinf(3042) 7 days ago [-]

There are many obvious cases that have made it to the Supreme Court. All it takes is a pig headed state or federal attorney general who insists on trying to uphold oppressive laws.

ISL(1625) 7 days ago [-]

'The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.'

Sounds like the Court got it right.

solotronics(10000) 7 days ago [-]

Now do guns next!

csours(4008) 7 days ago [-]

I suspect you may know this, but there exist many locations and circumstances wherein various rights are suspended.


andrewla(3201) 7 days ago [-]

> unreasonable

interlocutor(4211) 7 days ago [-]

This decision only covers electronic devices. All other items can still be searched without suspicion or warrant at the border, see here: https://en.wikipedia.org/wiki/Border_search_exception

paulddraper(4026) 7 days ago [-]

To be fair, these searches were prior to entering the United States.

In the United States, such an act would clearly be against the 4th Amendment. It was less clear here.

cies(3833) 6 days ago [-]

So a govt agency does something illegal. To be honest, im not thrilled. They seem to be acting against the laws they claim to uphold all the time.

Do they get punished? Are they put under extra monitoring now?

This is not a democracy people, this is not 'trias politica'. Police brutality rampant, this merely being one more case. These court rulings seem rather useless, when the same govt keeps illegal torture prisons (Guatanamo) open, where people's lives are ruined without due process.

yqx(10000) 6 days ago [-]

I didn't study law, but as far as I understand, illegal is not the right word here. They (govt) do things for which there is no legal precedence all the time. The court could also have found no problem with these searches. Now that they did take issue with it if the searches would continue then yes, that would be illegal.

SamuelAdams(3875) 7 days ago [-]

> the government's suspicionless searches of international travelers' smartphones and laptops at airports and other U.S. ports of entry violate the Fourth Amendment

So, does this not apply to people flying domestically?

cdmcmahon(10000) 7 days ago [-]

I don't have a source at the ready, but my understanding is that the existing law and/or case law determined that traveling domestically entitled one to such protection. Various federal agencies have determined via rule making that such protections do not extend to international ports of entry/airports for reasons of national security. Those determinations are what have been struck down in this case.

dreamcompiler(4034) 7 days ago [-]

No. TSA doesn't search smartphones and laptops, and the searches they do apply to everybody. When you search everybody equally, courts have ruled it's not a 4th amendment violation. See drunk driving checkpoints.

t0ddbonzalez(10000) 6 days ago [-]

Solution: Go through border control with blank phone/laptop and restore from backup later.

daliusd(10000) 6 days ago [-]

That's workaround - not solution.

cmurf(1645) 6 days ago [-]

Is this what the EFF, ACLU, etc. actually recommend these days? Or do they want regular Joe Citizen to be willing to subject themselves to duress by customs?

If most people erase their devices, it's a kind of giving up. Accepting data loss (hopefully temporarily, but not everything is stored in the cloud, e.g. SMS messages, at least in Android phones with the original texting app) is punative. And disproportionate. And resets are a PITA for non-technical people. As a work around to avoid being harassed, to me it proves we've lost the privacy fight.

sandebert(2602) 6 days ago [-]

I was under the impression that going through with blank phone/laptop would in itself be considered suspicious. Is that not the case?

coldcode(3422) 7 days ago [-]

Good. Also given there is no one running Homeland Security I wonder if it's even likely that this will be pushed to the SC.

1MoreThing(10000) 7 days ago [-]

Don't confuse the lack of a Senate-approved director with a lack of someone in power at a government agency. There are very much people running the Department of Homeland Security.

theandrewbailey(2094) 7 days ago [-]

OK, so what stops the TSA/DHS from saying 'We suspect you are a terrorist; please unlock your phone'? Their actions have effectively said that for almost 20 years.

JumpCrisscross(46) 7 days ago [-]

> what stops the TSA/DHS from saying 'We suspect you are a terrorist; please unlock your phone'?

The same thing that stops all tyranny. A vigilant and engaged citizen body.

Practically, this means donating to the ACLU and related groups. It also means, if you're party to injustice, suing or reaching out to organisation who can help you sue to assert our rights.

mikece(687) 7 days ago [-]

Ask them to explain what their reasonable suspicion basis for wanting to search your phone is. Without reasonable suspicion it's, by definition, an unreasonable search.

dragonwriter(4218) 7 days ago [-]

"suspicionless searches" refers to searches without reasonable suspicion; absent particular evidence providing grounds for reasonable suspicion particular to the searched individual, any search would remain "suspicionless" as the term is used in 4th Amendment law; the test has both subjective (was there actual suspicion) and objective (was that suspicion reasonably grounded) elements.

undersuit(4210) 7 days ago [-]

Hopefully you don't need to visit grandma, the onus is now on you to defend your rights. Like Ghassan and Nadia Alasaad and the nine other people who resisted similar requests from the government.

brundolf(2156) 7 days ago [-]

Presumably this gives you a much stronger case if you then decide to sue. If enough people do that, it starts to become a deterrent.

csomar(961) 7 days ago [-]

I though the 'reasonable suspicion' needs to be documented and so is the search. Otherwise, this law might reduce the number of searches but not the abuse of it.

refurb(2501) 7 days ago [-]

Nothing. What happens is if the cause for the search was insufficient, the case gets kicked out in court.

Historical Discussions: John Carmack: I'm going to work on artificial general intelligence (November 13, 2019: 1569 points)

(1570) John Carmack: I'm going to work on artificial general intelligence

1570 points 6 days ago by jbredeche in 1151st position

www.facebook.com | Estimated reading time – 1 minutes | comments | anchor

Starting this week, I'm moving to a 'Consulting CTO" position with Oculus.

I will still have a voice in the development work, but it will only be consuming a modest slice of my time.

As for what I am going to be doing with the rest of my time: When I think back over everything I have done across games, aerospace, and VR, I have always felt that I had at least a vague "line of sight" to the solutions, even if they were unconventional or unproven. I have sometimes wondered how I would fare with a problem where the solution really isn't in sight. I decided that I should give it a try before I get too old.

I'm going to work on artificial general intelligence (AGI).

I think it is possible, enormously valuable, and that I have a non-negligible chance of making a difference there, so by a Pascal's Mugging sort of logic, I should be working on it.

For the time being at least, I am going to be going about it "Victorian Gentleman Scientist" style, pursuing my inquiries from home, and drafting my son into the work.

Runner up for next project was cost effective nuclear fission reactors, which wouldn't have been as suitable for that style of work. 😊

All Comments: [-] | anchor

somewhereoutth(10000) 5 days ago [-]

Mathematically speaking, current computers run in discrete (integer/countable) space, and the human brain exists in continuous (real/uncountable) space. Cantor showed that continuous space is larger than, and thus cannot be represented in, discrete space (via diagonal method). I suggest that AGI and consciousness lie in continuous space, and thus are unreachable with our current discrete computation model, regardless of how sophisticated we make it. There exists a cardinality barrier.

hvasilev(10000) 5 days ago [-]

I wonder how these people have decided it is possible with a digital computer, given that the brain is analog. I wouldn't be surprised if the first step through AGI is to actually crack the analog computer hardware problem.

shrimpx(10000) 5 days ago [-]

Unrelated to AGI, but quantum physics doesn't admit 'continuity'. Continuity is an abstraction. There are no real numbers in reality.

hans1729(10000) 6 days ago [-]

Heh, from the comments:

>Congratulations on the new project, and may your hubris not doom us all.

shurcooL(3331) 5 days ago [-]

I wanted to find some discussion on this topic. I'm not pessimistic, but I am curious to consider what it may mean for humanity if/when AGI happens.

kotrunga(3891) 6 days ago [-]

That's the very comment that stood out the most to me as well...

ses1984(10000) 6 days ago [-]

What a boss.

mc3(10000) 6 days ago [-]

And now, without a boss!

ausjke(662) 6 days ago [-]

he is 49 this year, considered as one of the most genius programmer on earth. There are professors still doing real work at 90+ year old(yes, the UT professor goodenough for Nobel prize), John has a long way ahead, best luck!

streetcat1(10000) 6 days ago [-]

Just out of curiosity, how do you define a genius programmer? (vs regular programmer).

chasd00(4221) 6 days ago [-]

i've heard he was pretty good, i didn't realize he was this widely respected and admired. I remember him for that fast inverse square root hack but that's about it.

thrower123(3005) 6 days ago [-]

I was wondering why his tweeting about graphics and VR went to 0.

We've gone back into the part of the cycle where VR is an odd curiousity again, haven't we?

randomidiot123(10000) 6 days ago [-]

Unfortunately VR has turned out to be an intensely nauseating puke fest for many of us.

randomidiot666(10000) 6 days ago [-]

Unfortunately VR turned out to be an intensely nauseating puke fest for a lot of people.

laxatives(4207) 6 days ago [-]

Is he doing this under the Facebook umbrella? Or departing? Or more-or-less retiring from regular obligations entirely?

k__(3307) 6 days ago [-]

He said that he will work as a consulting partime CTO for Oculus and do GAI from home, sounded like non-Facebool work to me.

Someone1234(4213) 6 days ago [-]

I think semi-retiring. He's going to be 'consulting [on Oculus]' while:

> I am going to be going about it "Victorian Gentleman Scientist" style, pursuing my inquiries from home, and drafting my son into the work.

Which to me reads like part time work on Oculus, part time work on this AGI project. If it is with Facebook it isn't at all clear from the post (plus I'd assume it would be accompanied by marketing copy in that situation).

hans1729(10000) 6 days ago [-]

>Starting this week, I'm moving to a 'Consulting CTO" position with Oculus.

I will still have a voice in the development work, but it will only be consuming a modest slice of my time.

As for what I am going to be doing with the rest of my time: [...] For the time being at least, I am going to be going about it "Victorian Gentleman Scientist" style, pursuing my inquiries from home, and drafting my son into the work.

piinbinary(3016) 6 days ago [-]

What's a useful definition of what AGI is? If you have a computer that does some clever AI stuff, what criteria do you look for to decide that it has a general intelligence?

olalonde(147) 5 days ago [-]

It's generally used to mean human level intelligence (e.g. an AI that can learn any task a human can). An often cited criteria is passing the Turing test.

lonelappde(10000) 6 days ago [-]

AGI means it can solve any problem posed to it and interface with any sensory and motor peripheral.

jacquesm(43) 6 days ago [-]

AGI is not an engineering problem but a research problem. John Carmack is good at putting stuff together but how good he is at coming up with novel concepts for an open research problem remains to be seen. Even the rocketry example that is hailed here as a success mostly wasn't. That doesn't make me happy, it would have been far nicer if Armadillo had succeeded, more competition in that space is better. But for all the work done it was more of an advanced hobby project along the lines of those guys in the Nordics than something that moved the needle scientifically.

onion2k(2004) 6 days ago [-]

Even the rocketry example that is hailed here as a success mostly wasn't.

This demonstrates how hard the problem is. When you're tackling a really hard problem 'mostly not a success' is a success. Most people faced with the same problem would return 'no successes'.

soup10(10000) 5 days ago [-]

Such pointless gatekeeping, if Carmack turns his attention and resources to AI he will be able to make contributions to the field. Will they be groundbreaking? Maybe not, but why is everyone so eager to immediately discourage him.

Ajedi32(1903) 5 days ago [-]

Sounds like he agrees with you:

> When I think back over everything I have done across games, aerospace, and VR, I have always felt that I had at least a vague "line of sight" to the solutions, even if they were unconventional or unproven. I have sometimes wondered how I would fare with a problem where the solution really isn't in sight. I decided that I should give it a try before I get too old.

drcode(3307) 6 days ago [-]

Carmack is someone who has proven to be an almost unequalled productivity machine when working on medium-difficulty problems... Now, for the first time, we'll see if his approach to problem solving can also work on a truly difficult problem. I agree it's very much an open question.

brainpool(4042) 6 days ago [-]

AGI is not a research problem but an imagination problem. I can't vouch for how good John Carmack is in imagination, but striving to put things together with a goal seems like a good place to start.

carlosdp(4219) 6 days ago [-]

Bringing up armadillo wasn't meant as an example of success in business but rather as an example of ability to dive into new fields effectively. They did some cool stuff in their short run, and there's an offshoot company still going.

But yea, I agree with your general point. I'd just note that having that ability to be insanely productive in working on things people haven't done before means to me that if it's possible for someone like him to really get good at this field, he's probably gunna do it.

Who knows how far you can get with just 'putting stuff together.' That's what Edison did.

irjustin(10000) 6 days ago [-]

I would argue that this is exactly the space he should be in.

AGI is a very active research position that arguably lacks the engineering/real-world arm that I believe Carmack could provide.

His early work in 3D graphics and math are supportive arguments for that. Research ideas turned into viable real-world systems.

Is it too early? I think us armchair hnews users can go back and forth all day. But in the end, we'll only find out the answer after the fact.

I hope all the best for him in this. I think this is a perfect space for him to fit int.

dkural(4173) 6 days ago [-]

Please take this as nothing more than my subjective opinion: I believe that humans don't have GI but HI - we have a 'world-view' that is very idiosyncratic to being human, which is essentially heuristics all the way down - in other words, I don't believe there is a magical novel concept that explains HI, but that it is a collection of party tricks that evolved over time, i.e. hacky engineered system.

agumonkey(877) 6 days ago [-]

Wasn't his shadow volume technique quite hard and original ? He's not just plumbing things it seems.

program_whiz(10000) 5 days ago [-]

Maybe I am a naysayer, but John Carmack is going against other companies and teams with not only billions in funding for hardware and data, but also with big staffs of experts.

Not only are people who have been in the research for a while more likely to have good ideas, but having the support of engineers to write tests and data wrangling, Neuroscientists to bounce ideas off of, and a whole bevy of support staff is just more likely to produce results.

I'm also not a big fan of the kind of hero worship of 'well he wrote Doom, so this should be a cakewalk.' I'm not saying he won't, but he probably won't. What am I missing here that everyone seems all hyped up about?

coolassdude6941(10000) 5 days ago [-]

His "competitors" are irrelevant. It's obvious that the existing approach (dl/rl) is a dead end, at least for AGI. So the idea of a possible genius working on a new approach with no monetary incentive is exciting.

nwsm(3931) 6 days ago [-]

Yann LeCun's [0] comment on the post:

Welcome to the club, John.

A word of warning though: There is no such thing as AGI. Reaching human-level AI is a good goal. But human intelligence is very, very specialized.

[0] https://en.wikipedia.org/wiki/Yann_LeCun?fbclid=IwAR2e9mzCqS...

michannne(10000) 4 days ago [-]

Sounds exactly like what I would say if my life's work revolved around the idea that AI boils down to how many neurons we can simulate

drcode(3307) 6 days ago [-]

LeCun is more accomplished and smarter than I will ever be, but his thoughts on the term 'AGI' just seem like dumb pedantry regarding word definitions to me.

chj(4043) 6 days ago [-]

Bold undertaking even for someone as distinguished as John Carmack. I can't help thinking that this is too broad a goal and requires more theoretical work than engineering. Maybe making some self-replicating robots that can mate and give offsprings would be less ambitious (perhaps I am underestimating as well).

jobigoud(10000) 6 days ago [-]

He just gave the general domain of the project, it would be weird to start this kind of semi-retirement on a very specific domain. I see this as the broad topic of interest that he will be investigating, but any actual research will be done into narrow fields, wherever some interesting idea hasn't been fully explored yet.

leesec(10000) 6 days ago [-]

Wow, there are a lot of people in this thread arguing whether or not John Carmack has the right skills to help AGI, or about the specifics of his knowledge.

Do you all realize you're arguing about nothing?

Good for him for doing something he seems excited about. Maybe we should all stop gossiping and go do something we're excited about too.

voxl(10000) 6 days ago [-]

What is actually being argued is the cult worship of a particular engineer. I am against cult worship, so find the fact that Carmack is entering this space simply not news worthy. Yet here it is.

Do I wish him the best of luck and hope he cracks the problem? Of course, all the same I would wish that of an upstart PhD student. Yet, the announcements of a brilliant PhD student attending a university to work on AGI is somehow not on hacker news.

This is cult worship of the personality Carmack has amassed, perhaps completely accidentally. When Carmack actually achieves something interesting let us discuss it then, not the mere announcement that he will try, as if that means anything. Read: it doesn't.

joe_the_user(3788) 6 days ago [-]

'Wow, there are a lot of people in this thread arguing whether or not John Carmack has the right skills to help AGI, or about the specifics of his knowledge.

Do you all realize you're arguing about nothing?'

Absolutely. I should mention why. AGI is an open field. AGI is opennest of open fields. Advances in deep learning tell quite little about what AGI will look like. We don't know if AGI will be a hundred incremental innovations from deep learning, ten deep advances from deep learning or five incredible advances with only a slight relation to deep learning. We don't know if it will just appear when 100 super-computers are hooked together or if a genius at home on their laptop could cobble it together. Sure, you could extrapolate and say compute has mattered more than theory, so far. But you could also say impressive things have been done but they haven't approached robust generality and there's something we're missing. Pick the approach but then you'll have to see if it's possible.


baddox(4166) 5 days ago [-]

If he announced that he is retiring and sailing around the world, I'm sure you'd find people asserting that his skills as a programmer couldn't possibly translate to being a world-class sailor. Consider for a moment that he might just be very interested in AGI and wants to work on it.

hombre_fatal(10000) 5 days ago [-]

Yeah, this comments section is a bunch of gossiping hens trying to get it on record that they don't think his retirement hobby will amount to anything. Pretty sad in what's presumably a community of fellow craftspeople and makers.

The human brain really can't handle the crippling adversity of a fellow human announcing an aspiration. Hopefully our AGI replacement can.

libraryatnight(3917) 6 days ago [-]

My thoughts exactly. This is a man with a long and brilliant career. Let him work on what he wants. This comment thread is ridiculous.

TeamSlytherin(10000) 5 days ago [-]

I've been excited about AGI for a while, mostly because it wasn't a very competitive field, until now.

phyzome(10000) 5 days ago [-]

Ah yes, AGI: The idea that eats smart people. https://idlewords.com/talks/superintelligence.htm

marvin(3813) 5 days ago [-]

Working towards more general AI technology is not what the author of this is criticizing. They're criticizing focusing on super-human AI rather than ethical problems of current AI. 'The idea that eats smart people' is just a meme at this point, normally quoted outside its context.

But even so, history will have to judge whether the author's statements were true regarding the risk of super-human AI. Or whether a lot of _quite_ smart people weren't smart _enough_ to realize that there was a real likelihood of this being possible to achieve faster most thought.

Also, using ridicule as a rhetorical technique isn't the most sound type of reasoning, regarding the author of your link ;)

prvc(10000) 6 days ago [-]

>For the time being at least, I am going to be going about it "Victorian Gentleman Scientist" style, pursuing my inquiries from home

Who will fund the necessary computing resources? If not FB, then he will surely be joining or starting a different org

haihaibye(10000) 5 days ago [-]

In Victorian times, a 'gentleman' was someone who had so much money they didn't need to work.

He's been a major shareholder in 2 companies that have been acquired (Id and Occulus)

Animats(2071) 6 days ago [-]

This is encouraging. If you're going to work on artificial general intelligence, a reasonable context in which to work on it is game NPCs. They have to operate in a world, interact with others, survive, and accomplish goals. Simulator technology is now good enough that you can do quite realistic worlds. Imagine The Sims, with a lot more internal smarts and real physics, as a base for work.

Robotics has the same issues, but you spend all your time fussing with the mechanical machinery. Carmack is a game developer; he can easily connect whatever he's doing to some kind of game engine.

(Back in the 1990s, I was headed in that direction, got stuck because physics engines were no good, made some progress on physics engines, and sold off that technology. Never got back to the AI part. I'd been headed in a direction we now think is a dead end, anyway. I was trying to use adaptive model-based control as a form of machine learning. You observe a black box's inputs and outputs and try to predict the black box. The internal model has delays, multipliers, integrators, and such. All of these have tuning parameters. You try to guess at the internal model, tune it, see what it gets wrong, try some permutations of the model, keep the winners, dump the losers, repeat. It turns out that the road to machine learning is a huge number of dumb nodes, not a small number of complicated ones. Oh well.)

fastball(4141) 5 days ago [-]
the_af(10000) 5 days ago [-]

> This is encouraging. If you're going to work on artificial general intelligence, a reasonable context in which to work on it is game NPCs.

I don't think so. Game NPCs don't need AI, which would be way overkill; they just need to provide the illusion of agency. I think for general AI you need a field where any other option else would be suboptimal or inadequate, but in videogames general AI is the suboptimal option... more cost effective is to just fake it!

iandanforth(4008) 5 days ago [-]

Hi John! Animats was very cool! As you know game physics still kinda sucks for this work. Unity/Bullet/MuJoCo are the best we have and even they have limited body collision counts. Luckily we've now got some GPU physics acceleration, but IMO it's not enough.

What we really need is a scalable, distributed, physics pipeline so we can scale sims to 1000x realtime with billions of colliding objects. My guess is that Google/Stadia or Unity/UnityML are better places to do that work than Facebook, but if Carmack decides to learn physics engines* and make a dent I'm sure he will.

Until our environments are rich and diverse our agents will remain limited.

*More, I'm sure his knowledge already exceeds most people's.

aruggirello(4075) 5 days ago [-]

As a side note, if we're living in a simulation [0], I'd really like to know who's 'real' vs. who's an AI bot out there...

[0] https://en.wikipedia.org/wiki/Simulation_hypothesis

nikki93(4220) 5 days ago [-]

Yeah I believe in this game / simulated world NPC idea too. To get the kind of complexity we want we either need sensors in the real world or interfacing in a virtual world that humans bring complexity to (probably both -- the humans are part of the sensing technology to start). Things like AlphaZero etc. got good cuz they had a simulatable model of the world (just a chess board + next state function in their case). We need increasingly complex and intetesting forms of that.

In some sense you can think of interfacing w/ the online world + trying to win attention to yourself as the kind of general game that is being played.

PeterStuer(10000) 5 days ago [-]

I did my msc thesis in AI back then writing a dedicated simulater for a specific robot used in autonomous systems research. You find that especially when trying to faithfully reproduce sensor signals you need to dive deep into not just the physics of e.g. infrared light, but also the specific electronic operation of the sensor itself.

But that kind of realism is not needed for all AGI research.

I also spent some years on using evolutionary algorithms to evolve control networks for simple robots. The computational resources available at the time were rather limited though. Should be more promising these days now that your commodity gaming pc can spew out in 30 minutes what back then took all the labs networked machines running each night for a few weeks.

tomaskafka(3857) 5 days ago [-]

I love the idea of using the Sims as a platform, as it's a place where it will be blatantly obvious that 'effective' AI without built-in ethics is repulsively inhuman.

goatinaboat(10000) 5 days ago [-]

Simulator technology is now good enough that you can do quite realistic worlds.

It is nowhere near good enough to avoid running into Moravec's Paradox like a brick wall as soon as you try and apply it outside the simulator.

anotheryou(4164) 5 days ago [-]

It however has a huge bias towards human-like ai. Maybe it's not smart to narrow down to copying us so quickly.

I mean: maybe it's more efficient to have it read all of wikipedia really well before adding all the other noisy senses.

simonh(4121) 5 days ago [-]

I don't think that approach is going to work. For any clearly bounded and delineated task, such as a game, the most optimal, lowest energy and lowest cost solution is not AGI but a custom tuned specialist solver. This is why I don't think Deep Blue or Alphago are paths towards AGI. They are just very highly advanced single-task solvers.

Now Alphago and it's implementation framework are much more sophisticated than Deep Blue. It's actually a framework for making single-task solvers, but that's all. The fact it can make more than one single-task solver doesn't making it general in the sense we mean it in the term AGI. AlphaGo didn't learn the rules of Go. It has no idea what those rules are, it's just been trained through trial and error not to break them. That's not the same thing. It's not approaching chess or Go as an intelligent thinking being, learning the rules and working out their consequences. It's like an image classifier that can identify an apple, but has no idea what an apple is, or even what things are.

To build an AGI we need a way to genuinely model and manipulate objects, concepts and decisions. What's happened in the last few decades is we've skipped past all that hard work, to land on quick solutions to specific problems. That's achieved impressive, valuable results but I don't think it's a path to AGI. We need to go back to the hard problems of "computer models of the fundamental mechanisms of thought."[0]


guelo(955) 5 days ago [-]

Yes but it sounds weird to me because Carmack has spent his whole life involved with games but has not been known for an interest in game AI before.

fastbeef(4187) 5 days ago [-]

> Imagine The Sims, with a lot more internal smarts and real physics, as a base for work.

Sounds like the start of a truly horrifying Black Mirror episode

iamleppert(4220) 6 days ago [-]

It must be hard to be someone like him. Always chasing the original high that made him famous, the enormous pressure to succeed must be nauseating.

Instead of retiring and relaxing and looking back on an impactful and lucky career, it says something about how powerful the original emotions were that led him to his current point.

He will do anything to get back to that state, that place in time, even sacrifice what are supposed to be the good years of his life stuck behind a screen.

neonate(2396) 6 days ago [-]

How do you know he isn't just doing what interests him? Not everyone wants to retire, especially highly creative people.

mentat(4091) 6 days ago [-]

He invented modern graphics as a practical problem by himself as the sole researcher. Given the tools at the time that may have been a harder problem.

randomidiot666(10000) 6 days ago [-]

> He invented modern graphics as a practical problem by himself as the sole researcher

That is a ridiculous exaggeration. Carmack was clever enough to gain ~1 year advantage in performance over his competitors for the Doom engine, using Binary Space Partitioning, which was first applied to 3D graphics in 1969, before he was born. The Quake engine got a significant performance boost from Michael Abrash, who is a specialist in code optimization.

justin66(2881) 6 days ago [-]

> He invented modern graphics as a practical problem by himself as the sole researcher.

No, he didn't, and that is not a claim that he would ever make himself.

sterlind(10000) 6 days ago [-]

Yes, a practical problem. The math behind computer graphics (i.e. optics) had been around hundreds of years. The trick was using numerical analysis to optimize and approximate on limited hardware.

We don't have the laws of AGI like we had the laws of optics (Asimov notwithstanding.) Tons of research effort was poured into the wrong avenues in vision (hand-tuned HoG, transforms, optical flow analysis) and ML (support vector machines, computational learning theory) until a chain of breakthroughs hit on the right mathematical approach for vision and supervised learning more generally.

We have some mathematical approaches to try with AGI (e.g. policy optimization/max-Q in reinforcement learning), but they equations are plagued with fundamental issues (e.g. reward sparcity, easily-gamed artificial objectives.)

Carmack optimized some very difficult equations when he worked on graphics, but in AGI we still don't have the right equations to optimize.

eggy(3992) 6 days ago [-]

Great, AGI from the guy who brought DOOM to many. Do we ever learn?

chrisco255(4167) 6 days ago [-]

DOOM 4: AGI confirmed.

The_rationalist(10000) 6 days ago [-]

I hope he'll go the hybrid symbolic and neural network way (causal and statistical), instead of just statistical.

AGI needs a type system...

I hope I'll achieve AGI before him but it's nice to know there's some real competition! (because, reader, there are almost 0 researchers seriously trying to achieve AGI in a not totally bullshit way. Only opencog and Cyc comes to mind).

ilaksh(3227) 6 days ago [-]

Are you really that sure that the approaches to increasing the generality of AI being taken by LeCun (self-supervised model learning), Hinton (capsule networks) and Bengio (state representation learning) are all 'total bullshit'?

cr0sh(10000) 6 days ago [-]

You forgot 'he who shall not be named'...

/ok, maybe his project falls under 'total bullshit'...

asadlionpk(2532) 6 days ago [-]

Do you have a goto resource to watch/read for someone new and kinda interested in the field?

thundergolfer(4222) 6 days ago [-]

Where and how are you working on AGI? Are you at opencog or Cyx?

K0SM0S(10000) 6 days ago [-]

> AGI needs a type system.

My brain bit on that remark; would you care to elaborate?

cf(3292) 6 days ago [-]

I expect John Carmack to follow a trajectory of someone like David Ha who with little previous background started to write very creative and thought provoking papers (https://scholar.google.com/citations?user=J1j92GsxVUMC&hl=en)

It won't be AGI by most definitions but I bet it'll be pretty cool and I'm happy to have that.

0-_-0(10000) 5 days ago [-]

Dammit, I had the hypernetworks idea recently but he already did it 4 years ago! Nothing new under the sun...

xamuel(4198) 6 days ago [-]

I've been dabbling in AGI and it seems like the field has a lot of low-hanging fruit. I'll bet Carmack can offer some significant contributions.

I'll take an opportunity to plug a paper I recently published on comparing relative intelligence. The punchline will illuminate the low-hangingness of the fruit in this field.

Suppose X and Y are AGIs and you want to know which is more intelligent. For any interactive reward-giving environment E, you could place X into E and see how much reward X gets; likewise for Y. If X gets more reward, you can consider that as evidence of X being more intelligent. But there are many environments, and X might do better in some, Y in others. How can you combine those pieces of evidence into a final judgment?

The epiphany I had (obvious in hindsight) is that the above situation is actually an election in disguise. The voters are interactive reward-giving environments, voting (via their rewards) in an intelligence contest between different AGIs. This allows us to import centuries of research on voting and elections! In particular, by using theorems about elections published in the 1970s, I was able to provide an elegant notion of relative intelligence.

The notion I provided is elegant enough that some theorems can even be proved with it, for example, formalizations of the idea that 'higher-intelligence team-members make higher-intelligence teams'. Which emphasizes the low-fruit-hanginess of the field: as obvious as that idea seems, apparently no-one was able to prove it with previous formal intelligence measures, probably because those previous intelligence measures were too complicated to reason about!

Here's the paper: https://philpapers.org/archive/ALEIVU.pdf

omalleyt(10000) 5 days ago [-]

Unfortunately, your intelligent agents qualify as optimization algorithms and therefore the No Free Lunch Theorem applies:


I.e. across the space of all possible environments, all agents perform equally well

__s(10000) 6 days ago [-]

This is a neat idea. Seems there's room to weigh value of electorate based on how liberally they praise. Thanks for sharing

michannne(10000) 4 days ago [-]

I have also been researching AGI and what I found over the years is that a large majority of the work involved requires knowledge of outside fields that have been studied for centuries. Metaphysics, epistemology, macro/microeconomics, geometry --- dozens of fields that you would never guess should ever be related to AI in any way are actually pivotal when it comes to AGI.

sytelus(312) 6 days ago [-]

Could you summarize what exactly interesting insights did you gained by casting this problem as the election system? Unfortunately abstract fails to communicate this (I hate 'teaser-only' abstracts!). Also, it might be easy to go from relative metric to sort of absolute by comparing against a 'starndard' agent, for example, a random agent.

noelsusman(10000) 6 days ago [-]

But the fact that there's so much low hanging fruit is kind of the point. AGI as a field is still in its infancy. It's pure research and will be for a long time.

mycall(10000) 6 days ago [-]

How did you rule out other attributes in the voting, such as charisma, political environment, society wellness, etc?

jaster(10000) 5 days ago [-]

You might be interested in the No Free Lunch Theorem (https://en.wikipedia.org/wiki/No_free_lunch_theorem).

For what I skimmed from your paper, it looks like the LH agents may be viewed as discrete optimization processes trying to optimize an objective/utility function across an infinite space of possible environments (infinite voters).

If it is the case, and if each environment vote has the same weight, you may be in a case of no free lunch, where the performances of all possible agents (including the random agent) will average to the same across all possible environments.

Or, to restate the above, for each environment in which an agent is doing well, it is possible to construct an 'anti-environment' where the agent is performing exactly as bad.

My personal opinion on the topic of AGI is that it is actually a case of NFLT.

solipsism(10000) 6 days ago [-]

For any interactive reward-giving environment E, you could place X into E and see how much reward X gets; likewise for Y. If X gets more reward, you can consider that as evidence of X being more intelligent.

That's an odd definition of intelligence. By that definition, a bird is more 'intelligent' than a human at the task of opening a nut. Seems like 'fitness' would be a much more appropriate term.

It seems especially strange to consider this work in the field of general intelligence. Nothing about what you just described is general. By this definition, a chess bot is much more intelligent than the average person. I don't think we'd say a chess bot has general intelligence.

topmonk(4202) 6 days ago [-]

All environments are not created the same, and therefore not equal. If you feed 100 environments that are nearly identical to each other but favor A, and 4 other environments that all favor B but are all dissimilar to each other, then although B is more generalized, A would still win.

Since you have no formal way to compare environments to each other, you can't prevent this from happening. Therefore you just pushed the subjectively as to which AI is smarter to which environments are chosen by the user to run against.

reilly3000(4198) 6 days ago [-]

Arguably he got started on this years ago. The story goes that Quake 3's AI was so sophisticated, that when left alone for 4 years on a bot vs bot server, the units learned pacifism as a self-preservation strategy. https://www.forbes.com/sites/erikkain/2013/07/02/quake-iii-a...

moogly(10000) 6 days ago [-]

Mr. Elusive (Jean-Paul van Waveren) was the guy who wrote the bot code for Quake 3: Arena, not Carmack.

hombre_fatal(10000) 6 days ago [-]

Seems like this would be easy to actually test instead of rely on a single anecdote. And the simplest explanation is a bug though I don't believe the story at all. There are probably multi-year-uptime bot servers in the wild right now.

Reminds me of when I was new to Quake 3 and found an amazing server: it was always full of players and full of non-stop action. I played with these people all the time after school. Nobody said anything, they were 100% business which was cool. I would often congratulate them on nice kills or commentate on my victories. Everyone was about the same skill level.

Eventually I realized I was playing on a server that simply filled empty slots with bots. I was the only human player.

b0rsuk(4211) 6 days ago [-]

And I read that Quake 3 AI operates on a fairly simple stack of objectives. Like, bot goes to pick up yellow armor. Bot sees enemy and starts shooting him. Bot takes damage and starts to retreat. Retreat is now the objective at the top of the stack. If the bot finds a medkit, he pops the retreat objective off stack, and goes to the next one - shoot enemy. If he happens to kill the enemy, he resumes the 'get yellow armor' objective.

Qu3tzal(10000) 6 days ago [-]

'learned pacifism as a self-preservation strategy' this is an interpretation, not what the bot intended to do, in other words, you're reading into it

xeeeeeeeeeeenu(4131) 6 days ago [-]

That story is fake, its source is 4chan. Carmack himself debunked it[1]

[1] - https://twitter.com/id_aa_carmack/status/352192259418103809

foobiekr(10000) 5 days ago [-]

I've worked with legends in a specific space that is less consumer-y than games and graphics and less weirdly-desperate attention-seeking than ML and so the people therein are every bit Carmack-level but less visible. As they have aged out and wealthy'd out of working time, almost all of them have chosen to retire.

These are extremely high performing individuals who have made global impact. Shutting down for people like this is very hard and 100% of them have sent out mails much like Carmack's Facebook post when the end came. Even the style and verbiage are similar.

None of them made a dent in their tinkering-phase project and moved on to normal above-average low-engagement hobbies. They are done.

I read his FB post as a pretty standard retirement announcement as a result. I think he's telling us he is done.

jjoonathan(10000) 5 days ago [-]

I don't see any reason to be so dismissive.

Scientific pursuits have an extremely steep risk profile and are systematically underfunded because nobody knows how to capture the value that comes out of them. If someone wants to chase one on their own dime, we should celebrate that contribution to society rather than dwell on the fact that the median (even 99th percentile) outcome is that the project goes nowhere and the person continues on to retirement proper. The mean outcome could be very different and the benefit of the doubt costs us nothing, so why not give it?

mav3rick(10000) 5 days ago [-]

You cant project your colleagues on a third person.

staz(3818) 5 days ago [-]

by that standard Carmack retired a few times already and always came back with something interesting

carlosdp(4219) 6 days ago [-]

This doesn't surprise me at all. He went on a week long cabin-in-the-middle-of-nowhere trip about a year ago to dive in to AI (that's all this guy needs to become pretty damn proficient). (edit: I'm not claiming he's a field expert in a week guys, just that he can probably learn the basics pretty fast, especially given ML tech shares many base maths with graphics)

As recent as his last Oculus Connect keynote, he extolled his frustration with having to do the sort of 'managing up' of constantly having to convince others of a technical path he sees as critical. He's clearly the type that is happiest when he's deep in a technical problem rather than bureaucracy, and he likes moving fast.

On top of that, he likes sharing with the community with talks and such, and ever since going under the FB umbrella, he's had to clear everything he says in public with Facebook PR, which clearly annoyed him.

He's hungry for a new hard challenge. VR isn't really it right now since it's more hardware-bound by the need for hard-core optical research than software right now. With the Quest, he (in my opinion) solidified VR's path to mobile standalones. It's time to try his hand at another magic trick while he's on his game.

John's the very definition of a world-class, tried and true engineer/scientist. He's shown time and time again the ability to dive into a field and become an expert very quickly (he went from making video games to literally building space rockets for a good bit before inventing the modern VR field with Palmer).

If there's anyone I'd trust to both be able to dive into AGI quickly and do it the right(tm) way, it's John Carmack.

cyberjunkie(10000) 6 days ago [-]

I can trust John Carmack's words when he says in an interview, or on stage. There's a passion in his talks, a nervousness in blurting what he really feels, and those are really good traits, in my mind.

I genuinely felt a sense of disappointment when he moved to Facebook (via the Occulus acquisition). So yea, fuck you, Facebook and your manipulative, life values corrupting and PR machinery.

I place John Carmack miles above Zuckerberg.

throw231(10000) 6 days ago [-]

> about a year ago to dive in to AI (that's all this guy needs to become pretty damn proficient). (edit: I'm not claiming he's a field expert in a week guys, just that he can probably learn the basics pretty fast, especially given ML tech shares many base maths with graphics)

To be honest anyone who has a very good working knowledge of Linear Algebra can learn much of ML-math in a day. There really isn't anything mathematically super-sophisticated that is in popular use today.

AccordingDay(10000) 6 days ago [-]

But there's 100s of world-class researchers working on this problem already.

anonytrary(4070) 5 days ago [-]

AI today is comparable to physics in the 1700s. Back then, it was a bunch of people tinkering with prisms and apples. Today, it's a bunch of people tinkering with hyperparameters. I suspect that we know as little about AGI today as someone in the 1700s knew about QFT. Not only did they not know about QFT, but they didn't even know that they didn't know it.

deegles(309) 5 days ago [-]

If he invents (births?) an AGI, will it be Facebooks property? Sounds like the beginning of a dystopian novel.

drongoking(10000) 6 days ago [-]

> (edit: I'm not claiming he's a field expert in a week guys, just that he can probably learn the basics pretty fast, especially given ML tech shares many base maths with graphics)

This may be his biggest impediment. ML has gotten very far with looking at problems as linear algebraic systems, where optimizing a loss function mathematically yields a good solution to a precisely defined (and well circumscribed) classification or regression problem. These techniques are very seductive and very powerful, but the problems they solve have almost nothing in common with AGI.

Put another way, Machine Learning as a field diverged from human learning (and cognitive science) decades ago, and the two are virtually unrecognizable to each other now. Human learning is the best example of AGI we have, and using ML tech as a way to get there may be a seductive dead end.

mortenjorck(1098) 6 days ago [-]

Carmack is unquestionably a genius, but I think it's quite unlikely his solo work in a new domain will leapfrog an entire field of researchers.

I wouldn't, however, bet against some kind of insanely clever development coming out of his new endeavor. Something like an absurdly efficient new object classifier, that reduces the compute requirements for self-driving cars by a non-trivial factor, would be a very Carmack thing.

throwaway34241(10000) 6 days ago [-]

> he extolled his frustration with having to do the sort of 'managing up' of constantly having to convince others of a technical path he sees as critical

Yes, he seemed to put a lot of effort to try to get things through FB internal politics, and not always successfully. I really wish his experiments with a scheme-based rapid prototyping environment / VR web browser had been allowed to continue [1]. VR suffers from a lack of content, and VR itself is well-suited to creating VR content, and his VR script would surely facilitate closing that loop among other things. Although now four years later I guess FB has a large team working on a locked-down, limited world building tool (closed platform, no programming ability). Oh well.

I don't think this is the end of this wave of VR, but at this point I wouldn't be at all surprised if say Apple or someone else ends up bringing it to the mainstream instead of Facebook. [2]

[1] https://groups.google.com/forum/#!msg/racket-users/RFlh0o6l3...

[2] https://www.theverge.com/2019/11/11/20959066/apple-augmented...

buboard(3489) 5 days ago [-]

I dont know of his credentials as a scientist or mathematician to advance the field. But, he seems to be a ruthless optimizer, which can often leads to great leaps , even as a side effect. Neural networks are not difficult mathematically for any scientist to grasp really. And they are in actual need of compression and optimization. People are spoiled with general purpose tools that are not very efficient, even if computation is cheap.

LegitShady(10000) 5 days ago [-]

It's funny people talk about Palmer and carmack in vr but Oculus was built on appropriated valve tech and neither Palmer or Carmack have succeeded in making VR a thing.

As far as I can tell Carmack is an old engineer whose name gets thrown around for headlines. If there weren't articles about his stealing stuff to take to Oculus I don't think his presence there would be observable.

Now people are talking like Carmack switching topics is going to change the world. It's just going to change his schedule. There are smarter engineers already working on this problem.

nradov(886) 6 days ago [-]

Current ML technology probably has little or nothing to do with whatever technology will eventually be needed to produce true AGI.

tomaskafka(3857) 5 days ago [-]

> he extolled his frustration with having to do the sort of 'managing up' of constantly having to convince others of a technical path he sees as critical.

Sigh. I assumed the whole point of hiring John Carmack is that you trust him to identify critical problems - and to find the best way to solve them.

mochomocha(4191) 6 days ago [-]

> He went on a week long cabin-in-the-middle-of-nowhere trip about a year ago to dive in to AI (that's all this guy needs to become pretty damn proficient).

You must be joking, right? I'm as much of a Carmack fan as anyone here, but overstating the skills of one personal hero does no good to anyone.

randomsearch(4204) 5 days ago [-]

What scientific work has Carmack done?

friendlybus(10000) 6 days ago [-]

He had a lot of help behind the scenes and has been credited with things that aren't his. I respect his achievements more as a regular smart guy than a bonafide genius. He described the math in rocketry as being basically solved in the 60s and video games being far more complex as a project, so that was really a step down in difficulty. His VR role is the same field as his primary skills, impressive work but not an entirely unique role.

I'm glad to see he's aiming big with his billions and time. This is what rich people should be doing. Hl3 Gaben!

Zod666(10000) 6 days ago [-]

'With the Quest, he (in my opinion) solidified VR's path to mobile standalones'

Yes and I really wished he hadn't. Before he joined oculus they were working on the rift2, he steered them away from that to focus on mobile efforts.

I do see the appeal of mobile vr but at the end of the day it is basically an android phone in a vr headset.

PCvr is already 2 big steps back in graphical quality from desktop games. Mobile vr is like 10 steps back. 8 more steps than I'm willing to take even if it affords me mobility.

braindeath(10000) 6 days ago [-]

> especially given ML tech shares many base maths with graphics

I don't put learning state of the art ML past Carmack, at all. However, does ML tech of today lead to general AI? It's a strong assumption.

2OEH8eoCRo0(10000) 6 days ago [-]

I'm ready for this post to disappear, strange things start happening in cyberspace, and Carmack abruptly cans the project and doesn't want to talk about it.

zelly(10000) 6 days ago [-]


ekianjo(323) 6 days ago [-]

Other way to read this: big corporations are slowing down on VR. The market has not taken off as rapidly as they expected so we will see more moves away from heavy investment in VR.

baddox(4166) 6 days ago [-]

Carmack never struck me as the type to chase the money from fad tech to fad tech. On the contrary, he seems keen to chase his own interests, and by now I suspect he is financially secure enough to do so.

narrator(10000) 6 days ago [-]

Humans optimize for activating the opioid receptor. These receptors are distributed all over the brain and tied into all sorts of subtle neural networks. That's why opioid addicts don't do much when they're high. As far as the entire structure of the brain is concerned, an opioid addict's brain is done optimizing and the fitness function is pegged at 1.

I think an AGI will end up being like an AI that plays the Sims except we're the Sims and it's optimizing for our happiness probably by remotely monitoring our opioid receptor activation and some parameters of general health.

sneak(3056) 6 days ago [-]

So, opiate synthesis and delivery?

stevenwoo(3723) 6 days ago [-]

Isn't this backwards a bit or did I get it wrong - not sure if I am phrasing this right. But to simplify from a laymen's perspective a reading of The Selfish Gene - all creatures will work to continue their genetic line unless misdirected as you describe. So if in some way continuing their genetic offspring is something that activates their opioid receptor, that's what we'll do, creating and raising offspring or as grandparents/cousins/aunts/uncles, helping raise relatives. This can be applied generally to all life on earth.

chillacy(10000) 6 days ago [-]

I must be doing a poor job at being a human, given that I have passed the opportunity to activate my opioid receptors several times in my life so far (leftovers from surgeries).

yters(2024) 6 days ago [-]

He should first work on figuring out if AGI is possible. Why assume the human mind is computable?

jobigoud(10000) 5 days ago [-]

That's exactly what a scientist would say. Engineers invent things because nobody told them they were impossible.

phillco(4037) 6 days ago [-]

This is a very artfully written statement. It avoids the most more devastating headline of "John Carmack Stepping Down". (Yes, going part-time isn't the same as leaving entirely, but still).

nothis(10000) 6 days ago [-]

The real headline is that VR is finally confirmed to have hit a major roadblock. If John Carmack gets frustrated with its progress, there's something up.

I've long thought that the issue with VR is a conceptual one, not a technical one and maybe that frustration comes from there. 'Running forward' is an unsolved problem in room scale VR. For a seated experience, you're basically back to a neat display gimmick + accurate hand tracking.

Any real solutions need, on the one side, real-world physical constructions (think running threadmills) that soon hit holodeck-level limitations and, on the other, software that actually benefits from the real technology VR brings to interactive media: super accurate hand- and head-tracking. The first gets impractical/impossible soon, the second limits development to a few niche genres: Shooting ranges, cockpit sims, dance/party games and some vague 'experiences' where the actual tech is pretty much ignored and you just say 'but it feels so immersive!' (honestly, it does work for horror games!). It's basically motion controls 2.0.

The only place I could see the technology shine is, oddly enough, AR. It has way less mainstream hype to it but it makes much more sense because you actually benefit from the tracking of your real-world movement: You're still a part of it! The holo-lens demos that pop up on youtube might seem clumsy, but I can totally see a use case for replacing physical monitors with arbitrarily sized and positioned displays you can virtually move in any office space. There's rumors of Apple working with Valve on AR tech. If there's any technology that could follow the smart phone, AR is my bet. I'm honestly surprised Carmack didn't move in that direction rather than deciding to become a general AI guru.

wpdev_63(10000) 6 days ago [-]

Crap! When I thought VR was hitting its stride with the oculus quest.

For anybody who hasn't played an untethered VR experience, I highly recommend it. It makes a world of difference with games like Echo combat and Beatsaber. Tons of fun. It reminds me of the first time I played wii bowling.

brundolf(2156) 6 days ago [-]

More like 'John Carmack has well-balanced life priorities and has decided to do a really cool moonshot project with his son instead of making even more money'

daenz(538) 6 days ago [-]

Roko's basilisk[0] will remember this.

0. https://rationalwiki.org/wiki/Roko's_basilisk

mactyler(10000) 6 days ago [-]

Carmack might just be a basilisker after all!

antisemiotic(10000) 5 days ago [-]

I wonder what would happen if Carmack teamed up with Yudkowsky and his cult; Carmack's practicality with the imagination of people who non-facetiously talk about acausal deals with mulitiversal AIs could be a match made in heaven.

lazyjones(4140) 6 days ago [-]

Let's hope Boston Dynamics doesn't hire him. ;-)

red2awn(10000) 6 days ago [-]

Boston Dynamics isn't doing AGI.

xwdv(10000) 6 days ago [-]

Imagine inventing artificial general intelligences and then there's some public outcry for products powered by natural, organic intelligence instead.

0xdeadbeefbabe(3488) 6 days ago [-]


unityByFreedom(4218) 5 days ago [-]

One could imagine him beginning to work in gaming AI, but he says he wants to work on AGI, not simply gaming AI.

I'm personally not convinced that it is encouraging when someone bright sets their sights on AGI, particularly someone who appears to have never competed on Kaggle. It screams hubris.

de_watcher(10000) 5 days ago [-]

Artificial Gaming Intelligence (AGI).

bitexploder(4222) 5 days ago [-]

I am not convinced current AI is the approach to an AGI. I think it is at least feasible someone outside of this sphere has a reasonable shot at it. It feels like many AI researchers get caught up in refining existing techniques that amount to fancy statistics algorithms and data crunching, but not AGI. Current AI techniques may be synthesized or used in part in some AGI but it's clear there is a revolutionary step to be made. Kaggle is almost just an optimization fest, and not really advancing towards AGI.

vecplane(10000) 6 days ago [-]

I don't understand how he could contribute to the field of AGI research from home, by himself, and maybe with his son. It's the kind of problem that requires incredible amounts of data, hardware, and theory to make any progress.

Wouldn't it make more sense for him to join a cutting-edge team, like DeepMind or OpenAI?

DoctorOetker(4202) 6 days ago [-]

the recent result on 15% optimal learning error rate for binary classification, could have been derived with pencil on paper by anyone...

Impossible(287) 6 days ago [-]

Carmack built rockets and id bought $100Ks of NeXT machines to make Doom so I wouldn't put it past him to have incredible amounts of hardware... even at home. Considering his position at Facebook and that he is industry famous he probably has access to data and cloud resources that a researcher outside of OpenAI, Nvidia, Google, etc. normally wouldn't have access to. He could also raise money relatively easily to pursue more intense research.

whalebird(10000) 6 days ago [-]

> It's the kind of problem that requires incredible amounts of data, hardware, and theory to make any progress.

I wouldn't be surprised if the opposite was true, at least with the theory part. AI didn't really go anywhere for decades, because people focused too much on theory.

Otherwise, there's a lot of data and hardware at your disposal, even from the comfort of your home.

> Wouldn't it make more sense for him to join a cutting-edge team, like DeepMind or OpenAI?

You mean they guys that are training with videogames that people like John developed?

throwawayhhakdl(10000) 6 days ago [-]

Deepmind and openai are probably not on a reasonable track to AGI. IMO if we ever make an AGI it wont actually be especially good at things. An AGI, like humans, would probably be pretty bad at math naturally. You could get one to be great at math, but first getting great at math and backing into general intelligence is probably impossible.

Substitute math for anything you want.

dekhn(10000) 6 days ago [-]

Everything I've ever read about Carmack suggests he'll do his best on his own at home. Much of this work can be done on reasonable hardware, and he's always been really good at getting a lot out of reasonable hardware. Further, if he needs enormous compute resources, he can get it at any of several cloud providers.

pg_is_a_butt(10000) 6 days ago [-]

Might be a little burnt out from the 'cutting-edge team' at Oculus who painted him into a corner with technology that could never work, as it made every user sick.

thundergolfer(4222) 6 days ago [-]

It might make sense for him to join a team like DeepMind, but we could guess that the 'working from home by himself' bit was a lifestyle change he wouldn't compromise on.

drcode(3307) 6 days ago [-]

I think many people expect that a lot of the missing 'special sauce' for AGI (if anyone can figure it out at all) is going to be something for which massive GPU power isn't a key factor.

johnsimer(10000) 6 days ago [-]

My view is that you want many people working independent from each other towards the same goal, and that everyone working in one group could hinder creativity/lead to groupthink

nickjj(2568) 6 days ago [-]

John Carmack is practically a machine.

He's openly talked about his work ethic in a bunch of places. He's the type of guy who after a life time of coding calculated he's 100% efficient up until 13 hour work days and then he drops off[0]. Although he did mention working those long hours is often best working on multiple things instead of 1 topic but maybe with AGI there's a bunch of different avenues to explore.

[0]: https://www.youtube.com/watch?v=udlMSe5-zP8&t=4773

nikkwong(3608) 6 days ago [-]

I'm sure he knows his stuff, and is venturing into this knowing that there's somewhat of a 'hard problem' to be solved here; serious proponents of AI like Kai-Fung Lee have stated that AGI may never be possible.

The fact that he's making a public statement like this leads me to believe he may already have some novel solutions on how to tackle the problem. We won't be expecting to be seeing the same old parlor tricks coming out of John Carmack. He is already on the forefront of this stuff after all.

That's exhilarating but also terrifying. Our still-barbarian level human systems are still nowhere near ready to deal with the socioeconomic problems that may arise with AGI.

Voloskaya(4222) 6 days ago [-]

> leads me to believe he may already have some novel solutions on how to tackle the problem.

> That's exhilarating but also terrifying. Our still-barbarian level human systems are still nowhere near ready to deal with the socioeconomic problems that may arise with AGI.

I think you are reading way too much into his statement. It's extremly unlikely that he just magically figured out a way to tackle the problem (just knowing where to get started would be massive).

rhacker(4201) 6 days ago [-]

I think we just got 50% closer to making AGI happen.

ekianjo(323) 6 days ago [-]

If your odds were very low to begin with 50 % more wont make much of a difference.

soulofmischief(4222) 6 days ago [-]

I'm happy. Carmack has done a lot of good with Oculus, but it hurt seeing my hero working for Facebook. I understand the need to remain in a consulting position, but at least now he's no longer under the corporate leash.

If anyone can hack AI, it's Carmack, and so when I read this headline I had a moment of fright thinking this meant Carmack was working on AI for Facebook.

Voloskaya(4222) 6 days ago [-]

> If anyone can hack AI, it's Carmack

Pretty sure you don't get to AGI with some hacks.

bronz(3622) 6 days ago [-]

i say this with respect and humility, but i am very surprised at the naivete with which John addressed the subject of AGI. he is so casual about it -- not only the idea of working on it but also the idea of it existing at all. he seems oblivious to the gravity of that discovery. it is not just 'very valuable,' it will be earth shattering and probably wipe out humanity. and its his side-project. and his son will help out.

John is the perfect representation of what is wrong with peoples attitude toward AGI. aloof and naive.

asadlionpk(2532) 6 days ago [-]

I think the real naive are the people who are doing it as a day job.

McTossOut(10000) 6 days ago [-]

I'll level with you, if all I had heard was soundbites, I'd be skeptical myself; it's a bit like Neil DeGrasse Tyson telling you he's going to unify gravity or something. This guy can go and adlib a 2 hour talk about implementing subsurface scattering, and build it from the ground up in a commercially viable way.

This dismal, dimwitted 'advanced' filter and sort industry has recently started training all their employees in and vomiting all over every consumer with is nothing, worthless, and at best lunacy.

All it takes is patience, know-how, and insight. For that, Carmack fits the bill.

randomidiot123(10000) 6 days ago [-]

He might as well casually work on Faster Than Light travel, or a Grand Unified Theory.

Bizarro(10000) 6 days ago [-]

Stop staying up late to watch DUST on Youtube.

randomidiot666(10000) 6 days ago [-]

He might as well casually work on Faster Than Light travel, or a Grand Unified Theory.

whamlastxmas(4132) 6 days ago [-]

What is your source for saying AGI will probably wipe out humanity? How could we ever even attempt to guess at the motivations of something we can barely comprehend and doesn't even exist yet?

ageofwant(3853) 6 days ago [-]

You are saying 'probably wipe out humanity' is a bad thing ?

I know of several million species that would strongly disagree. Especially if AGIv1 decides to tune the genetics of say most mammals to append 'sapient' to the end of their species name.

Perhaps more constructively consider that AGI is simply the next iteration of 'humanity', yea sure the old versions are redundant anarchisms and apart from some living reserve specimens functionally extinct, but nobody cares as you can sim one up at almost no cost.

K0SM0S(10000) 6 days ago [-]

I'll say this: I'd prefer if the brightest minds approached the matter from the 'AI safety' angle (a sub field concerned with building not just AI but 'safe' AI, ie that we can control or understand in a practical manner).

Because that's really where the line of human history will be drawn if AGI and above becomes real. AI safety, how advanced we are in it, will directly map to civilization's progress or endangerment as a result of AI.

Edit: this is already true with regards to 'psychological safety' from undue influence or outright manipulation with motive (usually financial) by current 'ANI' algorithms (newsfeeds, 'recommendations', ads, etc). It's a real topic that reduces to human psychological freedom, freewill. It's a BIG topic.

zelly(10000) 6 days ago [-]

It would be easier to take the AI doomsayers' seriously if we were remotely close to AGI. For now it's treated the same way as some guy in a cape in Central Park trying to summon Satan. No one cares because everyone knows it's basically impossible.

AnimalMuppet(3739) 6 days ago [-]

> and probably wipe out humanity.

Probably? I do not think that word means what you think it means... or else I don't think the balance of probability lies where you think it does.

umvi(4218) 6 days ago [-]

There's no such thing as AGI in my opinion. There is no way to create a 'conscious' machine. We might be able to come up with some reasonably impressive imitations, but nothing that is conscious or actually thinking like a human.

ralusek(4210) 6 days ago [-]

Your opinion is very likely incorrect.

qwerty456127(4199) 6 days ago [-]

We don't have to. A general AI doesn't imply consciousness inner experience.

77pt77(10000) 6 days ago [-]

So what goes beyond physics in the human brain?

Because we can simulate physics.

wahern(10000) 6 days ago [-]

If you can explain precisely why we cannot build a conscious machine, you could become one of the most revered, and potentially wealthiest, researchers in the world, as I presume such a proof would necessarily introduce unknown and very useful science.

I personally don't believe modern machine learning is remotely close to AI, except perhaps the very lowest rung of the ladder of self-serving AI definitions. I base that belief on what seem to be the unknowns, reinforced by predictable failures[1]. But I have very little reason to believe it's impossible. Not even the possible necessity of quantum effects would seem to preclude it. Heck, we've already begun harnessing quantum effects in materials science, computing, biology, and other areas.

Unless you mean that whatever we could eventually come up with would be more biological than machine or that only a human could think like a human, but that seems more like word play, the kind of game AI believers play. (That said, that poses an interesting question: which is more likely to be achieved first--a designed-from-scratch, DNA-based cellular intelligence, or something not based on DNA or otherwise mimicking existing organic life? If at all, of course. Also presuming such a distinction isn't in fact hopelessly quaint and naive.)

[1] I'm not a naysayer. While I never believed self-driving cars were around the corner (not even 5 or 10 years out; you can Google my HN comments from years ago), I have no doubt the science has been useful and can and will and is put to great, largely unseen use, as is typical of most science.

yaseer(3916) 6 days ago [-]

John Carmack seems like a guy that would've made fundamental contributions to science, had he been born a century before.

Computer science now occupies the place physics once did, in its impact on moving the world forward.

Best of luck to him, I look forward to seeing what he produces!

ekianjo(323) 6 days ago [-]

> John Carmack seems like a guy that would've made fundamental contributions to science, had he been born a century before.

I see Carmack as a very(as in uniquely) talented Engineer. Usually, engineers are not the type who do very well in pure research topics. And AGI is certainly a pure research topic, since we don't have a clear leading us there. So while it's great to see he is interested in it, now sure if we should have any kind of expectation there.

JohnJamesRambo(4155) 6 days ago [-]

You guys really get high on your own supply don't you? It's a toss-up these days whether something from computer science or a programmer will either enhance or erode the human experience and quality of life.

glofish(10000) 6 days ago [-]

why would being born in the current century preclude one from making fundamental contributions?

You are making a common mistake of assuming that just because someone is good at something like programing computers the same skill would translate identically to a completely different domain.

If anything he is lucky to have been born in an era where his skill of programming computers could be put to use - otherwise his talents may have gone to waste, he may have ended up toiling fields his talent untapped and undiscovered, like that of millions before him.

pyentropy(4153) 6 days ago [-]

Progress in AI is due to data and computational power advances. I wonder what kind of advances are needed for AGI.

1. Biological brains are non-differentiable spiking networks much more complicated than backpropagated ANNs.

2. Ion channels may or may not be affected by quantum effects.

3. The search space is huge (but organisms aren't optimal and natural selection is probably local search)

4. If it took ~3.8b years to get from cells to humans, how do we fast-forward:

* brain mapping (replicating the biological 'architecture')

* gene editing on animal models to build tissues and/or brains that can be interfaced (and if such interface could exist how do we prevent someone from trying to use human slaves as computers? Using which tissues for computation is torture?)

* simulation with computational models outside of ECT (quantum computers or some new physics phenomenon)

Note: those 3.8b years are from a cell to human. We haven't built anything remotely similar to a cell. And I'm not claiming that an AGI system will need cells or spiking nets, most likely a lot of those are redundant. But the entropy and complexity of biological systems is huge and even rodents can outperform state of the art models at general tasks.

IMHO, the quickest path to AGI would be to focus on climate change and making academia more appealing.

JaRail(10000) 6 days ago [-]

> Progress in AI is due to data and computational power advances.

I think you'd be surprised how much progress is also being made outside those two factors. It's sort of like saying graphics only improve with more RAM and faster compute. We know there's more to it than that.

In many cases, the cutting edge of a few years ago is easily bested by today's tutorial samples and 30 seconds of training. We're doing better with less data and orders of magnitude less compute.

m0zg(10000) 6 days ago [-]

You forgot to mention, crucially, that neurons in close proximity affect each other, which is just one of the things that makes modeling of more than a few neurons in time domain a complete non-starter. It all results in enormous systems of PDEs which we don't know how to solve yet at all. You could say that we do not have the right mathematical apparatus to model any such thing.

debt(4089) 6 days ago [-]

We still haven't solved language nor intelligence.

Like what is language, what is intelligence? Some of the smartest linguists and philosophers would proudly declare they have no fucking clue.

Making Alexa turn on the lights or using Google Translate are cool party tricks though.

Idc how many Doom games ya made, but I'm sorry to say a bunch of software engineers aren't gonna crack this one.

andbberger(4169) 6 days ago [-]

> 1. Biological brains are non-differentiable spiking networks much more complicated than backpropagated ANNs.

Actually it's not so obvious that the brain is not differentiable. If you do a cursory search, you'll find quite a lot of research into biologically plausible mechanism for backpropagation. Not saying the brain does backprop, we just don't know and it's not outside of the realm of plausibility

catalogia(10000) 6 days ago [-]

From what I understand, quantum effects being essential to the process is a fringe belief. Penrose is probably the most famous 'serious person' (sorry Deepak Chopra) to espouse the idea, but I'm inclined to believe that might be a Linus Pauling/Vitamin C sort of scenario. Penrose started from the perspective of believing there must be quantum effects, then began fishing for physical evidence of it.

wahern(10000) 6 days ago [-]

I'm sure neural nets will herald AI right after the mechanical gears and pneumatic pistons that were envisioned as the secret sauce during the turn of the last century.

The key, of course, is redefining life and intelligence as whatever the current state-of-the-art accomplishes. (Cue explanations that the brain is just a giant pattern matcher.) It makes drawing parallels and prophesying advancements so much easier. Of all our sciences, that's perhaps the one thing we've perfected--the science of equivocation. And we perfected it long ago; perhaps even millennia ago.

air7(3724) 6 days ago [-]

This reminds me of a interesting armchair moral dilemma: Assume we have the tech to replicate/simulate a biological brain. Now say we want to study the effects of extreme pain/torture etc on the brain. Instead of studying living animals or humans we'd just simulate a brain, and simulate sending it pain signals and see what happens.

But, if this is a 100% replicated brain, doesn't that mean its suffering is just as real as a real brain's suffering, and therefor just as cruel? And if not, what's the difference?

Balgair(1959) 6 days ago [-]

> 2. Ion channels may or may not be affected by quantum effects.

In a sense, everything is affected by quantum effects. However, with neurons, they are generally large enough that quantum effects do not dominate. Voltage gated channels are dozens to hundreds of amino-acids long. Generally, there are hundreds to millions of ion channels in a cell membrane and the quantum tunneling of a few sodium ions in or out of the cell will generally not affect gestalt behavior of the cell, let alone a nervous system's long term state. Suffice to say, ion channels are not dominated by quantum behavior.

Largely, we have the building blocks to replicate neurons (as we currently understand them) in silico. However, as is typical with modeling, you get out what you put in. Meaning that how you set your models up will mostly determine what they do. Setting your net size, the parameters of you PDEs, boundary values, etc. are the most important things.

Now, that gets you a result, and it's likely to take a fair bit of time to run through. To get it up to real time the limiting factor really ends up being heat. Silicon takes a LOT of energy as compared to our heads, ~10^4 more per 'neuron'. If we want to get to real time, we're gonna need to deal with the entropy.

akira2501(10000) 6 days ago [-]

> gene editing

Gene expression is often tied to the environment the organism is in. Mere possession a gene isn't enough to benefit from it. Some expressions don't take effect immediately, but rather activate in subsequent generations.

Epigenetics is a whole equally large layer on top of this system. A single-focus approach may not be sufficient, and even if it is, it's not likely to cope with environmental entropy very well.

pron(440) 6 days ago [-]

> even rodents can outperform state of the art models at general tasks.

Rodents? Try insects [1]. In the late 40s and early 50s, when neural networks were first explored with great enthusiasm, some of the leading minds of that generation believed (were convinced, in fact) that artificial intelligence (or AGI in today's terms) is five/ten years away; the skeptics, like Alan Turing, thought it was fifty years away. Seventy years later and we've not achieved insect-level intelligence, we don't know what path would lead us to insect-level intelligence, and we don't know how long it would take to get there.

[1]: To those saying that insects or rodents can't play Go or chess -- they can't sort numbers, either, and even early computers did it better than humans.

buboard(3489) 6 days ago [-]

> even rodents can outperform state of the art models at general tasks

Rodents can't play Go or a lot of other humanly-meaningful tasks. We don't need to build an artificial cell. A cell is too many components that by blind luck happened to find ways to work together, this is as far from efficient design as can be. The same way we don't build two-legged airplanes, we don't need anything that's close to the wet spiky mess that happens in human brains. It's more likely that we have all the ingredients already in ML, and we need to connect them in an ingenious way and amp up the parallelism.

skokage(10000) 6 days ago [-]

>Runner up for next project was cost effective nuclear fission reactors, which wouldn't have been as suitable for that style of work.

I can't tell if he was serious about that comment or not... Considering he builds rockets with free time, it could go either way.

Voloskaya(4222) 6 days ago [-]

> I can't tell if he was serious about that comment or not... Considering he builds rockets with free time, it could go either way.

Nuclear fission shocks you but not AGI?

the_watcher(3996) 6 days ago [-]

I took it as 'not even I would experiment with nuclear fission reactors from my house'.

Jaruzel(959) 5 days ago [-]

Carmack's post in full:

Starting this week, I'm moving to a 'Consulting CTO" position with Oculus.

I will still have a voice in the development work, but it will only be consuming a modest slice of my time.

As for what I am going to be doing with the rest of my time: When I think back over everything I have done across games, aerospace, and VR, I have always felt that I had at least a vague "line of sight" to the solutions, even if they were unconventional or unproven. I have sometimes wondered how I would fare with a problem where the solution really isn't in sight. I decided that I should give it a try before I get too old.

I'm going to work on artificial general intelligence (AGI).

I think it is possible, enormously valuable, and that I have a non-negligible chance of making a difference there, so by a Pascal's Mugging sort of logic, I should be working on it.

For the time being at least, I am going to be going about it "Victorian Gentleman Scientist" style, pursuing my inquiries from home, and drafting my son into the work.

Runner up for next project was cost effective nuclear fission reactors, which wouldn't have been as suitable for that style of work.


We're at 500 comments at the time of posting this, and no-ones pasted his post in full to save us having to visit Facebook...

_hao(4221) 5 days ago [-]

I have FB blocked so thanks for sharing!

arcturus17(10000) 5 days ago [-]

Thanks it didn't so much save me as it enabled me, since I have it blocked.

geogra4(3811) 5 days ago [-]

Too bad he didn't go for his runner up. cost effective/mass produceable fission reactors could save humanity.

nwsm(3931) 6 days ago [-]

What a joke. Carmack is going to sit at home and solve what teams of scientists can't do in decades.

I'm complaining less about Carmack wanting to spend his time doing this and more about the comments here acting like he is some 10000x research scientist.

wanderer2323(10000) 6 days ago [-]

I doubt he will sit at home working on it alone for long.

lostmsu(3860) 6 days ago [-]

I recently 'retired' to do the same, and logic here is - there is no harm in trying, if you have resources (of course, he has magnitudes more).

You can get up to date in the field in under half a year of extensive reading. And many of those scientists are too busy solving more specific goals, that their labs set. I doubt there are more than 1,000 researchers in the world specifically working on AGI.

kyle-rb(10000) 6 days ago [-]

No one expects him to emerge with a fully formed AGI. By through experimenting he might contribute some new incremental but still useful improvements.

drcode(3307) 6 days ago [-]

There are few AI researchers (maybe around 5 or so) that could credibly claim technical accomplishments of any sort in the same ballpark as Carmack's.

People with this level of track record should not be underestimated, there aren't many of them out there... They matter.

starpilot(1917) 6 days ago [-]

I wouldn't be that... mean. But if it's anything like his aerospace pursuits, yeah, I wouldn't bet on any breakthroughs.

binarymax(2215) 6 days ago [-]

Does anyone have a non-facebook version of this story?

curiousgal(2871) 6 days ago [-]

Starting this week, I'm moving to a 'Consulting CTO" position with Oculus.

I will still have a voice in the development work, but it will only be consuming a modest slice of my time.

As for what I am going to be doing with the rest of my time: When I think back over everything I have done across games, aerospace, and VR, I have always felt that I had at least a vague "line of sight" to the solutions, even if they were unconventional or unproven. I have sometimes wondered how I would fare with a problem where the solution really isn't in sight. I decided that I should give it a try before I get too old.

I'm going to work on artificial general intelligence (AGI).

I think it is possible, enormously valuable, and that I have a non-negligible chance of making a difference there, so by a Pascal's Mugging sort of logic, I should be working on it.

For the time being at least, I am going to be going about it "Victorian Gentleman Scientist" style, pursuing my inquiries from home, and drafting my son into the work.

Runner up for next project was cost effective nuclear fission reactors, which wouldn't have been as suitable for that style of work.

nafizh(694) 6 days ago [-]

This is so vague as in borderline sarcasm. Okk, it's AGI but really, what is it? Reinforcement learning? Combining symbolic AI with modern advances? Deep learning theory? AGI is a vacuous term. And no one knows which path would lead to something similar to human intelligence. That itself is a matter of research.

carlosdp(4219) 6 days ago [-]

The things you all listed are possible pathways to the goal. All he said is he wants to start working toward the goal. This guy is serious business, he's never been one to spew BS. He's a real deal, no bullshit computer scientist with leagues of novel accomplishments under his belt.

I wouldn't bet against him when he sets his mind to something.

stupidcar(4193) 6 days ago [-]

Which is presumably the research he intends to do over the next few months? E.g. reading around the subject to determine where best to focus his efforts.

h5eath5eahj(10000) 6 days ago [-]

Hopefully short term includes going on Lex Fridman's podcast

Nimitz14(10000) 6 days ago [-]

Hm. I don't feel like additional engineering expertise is what's missing to achieve AGI, there's still a lot more science to do I think, and I'm not sure how good Carmack is at that.

MagnumPIG(10000) 6 days ago [-]

Honestly we don't even know enough in the psychology department to possibly arrive at AGI anytime soon.

BUT if anyone can clear a hurdle or two...

tus88(10000) 6 days ago [-]

What this title should have been: John Carmack gives up on VR.

mnd999(10000) 5 days ago [-]

Or John Carmack (pretty much) gives up on Facebook

Keyframe(3796) 6 days ago [-]

Following up on "John Carmack gives up on aerospace" and hopefully not followed up by "John Carmack gives up on AI".

I like the guy as much as most, but so far it seems like he has been wandering around. He's had much success in the early days of 3D video games and that's about it. A guy of his calibre would probably make a good impact if he joined one of the expert teams like DeepMind. No matter how smart he is, AI today is a completely different ballgame then what he was part of so far. I hope I'm wrong, but I don't see him making any sort of breakthrough on his own, with his son. Maybe he wants to spend more time with his family, which is great, or he drank koolaid about his own legend. Odds are against him, heavily so. Good luck, in any case.

(1469) Hacker Publishes 2TB of Data from Cayman National Bank

1469 points about 23 hours ago by andruby in 1770th position

twitter.com | Estimated reading time – 1 minutes | comments | anchor

Welcome home!

This timeline is where you'll spend most of your time, getting instant updates about what matters to you.

Tweets not working for you?

Hover over the profile pic and click the Following button to unfollow any account.

Say a lot with a little

When you see a Tweet you love, tap the heart — it lets the person who wrote it know you shared the love.

Spread the word

The fastest way to share someone else's Tweet with your followers is with a Retweet. Tap the icon to send it instantly.

Join the conversation

Add your thoughts about any Tweet with a Reply. Find a topic you're passionate about, and jump right in.

Learn the latest

Get instant insight into what people are talking about now.

Get more of what you love

Follow more accounts to get instant updates about topics you care about.

Find what's happening

See the latest conversations about any topic instantly.

Never miss a Moment

Catch up instantly on the best stories happening as they unfold.

All Comments: [-] | anchor

asymmetric(2582) about 8 hours ago [-]

A .torrent file hosted by the Internet Archive, since DDOSecret is apparently being DDOSed: https://web.archive.org/web/*/https://data.ddosecrets.com/fi...

black_puppydog(4220) about 2 hours ago [-]

This size of dataset does seem to me like a good case of why the current torrent format is lacking. Many people might be interested in generally seeding for this torrent, and maybe pulling some of the files. But few are actually going to download the full 2TB. At the moment, I can select individual files to download and seed, ignoring the rest, but I cannot say 'download what I want, and then seed random blocks up to a size of X GB.' It's quite conceivable that in a few months, the bigger, less 'popular' files will be very hard to come by.

That's (maybe the) one thing I think IPFS gets really right: full availability of the whole dataset, regardless of the popularity of the individual blocks. There were some BEPs about that IIRC, but they never got anywhere. I always figured archive.org and similar collection hosters would be interested in this sort of thing, so that people can just 'donate 1 TB to archive.org' or such.

Miner49er(10000) about 22 hours ago [-]

A news article on it with more info and the full HackBack: https://unicornriot.ninja/2019/massive-hack-strikes-offshore...

obilgic(686) about 22 hours ago [-]

offf, that scroll hack.

chelmzy(10000) about 22 hours ago [-]

Pastebin with translated guide on how they hacked the bank:


badrabbit(3537) about 21 hours ago [-]

It doesn't tell you how they infiltrated to begin with so that they can sniff cookies and keystrokes. But my guess is they phished an employee.

Edit: I missed that part,sorry.

Miner49er(10000) about 22 hours ago [-]

Also contains some info on how they hacked Hacking Team and Gamma Group.

Edit: Also a $100k bounty for future hacks on certain types of companies.

newnewpdro(4174) about 20 hours ago [-]

> Dell had setuid'd root on dos2unix, leaving the device easy to root.


52-6F-62(3611) about 22 hours ago [-]

This is a fascinating read... I hope it does make the news.

slovenlyrobot(3807) about 18 hours ago [-]

I had a quick peak at the web server log, it's tiny. The typical legitimate egress from this network is likely somewhere well under 1gb/day, yet somehow these people managed to upload at least 2TB without being noticed

In a well-maintained network an upload like this should have paged someone as it was happening, probably sometime in the first 5-10 minutes. Say your outbound pipe was 100mbit, if it is pegged for 15 minutes I can't see any reason why a small installation like this wouldn't want to know they've just spat out 11GB of data for some unknown reason

Even divvying things up over 14 days (per the age of the data), its still around 6GB/day. This should have shown up in a monitoring graph somewhere, or triggered a page

draw_down(10000) about 18 hours ago [-]

Maybe. Do we know how they exfiltrated it? They may have done it in a rate-limited fashion.

isseu(4188) about 18 hours ago [-]

> In a well-maintained network Bingo

m0zg(10000) about 22 hours ago [-]

This is going to be swept under the rug faster than Epstein, assuming it even surfaces in the mainstream press at all. Lots of rich people run their stuff through the Cayman Islands, probably more than through Panama. So far, as far as I can tell, no mention in the US. The entire front pages of major national newspapers (WaPo and NYTimes) are dedicated to their single-minded mission of telling everyone that orange man is bad.

hurrdurr2(10000) about 22 hours ago [-]

Regrettably the media needs to publish what gets the most clicks; unfortunately what gets the most clicks these days is partisan stories that gets readers riled up.

That Panama leak story was also quickly 'forgotten' by the MSM.

rmah(1180) about 21 hours ago [-]

I'm not sure of the details, but the Caymans have a reciprocal reporting arrangement with US tax authorities and financial regulators. That is, if you're an American, you can't really hide money there. What you can do is operate a subsidiary there (and it must have resident employees) that generates reasonable revenues with minimal costs, thus shifting revenue and profits to the Caymans low tax jurisdiction. Which will be reported to the US gov, so you better be complying with the letter of both US and Caymans law. Because of these reporting agreements, if there is activity in the Caymans that is illegal in a person's/company's homeland, it will likely be european or chinese, not the US.

tomohawk(1413) about 8 hours ago [-]

Consider this holocaust survivor.


His family is murdered by Nazis but he escapes to the UK. He sets up offshore funds in the event something like the holocaust happens again, but the tax authorities go after him.

This is one of many reasons for people to set up offshore accounts that have nothing to do with illegal activities.

How many people like this has this person hurt?

Where does this belief in the infallibility of the tax man come from?

IAmEveryone(3666) about 8 hours ago [-]

Pay your taxes and you can keep them as far away from any shore as you like.

And I don't really see why you feel the need to invoke the Holocaust in this manner? You're just using it as a sympathetic reason to justify tax evasion. Could I also rob some banks with that justification?

lm28469(4222) about 6 hours ago [-]

> How many people like this has this person hurt?

Could be for an holocaust fund, Bezos secret birthday party fund or for your new 4k TV it's all the same, pay your taxes, then do whatever you want with your money. It really isn't that complex.

jokoon(4214) about 22 hours ago [-]

After watching The laundromat, I think it's big news.

Although I will always be curious why countries are letting those small, insignificant islands and countries manage all this wealth, and why they're trusted to do it. This can't be secure or safe. I'm curious about the diplomacy and the political implications of this.

What if their office get robbed? What if the island gets attacked? What if one of those shady bank defaults? What if one of the people who manage it suddenly die? I mean apparently there is a long list of why it's risky, and I don't understand how this is even legal at all. How do you wire all this money to such a small place, and how can they trust so few people to hide so much money? I have so many questions.

conanbatt(4219) about 16 hours ago [-]

Most people don't know the US is the preferred tax haven of the world.

situationista(3454) about 21 hours ago [-]

Note that in his write-up of how the hack was accomplished (someone pasted a link in another comment) the hacker specifically mentions his attack was on Cayman National Bank of the Isle of Man. From cursory research it appears to be based there, not in the Cayman Islands.

jenxkfjeo0(10000) about 13 hours ago [-]

You're curious about them from the perspective of a normal citizen

Politicians don't know and have no desire to know, if you can't tell by their willfully ignoring official political processes (regardless of whether one agrees with the motivation or not, to simply say you're going to ignore it as folks like Sen. Graham have is nuts).

If they did they could be held accountable. Plausible deniability is a key legal position.

So politically zero fucks are given.

I mean we know more than one President was pals with a man the elite knew to be a statutory rapist. They don't give a shit about human lives themselves.

You think they give a shit about ephemeral shit like money when they don't worry about that?

The financial system has us mining bitcoin: ephemeral bullshit other people live big on.

How this forum can be so smart and so clueless is beyond me.

_Codemonkeyism(3889) about 5 hours ago [-]

Do you think the money is on those small islands? Or on Guernsey? Or the British Channel Islands? No, the money is in New York and London at the usual banks who earn money on fees and can work with the deposits. The banks on those islands don't have a data center, or a bank vault. They only legally have the money on paper, but 'their' money is in the banks in NYC (Caribbean) and London (former Empire).

Interesting read 'Treasure Islands: Tax Havens and the Men who Stole the World', Nicholas Shaxson

pas(10000) about 9 hours ago [-]

Big growing economies benefit from tax havens, because they get capital more easily, because the profits can be extracted at a low cost from other countries.


And naturally, as others noted, the security is quite good.

pprplt(10000) about 15 hours ago [-]

> Although I will always be curious why countries are letting those small, insignificant islands and countries manage all this wealth

Those small, insignificant islands and countries are current or former british or european colonies. And considering they are primarily located in the caribbean, the chances of them being successfully attacked is zero - good luck going up against the might of the US Navy.

> How do you wire all this money to such a small place

Whether it's $10 or $1000000, it's just a number (a few bytes) on transfer or just a few more digits on a check.

> and how can they trust so few people to hide so much money?

It's easier to trust a few rather than the many.

CaymanCruiser(10000) about 21 hours ago [-]

Using a throwaway for obvious reasons.

I have a fair bit of insight into this industry, both from friends, acquaintances, and family who use a variety of offshore structures to manage their wealth and from my own dealings (the company I founded and currently manage is offshored for tax and legal purposes).

What people don't realise is that aside from a couple outliers offshore jurisdictions aren't that shady at all and the major ones are usually under the protection and sometimes outright control of much larger and much more influential jurisdictions, with the United Kingdom being the leader here. If tax avoidance was as simple as just starting a business in a zero or low tax jurisdictions we would see a lot more of this kind of business being done in Africa (with the notable exception of Seychelles today) than the Caribbean overseas territories, but mature legal frameworks and more importantly reliable governments are incredibly important to both small time players who may stash some cash offshore as savings and large multinationals. If anything shady (civil unrest, wealth confiscation etc.) were to go down in the British Virgin Islands for example the United Kingdom has a nuclear option (as BVI is for all intents and purposes a Crown colony) to unseat the sitting government and bring order, which is incredibly valuable as an insurance policy.

Yes, it is possible to incorporate offshore corporations in less-reputable jurisdictions like Vanuatu or Seychelles, but doing business with these vehicles is next to impossible because no reputable bank in Singapore, Hong Kong, or Switzerland is going to want to have you as a customer if they don't already know you due to compliance reasons. Without a bank account your ability to funnel and store money in these opaque conduits becomes rather difficult and there really isn't any point any longer. Of course, there are small shady banks in certain jurisdictions (notably those that aren't under indirect British control) that will take you on as a client, but then you run into the risk of them running away with your money or defaulting.

The money itself isn't really managed by or from these island nations/colonies either, and in the cases where there are nominee directors the beneficial owner will usually have a power of attorney as well as an undated resignation letter from the nominee director which effectively gives them wherever they are total control of the company.

What is interesting is that crypto is changing all of this due it being completely outside the severely regulated financial sector, opening up a ton of opportunity for small island nations to compete on tax efficiency without being subject to the quite frankly draconian rules that usually places them on US/EU/OECD blacklists which can create severe consequences for the economies of said nations through sanctions.

Very interesting times indeed, and I think we will see more and more companies (particularly tech companies) being offshored than before in the next few years.

sbarre(2482) about 21 hours ago [-]

There is a great book you should read called Treasure Islands[0]. It's equal parts fascinating and infuriating, and makes you realize the system has never been set up fairly.

Rich people have always had their own financial systems that are outside of, and one step ahead of, the regulations and rules the rest of us are subject to.

[0]: https://www.goodreads.com/book/show/10197857-treasure-island...

zaro(4212) about 9 hours ago [-]

> Although I will always be curious why countries are letting those small, insignificant islands and countries manage all this wealth

This is easy, after all hiding cash is needed by everybody. And when it comes to finance , there are no countries only financial interest. Of course nothing bad will happen to these financial heavens, because they are protected by the richest and most powerful people and organisations.

'The Spider's Web: Britain's Second Empire'[1] gives a good overview of how this started.

[1] https://www.youtube.com/watch?v=np_ylvc8Zj8

corodra(4209) about 19 hours ago [-]

>What if their office get robbed? What if the island gets attacked?

Switzerland, WW2. There is safety in holding EVERYONE's money. Because the moment you try it, you enrage many, many, many people who have things that go boom. You can fuck with someone's politics, their car or even their wife. But once you fuck with someone's money, it's a whole new ball game.

> I don't understand how this is even legal at all

Bank fees are higher. The different 'islands' that do this, really don't have a choice. WTF else are they going to do as income generation? There's only so many happy endings you can give in a day spa. It's also easier to 'invest' and hide the money in tourist based enterprises on said islands. It totally makes sense to hire an interior designer for $10k an hour.

>how can they trust so few people to hide so much money

Because people don't like getting into car accidents. When you're 'trusted' with something on that level, suddenly people make sure you either don't have or do have car accidents that magically attract stray bullets.

>I mean apparently there is a long list of why it's risky, and I don't understand how this is even legal at all.

Homie, avocado farmers in Mexico get kidnapped and executed because avocados are cash crops. Shit, look up 'avocado stolen' or even 'avocado semitruck stolen'. No such thing as making bank and not having risk. Some of those truck drivers in the USA get killed. If you're going to get targeted for making money, might as well go all out.

lawlessone(10000) about 6 hours ago [-]

>Although I will always be curious why countries are letting those small, insignificant islands and countries manage all this wealth,

British Overseas Territory

sumedh(10000) about 18 hours ago [-]

> why countries are letting those small, insignificant islands and countries manage all this wealth

Some of the tax havens like Jersey, Isle of Man are British territories, Curacao is territory of Netherlands.

The big countries know what goes on on their little islands they just choose to ignore it for obvious reasons.

Treasure Islands is a great book if you want to learn more about this.

xibalba(10000) about 15 hours ago [-]

Most of these assets, although 'located' in obscure places, are actually warehoused right back here in the US of A.

Their status as being 'held' outside the US is mostly a fiction existing on paper which allows for privacy and tax avoidance.

a3n(3313) about 17 hours ago [-]

Because the people who run 'countries' use these services.

rolltiide(10000) about 21 hours ago [-]

The Cayman Islands has robust financial regulations and is a financial center.

It is also reliant on the UK for military assistance, and as you can see the bank records for this particular institution are digital.

You are grouping everything together with misplaced isolationist paranoia.

lemmox(10000) about 20 hours ago [-]

Many tax havens are actually quite stable and have very reliable banking systems. Canada encourages 'snow washing' with Limited Partnerships. Canada's banks are quite good and the corporate legislation allows a degree of anonymity. This structure has no obligation to file taxes in Canada. The partners in the structure have to pay taxes however if the partner is not a Canadian entity that obligation is in some other jurisdiction.


jbob2000(10000) about 21 hours ago [-]

The money isn't being kept there for safety, it's being kept there for anonymity and easy access. These countries don't ask any questions when you show up and ask to withdraw a million dollars. Maybe you're withdrawing that money to buy a yacht, maybe you're withdrawing it to buy drugs, maybe you're withdrawing it to pay for a seat on Epstein Air. Banks in developed countries ask questions, these banks in these countries ask 0 questions.

Aperocky(10000) about 21 hours ago [-]

You'd think it's the small island and countries managing those bank, but it's not. The banks are from the rich and for the rich, the country is just a ruse.

Accujack(10000) about 21 hours ago [-]

>why countries are letting those small, insignificant islands and countries manage all this wealth

Enlightened self interest. Those 'insignificant' islands are generally speaking countries. By showing respect for the rule of law even in situations where they could easily overwhelm with force or economic power smaller nations, the larger ones reinforce that the law matters.

The larger countries use as a defense the idea that something is an 'internal matter', IE it's no one else's business. If they bully their way into other smaller countries' business, then they open themselves up to the same interference.

Might does not make one right. Law does.

StuffedParrot(10000) about 12 hours ago [-]

> Although I will always be curious why countries are letting those small, insignificant islands and countries manage all this wealth, and why they're trusted to do it.

Probably because the people who run the state are also people who take advantage of these tax loopholes. I don't believe the US even needs foreign countries to hide taxes this well, we have better mechanisms in e.g. South Dakota.

gxqoz(10000) about 21 hours ago [-]

A recent piece in The Guardian argues that South Dakota and other US states are actually becoming the go-to place to store anonymously your illicit cash and make sure that your heirs can inherit it without taxes forever: https://www.theguardian.com/world/2019/nov/14/the-great-amer...

Partly because the US has compelled other countries to provide more transparency without joining that transparency organization itself.

'Congress responded with the Financial Assets Tax Compliance Act (Fatca), forcing foreign financial institutions to tell the US government about any American-owned assets on their books. Department of Justice investigations were savage: UBS paid a $780m fine, and its rival Credit Suisse paid $2.6bn, while Wegelin, Switzerland's oldest bank, collapsed altogether under the strain. The amount of US-owned money in the country plunged, with Credit Suisse losing 85% of its American customers.

The rest of the world, inspired by this example, created a global agreement called the Common Reporting Standard (CRS). Under CRS, countries agreed to exchange information on the assets of each other's citizens kept in each other's banks. The tax-evading appeal of places like Jersey, the Bahamas and Liechtenstein evaporated almost immediately, since you could no longer hide your wealth there.

How was a rich person to protect his wealth from the government in this scary new transparent world? Fortunately, there was a loophole. CRS had been created by lots of countries together, and they all committed to telling each other their financial secrets. But the US was not part of CRS, and its own system – Fatca – only gathers information from foreign countries; it does not send information back to them. This loophole was unintentional, but vast: keep your money in Switzerland, and the world knows about it; put it in the US and, if you were clever about it, no one need ever find out. The US was on its way to becoming a truly world-class tax haven.'

coliveira(3316) about 20 hours ago [-]

> I don't understand how this is even legal at all.

A lot of the process is not legal. That's why it is a scandal when they find this kind of information. But the existence of this type of service is practically guaranteed for the same reason: a lot of the money in these accounts was acquired in shady, if not totally ilegal ways. Even if countries make it completely illegal, there will be a lot of clients for offshore accounts. You can think of this as a modern version of pirates hiding their illegal treasures in some Caribbean island.

noobermin(2912) about 11 hours ago [-]

Hanlon's razor. Also, a Michelle Obama quote[0].

[0] https://www.newsweek.com/michelle-obama-tells-secret-i-have-...

badrabbit(3537) about 20 hours ago [-]

From what I read the hack was not sophisticated at all. If they had decent endpoint protection this would not have happened. Maybe they disabled defender after the initial access and privesc because defender blocks Get-Keystrokes and other things in empire,even with significant modification.

Anyway, I don't care about the hacktivism and propaganda (don't really get it -- tbh,rich people bad/corrupt?) But would very much be interested in any post-incident analysis of what happend.

But here's what stands out to me: they're talking about using empire and hvnc. Empire is by design detectable,you could use the techniques in empire in your own malware though. Most rats and bots (take trickbot) let you sniff key strokes, run commands,etc... So I guess they used psexec or winrm to access the hosts(no vlan segmentation or firewall on endpoints!?). I ask all this because the specific MO is significantly different than what is seen with criminal actors and crimeware(like Carbank which She mentions). And to me, it does corroborate the story that this was likely an independent hacker motivated by hacktivism as opposed to a bigger conspiracy to burn some group and plant false records in the leak (which could still less likely be the case)

Very exciting,righteous hack.

I think overall, the bank cheaped out on IT and paid big time for it.

nyolfen(4065) about 19 hours ago [-]

> privacy for the weak, transparency for the powerful

arminiusreturns(10000) about 15 hours ago [-]

Here's the thing. While most people on HN tend to be in the global top performers when it comes to IT, the vast majority of businesses are simply ripe, low hanging fruit just waiting to be breached, and almost every time it is because of a failure of management to understand the importance of those things. Way too many 60 year old men who barely know how to use a computer are at the top of orgs, and are stuck in the janitorial cost sink fallacy of IT. I also put a lot of this on the heads of IT directors or senior sysadmins, who are failing to convey the importance of the matter to those C-levels in terms they understand.

I have learned this the hard way, having started up an MSP at one point and done a lot of contracting to help orgs unfuck their infra, I have seen it all first hand, from 5-30 person lawfirms to fortune 500 oil companies to unicorn startups. I have been the senior sysadmin who failed to convey things in a way the got through C-level thick skulls, and most of my career trajectory has been angled towards keeping up with the sysadmin transition to devops while learning how to fill that gap so I don't repeat those mistakes. I've failed multiple times, but each one is a lesson I learn from and try to apply to the next place, and sometimes C-levels literally just don't care and can't be reached, and nothing happens till a major breach or lawsuit costs lots of money.

To me, this is the importance of the CTO/CIO roles. The problem is, again, the vast majority of America isn't SV. I would say ~%80 of companies I saw didn't even have those roles, and if they had someone in that role but without the title (like IT director), they often didn't have a seat on the board or any real influence with the C's. I have also seen those roles taken by people who should be in other positions but enjoy the 'C' title too much, to the detriment of the org.

Sorry, this got kind of ranty, but it's an issue I obsess over in trying to find better solutions for. Basically I'm learning how to hack management instead of computers these days.

joshypants(10000) about 20 hours ago [-]

The future of the species is literally imperiled right now because we've allowed rich people to hoard wealth, so that's a reason to care.

georgeglue1(4198) about 20 hours ago [-]

>> (don't really get it -- tbh,rich people bad/corrupt?)

it's not so much that wealth is bad, but tax evasion and money laundering (from organized crime, corruption, etc.) is bad.

xmly(4216) about 21 hours ago [-]

Where could I download!

unnouinceput(10000) about 12 hours ago [-]

Search through twitter, there is a torrent as well.

wyxuan(10000) about 20 hours ago [-]

Link is on their website, just click though

v4dok(10000) about 20 hours ago [-]

Is there a forum/IM where someone has already found out any interesting parts? I am unable to download 2TB of data

danaos(4218) about 17 hours ago [-]

You don't have to download the entire folder. Some parts are less than 10 GB.

Nextgrid(10000) about 10 hours ago [-]

I've read the Pastebin and I wonder, where did they send the 200k to be able to cash them out without getting in trouble? I've always assumed that diverting electronic money is relatively easy, the hard part is to convert it to a physical or untraceable form without getting caught; I assume the receiving bank would have kept a record of the incoming transfer and what happened with that money afterwards.

pearjuice(1554) about 6 hours ago [-]

Plenty of people on the darkweb allow for flushing of bank transfers to untraceable crypto currencies. Typical scenario: use an unknown (homeless, hacked account whatever) person to register bank account, send money to bank account, withdraw in cash or use digital service to convert to cryptocurrency. Money is gone and owner of bank account might not even know and can probably not be held responsible. The reason the hacker got caught was because he made invalid SWIFT transactions to accounts in Mexico and someone found out the reported errors and checked other transactions.

ptah(10000) about 7 hours ago [-]

This will have as much of an effect as the panama papers: other than some journalists involved in reporting it getting killed, nothing will change.

stef25(10000) about 5 hours ago [-]

Supposedly the hacker distributed the money to various good causes.

cookie_monsta(10000) about 20 hours ago [-]

Strangely under-reported is that the same hacker is offering up to 100k to others who perform similar acts of hacktivism:


koheripbal(10000) about 6 hours ago [-]

Probably a nation state actor

dmix(1368) about 16 hours ago [-]

So this activist has somehow profited from his "hacktivism" and he wants to spread his wealth to other activists?

Historical Discussions: Build Your Own React (November 14, 2019: 1458 points)

(1465) Build Your Own React

1465 points 5 days ago by pomber in 3803rd position

pomb.us | Estimated reading time – 8 minutes | comments | anchor

function createElement(type, props, ...children) {

children: children.map(child =>

typeof child === 'object'

: createTextElement(child)

function createTextElement(text) {

function createDom(fiber) {

fiber.type == 'TEXT_ELEMENT'

? document.createTextNode('')

: document.createElement(fiber.type)

updateDom(dom, {}, fiber.props)

const isEvent = key => key.startsWith('on')

const isProperty = key =>

key !== 'children' && !isEvent(key)

const isNew = (prev, next) => key =>

const isGone = (prev, next) => key => !(key in next)

function updateDom(dom, prevProps, nextProps) {

isNew(prevProps, nextProps)(key)

.filter(isGone(prevProps, nextProps))

.filter(isNew(prevProps, nextProps))

dom[name] = nextProps[name]

.filter(isNew(prevProps, nextProps))



function commitWork(fiber) {

const domParent = fiber.parent.dom

fiber.effectTag === 'PLACEMENT' &&


fiber.effectTag === 'UPDATE' &&

} else if (fiber.effectTag === 'DELETION') {



function render(element, container) {

let nextUnitOfWork = null

function workLoop(deadline) {

while (nextUnitOfWork && !shouldYield) {

nextUnitOfWork = performUnitOfWork(

shouldYield = deadline.timeRemaining() < 1

if (!nextUnitOfWork && wipRoot) {



function performUnitOfWork(fiber) {

fiber.dom = createDom(fiber)

const elements = fiber.props.children

reconcileChildren(fiber, elements)

nextFiber = nextFiber.parent

function reconcileChildren(wipFiber, elements) {

wipFiber.alternate && wipFiber.alternate.child

index < elements.length ||

const element = elements[index]

element.type == oldFiber.type

if (element && !sameType) {

if (oldFiber && !sameType) {

oldFiber.effectTag = 'DELETION'

oldFiber = oldFiber.sibling

wipFiber.child = newFiber

prevSibling.sibling = newFiber

function App({ props }) {

return <h1>Hi {props.name}</h1>

const element = <App name='foo' />

const container = document.getElementById('root')

Didact.render(element, container)

All Comments: [-] | anchor

pier25(3433) 5 days ago [-]

Very nice. This guy is going places.

Hey Rodrigo so how performant do you think this didactic example would be compared to React? Maybe 80% there?

pomber(3803) 5 days ago [-]

Much less than 80%. Half of React code is optimizations, and not only speed, for a medium sized app I guess React uses less than 20% of the memory that Didact uses.

franky47(10000) 5 days ago [-]

your* ?

pomber(3803) 5 days ago [-]

fixed, thanks

tabtab(4131) 5 days ago [-]

Next article: 'Build yore own grammar checker' :-)

level(4204) 5 days ago [-]

Along a similar vein, Sophie Alpert did a talk[1] at React Conf this year on building your own React custom renderer by using implementing the interfaces that React exposes. Sophie explains how you could use React as a declarative API for pretty much any platform if you wanted, similar to how react-native works.

[1] https://www.youtube.com/watch?v=CGpMlWVcHok

_hardwaregeek(10000) 5 days ago [-]

Which is actually a little funny, because that's essentially turning React into a domain specific language for UI rendering. The Racket people are probably facepalming that we had to go via JavaScript and Babel to get to a UI DSL.

yboris(3846) 5 days ago [-]

The presentation is stellar! I love the animated code changes on the side <3

jaequery(2803) 5 days ago [-]

Forget building my own React, I want to build how you did the presentation! It's a great concept!

jolmg(4201) 5 days ago [-]

I agree, though, it's a shame there's no fallback for when there's no javascript.

petetnt(4185) 5 days ago [-]

Here's the Gatsby theme https://www.gatsbyjs.org/packages/gatsby-theme-waves/ (and on GitHub https://github.com/pomber/gatsby-waves)

Great article and awesome storytelling there!

musicale(10000) 5 days ago [-]

I really hate the layout and the annoying, distracting, animated garbage on the left side of the page.

Maybe it isn't rendering properly in my browser, but I would greatly prefer a simple, wide, vertical layout.

spectramax(4089) 5 days ago [-]

I agree, it is simply astounding that the entire thread is filled with praising the presentation style which only gets in the way of the user. Anytime there is a need to add effects, animations, snazzy things - question ruthlessly as to why is it needed. Then after justifying it, ask again.

This is what good documentation looks like: https://golang.org/pkg/

And this: https://www.mpfr.org/mpfr-current/mpfr.html

reggieband(10000) 5 days ago [-]

Excellent content. I did find that the code previews in the side bar occasionally changed before or after I expected. I understand the intent with having them adjacent rather than inline ... but I'd probably prefer inline. It would be nice to make this an option of the presentation format.

pomber(3803) 5 days ago [-]

Is it because you read closer to the top or bottom of the viewport?

I want to add a print mode that would have each code step inline.

jjcm(3841) 5 days ago [-]

Presentation is amazing here.

That said, I recently wrote a webcomponent based router and I was surprised at how easy it was to get to a very usable point at such a minimal amount of code. All too often I get in the loop of just `npm i`'ing whatever it is I need, even if the functionality I really need is just a tiny subset of whatever huge library I'm importing.

Rodrigo's right that something like this helps you understand React and how to contribute to the codebase, but I think it goes further than that - it lets you understand that the library isn't magic. It gives you the confidence you need to think that maybe you don't need that full 560kb library.

grrowl(10000) 5 days ago [-]

Especially when you consider a non-trivial amount of the react-dom codebase is actually Synthetic Event handling (smoothing out differences in browser implementations). If you're using 'the platform' and understand native events, that's a huge saving for you and your users.

_hardwaregeek(10000) 5 days ago [-]

Redux is a very good example of this. If you watch the original talk or just understand primitives like Observables, you can realize that Redux almost isn't a library. It's basically a pattern. Where libraries are actually required is when you connect Redux to React, or need async actions. But even then it's not super hard to see how they work.

rajeev-k(10000) 5 days ago [-]

> webcomponent based router

Routers can be (and should be, IMO) written to be independent of UI component libraries such as React and webcomponents. See an example here: https://github.com/Rajeev-K/mvc-router You can use it with React---and in some ways it works better than react-router.

interlocutor(4211) 5 days ago [-]

> you don't need that full 560kb library

In fact, most web applications only need this 200-line library: https://github.com/wisercoder/uibuilder

React started out as a simple library, but it is getting more and more complicated (see concurrent mode). 90% of applications don't benefit from React's complexity. This tiny lib has the same templating technology as React, but none of the complexity.

proc0(10000) 5 days ago [-]

Wasn't there a very similar article recently? Wouldn't mind if this is a new tutorial trend. Regardless the presentation of this article is great! I hope this also catches on, it has maximum scanning efficiency.

kaycebasques(3013) 4 days ago [-]

What's the trend here?

leetrout(4073) 5 days ago [-]

Very, very nice.

His presentation library is available as well!



stockkid(2012) 5 days ago [-]

Yeah the presentation effect on code is super cool. Yet on mobile I didn't feel that it was particularly helpful as it covered about half the screen height and I couldn't clearly skim the content.

spectramax(4089) 5 days ago [-]

Please don't perpetuate this type of animated presentation style. Why?

1) Vertical layout is easier to scroll

2) Vertical layout puts the code blocks in-line to know where they belong in the text

3) Author's presentation style has animations that slow down user's ability to scan quickly

4) Scroll position determines the code block's visibility (!!! WTF)

5) Either have the entire page white or black. The contrast changes between reading the document and then reading the code tires the user

6) Scroll wheel feels highjacked because the left pane is static

I could go on and on. I strongly discourage people from using this type of documentation. Your users, will get tired of the novelty and when push comes to shove, they will want something like this:


agumonkey(877) 5 days ago [-]

This might even become standard for most documentations.

ghego1(4221) 5 days ago [-]

Is it just me or on mobile it does not render properly?

jermaustin1(3289) 5 days ago [-]

I love this effect, I was actually just writing a blog post that I didn't save before my tab crashed. I'm actually now GLAD it didn't!

This is a lot more effective a medium for the content!

I even went ahead an set up a monthly donation to their open collective project [1] because it is so beautiful.

1: https://opencollective.com/code-surfer

palerdot(3410) 4 days ago [-]

This is basically unusable in mobile ... but looks nice in desktop though ...

tomcooks(10000) 5 days ago [-]

WOAH amazing presentation, this is how code should be explained.

Quick validation: would you pay for good bits of code explained well by the author?

kybernetikos(4215) 5 days ago [-]

I have once already: https://www.amazon.com/Beautiful-Code-Leading-Programmers-Pr...

I liked it, but I'm not going to be paying for this sort of thing often. I think a lot would hinge on having content that was high quality, and being able to persuade me that it would be high quality before I paid for it might be challenging.

Historical Discussions: Brave launches 1.0 (November 13, 2019: 1005 points)

(1005) Brave launches 1.0

1005 points 6 days ago by etr71115 in 2662nd position

brave.com | Estimated reading time – 7 minutes | comments | anchor

The Industry's Most Advanced Browser Offers 3-6x Faster Browsing and Ends Surveillance Capitalism with a Private Ads and Payment Platform that Benefits Users, Advertisers, and Publishers

Brave Rewards Now Available on iOS, Completing Cross-Platform Availability

San Francisco – November 13, 2019 – Brave Software, makers of the innovative Brave browser which combines privacy with a blockchain-based digital advertising platform, today announced the official launch of Brave 1.0. The Brave open source browser fundamentally shifts how users, publishers, and advertisers interact online by giving users a private, safer, and 3-6x faster browsing experience, while funding the Web through a new attention-based platform of privacy-preserving advertisements and rewards. In addition to desktop and Android devices, Brave's private ads and payment platform (Brave Rewards) is now available for iPhones and iPads.

"Today's Internet is broken, and users suffer the most. They are being tracked, tagged, and exploited; this not only violates privacy, but slows down page loads, drains batteries, and makes for a miserable experience. Meanwhile, publishers are losing revenue at a record pace due to a few gigantic super-companies and too many ad tech intermediaries. Advertisers are wasting time and money on an industry filled with fraud. Everyone with a legitimate stake in the open Web loses in this environment," said Brendan Eich, co-founder and CEO of Brave Software. "Surveillance capitalism has plagued the Web for far too long and we've reached a critical inflection point where privacy-by-default is no longer a nice-to-have, but a must-have. Users, advertisers, and publishers have finally had enough, and Brave is the answer. Brave 1.0 is the browser reimagined, transforming the Web to put users first with a private, browser-based ads and payment platform. With Brave, the Web can be a rewarding experience for all, without users paying with their privacy."

"Either we all accept the $330 billion ad-tech industry treating us as their products, exploiting our data, piling on more data breaches and privacy scandals, and starving publishers of revenue; or we reject the surveillance economy and replace it with something better that works for everyone. That's the inspiration behind Brave," he added.

The Brave browser was previously in beta release and already has 8.7 million monthly active users across the globe. As data privacy and security issues take center stage around the world, users turn to Brave for its privacy-first approach that natively blocks trackers, invasive ads, and device fingerprinting – leading to substantial improvements in speed, privacy, security, performance, and battery life. Default settings block phishing, malware, and malvertising. Embedded plugins, which have proven to be an ongoing security risk, are disabled by default. Browsing data always stays private and on the user's device, which means Brave will never see or store the data on its servers or sell user data to third-parties. With the launch of Brave 1.0, users have access to a full set of unique features, including:

  • Brave Rewards to fund the open web – Users can support their favorite publishers and content creators by activating Brave Rewards and its integrated Brave wallet on both desktop and mobile. This feature allows users to send BAT as tips for great content, either directly as they browse or by defaulting to recurring monthly payments to continuously support the over 300,000 verified websites, including The Washington Post, The Guardian, and Wikipedia, or creators on YouTube, Twitch, Twitter, GitHub and more.
  • Brave Ads that compensate users for their attention – Brave is built on top of the first global private ad platform, designed from the start to value users' attention and privacy. Brave has pioneered a new blockchain-based advertising model that reforms the current system with privacy by design and 70 percent revenue share to users in the form of Basic Attention Tokens (BAT). Brave Ads, the world's first private ad network and part of Brave Rewards, allows users to opt-in to view relevant privacy-preserving ads in exchange for earning BAT. When users opt into Brave Rewards, Brave Ads is enabled by default. Ad matching happens directly on the user's device, so their data is never sent to anyone, and they see rewarding ads without web-wide mass surveillance. To date, Brave Ads has delivered nearly 475 privacy-preserving ad campaigns, sparking 130 million ad confirmation events, from brands such as Intel, Pizza Hut, Home Chef, and more. Users can transfer earned BAT out of their Brave Rewards wallet and convert the tokens to many digital assets and fiat currencies, after completing a verification process with digital money platform Uphold. More significantly, they can contribute to a better Web by keeping the BAT in their browser to automatically support online creators via Brave Rewards.
  • Brave Shields for automatic ad and tracker blocking – Brave Shields, on by default and customizable from the address bar, block invasive third-party ads, trackers, and autoplay videos immediately – without needing to install any additional programs. Users can now confidently browse the web knowing that their interests and reading habits are not being tracked. Because Brave Shields removes the friction of unwanted content, Brave loads websites up to 3 to 6 times faster than other browsers and introduces significant memory and battery savings on desktop and mobile. In fact, Brave saves1 an average of 27 seconds per page load against Chrome on macOS and 22 seconds per page against Firefox, and Brave uses 58% less data than Chrome to load those same pages. Brave also uses less memory than other browsers, with an improvement of 40% over Chrome and 47% over Firefox.

"As the conversation around data privacy and transparency moves increasingly into the forefront, Brave shows us that it's possible to create a more equitable value exchange between brands, content creators and consumers," said Nick Coronges, Global Chief Technology Officer at R/GA and Partner at R/GA Ventures. "We're excited to see Brave's continued innovation in the space with the 1.0 release, and looking forward to partnering on future collaborations that put people in charge of their data and their time spent online."

Brave comes with a modern take on a streamlined user interface that users will find comfortable when switching from Chrome or other browsers, offering extension support as well as an easy system to import bookmarks, passwords, cookies, and browsing history, making the transition to Brave seamless.

Brave is available for download for Windows, macOS, Linux, Android, and iOS, in 52 different languages. To download for free, visit https://brave.com/download, Google Play or the App Store. To find out more about Brave 1.0's features and performance, please check out the Reviewer's Guide (https://www.brave.com/press/1.0-REVIEWERS-GUIDE.pdf).

1 Load time was measured by visiting a sample of news articles from different sites and measuring time until the point after which there's no network activity for five seconds. This test was run from a cold-cache start on a high-speed connection.

About Brave SoftwareBrave Software's fast, privacy-oriented browser, combined with its blockchain-based digital advertising platform, is reinventing the web for users, publishers and advertisers. Users get a private, speedier web experience with much longer battery life, publishers increase their revenue share, and advertisers achieve better conversion. Users can opt into privacy-respecting ads that reward them with a frequent flyer-like token they can redeem or use to tip or contribute to publishers and other content creators. The Brave solution is a win-win for everyone who has a stake in the open Web and who is weary of giving up privacy and revenue to the ad-tech intermediaries. Brave currently has 8.7 million monthly active users and over 300,000 Verified Publishers. Brave Software was co-founded by Brendan Eich, creator of JavaScript and co-founder of Mozilla (Firefox), and Brian Bondy, formerly of Khan Academy and Mozilla.

All Comments: [-] | anchor

MentallyRetired(10000) 6 days ago [-]

As a Firefox user, is there a reason to convert? Seems like a step backwards to me, even though I applaud what they're doing.

gnicholas(1393) 6 days ago [-]

In my experience (2017 MBP running Mojave), it is much, much faster than Firefox. I have lots of tabs open, and my fan almost never kicks on anymore. With FF, even using ublock and even a pihole, the fan was constantly at full throttle. Maybe I was doing other things wrong, but switching to Brave has made a huge difference for me. I say this as someone who stood by FF for a decade and a half.

Lordarminius(3903) 6 days ago [-]

Here are two problems I have noticed with brave-browser : The layout is great on widescreen monitors but breaks on my 15'' laptop. Sites like YouTube render as a garish mess of incongruously placed whitespace. This is a major issue. Also, the brave icons for receiving 'rewards', when activated, are too large obstrusive when displayed on websites. I had to turn them off in settings.

On the good side, I like that I don't have to ever watch pesky ads on YouTube.

Overall however, brave-browser is great and offers a welcome change from traditional browsers.

jonathansampson(4122) 6 days ago [-]

Thank you for the feedback! Would you be able to share a couple screenshots? I'd love to share this information with our design team.

caiocaiocaio(4145) 6 days ago [-]

Use an ad blocker.

olalonde(147) 6 days ago [-]

Getting a 504 error on https://brave.com/download/

jonathansampson(4122) 6 days ago [-]

Apologies for that; server is getting a little more attention than we had expected. Should be working now, though perhaps a bit sluggish at times.

chilighost(10000) 6 days ago [-]

Brave was initially an interesting browser, but without any options to disable the credits/money add-ons the browser have no future.

colordrops(10000) 6 days ago [-]

They are opt-in and you can disable them. It seems you never tried the browser. Odd that you'd need a new account to post this.

jonathansampson(4122) 6 days ago [-]

Brave Rewards is off-by-default. It's up to you to opt-in. And, once you do, you can set the number of ads to be shown each hour (upper-limit). The user is always in control.

fuu_dev(10000) 6 days ago [-]


#1 Brave advertise with privacy and speed while all they do is adding adblock.

#2 Brave replaces adds with their own adds. This basically means you get the same experience but cut off the income of the content creator entirely.

#3 Brave allows you to reward 3rd party sites. This system is so nice they even got legal issues as it was assumed as fraud.

#4 BAT. Brave advertise with consumer first. The best choice for the consumer would be Paypal. If there is an incent to stay with crypto currency Bitcoin or Ethereum.

Given Brave created their own crypto currency and base every model around that they can profit from Transactions/services with the currency too on top of the share they take when you use their add platform/feature.

Given the criticism its very hard for me to understand users sympathizing with the browser.

fastball(4141) 6 days ago [-]

#1 Their ad-blocker is also a tracking blocker and a pretty strict one at that. They also spend a lot of time optimizing the performance[0] of this ad-blocking. This improves the browsing experience.

#2 Brave Ads are opt-in. You can alternatively fund the sites you visit on the web by putting BAT into your wallet directly and having the browser automatically distribute funds based on which content you spend time on per month. The entire idea of Brave is that people were already using ad-blockers, so I'm not sure how this is worse for content creators.

#3 ???

#4 BAT is an ERC-20 token - it exists on the Ethereum blockchain. Not sure why you think BTC would be better than that. Honestly not sure why you think Paypal would be better than that. At this point, anyone can transact the currency easily and quickly without any intervention from Brave. I can convert BAT to DAI right now on a decentralized exchange in less than a minute and have a virtual currency pegged to USD even if I don't bank with USD. At the moment this process is somewhat technical, but it overall seems very pro-consumer to me.

[0] https://brave.com/improved-ad-blocker-performance/

siegecraft(10000) 6 days ago [-]

Re: criticism 1, this is mostly true. However the fact that it's 'not google' adds some privacy, or at least diffuses vulnerabilities.

Regarding criticism 2, you have to opt-in. Otherwise brave is just a browser with adblocking capabilities built-in.

I don't understand criticism 3, but then again I have just been using Brave without opting-in to the BAT portion so I'm out of the loop on that.

Regarding 4, why would Paypal be the best choice? Paypal has shown they will close your account and confiscate your money pretty much at their whim. Using a non-BTC/Ether crypto token at least means your Brave wallet isn't a big fat target for crypto thieves.

I think the success of patreon et al have shown that people in fact are perfectly willing to pay content creators, they just hate ads.

bobowzki(3784) 6 days ago [-]

I wonder how much brave.com was?

bsclifton(4207) 6 days ago [-]

I don't think it says a price, but it was previously used by a 'nuclear polka' band :) https://brave.com/the-road-to-brave-one-dot-zero/

maelito(3288) 6 days ago [-]

What if I really don't want to see ads at all ? Can I pay Brave in € and let it distribute this money to the content creators ?

AgentME(10000) 6 days ago [-]

Yes-ish. You can put your own BAT (or Bitcoin or ETH) into your Brave wallet. You can buy BAT from Coinbase/Uphold/other exchanges.

mind_half_full(10000) 6 days ago [-]

Great job Brave! Privacy protecting ads and you get paid to browse. Simply amazing!!

Sohcahtoa82(10000) 6 days ago [-]

A brand new account praising a business.

This reeks of being an astroturfing account.

xyst(10000) 6 days ago [-]

Front end developers: "yet another browser to support"

cheald(3050) 6 days ago [-]

It's just Chromium, so it's not particularly anything new to support.

caiocaiocaio(4145) 6 days ago [-]

Brendan Eich named his browser 'brave' because he thinks that he is 'brave' for attacking a minority group.

When I was young, in a trashy town, big groups of guys would get together and go 'fag-bashing', which is when a bunch of grown adults beat someone up because they suspect he's gay. Often, the target is a little kid. Since a lot of cops were former fag-bashers, there wasn't much to do about it. Fag bashing stopped for about a year when one 13-year-old had to get multiple surgeries. Even after all the surgeries, his face is deformed to this day, about 25 years later.

The fag bashers were still much more brave than Brendan Eich, because they had the courage to look their victims in the face. Anyone who uses Brave should seriously reconsider what kind of person they've become, and what kind of person they want to be.

brundolf(2156) 6 days ago [-]

I'm sorry- I'm extremely confused about what this has to do with this web browser.

johnpowell(4220) 6 days ago [-]

They still aren't brave enough to use a unique user agent string. Brave on left, Chrome on right.


aembleton(4182) 6 days ago [-]

Reduces fingerprinting.

viach(3570) 6 days ago [-]

> You set a budget (e.g. 10 BAT/month), and Brave distributes it the sites you use most, e.g. if you watch a particular YouTube channel 30% of your browsing time, it will send 30% of 10 BAT each month to that content creator.

If you would own a video streaming service (a startup!) and some company started to pay your users for viewing ads on your platform in some MagicMoneyCoins putting your pocket outside the cash flow, what would you do?

If you would want to receive organic traffic from your ads and instead start to experience crazy stream of incentivized ad clickers with near zero conversion rate, what would you do?

Imo the answers to these questions are not that pleasant for Brave. Maybe I'm wrong of course, like the most of times.

fastball(4141) 6 days ago [-]

Yeah, having seen all the useless previous attempts at paying users to watch/click ads, I'm not particularly hopeful for the long-term prospects of that system.

Seems like it might be better to just force the user to disburse those funds to the sites they visit.

83insfnadsjfn(10000) 6 days ago [-]

Can you login to the StackDriver metrics console with Brave yet? I had to disable everything, literally all protections, to get this to non-endless redirect, its difficult to whitelist as there is dozens of domains from (gcloud to accounts.google.com, to some sd/stackdrivers one) at this point it I gave up

throwaway-9320(10000) about 9 hours ago [-]

I had this issue, but on Firefox and with Privacy Badger and uBlock Origin enabled. Some cookie seems to get blocked, leading to the infinite redirect loop. At least in this case I could just disable two addons temporarily and was good to go.

simon_weber(3617) 6 days ago [-]

I used Brave for a while on Android and found it to work well. However, in the last month or so something changed and it started freezing constantly.

I've since switched to Firefox, which is much better on Android than when I last tried it.

mattlondon(10000) 6 days ago [-]

I've been using Brave on android for a while now and it has been really good - it is basically the same as chrome, except with a built-in ad blocker. Highly recommend it as your daily driver.

I use firefox on desktop, but found firefox on android to be a bit clunky the last time I tried it (years ago)

Aaronstotle(10000) 6 days ago [-]

I don't understand why they didn't use a stable coin/transition BAT to a stable coin, the volatility of BAT probably keeps creators uneasy. Why sign up for a service where contributions can lose half their value overnight.

jonathansampson(4122) 6 days ago [-]

When did you see that type of volatility? Advertiser dollars go into the system through purchases of BAT by the team. This helps to maintain steady value. You can see previous transactions (and more) at brave.com/transparency.

mk89(10000) 6 days ago [-]

I don't know if I understood it right, but BATs are only given when users see the ads, right?

If this is the case, it seems that Brave fosters the use of ads, not reduce it.

Is my understanding right?

buboard(3489) 6 days ago [-]

I think they want users to be able to control the web through their attention. It's a great idea that acts as indirect pressure on websites to behave. As things are now, users have no way to slap a site on the wrist for being horribly slow and invasive. Brave seems to want to change that.

jonathansampson(4122) 6 days ago [-]

Ads aren't inherently bad, but modern digital advertising has been co-opted by bad actors. Brave offers an ad-free experience by default; this is necessary for your privacy and safety online.

But users are increasingly more aware that protecting themselves from harmful ads also means stripping their favorite creators of support. This is where Brave steps in to offer a complete solution, rather than the partial solution of 'just block and forget'.

Brave Ads are opt-in, entirely private (data never leaves your device), and pay the user 70% of the ad revenue. By default, that 70% will flow out to the sites/properties you visit on a monthly basis. If you like, you can choose to keep some (or all) of it for yourself.

So we don't necessarily want a Web full of ads. We want a Web full of empowered users who have control over their data and attention.

notyourwork(10000) 6 days ago [-]

Did Brave ever claim to reduce the use of ads? As I understand, they want to shift the revenue model and put control back in hands of users. You can use their browser and ignore ads (also not receive Bats) or you can see ads and receive compensation for it.

chops(3219) 6 days ago [-]

It blocks the on-page ads, but, if you opt-in will show you notification ads (non-invasive notifications), and will pay you for them.

aloisdg(3923) 6 days ago [-]

I am all for fighting capitalism, but I wont tolerate any form of ads. Ads are a threat for free thinking. They are pollution, poison, for the mind. I dont want to face any form of manipulation.

Beside, Brave is chromium based. If you want to fight the chrome monopoly, go somewhere else.

I could be fine with a fork of Firefox with a preinstall uBlockOrigin, Privacy Possum, facebook container, etc.

Vaslo(1607) 6 days ago [-]

Your first comment doesn't make sense or maybe I am misreading. Do you mean you are fighting for capitalism or fighting against it? You are implying you are against capitalism by banning advertising which is a very capitalist thing. I'm confused

topspin(4101) 6 days ago [-]

> I wont tolerate any form of ads. Ads are a threat for free thinking.

Then turn off the ad feed in Brave and enjoy the built in ad blocking. If you would still like to fund participating sites through Brave you can fund your BAT account with your own money.

octocop(10000) 6 days ago [-]

If not Ads, how would a 'free service' generate money?

judah(3518) 6 days ago [-]

I've been using Brave rewards, both as a user and a content maker. It's really great, and I feel this may be a reasonable alternative to the invasive trackers+ads we have today.

For the uninitiated, Brave lets users opt-in to Brave rewards:

- You set your browser to reward content creators with Basic Attention Token (BAT). You set a budget (e.g. 10 BAT/month), and Brave distributes it the sites you use most, e.g. if you watch a particular YouTube channel 30% of your browsing time, it will send 30% of 10 BAT each month to that content creator.

- As a user, you can get paid in BAT. You tell Brave if you're willing to see ads, and how often. If so, you get paid in BAT, which you can then distribute to content creators. Brave ads are different: rather than intrusive in-page ads, Brave ads show up as a notification in your operating system outside of the page. This prevents slow downs of the page, keeping your browsing focused, while still allowing support of content creators. And of course, Brave ads are optional and opt-in.

sadturnip(10000) 6 days ago [-]

Can someone explain to me this, as i could not find any real answers from my few minutes of searching. Lets say i have a blog and suddenly people start sending me BAT.

1. How does this impact my taxes? Wouldn't this be considered an income?

2. If i don't collect it where does it go?

3. If my content is being hosted on something like youtube or github do i get it or does the site hosting it get it?

4. How do i go about claiming that i own this, and how is this even verified?

After reading their FAQ, basically to collect any money you need to sign up for an uphold account. In order to become verified on Uphold i need to provide a random company a copy of my passport/drivers license/etc to verify my identity. On top of this they also take 1.95% conversion fee for working with BAT. Ontop of the 5% that Brave already takes by default.

On top of this if you are lets say a Twitch streamer sign up for Brave Rewards, but Twitch doesn't sign up as a publisher on Brave. According to the documentation you apparently get nothing? Where do those tokens go if someone donates?

a3_nm(4031) 6 days ago [-]

Sorry, this must be very basic but I've never been able to make sense of one thing about Brave's model. You say that users can get paid in BAT to watch ads. But Brave is open-source. So is there anything preventing me to fork the browser to pretend to display the ads and reap the rewards without actually displaying them? Doesn't this break the whole system?

djsumdog(1073) 6 days ago [-]

I preferred the Flattr model, of just paying the content creators directly without the ad middle ground. I don't really see why I should use Brave over something like Firefox with uBlock Origin.

I actually drop people on Patreon when they start adding ads or sponsors into their podcasts or videos. I would rather people be fully supported by their fans instead of double dipping into both fan support and adverts.

thefreeman(3229) 6 days ago [-]

doesn't Brave have to track your browsing history in order to accurately award sites 'you visit the most'? Isn't that exactly the issue a lot of people have with ads in the first place?

spir(4088) 6 days ago [-]

I ran an information site for an Ethereum project[1] and was pleased to see that our first revenue came from visitors' BAT donations. It wasn't a lot of money. But, it was fully automatic and easy to use. Our site earned BAT before we even signed up as a publisher. I signed up to claim the money.

[1] https://predictions.global/

mdorazio(10000) 6 days ago [-]

I would personally much rather allocate actual dollars instead of CryptoBuckOfTheDay backed by ads I have no desire to ever watch. Say, $20/month that gets allocated to sites I visit. The whole 'reward people for watching ads' model just seems... wrong.

notadoc(4114) 6 days ago [-]

> Brave ads show up as a notification in your operating system outside of the page

LOL WTF, no!

smileysteve(10000) 6 days ago [-]

> if you watch a particular YouTube channel 30% of your browsing time, it will send 30% of 10 BAT each month to that content creator.

But does YouTube get anything? To support their business

onyva(4150) 6 days ago [-]

if you care about privacy you'd be better off with Mozilla Firefox. Brave is just a Trojan horse for Eich's ambition to replace google as an ad broker. With a meaningless market share it's not likely but still, it's important to know what Brave is about, and it's not about the users.

jeena(2007) 6 days ago [-]

Sounds awfully close to flattr which didn't fly.

breck(365) 6 days ago [-]

I sort of like this concept and was playing with Brave but I still can't explain to you how it works. I wish there were a page where I could see an ad, or something, a page that fully demos what is going on. The wallet page is a little confusing (I need to verify a wallet or something, which seems like it is a laborious process, etc). Seems very cool, but I hope they can keep simplifying it.

apazzolini(10000) 6 days ago [-]

> rather than intrusive in-page ads, Brave ads show up as a notification in your operating system

I'm sorry, how is that not orders of magnitudes MORE intrusive?

ouid(10000) 6 days ago [-]

Alright, so now instead of mining for bitcoin, everyone is going to be making money directly from advertisers by running virtual machines where they simulate a human interacting with the internet for BAT?

Do I have to solve captchas to qualify for brave rewards?

Can I make more money from being advertised to by being wealthy, and thus more valuable to advertise to or does brave capture all of that? Does no one capture that? Do I still get paid if I don't click through stuff? Do I get paid more if I do? How does brave verify that an ad has even been rendered? I doubt that it's terribly difficult to fool brave into believing that it has successfully rendered an ad at the maximum rate. Is it merely a violation of the ToS and if they 'catch you' your bitcoins are forfeit? Is BAT a currency? When they confiscate your BAT, what is the legal process involved?

What if I want to launder a tremendous amount of money? Is brave going to suddenly make that a lot easier?

What if I have a massive botnet on the home computers of the elderly and I want to monetize it, does Brave help me do that?

blondin(10000) 6 days ago [-]

my issue with brave is the false pretense of caring about our privacy and being better than what's out there they started with then at some point they did a u-turn and went on selling ads... i just can't shake the feeling of that nothing good is coming out of this.

Ajedi32(1903) 6 days ago [-]

This seems like a great alternative funding model for the web. Modern online advertising has a lot of issues (privacy, intrusiveness) but most users don't want to directly pay for the sites they browse either (i.e. the subscription model). By displaying ads controlled by the browser rather than the website, the user gets a lot more control over the ads they see (solving the privacy and intrusiveness problem), while still supporting the websites they visit without paying for them directly.

The only major problem with this funding model that I can see is that it provides no incentive for users to contribute to the websites they visit at all. They can just as easily block ads and not replace that revenue with anything, which is actually the default behavior. It's essentially moving from a web funded by advertising to a web funded by donations; and based on past experience in using donations to fund open source projects I can't really see that model working out well for anyone but large, popular websites with minimal operational costs.

Though on the other hand, Brave does eliminate a lot of the friction associated with the typical donations model. Maybe they'll be more effective at soliciting donations on behalf of websites than previous efforts have been. It's hard to know for sure.

melvinram(3556) 6 days ago [-]

Give me an ad-free option. I'd rather pay a monthly fee to fund the BAT that gets distributed to content creators than to view ads.

bvda(10000) 6 days ago [-]

I wonder why they chose to use (yet another) crypto currency for this.

elsurudo(10000) 6 days ago [-]

Brave 'rewards' started appearing in my Android's notification center without my consent. Seems pretty intrusive to me, and I defiitely did not opt in...

pergadad(10000) 6 days ago [-]

Ah there you go. That's exactly why I deleted brave. Make it more annoying, splendid! And in the process demonetise websites and claim it's all for the greater good.

iamaelephant(2769) 6 days ago [-]

So Brave is tracking all of the content I consume? That will be a hard pass from me, thanks.

hinkley(4121) 6 days ago [-]

You had me...

> Brave ads show up as a notification in your operating system outside of the page.

and then you lost me. Hitting people with notifications is an escalation of ad hostility, not a reduction.

Dolores12(10000) 6 days ago [-]

You disable competitors ads to show your own. Something is wrong with it.

Waterluvian(4118) 6 days ago [-]

Can I just buy BAT with cash and never ever see ads?

n3k5(10000) 6 days ago [-]

In some ways, this looks promising: Maybe in a few years, they'll be supporting many more countries and payment providers. I'm not overly skeptical about Brave being able to become a viable alternative to my current go-to browsers either — though I hope Safari and Firefox will keep ramping up privacy-protecting features, and I think for the purpose of getting Brave Rewards to critical mass, it would be best to provide plug-ins for integrating that functionality with other browsers as well.

But I see one big problem: For a significant amount of media, a browser isn't the most desirable content delivery mechanism. Take Youtube, for example. The web site is great on PC, but on mobile, the native apps are much better. There's no way those will add support for Brave Rewards, and the chances of the browser-based version catching up don't look so good either. Similarly, there's music streaming, podcasts, F2P games ...

I don't want to be in a situation where I have to choose between accessing content in the way I prefer and jumping though hoops for the sake of shuffling those precious BATs around.

Equating the value I'd assign to a creator's work with the time I spend on perusing it seems quite iffy, too.

Meanwhile, Patreon already has most of these issues solved; in a super simple way.

I used to be interested in Brave because of their pro-privacy stance. The more I think about that BAT stuff, the less attractive I find the entire project. I can give them the benefit of the doubt and assume that they're idealists striving to build an ecosystem where everyone can help to create valuable on-line content, rather than entrepreneurs trying to syphon a portion of existing revenue streams into their accounts. But then I'm inclined to question their competence.

bilbo0s(4207) 6 days ago [-]

>You set a budget (e.g. 10 BAT/month)

What if a user doesn't earn 10 BAT a month browsing?

It seems like that system would break down over time.

leitasat(10000) 6 days ago [-]

so instead of spying on the level of websites, it is moving to the level of a browser? And not only aggregated clicks, but detailed statistics of browsing time per user?


annoyingnoob(10000) 6 days ago [-]

Fuck ads. Fuck tracking. Fuck making me part of your 'audience'. There has to be a better way. I'm going with blocking until there is a better answer. Brave may be a step in the right direction but its not the solution.

AgentME(10000) 6 days ago [-]

If you don't opt into seeing ads, then Brave is a browser with ad-blocking built in (and a donate-to-sites-you-use button).

foob4r(10000) 6 days ago [-]

I understand 'fuck trackers'. Why 'fuck ads'?

degenerate(4106) 6 days ago [-]

The main reason I block ads is because ad networks make so many requests (and the ads load so slow) that it always affects my browsing. I am either waiting for the page to load because of poorly engineered synchronous ad network code, or I'm having my content fumbled around at the tail end of pageload. Both situations are obtrusive, aggravating, and slow me down.

In the early 2000s ads were simple images that loaded just as fast as everything else, and I didn't mind them much, even the cheesy animated ones. It's the delivery, not the content, that I abhor.

dkhenry(3866) 6 days ago [-]

edit2: Nevermind all this, Brave did in fact implement protected content this earlier this year


I still can't use brave to watch netflix on Linux. Its a great product and I love shifting the incentive of advertising away from intrusive data harvesting, but treating Linux like its a second class citizen means I won't use your product. Whats worse is that Firefox is 100% open source and Chromium is 100% open source and they both manage to get this working out of the box. There is no reason for Brave to not have working Netflix

edit: Also I know the library that decodes the DRM is proprietary, my point is the integration with it is open and available, no reason it can't be duplicated.

colordrops(10000) 6 days ago [-]

Are you referring to Widevine? Brave asks you if you want to install it rather than just assuming you want DRM out of the box.

brundolf(2156) 6 days ago [-]

I tried out Brave ~3 years ago when it was still Electron-based, and it was unfortunately too laggy to be a daily driver. Then I tried it out again ~6 months ago after they'd switched it to being a Chromium fork. The speed was much-improved, but it suddenly started spamming me with push notifications about various features, Brave rewards, etc. I felt like I was blocking web ads just to get native ads instead. You can turn them off deep in the settings, but I got sketched out and stopped using it.

I think Brave's payment model concept is a very interesting idea for freeing the open web from the need for advertising, but I haven't been impressed with the way the project has been executed so far.

BrendanEich(10000) 6 days ago [-]

You turned on Brave Rewards at some point, it seems. If you want to tip and contribute anonymously but not get ads, just turn off ads (upper left of rewards settings).

spurgu(3735) 5 days ago [-]

This is interesting, I've been using Brave for well over a year and I've never seen a single push notification...

etr71115(2662) 6 days ago [-]

You can turn off ads in your browser settings under 'Brave Rewards'. You won't earn any tokens though.

abhinavsharma(4052) 6 days ago [-]

What is Brave doing under the hood to get these claimed performance improvements over Chrome, Firefox and Safari?

bityard(10000) 6 days ago [-]

I would guess they block the ads

300bps(3771) 6 days ago [-]

By default, they fail to load stuff like trackers, ads, etc.

etr71115(2662) 6 days ago [-]

Not rendering ad networks and trackers.

Sargos(10000) 6 days ago [-]

Probably the built-in ad and tracker blocking. All of those scripts and content add up.

nissimk(3806) 6 days ago [-]

Blocking ads?

adambyrtek(1338) 6 days ago [-]

I guess just blocking ads and trackers gives a significant performance boost they leverage in their marketing. I bet benchmark against any of the other browsers with ad-blockers installed would give a different result.

emptysongglass(4102) 6 days ago [-]

Two big issues for me that have kept me on Firefox despite being a huge cryptocurrency nerd:

1. The decision to base on Chromium is a bigger negative for me than any feature. We're all going to regret contributing to the One True Engine someday and it's going to be so painful to fix.

2. BAT is a pure money play. There's no inherent utility outside buying and spending. And Ethereum is nothing if not the biggest utility crypto out there. This is always one of those things in the Ethereum ecosystem where I point out that you didn't have to make yet another ERC-20 if your only play is money.

It's too late now, of course. But man, it would have been terrific to recommend a browser that blocked ads out of the box and wasn't part of the engine hegemony.

jammygit(3596) 6 days ago [-]

I used brave for about a year but stopped. There were issues with the Linux version - eg, it wouldn't start after a system boot without using the terminal for the first time

eigenvalue(4179) 6 days ago [-]

What would you have the Brave team do? Create a new html rendering engine from scratch? That is an incredibly challenging engineering project that would probably require years and many tens if not hundreds of millions of dollars to do well, and what would be the benefit to Brave's users and investors? It would just lead to loads of performance and compatibility issues, and it is fundamentally irrelevant to the core value proposition of Brave, which is to monetize ads on behalf of users and content creators.

sneak(3056) 6 days ago [-]

There is no meaningful technical criticism of Chromium that I have seen. All of the anti-engine folks' main argument seems to be "it has the Google stink on it", but I can't see how that is relevant in an application explicitly designed to avoid their ad/tracking/cloud services. It's just a renderer.

ropiwqefjnpoa(10000) 6 days ago [-]

Did they fix their sync issue yet? I couldn't get my laptop and desktop to work together so I switched over to Firefox.

jonathansampson(4122) 6 days ago [-]

Happy to help investigate Sync issues. Able to share more details?

Hamcha(10000) 6 days ago [-]

I've used Brave since March, and I'm really not impressed. For a browser that tries to be different, it's incredible how little consideration has been given to things that annoy users and that could probably be trivial to remove.

To be more specific, my issues with Brave are:

- Still uses the Chrome Web Store as main store

- Does not have an easy way to load packaged extensions from 3rd party sources

- Even after adding them as unpacked, you still get the same annoying popup every time you open the browser.

- I received notices that 'My account was waiting for a deposit' the entire time, even though I never allocated any money to anyone

- Tipping amounts are fixed

I always saw Brave as 'not chrome, and not as much of a change as Firefox', but their approach seem lazy, features halfassed at best and the sneaky anti-features that Chrome has been adding to push users to lock into Google stuff (see all my remarks about extensions) not addressed at all.

I'm using Firefox as main browser now, and I haven't been missing brave once.

colordrops(10000) 6 days ago [-]

It's perfect for my use case: Chrome but with all the google bits removed and built-in ad-blocking. Can't say any of those things you listed have bothered me. I don't use BAT.

Edit: Why do so many new accounts (green text) post to any thread about Brave?

hunterloftis(3269) 6 days ago [-]

I had the opposite take on the same product decisions.

As a long-time Chrome user, I have a set of useful extensions that have become a part of my daily browsing experience. I found converting to Brave an easy transition since everything I'm used to in Chrome is still available, right down to the developer tools.

As far as I can tell, Brave is basically Chrome with built-in Adblock and anti-tracking, plus an innovative mechanism for user-controlled advertising and content-author micropayments. I'm very impressed.

fastball(4141) 6 days ago [-]

Sooooo, basically all of your issues are related to extensions.

Also don't know what you're talking about with items 3-4, I have 3rd party extensions loaded, it was pretty easy, and I don't get any error messages on startup.

Also I love the fact that all the Chrome extensions work in Brave.

DeckOfSardines(10000) 6 days ago [-]

Have you had the same experience with their mobile browser? I found the desktop experience lackluster but the only browser that's run faster on my phone is Firefox Focus.

mistrial9(10000) 6 days ago [-]

I guess that a big amount of time and energy is used to bicker about money and legalities, instead of improving the GUI and UX. However, this may simply be the price for something new.. the net had twenty years to decide on micropayments, and now its ad-infestation and surveillance capitalism instead. So, the GUI whining is not convincing here...

uncletammy(4202) 6 days ago [-]

Here's an idea. Go help them make it better. Fixing our broken ass network and the (bad) incentives that keep is afloat is one tough job. We need all the help we can get!

You can start here: https://github.com/brave

nostrademons(1712) 6 days ago [-]

I've also been using it since about March, and been generally happy with it. It's faster than Chrome, and the built-in adblock is great, and the easy incognito/TOR modes are handy. That's basically all I want out of it - it's got all the other nifty Chrome goodies that I use (devtools etc.), the only Chrome extension I used regularly was Adblock, I don't use a whole lot of other power-user features. I've had some stability issues (it crashes about once every 2-3 days), but that could be because I'm running several versions behind. Looking forwards to trying 1.0.

the_duke(3525) 6 days ago [-]

These mostly seem like very specific power user concerns that wouldn't be relevant to most users.

(not advocating for Brave, I'm a happy Firefox user. Just saying...)

JohnJamesRambo(4155) 6 days ago [-]

You've used a browser you aren't impressed with for seven months?

wjoe(10000) 6 days ago [-]

> I always saw Brave as 'not chrome, and not as much of a change as Firefox'

It seems more like 'Chrome, with some plugins' than anything else. It's a shame really, I like the concept of the BAT project, I'd be interested in using it as a plugin for Firefox, but I'm not interested enough to switch browsers and lose all the features and plugins I'm used to in Firefox.

Angeo34(10000) 6 days ago [-]

Whoever uses Brave for 'privacy' reasons should understand that Chrome offers more privacy due to fingerprintability.

I see the downvotes coming yet it is true check it for yourself. Brave is so easy to fingerprint compared to Chrome it defeats any purpose ever to use Brave in the first place if privacy is your main goal. Removing all the Google bits from the Chromium base doesn't help since you are so fingerprintable.

songshuu(10000) 6 days ago [-]

If you're talking about the canvas id fingerprint which is identical across all instances of the browser, that lets Brave users hide in the crowd, so it's literally more private.

Panopticlick needs to update their docs to make that clearer.

blacksmith_tb(10000) 6 days ago [-]

Hmm, I get identical results from Chrome and Brave on my Ubuntu desktop in Panopticlick, which doesn't disprove your argument, but it looks like Chrome is fingerprintable-enough that the difference may not matter.

emiliovesprini(10000) 6 days ago [-]

Eternal reminder by @paulskallas (Twitter):

> there's never been a good ad in the history of advertising. 120 years and there's never been an ad that has stood the test of time and was fondly remembered like literature or a song. It is a profoundly anti lindy medium that exists in the now, forever

toxicFork(10000) 6 days ago [-]

There was a Turkish tv channel that highlighted funny or cool advertisements.

techntoke(4220) 6 days ago [-]

Offers 3-6x faster browsing than what exactly, browsing with IE or Edge without any ad blocking?

have_faith(4166) 6 days ago [-]

If people like Brave's ad model, all the power to them. I would personally never touch it. I'm not philosophically on board with the idea that paying people to look at ads is a pragmatic way forward. It doesn't matter if it's opt-in, free-to-play games with loot crates are opt-in, but the decision to go down that road will effect their decision making process when designing new features and I don't want to be part of it even indirectly.

NelsonMinar(1080) 6 days ago [-]

Brave is unethical. The key problem is they insert themselves into the revenue stream without the consent of the web publishers. It's fine to block ads entirely, but substituting your own ads and collecting money from that is wrong.

the_duke(3525) 6 days ago [-]

I'm skeptical of the model too, and have no intention of using Brave.

But: you can also not enable ads and buy BATs yourself. This gives you a single system to distribute donations to content you enjoy.

Which is actually quite an interesting concept, since it is low effort/automatic for the user and yet allows small contributions.

Of course it also gives Brave a potentially huge position of power, which is my biggest problem.

If the BAT system was run by a independent non-profit and not browser specific it would be more appealing.

bilbo0s(4207) 6 days ago [-]

In all honesty, I'm not sure how I feel about this model? I'm one of those people who has been out saying that the web needs a new business model. Well, here one is, and yet I find myself a bit hesitant to embrace it? It's probably just a matter of needing more information to fill in the blanks and explain the contradictions.

That said, my mind tends to automatically start strategizing the long game, and I'm uncertain how valuable people who have to be paid to look at ads are to advertisers? I mean, Certainly some advertisers would still be interested. But enough to displace the current model? I'm just not sure.

stefan_(4222) 6 days ago [-]

The Brave approach seems to be the same as the infamous Adblock Plus, i.e. insert themselves as a middle-man between websites & ad networks on the one side and the user on the other and under the guise of 'ethical ad blocking' extract a ransom.

buboard(3489) 6 days ago [-]

If people aren't willing to look at ads, what will news articles become? ads. What will HN comments become? Ads (many blog posts already are). Advertising covers a need

iseeyou(10000) 6 days ago [-]

I use Brave on iOS with javascript blocked as it is the only* iOS browser which gives me the ability to easily use the Web without JS while giving me the ability to quickly enable it for the site where it makes sense, and still retain the ad blocking feature.

A surprisingly big part of the Internet works as well or better and faster with javascript disabled, but sometimes it is actually needed and useful so I find this whitelisting method works well.

I tried playing around with the script blocking ability of native content blockers but never got it working properly, for one I think it can't block script tags only externa scripts.

But my recommendation, if you are giving Brave a try is to change the default to disallow scripts, and enable them as needed, it's refreshing to browse the Web without Popups 2.0 and all the other not so nice uses of JS.

(* only I've found with my limited searching)

pcnix(10000) 5 days ago [-]

You might also like umatrix for this, it gives unmatched control over blocking resources.

DHPersonal(10000) 6 days ago [-]

SnowHaze seems to have a quick toggle available. https://snowhaze.com/en/index.html

dsincl12(10000) 6 days ago [-]

Sorry I've tried to give Brave a shot for a couple of months and the 3-6x faster browsing is nonsense. It's really slow compared to Chrome or Firefox. Just do a quick comparison yourself, you can probably count seconds on a couple of sites you visit regularly whereas Chrome loads instantly.

Wheaties466(10000) 6 days ago [-]

I run a high spec machine with 3 browsers(Brave and Google Chrome, Firefox) 8-10 hours a day every work day. I respectfully completely disagree. I do not find it slow in comparison to the others, I do find it much much quicker on 90%+ sites.

I do not work for brave, mozilla, google or apple. I also have no stake in any of those companies nor do I know anyone that works for those companies. I really wonder how so many people in this thread can have a negative response when I personally have felt the complete opposite.

I do suspect it has something to do with employer bias.

lacker(1694) 6 days ago [-]

Is Brave making any amount of money from their advertising system? I can't find any public communications about it, and I would expect them to be touting if the numbers are good.

The weird thing about Brave is that they raised $35 million from their crypto ICO, but that isn't like normal fundraising because you can only do it once. So they really need their real business to kick in and start working before they run out of that money, more than most venture-backed startups need to start making money.

Getting paid money to watch ads reminds me of AllAdvantage in the first dotcom boom. It seems cool at first when the system is juiced by investor money, but once advertisers realize the ROI is low, the money dries up.

BrendanEich(10000) 6 days ago [-]

https://brave.com/transparency shows revenue share BAT buys to pay users their 70% from invoices paid in fiat. Some ad buyers use BAT, and we do some barters.

atombender(2977) 6 days ago [-]

I've been using Brave for a few months, and I'm basically using it as an un-Googled Chrome. Not using the ad rewards system.

It's pretty good, though I sometimes miss Safari with its amazing location bar autocompletion and ease of syncing with the iOS version. (Brave supports syncs, but it's not great, and the iOS app is nowhere as good as Safari on iOS.)

Brave does have some bugs. Since July or so, several sites (including my bank) don't work because Brave incorrectly [1] blocks their cookies, even with the 'shield' turned off.

[1] https://github.com/brave/brave-browser/issues/6099 and several others.

csharp(10000) 5 days ago [-]

What do you like about Safari that iOS Brave is missing? We are quite a bit faster than Safari, and our private mode is more aggressive against things like cookies and writing data to disk.

FpUser(10000) 6 days ago [-]

https://brave.com/malvertising-homeland-security/ 'Brave warns US Senate & Congress: foreign state actors can use targeted ads to run code on US government computers, exploiting conventional browsers'

Oh, my. I am so scared. And Brave is of course a knight in shining armor ready to protect our shaken foundations.

300bps(3771) 6 days ago [-]

I got a virus from an ad on reddit in 2008.

It actually does happen, and I've used ad blocking software consistently ever since.

chilighost(10000) 6 days ago [-]

Initially I liked the concept of brave, but they destroyed themselves by bloating it with this coin/payment stuff that is not possible to remove 100%. Back to Firefox now.

Sohcahtoa82(10000) 6 days ago [-]

Curious why you created an account just to make two comments, both of which are entirely incorrect.

What's your agenda? Who's paying you to slander Brave?

ulzeraj(10000) 6 days ago [-]

It's opt in. You never really tried brave didn't you?

(962) The cognitive costs of air pollution

962 points 1 day ago by jseliger in 15th position

patrickcollison.com | Estimated reading time – 5 minutes | comments | anchor

Air pollution is a very big deal. Its adverse effects on numerous health outcomes and general mortality are widely documented. However, our understanding of its cognitive costs is more recent and those costs are almost certainly still significantly under-emphasized. For example, cognitive effects are not mentioned in most EPA materials.

World Bank data indicate that 3.7 billion people, about half the world's population, are exposed to more than 50 μg/m3 of PM2.5 on an annual basis, 5x the unit of measure for most of the findings below.

  • Substantial declines in short-term cognitive performance after short-term exposure to moderate (median 27.0 μg/m3) PM2.5 pollution: 'The results from the MMSE test showed a statistically robust decline in cognitive function after exposure to both the candle burning and outdoor commuting compared to ambient indoor conditions. The similarity in the results between the two experiments suggests that PM exposure is the cause of the short-term cognitive decline observed in both.' [...] 'The mean average [test scores] for pre and post exposure to the candle burning were 48 ± 16 and 40 ± 17, respectively.' – Shehab & Pope 2019.
  • Chess players make more mistakes on polluted days: 'We find that an increase of 10 μg/m3 raises the probability of making an error by 1.5 percentage points, and increases the magnitude of the errors by 9.4%. The impact of pollution is exacerbated by time pressure. When players approach the time control of games, an increase of 10 μg/m3, corresponding to about one standard deviation, increases the probability of making a meaningful error by 3.2 percentage points, and errors being 17.3% larger.' – Künn et al 2019.
  • A 3.26x (albeit with very wide CI) increase in Alzheimer's incidence for each 10 μg/m3 increase in long-term PM2.5 exposure? 'Short- and long-term PM2.5 exposure was associated with increased risks of stroke (short-term odds ratio 1.01 [per μg/m3 increase in PM2.5 concentrations], 95% CI 1.01-1.02; long-term 1.14, 95% CI 1.08-1.21) and mortality (short-term 1.02, 95% CI 1.01-1.04; long-term 1.15, 95% CI 1.07-1.24) of stroke. Long-term PM2.5 exposure was associated with increased risks of dementia (1.16, 95% CI 1.07-1.26), Alzheimer's disease (3.26, 95% 0.84-12.74), ASD (1.68, 95% CI 1.20-2.34), and Parkinson's disease (1.34, 95% CI 1.04-1.73).' – Fu et al 2019. Similar effects are seen in Bishop et al 2018: 'We find that a 1 μg/m3 increase in decadal PM2.5 increases the probability of a dementia diagnosis by 1.68 percentage points.'
  • A study of 20,000 elderly women concluded that 'the effect of a 10 μg/m3 increment in long-term [PM2.5 and PM10] exposure is cognitively equivalent to aging by approximately 2 years'. – Weuve et al 2013.
  • 'Utilizing variations in transitory and cumulative air pollution exposures for the same individuals over time in China, we provide evidence that polluted air may impede cognitive ability as people become older, especially for less educated men. Cutting annual mean concentration of particulate matter smaller than 10 μm (PM10) in China to the Environmental Protection Agency's standard (50 μg/m3) would move people from the median to the 63rd percentile (verbal test scores) and the 58th percentile (math test scores), respectively.' – Zhang et al 2018.
  • Stock market returns are lower on polluted days. 'This estimate indicates that a one unit increase in PM2.5 decreases the daily percentage returns by 1.7%. Put differently, a one standard deviation increase in PM2.5 decreases the daily percentage returns by 11.9%, a substantial effect on daily NYSE returns.' Hayes et al 2016.
  • Baseball umpires make worse decisions on polluted days. 'Unique characteristics of this setting combined with high-frequency data disentangle effects of multiple pollutants and identify previously under-explored acute effects. We find a 1 ppm increase in 3 hour CO causes an 11.5% increase in the propensity of umpires to make incorrect calls and a 10 μg/m3 increase in 12-hour PM2.5 causes a 2.6% increase.' Archsmith et al 2018.
  • Politicians use less complex speech on polluted days. 'We apply textual analysis to convert over 100,000 verbal statements made by Canadian MPs from 2006 through 2011 into—among other metrics—speech-specific Flesch-Kincaid grade-level indices. This index measures the complexity of an MP's speech by the number of years of education needed to accurately understand it. Conditioning on individual fixed effects and other controls, we show that elevated levels of airborne fine particulate matter reduce the complexity of MPs's peeches. A high-pollution day, defined as daily average PM2.5 concentrations greater than 15 μg/m3, causes a 2.3% reduction in same-day speech quality. To put this into perspective, this is equivalent to the removal of 2.6 months of education.' Heyes et al 2019.
  • 'Exposure to CO2 and VOCs at levels found in conventional office buildings was associated with lower cognitive scores than those associated with levels of these compounds found in a Green building.' – Allen et al 2016. The effect seems to kick in at around 1,000 ppm of CO2.

All Comments: [-] | anchor

rbavocadotree(10000) about 24 hours ago [-]

From the original source:

> Stock market returns are lower on polluted days.

I don't want to dismiss the entire article, but I think this points to possibility that there is some bad statistics going on here. I don't see how daily stock returns are proof of, or even related to, cognitive ability on a daily basis.

This is exactly the type of finding you'd expect with p-hacking.

andy_ppp(4121) about 23 hours ago [-]

Maybe people are more negative because of traffic, noise and frustrated about getting to work later. I find it hard to imagine a way to control for this short of making people breathe these substances separately and seeing if they actually do have lower IQs. You can then control for the other factors. These all seem to be epidemiological studies that we can't show the pollution is for sure the root cause. It could, for example, be that pollution is 25% of the result for example with more noise, traffic, irritation etc. being the other 75%.

alexandercrohde(4086) about 22 hours ago [-]

'I don't want to dismiss the entire article'

Good. Because you can't cherry-pick one dubious-seeming journal article and use it as a justification to ignore several completely unaffiliated articles.

sgustard(3080) about 21 hours ago [-]

To clarify, the paper studied 'air quality in the vicinity of Wall Street'.

lazyjones(4140) about 24 hours ago [-]

This is probably just a coincidence. Stock markets perform lower on Mondays: https://www.macroption.com/stock-market-performance-volatili...

Traffic is probably generally higher on Mondays, though I have only anecdotal evidence.

ajross(10000) about 24 hours ago [-]

It's a review article. The paper you mention was just one of many cited. And sure, those are all different papers written by different authors and it's surely not unlikely that there are errors in there. But the reason for reviews like this one is to point out that all these different results support the same basic hypothesis.

As far as stocks, specificaly: really? People with cognitive trouble make bad stock picks all the time. Most of us have grandparents who have exhibited exactly this kind of mistake. While it's surely true that the bulk of analyst-driven trades are checked by methods that aren't sensitive to pollution (or just by analysts in different climates), there are certainly enough single-decider trades going on to show a small effect like this if it exists.

I mean, no, one oddball stock market paper doesn't prove much of anything. But in combination with a bunch of other research like this, it's worth taking seriously.

cowsandmilk(10000) about 24 hours ago [-]

Read the linked journal article; the authors don't actually claim it is about cognitive ability, they attribute it to lower risk taking on those days.

known(222) 1 day ago [-]

Poverty also reduces IQ by 13 points https://archive.is/bmj11

larnmar(10000) 1 day ago [-]

Has this study been replicated? It seems a lot like one of those terrible "priming" studies that won't.

Statistics aside, the experimental methodology is terrible. You ask a bunch of people to think about being poor, or not, and that's supposed to be a simulation of being poor or not?

igor47(4189) about 22 hours ago [-]

Plug: if you care about this topic and are interested in working on this problem, we're hiring at Aclima. We've got contracts with the major air quality regulators in California, and are looking to scale. Ping me at [email protected]

maelito(3288) about 22 hours ago [-]

Do you work with https://plumelabs.com ?

brenden2(10000) 1 day ago [-]

This is blogspam. Here's the original post: https://patrickcollison.com/pollution

CelestialTeapot(10000) 1 day ago [-]

That would be a much better link than the one provided. Wasting energy on the 'IQ' issue is not productive to the problem presented: that air pollution does, indeed, cause (hopefully only temporary) cognitive decline. It can be measured on a small scale, for example, see the Allen et al. [1] article showing cognitive decline in an office environment (conventional building vs Green+ building).

1. https://dash.harvard.edu/bitstream/handle/1/27662232/4892924...

wizzwizz4(10000) 1 day ago [-]

Should I upvote despite that, or wait for somebody to post the original? I'm not sure of HN's norms here.

thedudeabides5(10000) about 22 hours ago [-]

Anyone seen analysis or similar data for noise pollution?

yagurastation(4135) about 21 hours ago [-]

This open source air quality citizen *science project now also has noise sensors going: https://luftdaten.info/laermsensor-bauen/

dang(179) about 23 hours ago [-]

Url changed from https://marginalrevolution.com/marginalrevolution/2019/11/ai..., which points to this.

ng12(10000) about 22 hours ago [-]

Things like this make it really hard to follow a discussion as it evolves over time.

PeterStuer(10000) about 24 hours ago [-]

So does this mean we can now expect populist politicians to crank up PM2.5 emitters at a massive scale prior to elections?

hombre_fatal(10000) about 22 hours ago [-]

No need. Some people gleefully accept and partake in air pollution.

Out in the countryside where my parents live in Texas, people are even proud that they burn their trash as if they're sticking it to some sort of treehugger group rather than to the inner lining of their (and all their neighbors' and loved ones') lungs.

WhompingWindows(4217) about 24 hours ago [-]

You mean diesel engines? They're trying that in France.

digitalsushi(4052) about 21 hours ago [-]

short of plants, i want to find a device i can put in my area that reduces the co2 in the air. (my space doesn't support plants, it's too cold).

dheelus(10000) about 21 hours ago [-]

There's a decent amount of research that indicates that indoor air quality doesn't significantly improve with indoor plants. https://time.com/5105027/indoor-plants-air-quality/

driverdan(1482) about 19 hours ago [-]

You want heat recovery ventilation: https://en.wikipedia.org/wiki/Heat_recovery_ventilation

cwkoss(4217) about 21 hours ago [-]

Most CO2 scrubbing systems either use plants (sometimes algae) or expend chemical ingredients (not very practical for long term daily use). Id recommend looking at systems designed for survivalist bunkers.

I did find this interesting project for using a thermosiphon to separate CO2 from air which looks pretty cool, but would likely require a good bit of sun to work well. I imagine it could work with an electric heating element instead of the sun, however.


ghufran_syed(4221) about 18 hours ago [-]

I think this is an important question to answer, but the shehab paper (the first link) provides vertically no useful information. They didn't blind the assessors carrying out the cognitive studies, which on its own invalidates their conclusion. They also didn't randomize the ordering, but appear to have 'investigated for order effects', found 'no statistically robust difference', then proceeded to have every participant do the 'post exposure' test second, making it highly likely that fatigue alone could explain their effects. With only 30 subjects, their tests for ordering effects are likely underpowered.

Also note that they did 11 different tests, but didn't appear to correct for multiple comparisons [1]

This would be a really interesting problem to study, but you'd probably want to look at academic performance in college students before and after transfer between universities e.g urban vs rural (presuming you can measure differences in pollution) between the two, you'd expect to see a relative drop in academic performance going from rural to urban, now that would worry me. But this study is unhelpful.

[1] https://en.m.wikipedia.org/wiki/Multiple_comparisons_problem

radium3d(10000) about 14 hours ago [-]

If the effects are permanent then your results would be inconclusive if studying the same people transferring between polluted and non polluted universities.

pliny(4062) about 6 hours ago [-]

Just want to say it's impossible to tell which paper you're referring to.

The first link on the page isn't a paper. The word shehab doesn't appear anywhere on the page. The word shehab doesn't appear anywhere in the first paper linked. The three quoted sentences / fragments you provide don't appear anywhere in the first paper linked. I opened some of the papers and didn't see any with n=30.

WalterBright(4126) about 19 hours ago [-]

> 'Exposure to CO2 and VOCs at levels found in conventional office buildings was associated with lower cognitive scores than those associated with levels of these compounds found in a Green building.' – Allen et al 2016. The effect seems to kick in at around 1,000 ppm of CO2.

The last time this came up I bought a CO2 meter for my office. I was quite surprised how often it went over 1,000. Outside air runs around 450.

I increased the duty cycle of the house fan, and open a window when it reaches 800. Right now I'm at 505.

Buying the meter was a really good investment.

techiferous(3815) about 16 hours ago [-]

After watching this Tom Scott video https://www.youtube.com/watch?v=1Nh_vxpycEA I bought this CO2 monitor: https://amazon.com/gp/product/B00MB93CK8 It worked quite well and showed that I don't have anything to worry about. Now I'm trying to pass it on to someone else. I'll be willing to sell it for half price if anyone is interested.

amelius(879) about 18 hours ago [-]

Curious, did you notice the cognitive effect yourself?

amelius(879) about 18 hours ago [-]

Perhaps when we commit code, we should have a script that adds 'this code was written at x ppm CO2' to the commit message :)

sxates(3625) about 19 hours ago [-]

I did the same thing at my office and brought it into a conference room with 7 other people. We watched the CO2 climb to about 900 and then it started going down. Seems the climate control system is also tracking CO2 and it started pushing fresh air in when the level got too high. A nice surprise that we have systems managing this - I don't know how common that is in offices though.

popotamonga(10000) about 19 hours ago [-]

Where please? What model.

jobigoud(10000) about 17 hours ago [-]

Reminded me of the videos by Kurtis Baute: 'I sealed myself in a Jar'.


jonstewart(10000) about 17 hours ago [-]

One of the big trends in green/sustainable building is creating relatively air tight buildings in order to cut down on heating and cooling costs (see: Passiv Haus, Passive House, Pretty Good House). In order to do that, though, mechanical ventilation is needed. As a result, Heat-Recovery Ventilation & Enthalpy-Recovery Ventilation residential units can be installed for only $2k-$4K. Seems like a worthwhile investment (for when it's not convenient or possible to open a window).

saberdancer(10000) about 10 hours ago [-]

Does anyone know of a cheap IoT type sensor for CO2? It can be any type, even RasberryPi or Arduino based. I'm not interested in sensors without connectivity options.

About the levels, I'm working on an IoT project and I've seen data from a public school that has heating with radiators (so no real ventilation). One classroom with the sensor would routinely rise above 1000 ppm, with peaks over 3000ppm. Then you'd observe a sudden drop in CO2 level and temperature as they would open up a window. It's insane how high it can get especially if the building lacks proper ventilation.

initself(3677) about 13 hours ago [-]

Which meter did you buy?

pastor_elm(10000) about 18 hours ago [-]

I see low CO2 spaces becoming the next big thing for the privileged classes as CO2 levels rise. Weworks of the future will be advertising this as a definite selling point. Mansions with scrubbers getting it down to 200ppm

moultano(3086) about 18 hours ago [-]

I put one in my son's bedroom and was blown away. He's 3, and calls me in to sleep with him sometimes. It's a small room, and we keep the doors closed so his sister doesn't wake him up and I was blown away by how high the CO2 got with both of us in there: https://www.reddit.com/r/dataisbeautiful/comments/cek4dm/co%...

Psychlist(10000) about 19 hours ago [-]

I have a slightly more capable one but the results are similarly motivating. Right now I'm more concerned with the particulate counts... if 27 is bad, does 120 worse and going outside into 700+ me dumberer?

inside: https://www.uradmonitor.com/tools/dashboard-04/?open=8200009...

outside: https://www.uradmonitor.com/tools/dashboard-04/?open=8200011...

In the office at work is better: https://www.uradmonitor.com/tools/dashboard-04/?open=8200011...

When I was living in Lakemba this prompted me to rig a HEPA filter outside and run my bedroom at positive pressure. That helped a lot. But I've had to move for work so now I'm renting a cardboard box (ok, technically it's 'fibro' on piles with a leaky wooden floor. Outside ~= inside)

cylinder(4097) about 17 hours ago [-]

Been meaning to do this; do you have a link or recommended model?[EDIT: I see you already responded with the model below - https://www.amazon.com/gp/product/B001PDGFR8/ ]

I'm very sensitive to lack of oxygen in a room. Our office is old and some days they don't run the A/C if it's not hot enough outside, and I feel like I'm suffocating and my productivity suffers.

QuanticSausage(10000) about 22 hours ago [-]

I don't know you, but in every lecture in college that was done in a room with no air flow, I got distracted every five minutes or so. If the proportion of C02 had a cognitive effect that I could feel, then I wouldn't be surprised if other kinds of contaminants do the same.

ketzo(4216) about 21 hours ago [-]

Can't find a source, but there's a similar thing with long meetings — after two hours in a conference room, you've added enough CO2 to affect cognitive function.

mberning(10000) 1 day ago [-]

I'm confused - is IQ a reliable way to measure intelligence between individuals, or is it a culturally biased measure that is scientifically bankrupt? I guess it depends on the context.

glofish(10000) 1 day ago [-]

whatever skills IQ tests measure these do correlate (and better than other readily measurable predictors) with various long term successes

what throws most people off is that the skill that IQ test measure is not a just an innate ability but a learned skill that is in demand and often needed for success in that society

yagurastation(4135) about 21 hours ago [-]

Anyone interested should also check out the open source citizen science project https://luftdaten.info/en/

Build your own inexpensive sensor, contribute to the global network. See also https://github.com/opendata-stuttgart

Psychlist(10000) about 19 hours ago [-]

It frustrates me that there are so many of these networks and they don't talk to each other. I deliberately went with a system that measures multiple values, but then I see other people going 'we only measure one thing instead of nine so our units cost half as much'.

I should try to script at least my systems so they write into as many of the open networks as I can manage.


simonebrunozzi(460) about 21 hours ago [-]

For people living in the US like me, this is where you find US pollution maps: https://usa.maps.luftdaten.info/

gerbler(10000) about 11 hours ago [-]

I'd love to set up something like this in my garden. Could I figure this out without speaking German?

H8crilA(10000) about 18 hours ago [-]

Sorry for a pointless post, but this project gives me the CCC vibes. Y'all should attend one day, it's brilliant! In fact it's so rich in things and hacker exhibitions that you don't even have to go to the talks!

rahuldottech(1424) 1 day ago [-]

This really bothers me. I live in a VERY polluted place, and unfortunately moving away won't be an option for me for at least half a decade. What can I do to minimise the health effects of such an atmosphere? I'm planning on getting an indoor air filter for my home.

dmos62(4046) about 23 hours ago [-]

I'm not sure what _VERY_ polluted means, but you might consider taking up more physical activity. On GCN youtube cycling channel they argued (including interviews with some health professionals) that even if you live in a city with considerable pollution, the positive effects of commuting on a bicycle outweigh the negative effects of air pollution. They were primarily talking about Britain-scale pollution of course. The principle might change if transfered to a more extreme environment. Something to look into.

cameldrv(4206) about 22 hours ago [-]

Buy a cheap PM detector (There are a few on Amazon using the Plantower sensor that are ~$50.) Buy any HEPA air filter for your home. Buy the best cabin air filter you can for your car (Bosch makes HEPA filters for some cars), and try to get your office to install something, or just buy one for your desk.

The hard math of it is that no matter what you do, you aren't going to be able to reduce your exposure by more than 75% or so, because you have to go outside sometimes, so going super high end on filtration makes essentially no difference in terms of total exposure.

4ec0755f5522(10000) 1 day ago [-]

Find one that can filter nano particles.


graeme(2852) 1 day ago [-]

Starts with monitoring. Awair or laseregg are good quality monitors. Then you can assess how well filters work.

I think there are masks you can use outside too. You can also check which times of day/days are worst. And which streets. You can carry the laseregg around for local measures.

seanmcdirmid(2212) about 20 hours ago [-]

Do you live in Beijing?

We got an expensive swedish air purifier but it still sucked hard (noisy, made the air smell very stale). You might be able to mitigate some of the effects, but not all of them. And if young kids are involved (or you are expecting), screw the career, move.

puranjay(3888) 1 day ago [-]

I'm assuming Delhi?

Not much you can do except get out. I've already started moving out for the first 2-3 weeks of November and working from a city like Panjim

Richard_East(10000) about 23 hours ago [-]

Its not difficult, firstly buy a monitor which tracks CO2, PM2.5, VOC:


Tracking CO2 is very useful for internal spaces.

And an Air purifier: https://www.amazon.com/Purifier-Display-Formaldehyde-Sterili...

mattjaynes(1281) 1 day ago [-]

David Heinemeier Hansson (DHH of Rails fame) gave a great talk on this recently, though he focused more on indoor air quality and its cognitive impact:


hinkley(4121) about 22 hours ago [-]

My takeaway from this: If you live in new construction (owner, or renter), buy new furniture in the spring/fall when you can leave the windows open. Same for paint.

olivermarks(3031) about 24 hours ago [-]

So logically living and working in the countryside is optimum for your IQ rather than polluted cities and areas?

Psychlist(10000) about 18 hours ago [-]

The bushfires act as a counterbalance in Australia ...

electricslpnsld(3996) about 23 hours ago [-]

The countryside can have some bad environmental problems too (expansion of fracking in the rural Midwest of America, etc).

pkaye(10000) 1 day ago [-]

I thought IQ was not a reliable measure of anything?

mrandish(10000) about 20 hours ago [-]

It depends on how one defines intelligence and what specific things are considered material to the measurement. IQ tests measure the things that IQ tests measure. The challenge is that the things we often want to determine in trying to assess 'intelligence' are multi-dimensional, multi-variate and have complex interactions.

Personally, I consider most typical assessments of a person's general 'smartness' to approach zero utility.

ytNumbers(646) about 24 hours ago [-]

I wouldn't call IQ a completely useless yardstick. Likely through examining all of the evidence regarding IQ and military performance, the United States government has come to the conclusion that an IQ below a certain level disqualifies the person for military service.

Unsurprisingly, US law is a bit complicated on this, but to summarize, it implies that an IQ below 81 ("tenth percentile") is disqualifying, and dictates that persons with IQ's between 81 and 93 ("thirty-first percentile") cannot comprise more than 20% of all enlistees.

larnmar(10000) 1 day ago [-]

If the effect were present, and as large as implied, you'd expect low IQ test results in ultra-polluted Chinese cities such as Shanghai and Beijing. And yet... http://www.city-data.com/forum/world/2348902-china-iq-map-pr...

glofish(10000) 1 day ago [-]

No that is not what it is claimed. Pollution reduces your IQ, not that it makes your IQ lower than someone else's.

The people living in those cities form a so-called 'self-selecting' sample. They are probably, on average better educated, and they recognize the value of education much better. They already would score higher than the others, that's why they moved to the city.

It might just as well be that the same people would score far higher in a non-polluted city.

TrainedMonkey(10000) 1 day ago [-]

Large cities tend to concentrate IQ as smart young adults come to seek better fortune. Furthermore Chinese party selection system, which involves standardized tests and relocation of gifted children to better schools in big cities also contributes heavily.

ErikAugust(4063) 1 day ago [-]

Pardon my ignorance, but how is this IQ data obtained? It is reported by the government? Are IQ tests standardized?

BurningFrog(10000) 1 day ago [-]

How long have they been that polluted though?

Looks like that map is from 2005.

pmart123(4201) 1 day ago [-]

Reading his bullet points, I am not sure why the correlation between daily stock market returns and air quality supports the argument that air pollution reduces IQ.

PeterisP(10000) about 19 hours ago [-]

One hypothesis is that part of the short-term stock movements is causing about publicly visible stupid acts, causing either physical or PR disasters of various sizes. That would imply some correlation, but a relatively small one.

acd(4132) about 22 hours ago [-]

Most small particles are bad for health.

Small particle emissions can be anything, diesel cars emissions, stone dust and flour from bakeries. All small particles are bad for the lungs. Long term exposure of air pollution will lead to health problems.

geomark(4053) about 12 hours ago [-]

Although all small particles are bad for the lungs, there must be a difference in their effects on brain and organs, depending on what those small particles are made of since PM 2.5 particles pass through lung lining and into the blood to be distributed around the body.

bryanmgreen(4171) 1 day ago [-]

I wish the argument against pollution would be the simplest one.

Go to a massively unpopulated area and just breath. Feel the difference? Isn't it intoxicating?

Who gives a crap if umpires make worse calls? Just fight pollution because it's obvious pollution is bad for humans and the planet.

Normally I'm all for extra evidence, but in a situation like this come on. It's basically like immaterial evidence against genocide by saying, I don't know, saying the increased production of bullets to commit genocide reduces national science and health budgets by 12% (fake stat just for context). Who cares! It's just plain bad.


(ps - rant isn't against the article, but a broader frustration about the topic)

zaptheimpaler(3981) about 13 hours ago [-]

Agreed. In a data-driven world, any claim not backed by data will be quickly dismissed :)

swiley(4061) about 23 hours ago [-]

We need to quantify how bad pollution is exactly. How many people should be allowed to die because the ICE ambulance can't reach them or the doctor isn't able to travel to the hospital or food isn't able to be delivered for a reasonable price etc.

larnmar(10000) about 22 hours ago [-]

> I wish the argument against pollution would be the simplest one

Arguments against pollution would be valuable if someone were actually arguing in favour of pollution.

dmos62(4046) about 23 hours ago [-]

I like thinking of it as waste, as opposed to bad. Good-bad is the moral realm. Waste is the realm of practicality. The danger is in causing defensiveness, due to judgment, in people whose open-mindedness would be beneficial. I'm not being apologetic of anyone. It's just that this doesn't have to happen through a dramatic conflict of good v. evil (which probably wouldn't work anyway); there are much better alternatives.

I agree with everything you said. Just wanted to stack my rant on yours there.

konstmonst(10000) about 4 hours ago [-]

George Carlin - 'The Planet is fine, the people are fucked ...'

734129837261(10000) about 19 hours ago [-]

From Western Europe I moved to Mexico City. I work and live here now, and this city suffers intense amounts of pollution almost every day. When you take an Uber somewhere they all drive with their windows open (airco's are expensive to run) and you get the full blast of diesel trucks and older cars' fumes.

I came here for a girl. I'm leaving because of pollution. The city is amazing in its own right, but I feel like I'm breathing lung cancer.

zachkatz(4221) about 18 hours ago [-]

Damn, I'm about to go to Mexico City—was planning to stay for a few months, but I hate dirty air. Is it like that all over the city, or just certain parts?

mc3(10000) about 21 hours ago [-]

Sydney is having a bad day :-( https://aqicn.org/city/sydney/

seanmcdirmid(2212) about 20 hours ago [-]

We would call 159 a clean air day in Beijing.

Forest and bush fires suck. Even Seattle air can smell bad when the wind blows in smoke from a big forest fire up in BC.

Scene_Cast2(10000) 1 day ago [-]

How do you all monitor air quality?

I personally have a couple of devices (an Awair and a Flow). They seem to under-report PM2.5 compared to purpleair, airnow, and waqi.

ng12(10000) about 23 hours ago [-]

I have an Awair. I don't really trust the baseline values but I do trust the deltas. I mostly use it to inform me when I should open a window or turn the air purifier or humidifier on.

fossuser(3926) about 22 hours ago [-]

I have an Awair and two Foobots, they're basically completely unreliable and the software for them is pretty bad.

I've been tempted to get the purple sensors because at least the purple map seems reliable (so maybe their stuff is better?).

doitLP(4191) about 22 hours ago [-]

I have a uHoo and it's been great. The readings are inline with some control sensors I got to test the readings.

vrodic(10000) about 21 hours ago [-]

I have pms3003 you can get for about 12 USD on AliExpress.

I bought a small usb-to-serial adapter connected to Raspberry Pi Zero W.

Here's a combo with a similar device here:


There are plenty of comparisons available online, but there's a good overview site here: https://aqicn.org/sensor/

leonroy(4162) about 22 hours ago [-]

Any idea how good the Awair monitors are? I recall a report saying they were just so-so but Awair recently updated them to be more accurate.

I personally use an IQAir Air Visual which is highly rated and meant to be very accurate as far as consumer grade stuff goes. It's solid, reliable and the iOS app has been great.

That said after installing piHole I noticed it making 17000 DNS requests a day to a non-existent host: https://imgur.com/tPgwGoK

Explains why the DNS Resolver on my router (pfSense) kept crashing at least.

tasty_freeze(10000) about 22 hours ago [-]

I have a uHoo monitor (yes, it is a dumb name).


It logs temperature, relative humidity, air pressure, CO2, CO, TVOC, PM2.5, NO2, and ozone. They are about $300. The device itself doesn't have any UI; you need to connect to it via a phone app.

yagurastation(4135) about 21 hours ago [-]

The inexpensive SDS011 holds up very well and works for this global citizen science project very well: https://luftdaten.info/evaluation/

spladug(3998) 1 day ago [-]

The South Coast (California) Air Quality Management District does some really thorough tests on the quality of the various low-cost sensors: https://www.aqmd.gov/aq-spec/evaluations/summary-pm

fizwhiz(3449) about 23 hours ago [-]

Dylos AQM is the gold standard. It's pricey though and has an old school interface (no fancy UI, IoT or 'AI').

Youden(10000) about 9 hours ago [-]

I've built my own 'IoT' Sensors by hooking up a bunch of components to an ESP32 on a breadboard. I ended up going with:

- CO2: Sensirion SCD30

- Particulates: Amphenol SM-UART-04L

- VOC: Sensirion SGP30

- Temperature/humidity/pressure: Bosch BME680

After hooking everything up I just wrote some code to pipe sensor readings to InfluxDB, which is connected to Grafana.

In the end I have no way of properly testing accuracy but the sensors are consistent between units and behave as expected when exposed to fresh air, which is the most I can look at right now.

yissp(10000) about 21 hours ago [-]

Do you act on this information? Is there really much you can do to mitigate the effects?

cdiamand(3856) about 14 hours ago [-]

I have an arduino uno hooked up to CCS811 (a tiny air sensor chip). We're floating around 550ppm C02 right now, but earlier I was terrified as I watched the levels spike to 4500.

briefcomment(10000) 1 day ago [-]

Measurements from PurpleAir's network of public sensors are available for free.

It's a useful tool if the locals in the area that you're interested in are using PurpleAir's sensors.


skocznymroczny(10000) about 21 hours ago [-]

I have Xiaomi Mijia Air Detector and Xiaomi Mi Purifier 2S. The air detector reports CO2, PM2.5 and TVOC, Purifier reports only PM2.5.

Historical Discussions: AI Clones Your Voice After Listening for 5 Seconds (2018) (November 13, 2019: 907 points)

(909) AI Clones Your Voice After Listening for 5 Seconds (2018)

909 points 6 days ago by lukeplato in 4015th position

google.github.io | Estimated reading time – 5 minutes | comments | anchor

Audio samples from 'Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis'

Paper: arXiv

Authors: Ye Jia *, Yu Zhang *, Ron J. Weiss *, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu. (*: equal contribution.)

Abstract: We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of many different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech from thousands of speakers without transcripts, to generate a fixed-dimensional embedding vector from seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2, which generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a sequence of time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the new task, and is able to synthesize natural speech from speakers that were not seen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.

Click here for more from the Tacotron team.

Speaker Adaptation for Unseen Speakers

Each column corresponds to a single speaker. The speaker name is in 'Dataset SpeakerID' format. All speakers are unseen during training. The first row is the reference audio used to compute the speaker embedding. The rows below that are synthesized by our model using that speaker embedding.

These examples are sampled from the evaluation set for Table 1 and Table 2 in the paper.

VCTK p240 VCTK p260 LibriSpeech 1320 LibriSpeech 3575 LibriSpeech 6829 LibriSpeech 8230
0: Take a look at these pages for crooked creek drive.
1: There are several listings for gas station.
2: Here's the forecast for the next four days.
3: Here is some information about the Gospel of John.
4: His motives were more pragmatic and political.
5: She had three brothers and two sisters.
6: This work reflects a quest for lost identity, a recuperation of an unknown past.
7: There were many editions of these works still being used in the nineteenth century.
8: Modern birds are classified as coelurosaurs by nearly all palaeontologists.
9: He was being fitted for ruling the state, in the words of his biographer.

Example Synthesis of a Sentence in Different Voices

We compare the same sentence synthesized using different speaker embeddings.

These examples correspond to Figure 2 in the paper. The mel spectrograms are visualized for reference utterances used to generate speaker embeddings (left), and the corresponding synthesizer outputs (right). The text-to-spectrogram alignment is shown in red. Three speakers held out of the train sets are used: one male (top) and two female (center and bottom).

Fictitious Speakers

Brand new voices created by randomly sampling from the embedding space. The synthesizer is trained on the LibriSpeech dataset. These examples correspond to Figure 5 in the paper.

Transferring Across Language

Reference audios are recorded by paper authors. The synthesizer is trained on the LibriSpeech dataset. The speaker encoder is only trained on US English dataset.

Reference Synthesized Text
a long time ago, the second December, a cup of water.
a long time ago, the second December, a cup of water.
a long time ago, the second December, a cup of water.
a long time ago, the second December, a cup of water.

All Comments: [-] | anchor

Lucadg(4121) 6 days ago [-]

At first glance there seems to be more more malicious uses than good ones with voice. Yes, hearing someone dear to me say things he/she never said maybe comforting. Anything else?

Maybe some movies with the deceased actor's voice?

But what if someone who wants to hurt me sends me files (or phone calls) from the deceased person saying horrible things like:

- 'I am still alive but left as I was tired of you'

- 'oh Jan, I love you' [fake phone call from the past, where Jan is a lover which never existed]

or even from alive people:

- 'I am leaving you'

- or my live voice saying stuff which gets me fired or in prison.

We will never be able to believe voice again...how will we adapt?

danaris(10000) 6 days ago [-]

It doesn't have to be about creating audio of specific text with <specific person>'s voice; it can be much more about creating audio of specific text with a wide variety of believable people's voices.

I could see this, if it becomes commercially viable, potentially being a huge boon to indie game creation, for instance, since hiring a load of voice actors to record the dialogue for an entire game is vastly more expensive than, say, hiring a bunch of different people to record their voices for 5 seconds—or even, if this ever took off, buying a bunch of samples pre-recorded (or networks pre-trained) for the purpose.

anextomp(10000) 6 days ago [-]

I'm looking forward to an AI assistant that can use my voice to make phone calls

I'm sure some people with selective mutism that would like to use text to speech with their own voice

> How will we adapt?

Digitally signing audio clips

Defenestresque(10000) 6 days ago [-]

People had the exact same concerns when digital image manipulation software became popular, including the 'we will never be able to trust an image again' question.

To answer your question, I think the biggest step we took in adapting to the ever-present risk that an image may have been manipulated is acknowledging that it's possible. As soon as people knew that something could be faked, they realized that having a purported photograph wasn't irrefutable proof that it happened and learned to ask for corroboration before making assumptions.

I think we'll learn to deal with this new development too.

MagnumPIG(10000) 6 days ago [-]

We can barely trust pictures anymore either.

It seems we'll be going back to judging the likelihood of one's actions based on one's reputation, for better or for worse.

There soon won't be such a thing as unreasonable doubt.

a_f(10000) 6 days ago [-]

Maybe a little less insidious than your examples, my first thought was being able to generate voice actor lines for videogames. Maybe not main characters, but background NPCs and the such. Might make the VA union a little nervous going foward though!

willmadden(10000) 5 days ago [-]

>We will never be able to believe voice again...how will we adapt?

'Say, Grandma, before I wire the money, what's the name of your cat?'

shrimp_emoji(10000) 6 days ago [-]

But these attacks are hilarious.

Your question, rephrased: if technology can be used in ways I find psychologically hurtful, should IT BE ALLOWED??? xD

CraneWorm(1902) 6 days ago [-]

Now we can have audiobooks read by anyone we like!

They can direct us to our destination!

They can speak at our funeral, being long dead themselves (as long as there is sufficient training material recorded).

The future is awesome.

taf2(3748) 6 days ago [-]

Imagine when they can also generate the visuals of the book to show you the book as an auto generated video

thomascgalvin(10000) 6 days ago [-]

I legitimately think this could be huge for self-published authors. It takes a skilled professional about forty hours of work to produce an audiobook from a novel-length manuscript. Tacotron could do it in minutes.

jakob223(4064) 6 days ago [-]

The emphasis on a lot of these sentences is all wrong; I wouldn't want to listen to an audiobook by this engine. It's still super impressive/terrifying though.

ccsnags(10000) 6 days ago [-]

I did something like this before my grandmother passed. She was a teacher and loved reading books to kids. I recorded her reading Dr Seuss and the Giving Tree to my cousins so I could give my future children a glimpse of that wonderful woman.

It seems that we aren't far from being able to take those recordings and spin it into a reading of anything. Fascinating. It's kind of scary though. Grandma's voice can read anything. Anything.

jsnk(3378) 6 days ago [-]

I am currently working on an audiobook project, called Odiobooks.com. I hope to release something soon.

If anyone's interested in the project, feel free to contact me at [email protected]

Lordarminius(3903) 6 days ago [-]

AI can make decisions, create deep fakes, and now, clone voices.

It may be that the next big business opportunity lies in creating 'anti-AI' technology just as it did with antiviruses in the 90's and 2000's

sangnoir(10000) 6 days ago [-]

Is it that difficult to create/retrofit an AV container format for cryptographically signed audio and video streams? Key management & revocation could be a pain, but it's something that consumer electronics companies like Apple could do: think of it as the MPEG LA, but with signature checking & non-repudiation.

pcmaffey(3595) 6 days ago [-]

AI that detects AI seems entirely plausible. And like all "anti" measures, is another arms race (and if I put my scifi hat on, is what may lead to AI self-awareness).

filoeleven(4216) 6 days ago [-]

Better provenance is the way to guard against fakes.

For video and audio, I imagine a combination of hardware signing, perhaps with the camera itself living on an isolated, Secure Enclave-like chip, and sending hashes of (incoming images/video * deviceID * trustedTimestamp) to a blockchain or some other public distributed ledger. Getting the timestamp from a service that keeps its own record adds further security.

This obviously requires an internet connection, and would likely be useful mostly for news and government agencies, law enforcement. But if the culture is affected enough by deepfakes, I can imagine it becoming more ubiquitous. The parts are all there, it's a question of utility.

rojobuffalo(4205) 6 days ago [-]

Wouldn't an AI-detector just end up being used like a benchmark for fine-tuning AIs though?

andrewfromx(2895) 6 days ago [-]

ah yes, if u listen carefully to the samples, you can always tell subtle things that make it seem a little off. Maybe if you look at the binary data very carefully, it would be easy to show HUMAN_AUTHENTIC or CREATED_BY_MACHINE and sell this service. Someone have a recording of something you never said? For $99.99 get it checked at AreYouHumanDotCom!

b0rsuk(4211) 5 days ago [-]

The only way to stop a bad guy with an AI is a good guy with an AI...

hans1729(10000) 6 days ago [-]

Good luck with that. The best product you'll come up with is some sort of snake oil. The whole point of GANs is that you can't really 'detect' the synthesized components anymore. Not that this would/will stop people from claiming otherwise in the spirit of profitablity :-)

kevinwang(4097) 6 days ago [-]

Like the booming anti Photoshop industry of the early 2000s?

jacquesm(43) 6 days ago [-]

One more scene from Terminator that we can now do for real.

martin1b(10000) 6 days ago [-]

What's wrong with Wolfy?

thimkerbell(10000) 6 days ago [-]

Ok, what happens to society when this gets to be really good?

fasturdotcom(10000) 5 days ago [-]

computers will have unique voices.

someone will invent a conversational interface for programming

the worlds demand for human programmers will go near 0

everyone will call themselves an engineer and live on basic income

tiborsaas(3886) 6 days ago [-]

Better memes, better apps, better scams

hurrdurr2(10000) 6 days ago [-]

Autocratic regimes can rejoice...they can extract public confessions so much easier now...

blacksmith_tb(10000) 6 days ago [-]

Only ones that care about the appearance of propriety though, presumably most never had to try to produce convincing fakes when their word was already law?

hakanito(4014) 6 days ago [-]

It will be interesting to see if these made-up AI voices can deliver jokes with the same tonality and delivery as good comedians can. I'm just a layman but it feels like a hard problem to solve.

NoodleIncident(10000) 6 days ago [-]

The furthest right column in the first table shows that they might be a long way off from getting timing right. The 5-second sample happens to have a comma, at which the speaker pauses; this pause is in most of the generated output, at seemingly random places in the sentence. The one sentence that does have a comma doesn't use the pause, either.

arkades(4187) 6 days ago [-]

I once got a call while I was lecturing some students. It was repeated three times in three minutes - I assumed it was an emergency and stepped out.

I was greeted by someone explaining that my father had caused a car accident, and they were calling on his behalf. That someone would need to send over some money for repairs or they'd call the police.


They added that their cousin, the driver, is a parolee now holding my father at gunpoint. That if I don't send them money to make them whole, they'll kill my father.

This was super fishy, you know? But still, with things like "life of a loved one" at stake, it's hard to call a bluff.

I can only imagine what I'd have done if I'd heard my fathers voice pleading for help. They might have been able to get any amount of money out of me.

Well, if my father hadn't passed away nine months prior. They were not delighted to hear that.

x2f10(10000) 6 days ago [-]

How were you to send money?

headcanon(4158) 6 days ago [-]

First off, Awesome story.

I have a friend who had something similar happen, he got a frantic call from his grandmother who learned via a scam call that he was in jail across the country and needed bail money. This was a few years ago, so they couldn't have used a duplicate of his voice, but possible they were relying on imperfect memory.

Sweeping generalization, but elderly are and would likely prime targets of this kind of scam in the future since they likely have funds and are less likely to be educated in the state of the art for this kind of tech, not to mention a protective instinct.

mrtweetyhack(10000) 6 days ago [-]

What's your number?

toxicFork(10000) 6 days ago [-]

How do you even prepare for something like that... Do we need to assign identifying keywords to each other when we leave home so we know we are really ourselves? Like a vocal pgp?

throw1234651234(10000) 6 days ago [-]

'They were not delighted to hear that.' - I used to do that, be a smartass. Then I realized at worst they get more info, at best, you are training them and wasting your time.

Then there is that whole thing where they are getting your voice.

inetknght(4170) 6 days ago [-]

I live in Houston. I recently read a Houston Chronicle article describing very similar scenario. I don't have the exact link (it was from 2019), but here's one from 2013 [0].

Combined with the inability to verify the actual phone number displayed on caller ID has led me to tell all of my family to not ever accept a phone call from a number they don't recognize. There's literally zero trust in our phone system upon which we've built our modern economy.

Unfortunately that's not possible for everyone. Some people are legally required to answer the phone, always, even for numbers they don't recognize.

[0]: https://www.chron.com/news/houston-texas/houston/article/Hou...

jancsika(10000) 6 days ago [-]

> I can only imagine what I'd have done if I'd heard my fathers voice pleading for help.

Hm... that certainly gives me pause, and my first reaction was to be very afraid.

On the other hand-- it still doesn't hold a candle to pyramid scheme sales techniques. I mean in a lot of cases those involve your actual loved ones betraying your trust and love in order to sell merchandise for a third party. Yet somehow in the face of a rising tide of those we still have functioning communities in the U.S.

ehsankia(10000) 6 days ago [-]

It was a real rollercoaster reading this comment! Also now, you have to also be worried about talking back being a smartass, because now they will record YOUR voice and use it to contact another loved one...

rolltiide(10000) 6 days ago [-]

I know someone that had a fling abroad and their fling began asking for money for treatment over facebook

The American assumed it was a scam and the person did die

I have often found that truth is stranger than fiction, and people are too conditioned for fiction that they can't perceive truth

andrefmoniz(10000) 6 days ago [-]

It means we can talk to anyone forever

seph-reed(4206) 6 days ago [-]

It means you can make someone talk, non-stop, for an indefinite amount of time.

Now I want to make an art piece that's just a valley girl droning on, and on, and on, and on about believable and obnoxious life experiences. The stores they go to, how they feel about certain colors, what 'too spicy' is. It just never stops.

SubiculumCode(3521) 6 days ago [-]

If acting was ever a good career choice, it isn't now. I am becoming convinced that actors will be replaced wholesale.

Verdex(10000) 6 days ago [-]

I wonder if what we'll see is actors being deep faked etc over by other actors. So of the new actors we will say, 'He plays a good Pitt, but his Clooney leaves something to be desired.'

Movie analysis will get arbitrarily complex. 'Robertson playing Dwayne Johnson playing James Bond really wasn't the right choice for this film. Robertson playing Ferrell playing Bond OR Strong playing Johnson playing Bond would have both been better for the following reasons.'

visarga(3716) 6 days ago [-]

No, actors are still going to act. But their voices will be just an input feature to the speech engine.

AbbasHaiderAli(10000) 6 days ago [-]

Wow, impressive results! Already a few examples in the comments of what bad actors could do this tech. I wanted to share an example of something good.

I lost my dad about 6 years ago after a Stage 4 cancer diagnosis and a 3 month rapid diagnosis. I have some, but not a lot of video content of him from over the years. My mom still misses him terribly so for her 60th birthday I tried to splice together an audio message and greeting from her saying what I thought he would have said.

The work was rough and nowhere near what this Google project could produce. She listens to that poor facsimile every year for her birthday. It's therapeutic for her. With some limits for her mental health of course, I'm sure she would love to hear my dad again with this level of fidelity.

And so would I.

etskinner(10000) 6 days ago [-]

Would be interesting to have a community where people looking to be comforted by their loved one's voice would post whatever snippet of recording they have, then others would listen and see if they know someone who has a similar voice and have them record a message.

dejj(10000) 6 days ago [-]

Seems like,

> This is just

>> All can be returned, all can be taken away

> with extra steps.

ada1981(3708) 6 days ago [-]

When I interviewed Ray Kurzweil we talked about the obvious-in-hindsight insight that his life's work was essentially trying to build an AI to bring his father back to life.

xfkechyk(10000) 5 days ago [-]

Reminds me of the LifeAfter podcast: https://www.stitcher.com/podcast/panoply/the-message

pmoriarty(51) 6 days ago [-]

Philip K Dick wrote about people going to commune with artificial personality constructs of their deceased loved ones.

Unfortunately, it's been a long time since I read it, so I don't remember which book it was in. Maybe someone who's read him more recently can remember.

Update: Apparently, lots of other people wrote about this too, but PKD wrote about this before any of the ones mentioned so far, as he wrote about this in the 1950's or 60's. I'm not sure if he was the absolute first, however. So if anyone knows of any earlier references, it would be interesting to learn about them.

cayblood(4073) 6 days ago [-]

Beautiful. Thanks for sharing. It's good to point out the positive potential uses as well as the negative.

Aaronstotle(10000) 6 days ago [-]

Sorry for your loss, thank you for providing an optimistic example.

nradov(886) 6 days ago [-]

That was part of the original idea behind the Infibond start-up in Israel. Not clear how real it ever was.


drbawb(10000) 6 days ago [-]

When I was doing a computer repair: I remember a woman coming in with a digital answering machine; the kind that stored its recordings in volatile flash. During a thunderstorm the night prior the machine lost power, and subsequently lost all the stored recordings. As it happens some of those lost recordings included messages from the woman's late mother.

That moment has stuck with me for many, many years. The heartbreak on her face, combined with my own frustration of knowing that no amount of luck (or skill) will ever be able to flip the bits of that flash chip back to a permutation which contains samples of her loved one's voice.

Fast forward to the present, my own grandmother passed away shortly after the start of 2019. I was able to salvage some of the many voicemails she had left me over the years, despite having had probably five or six cellphones during that period. Why? I used Google Voice, which is part of their Google Takeout data exfiltration program. I was able to download all those voicemails as MP3s, neatly categorized by caller. My grandma was very terse, so most of them are exactly the same: 'Robert, can you please call me?', but in spite of that each one is unique and precious to me. A lot of developers think about getting data into their platform, but it seems to me that not as many think about users getting their data, sometimes precious & irreplaceable, back out of the platform.

mottiden(3771) 6 days ago [-]

I'm deeply sorry for your loss. Thanks for sharing your story.

mcdoh(3900) 6 days ago [-]

It could also be used for recreating voices for people that have lost theirs, like Roger Ebert. I think he benefited by having so much of his voice already recorded, this would make it much easier for regular people.


ehsankia(10000) 6 days ago [-]

This podcast explores how similar tech is used to give voice back to people who have lost it due to voice impairment. Basically allowing all the people using machines that sound like the classic Hawking computer voice to have their own voice instead.


daenz(538) 6 days ago [-]

I'm imagining Siri with the voice of your partner.

rshi(10000) 6 days ago [-]

I wonder what the legal implications of this alongside similar developments like deepfakes are going to be in the next couple years. We're already having fraudsters impersonate CEOs using Deep learning-aided Voice generation[1] due to just how low the barrier of entry is now. There's already a public implementation of the paper out [2]!

[1]: https://www.wsj.com/articles/fraudsters-use-ai-to-mimic-ceos... [2]: https://github.com/CorentinJ/Real-Time-Voice-Cloning

qznc(1788) 6 days ago [-]

Currently watching UK: https://mobile.twitter.com/FutureAdvocacy/status/11942824810...

Sorry for the Twitter link but Future Advocacys website seems to be down.

jermaustin1(3289) 6 days ago [-]

The latest episode of Blacklist had a dark plot based on deep-fakes.

IshKebab(10000) 6 days ago [-]

CorentinJ's implementation isn't quite as good as Google's - I think with some of Google's samples I couldn't tell that they weren't real, especially over the phone. But I could easily tell with CorentinJ's.

That seems to be common with open implementations of Google's voice synthesis and speech recognition work. I guess they hold back some of the secret sauce, or can afford to train it more.

jchook(4178) 6 days ago [-]

Lyrebird has had similar technology for a few years now.


Conlectus(10000) 6 days ago [-]

Worth noting that Lyrebird is very rough -- at least last I tried -- and produced extremely robotic sounding (though recognizable) audio.

This method has much clearer audio, but seems to lack generality / TTS capability.

0xcafecafe(4222) 6 days ago [-]

An apt name for the technology considering the marvel of nature that the lyre bird is.


wenc(4191) 6 days ago [-]

Yes, I remember seeing this a while ago. Nice work by the MILA group in Montreal.

frandroid(3219) 6 days ago [-]

The synthesized voices sound similar but would probably not fool a good voice-print system.

macca321(3574) 6 days ago [-]

No, but they would probably fool a friend or family member over a phone line. Yikes.

Riccardo_G(10000) 6 days ago [-]

What it is doing is not really cloning, but because it was trained on 18k different voices, it actually finds one that is closest to yours, and uses that one. It can do a bit of interpolation to create an embedding which is closer to your own, but only if it is well represented by a mix of other voices. Real voice cloning like at https://replicastudios.com/ can take just a minute or two of audio, and it does a fairly good job, and it is always improving. With more audio you start being able to also play with emotion and styles, which is very cool!

JaRail(10000) 6 days ago [-]

I'm not really sure where you're getting this. It doesn't pick a specific voice from a database to use.

From their introduction: 'Our approach is to decouple speaker modeling from speech synthesis by independently training a speaker-discriminative embedding network that captures the space of speaker characteristics and training a high quality TTS model on a smaller dataset conditioned on the representation learned by the first network.'

Section 2 of the paper explains how it works. Two minute papers also goes through it if you'd prefer a video. Link: https://youtu.be/0sR1rU3gLzQ

ehsankia(10000) 6 days ago [-]

Was it 18k voices or samples? Also, is it finding the closest voice, or is it a continuous parameter space formed from the voices?

ihm(4016) 6 days ago [-]

Reading this headline I begin to understand certain people's worry about having their soul stolen upon being photographed.

code_code(10000) 6 days ago [-]

Even decades ago I gave such reported concern with photographs more credence than the typical western account of it. I also wondered if the translation was precise enough -- could it (in some cases) have reflected a concern with 'essence' more generally? Even without reference to a soul the concern can be a bit immaterial.

jmann99999(10000) 6 days ago [-]

You mean the abo-diginals? :-)

Dramatize(3711) 6 days ago [-]

I'm looking forward to moving past just talking about the AI and concentrate on the new products the tech enables.

I guess it's similar to how most photos are a means to an end now, rather than the final product. ie satellite imaging or Instagram.

nwsm(3931) 6 days ago [-]

Images of us can be sourced from any number of places: social media, government surveillance, private surveillance. Video less so but from the same sources. Audio from phone companies, VoIP services, surveillance, etc. Health data easily from a number of private companies if you use new-age 'health' services, or less easily (illegally) from health records.

Maybe we can find solace in the fact that is or will soon be infeasible to avoid, so we needn't try to avoid it.

jacquesm(43) 6 days ago [-]

I had that exact same thought the other day in respect to biometric data from photographs.

nsxwolf(3693) 6 days ago [-]

A staple of science fiction comes to life.

teraflop(10000) 6 days ago [-]

And conversely, Star Trek's frequent use of voiceprint authentication looks sillier by the day.

maxwell(1868) 6 days ago [-]

Everyone's really pitching in doing their part to get the T-800 ready within 10 years.

pier25(3433) 6 days ago [-]

In a couple of years we won't be able to trust any media.

I wonder what the cultural implications will be, much like photoshopped models and actors have change the beauty ideals.

gknoy(3924) 6 days ago [-]

I'm sure we can always trust Eliza Cassan.

Riccardo_G(10000) 6 days ago [-]

There are a lot of security features that are being put in place to help us all understand what is real and fake. Of course there is still a lot of work to be done and the technology is very new, but at https://replicastudios.com/ work in watermarking audio, as well as authentication of Replica voices and detection of fake, non-authorised replicas is already in progress.

Facebook, YouTube, Twitter, and the likes, will then be able to let users know what is using real (actual real voices, or the authenticated Replica voice) or fake voices.

badrequest(4054) 6 days ago [-]

Has anybody tried making an AI that generates 5 seconds of arbitrary speech to feed into this AI?

kleer001(3895) 6 days ago [-]

No to create an Ai for that. Just like with any neural network random noise can be fed into the detector networks, fedback to its self then used to create novel maps. Like deepdream.

moyix(2631) 6 days ago [-]

This is from 2018 – does anyone know if there are pretrained models and code for this? I found https://github.com/CorentinJ/Real-Time-Voice-Cloning , but the generated audio quality was much worse than the samples here.

JaRail(10000) 6 days ago [-]

The biggest missing piece is WaveNet, which is Google's proprietary voice-synthesizer. With only the models trained for this paper, the best you could build is a voice-recognizer. As far as I know, Google only allows people to do TTS with one of their provided voices.

I don't expect them to open it up until other companies/academics have achieved similar results. It's too much of a competitive advantage right now. Alexa, Siri, etc all sound like robots compared to WaveNet (google assistant).

werds(10000) 6 days ago [-]

Anybody else notice how that Scottish male reference voice sounds considerably more English in the synthesized versions.

VBprogrammer(3337) 6 days ago [-]

Yeah, maybe it's just having a more sensitive ear to the Scottish accent but that to me was the furthest from the reference by far.

jonplackett(4184) 6 days ago [-]

Do papers like this have code to play with anywhere?

lukeplato(4015) 6 days ago [-]

from two minute papers video description: > An unofficial implementation of this paper is available here. Note that this was not made by the authors of the original paper and may contain deviations from the described technique - please judge its results accordingly! https://github.com/CorentinJ/Real-Time-Voice-Cloning

grawprog(3566) 6 days ago [-]

So....I'm going to paste the abstract here because the headline is incredibly misleading and should be changed.

>Abstract: We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of many different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech from thousands of speakers without transcripts, to generate a fixed-dimensional embedding vector from seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2, which generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a sequence of time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the new task, and is able to synthesize natural speech from speakers that were not seen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.

aerovistae(2940) 6 days ago [-]

Do you want to elaborate on how the title is misleading? From reading the abstract it seems accurate to me.

godelzilla(10000) 6 days ago [-]

Thanks... Too long of a scroll to find somebody posting the actual science behind the click-bait.

lukeplato(4015) 6 days ago [-]

I took the headline from this Two Minute Papers video: https://www.youtube.com/watch?v=0sR1rU3gLzQ

agentultra(4126) 6 days ago [-]

Impressive but they still sound like robots imitating humans. I can only imagine the chaos this could cause, if used by bad actors, as it continues to improve. If someone took my voice I'm not sure that my partner would know it wasn't me. That would enable all kinds of social engineering attacks.

Enginerrrd(10000) 6 days ago [-]

Exactly what I was thinking. We already have a problem with scammers claiming to be relatives who need some quick cash wired over. Imagine how much more effective that could be if it actually sound indistinguishable from a loved one.

anoncow(4019) 6 days ago [-]

Mission Impossible is now Mission Possible.

onion2k(2004) 6 days ago [-]

It always was. IMF never failed in the TV shows or the more recent films. :)

irrational(10000) 6 days ago [-]

So, now we can't trust text, images, videos, or audio. Any of them could be fake. What is left?

dole(10000) 6 days ago [-]

AI-generated scents. After you die, your body odor, sprayed on an article of clothing, as a monthly subscription to your loved ones.

costcopizza(10000) 6 days ago [-]

I was born in the wrong era. Christ.

ttul(10000) 6 days ago [-]

You were born in 0 CE?

nmeofthestate(10000) 6 days ago [-]

VCTK p240: duplicates the (a) north of England accent well.

VCTK p260: all over the place accent-wise.

LibriSpeech: can't really comment on the American examples, but they seem decent.

jdbernard(10000) 6 days ago [-]

Sample 9 is a good example. The pronunciation of 'biographer' is consistent even when it should be very different. All of the examples stress the first syllable but an American would stress the second.

goodmachine(3549) 6 days ago [-]

The malign applications of this technology greatly outweigh the benign. Discuss.

telesilla(4203) 6 days ago [-]

On a benign level, many VO artists will find themselves out of work now that we can have Don LaFontaine back.

On more positive outlook, perhaps this, along with deepfakes, propels us faster towards an evidence-based society.

tgsovlerkhgsel(10000) 6 days ago [-]

I think there are many benign applications, and definitely a massive potential for abuse. In practice, it will be used mostly for benign applications, I think, but due to the outsized impact, you could still say the malign applications outweigh.

However, what I found reassuring is that the paper actually addresses these concerns:

'However, it is also important to note the potential for misuse of this technology, for example impersonating someone's voice without their consent. In order to address safety concerns consistent with principles such as [1], we verify that voices generated by the proposed model can easily be distinguished from real voices'

This doesn't mean it won't fool humans, especially when used in a carefully crafted setting (low-quality phone call with distressing content).

SamBam(10000) 6 days ago [-]

Con: It's now easier than ever to fake someone saying something outrageous, and have that lie spread across the world long before the truth can get its boots on.

Pro: Humphrey Bogart can direct you to your destination!

I admit, it's a hard choice.

mistermann(4181) 6 days ago [-]

Here's an outlandishly optimistic take on the possibilities.....as the internet and media is flooded with increasingly convincing but false representations of reality, a widespread habit of greater skepticism of 'the facts' starts to spread throughout society, leading people to alter the speed at which they form new opinions on various issues (possibly degrading confidence in preexisting opinions), and the manner in which they construct their personal mental model of reality. As the frequency with which an individual's brain is fed data inconsistent with directly observable reality increases, might a tipping point be eventually released where it refuses to continue making snap decisions, and instead delays judgement until a later point in time when more information is available?

Perhaps loosely similar to being forced into a stature of 'noting' in meditation: https://www.insightmeditationcenter.org/books-articles/menta...

carapace(2884) 6 days ago [-]

Yes, I think there are almost no legitimate uses for the Farnsworth Device. https://theinfosphere.org/A_Device_That_Makes_Anyone_Sound_L...

My personal hell: My mother has dementia and a land line telephone.

Scammers call all the time. All day long. (Although the last few days have been pretty good, I assume somebody somewhere is doing their jobs. The scammers will adapt.) One thing they do is spoof their number to have the same area code and prefix as the one they're calling, so it's like 'Oh, is this a neighbor?' or something, but of course it's not. It's an automated machine abusing the telephone network to try to steal money from a little old lady with dementia.

Evil men with robots are attacking my mom. Another one called while I was writing this post!

This is a goddamned sci-fi dystopia.

And now the robo-thieving bastards can imitate my voice!?

I'm going to have to get her one of those satellite-linked walkie-talkies or something. Thank God she doesn't use the internet.

danShumway(4077) 6 days ago [-]

I'm currently working by myself on a game that will likely launch without voice acting (text only) because I don't have the money or skill to find and pay voice actors.

If I could act out the dialog myself and then purchase or generate voices other than my own to overlay on top of those performances, the quality and accessibility of my finished product would go up dramatically.

That would also open up the door for more people to be able to mod the game and add additional dialog options. A big complication with voice acting is that it's essentially static. Even though a big focus of my game is modability, if I do voice acting no-one else can add additional levels or areas or expand on the characters without breaking the recorded dialog.

It would be amazing if I could ship some kind of compiler so that modders could record themselves talking through new/changed dialog, and then insert it seamlessly into the game with the correct character's voices.

AnIdiotOnTheNet(3914) 6 days ago [-]

Con/Pro, depending on perspective: people will have to give up the illusion that they ever really could definitively tell truth from fiction.

TomMckenny(10000) 6 days ago [-]

Are you asking about cordite or gunpowder in genneral?

clort(10000) 6 days ago [-]

There is the obvious, using this technology to put words in somebodies mouth. The more nefarious though, is that certain people who lie all the time can easily claim that a lie you recorded them saying wasn't them and now you can't prove otherwise. Certain people can say whatever they want and just deny it later. 'Fake News' indeed..

Angostura(3660) 6 days ago [-]

My bank's (HSBC) telephone banking offers the option do away with a PIN and instead a 'my voice is my password'phrase system.

I'm glad I never opted in.

ropiwqefjnpoa(10000) 6 days ago [-]

Oh god, Sneakers

bidluo(10000) 6 days ago [-]

The Tax Office in Australia does the same thing and push it every time I call, I imagine it branches out to the other gov bodies too. It's fun listening to the whole spiel about 'Like a fingerprint your voice is unique to you'.

fasturdotcom(10000) 5 days ago [-]

unchecked, the progress of this technology and the staleness of banking security might cause entire institutions to fail

shostack(4217) 5 days ago [-]

I recently called Chase and got some message like 'your voice may be recorded for verification purposes' or something to that extent. Creeped me out and I don't recall ever opting into that specifically, so I'm guessing it is an opt out buried in some agreement.

kleer001(3895) 6 days ago [-]

Saw this on Minute papers last night and had a discussion with my partner about if we needed a secret password or not to tell if it were really one or the other on the phone. I figured that we had enough shared history that that wouldn't be a problem. Then we realized that there's no such thing as a simulated sense of humor yet and that that would be the best natural encryption to any communication.

nighthawk454(10000) 6 days ago [-]

Here's the link to the Two Minute Papers video if anyone else is curious: https://www.youtube.com/watch?v=0sR1rU3gLzQ

toxicFork(10000) 6 days ago [-]

2023: ai identifies your sense of humour after hearing you breathe for 0.7 milliseconds

undershirt(3045) 6 days ago [-]

"When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success."


robertjwebb(10000) 5 days ago [-]

The thing is, these topics have already been discussed by philosophers! Questions of authenticity, human subjectivity, reproducibility etc are not new. But for the average joe and the non-philosophically-inclined techie, the thing has to actually exist before they start really talking about it.

kingkawn(3320) 6 days ago [-]

This is part of the process of truth being removed from all recording. Soon we will be back to a state where the only certainty is the person we speak to in person

Hoasi(4129) 6 days ago [-]

> Soon we will be back to a state where the only certainty is the person we speak to in person

That is until we are able to tell whether that 'reality' is synthesized or not.

Historical Discussions: PayPal stops payouts to models on Pornhub (November 14, 2019: 870 points)

(870) PayPal stops payouts to models on Pornhub

870 points 5 days ago by protomyth in 96th position

www.vice.com | Estimated reading time – 4 minutes | comments | anchor

YouTube this week updated the company's terms of service to accomplish what's now an obvious goal of any giant corporation: giving itself more leeway to do whatever it wants.

The latest update to YouTube's TOS, unveiled over the weekend, makes it clear that the company "is under no obligation to host or serve content" if it doesn't want to. Another section of the update further clarifies that the company is free to boot users off any of its platforms—from YouTube to Gmail—if they're not worth the expense.

"YouTube may terminate your access, or your Google account's access to all or part of the service if YouTube believes, in its sole discretion, that provision of the service to you is no longer commercially viable," the updated language reads.

The changes are the third TOS revision in the last year and weren't greeted warmly by many content creators or Reddit users. Many expressed concern the updated changes could be used to unfairly drive users off the platform or even block Adblock users.

Google for its part told Motherboard there was nothing nefarious about the update.

"We updated the language in our Terms of Service to make it more transparent and easy to understand," a Google spokesperson told Motherboard. "All pretty standard practice," the spokesperson added, noting that there was no corresponding changes to company products, users settings, or how the company works with creators.

Google was also quick to point out that the references to "commercial viability" aren't entirely new, and some variant of the same language has existed in its TOS since around 2018. The latest update just further clarifies YouTube's right to kick you offline at its discretion.

YouTube has faced widespread criticism over the company's moderating of online content. Some believe the company's policies are arbitrary and ill defined, and others insist the company isn't going far enough to police things like internet hate speech. The company has come under particular fire for exploitative content that targets children.

Standard practice or not, copyright and policy lawyer Dylan Gilbert told Motherboard in an email that such TOS updates pretty routinely only favor one side of the corporate consumer equation.

"Dominant tech platforms like YouTube can harm users both economically and non-economically by taking action against them for alleged misconduct, or even for not being 'commercially viable' in this case, by terminating their accounts or de-monetizing content," Gilbert told Motherboard.

Gilbert was quick to point out that the updated TOS doesn't mean that YouTube will terminate any accounts, just that it can. The real problem is that consumers don't really have much recourse if they're targeted unfairly. However flawed copyright laws like the Digital Millennium Copyright Act (DMCA) are, there's at least some degree of recourse for bogus takedowns.

In this case, ever-shifting fine print means YouTube and Google can do pretty much whatever it likes with little to no public recourse, banning users or channels at its discretion. There's no rules ensuring the process is fair, and there's no neutral third party who'll independently audit the process to ensure users aren't being unfairly screwed.

"Folks are justified in being alarmed at the broadening of the language in the TOS, but it doesn't necessarily mean that things are going to turn into YouTube starting to terminate left and right," Gilbert said. "The biggest issue in our view is that the users don't have any recourse if the power gets abused."

All Comments: [-] | anchor

unreal37(3520) 5 days ago [-]

17 years ago, my business partners and I were approached by a porn website looking to get their website redone. (We didn't take the job, but it was an interesting meeting.) He said that his biggest problem was 'refunds'.

It seems it was a common thing for people to buy his service (access to porn), and then a few days later claim to the credit card company that it was an unauthorized charge.

He told a story of an irate wife calling him on the phone one day and demanding the charge be taken off her husband's credit card bill. She was insistent that there was 'no way' her husband would sign up for a porn website. Well of course he did (or was it their son?). But he couldn't admit it to his wife. And so she called the office of the porn website to complain.

I'm sure that hasn't changed 17 years later. Credit card chargebacks must still be a huge problem.

Chargebacks make it a customer service nightmare. One of the highest refund rates that credit cards get, the highest incidence of actual fraud (charges that really weren't authorized), undisclosed recurring charges, etc.

Still, cutting off service to the models seems unfair. But refusing to deal with the websites themselves actually has underlying business reasons other than morality.

johnpowell(4220) 4 days ago [-]

I have signed up for trials just because I wanted to grab a certain clip. And while I was in there I would download as much as I can and then cancel. And there is also that once you are done watching the last thing you want to do is watch more. And did I just pay for that? Wasn't worth it.

Family Guy sums it up nicely.


habosa(3609) 5 days ago [-]

I hate that any online company can set arbitrary Terms of Service which they know for a FACT I have not read (I didn't click) and then decide to pull the rug out from me at any time because of that.

These companies are seriously important in our lives. If you take away my payment source or my email or whatever you are potentially setting me on a path to ruin.

I don't know what the answer is but I do think these companies should be required to have an appeal process of some kind that's regulated. They have so much power.

gorgoiler(10000) 5 days ago [-]

Even if you did click to accept the terms of service, their server knows you probably didn't read it. They know exactly how long it was between you being served the contract, and you accepting it.

Clearly if that time is anything under a number of minutes or hours you can't possibly be agreeing to the contract and so they should be duty bound to refuse service.

I look forward to a test case where a judge rules a terms-of-service contract unenforceable because the vendor logged that I read it in 3.2 seconds and accepted that as me having read it.

Or maybe it's time to start being that person again, who reads contracts slowly and in full before signing.

mola(10000) 4 days ago [-]

Talk about one sided agenda driven reporting... Vice is such a sad excuse of journalism. Too bad it is so representative of the entire media world. Most of us forget something different can exist.

mola(10000) 4 days ago [-]

To be clear, there are many issues with this industry, most players are very sketchy especially how they treat the performers. Thats part of the reason why mainstream businesses don't want anything to do with it. Maybe if they'll clean up their act and get rid of the criminal elements, an article with this tone would be warranted.

fooxbar(10000) 5 days ago [-]

Bitcoin Cash to the rescue!

arcticbull(4024) 5 days ago [-]

Remind me again how you plan to pay your employees when all the fiat gateways also preclude this kind of transaction?

linuxhansl(4213) 5 days ago [-]

Whatever you might think of the various professions... As long as it is legal, it seems the major payment service providers should not pass on their moral code onto everybody else.

What's next? You get no water/power/whatever if you run a business that the service providers simply do not agree with?

draw_down(10000) 5 days ago [-]

It's not obvious, but morality and fraud are linked. People buying something considered immoral, when confronted by a spouse or other loved one, lie and say the charge is fraudulent. In this way "immoral" businesses rack up lots of chargebacks, which pisses off financial institutions. Payment providers need to remain on good terms with those institutions. So dealing with these businesses can create more risk than is worth it.

Nobody who runs a payment processor is wagging their finger at camgirls or dispensaries or whatever. They could not care less.

I'm not saying it's fair or good, I'm saying it's about risk and financial institutions not morality.

httpsterio(10000) 5 days ago [-]

I think PayPal might've publicly said that this is solely because porn poses a greater risks and SFW businesses just due to the amount of actually fraudulent payments and account hacking performed to pay for porn. Also, there's the less obvious issue with charge backs and people lying about getting hacked and having to deal with all that.

JackFr(3299) 5 days ago [-]

Talk to tobacco about being deplatformed. Tobacco companies have banking, but fewer places sell cigarettes or allow smoking. It's not discrimination —- neither cam girls or tobacco farmers are a protected class. Payment processing isn't a right.

zamadatix(10000) 5 days ago [-]

Water/power are utilities and are regulated differently than other industries.

But your general question is an open debate. If you run a hosting company do you have to host things you disagree with? If you run a transaction company doy you have to exchange with parties you disagree with? What if a certain type of transaction is too high risk, do they have to do it anyways?

Personally I'd like to see things lean more towards being agnostic to avoid amplification of the popular opinion but it isn't a cut and dry topic without corner cases or hard to define regulations.

xtracto(10000) 5 days ago [-]

The problem for those providers is often not their own morals per se. But the fear of being related to certain verticals. This oftentimes has a real impact on lost deals from other companies that do have a moral judgment and wont do business with you if you are related say to marijuana selling companies.

ElCapitanMarkla(10000) 5 days ago [-]

I totally agree with you but I wonder if it's a moral thing (like those assholes have any) or if it's a fraud thing. I've worked in the online dating space and we always had trouble with payment providers due to a really high rate of fraudulent transactions.

gorgoiler(10000) 5 days ago [-]

What are the biggest cost centers for a new payment provider? Software engineers? Rack space? Dispute (fraud) resolution? Marketing?

cududa(3408) 5 days ago [-]

Getting certified as a payment processor can take a year or two AFTER your product is built.

arcticbull(4024) 5 days ago [-]

Risk and compliance, honestly. Margins are razor thin, for instance Square's most recent quarterly filings where they indicated transaction margins had them at ~1%. You mess up, you've gotta process $99 more for every $1 you lost. Compliance paperwork is expensive enough but you make a mistake and prepare to have some uncomfortable chats with the government.

user5994461(2914) 5 days ago [-]

It's not about costs. It's about trust and scale, neither of which you can buy.

Payment providers earn a fraction of a percent on transactions, so they must have extremely high volume in payments to be sustainable. It takes hundreds of millions just to be able to afford some workers and office.

Which can't happen because trust and fraud. A legit customer, especially with any amount of volume, would never adopt a small or unknown provider. Corollary, the provider only attracts low grade or fraudsters, assuming they ain't the fraudster themselves.

spookthesunset(10000) 5 days ago [-]


One of the factors in determining your interchange fees as a merchant is your risk profile. It is why a grocery store has a different fee structure than a gas station or a restaurant.

Almost the entire industry is driven by risk profiles.

magashna(10000) 5 days ago [-]

Would be interesting to see if this pushes PH to build their own infra top to bottom

Justsignedup(4205) 5 days ago [-]

necessity breeds innovation

on_and_off(10000) 5 days ago [-]

It would be awesome !

There is a long history of people getting screwed to the point of having their company go out of business because paypal decides that they will freeze their funds for 6 months or that they no longer want to be their payment platform.

It would not be a big issue if there were 3 or 4 competitors .. but as things are, paypal is what people use to buy things online or to send money to a friend.

thrower123(3005) 5 days ago [-]

I'm not sure how they would avoid getting frozen out by Visa/Mastercard/Amex in turn

mbdesign(10000) 5 days ago [-]

I know of Spankchain[1] who are working on solving this on the Ethereum network.

Otherwise, I see DAI token being ideal for this coupled with locking your DAI in MakerDAO [2] to earn interest.

[1] https://spankchain.com/ [2] https://cdp.makerdao.com/

pentae(3876) 2 days ago [-]

It's a shame nobody cares

mjevans(10000) 5 days ago [-]

A chilling effect on free speech. Real alternatives to paypal and credit cards need to enter the market.

It sadly looks like my favorite hope is still not ready, but I am hopeful that this is due to slow and careful development that will lead to a stable long term outcome. https://taler.net/en/faq.html

rvz(4080) 5 days ago [-]

> A chilling effect on free speech. Real alternatives to paypal and credit cards need to enter the market.

This would have been a huge problem in 2012, Now there are realistic solutions to avoid this. Just pay by cryptocurrencies on to a Coinbase Card and you can withdraw the cash and pay anonymously.

Job done.

dahart(3788) 4 days ago [-]

> A chilling effect on free speech.

It will have a chilling effect on Pornhub for sure, but porn and commerce speech are already both separately and specifically quite limited in their free speech protections in the U.S. https://en.m.wikipedia.org/wiki/United_States_free_speech_ex... Put them together and it might be difficult to demonstrate that there was much expectation of free speech in this case in the first place.

grzte(10000) 5 days ago [-]

Free speech? Corporations can set their own rules!!! The same thing they say to stop the evil nazees.

miohtama(3499) 5 days ago [-]

There are alternatives. Even obvious ones like wire transfers (ACH, SEPA) which have less fraud due to enforced two-factor and better online banking cyber security. PayPal, on the other hand, is free game after you infect a Windows PC with any malware.

The problem is that your CC number is both username and password. Cardsters are looking ways to convert soft credit card money to something without chargebacks. These services are fraud and money laundering magnets. Think it as a same as publishing fake eBooks on Amazon and then sell them to launder your income clean.


toomuchtodo(2025) 5 days ago [-]

Stripe won't touch adult content, I've tried (chargeback ratio is very high, I don't blame them). Very hard to find processors who will work with adult content of any sort.

The real problem is that person to person money transfers shouldn't be prohibited unless by law. Perhaps Pornhub, in the US at least, could use something like Zelle to mediate transfers between patrons and performers (using their bank account to make Zelle requests, and then distributing that out to performers with Zelle). At least until the Fed gets their ACH modernization act together and we get near real time payments.

ur-whale(4149) 5 days ago [-]

> Real alternatives to paypal and credit cards need to enter the market.

They already have. They're called cryptos.

[edit]: judging by the downvotes, looks like the sheep instinct is strong on HN

lazzlazzlazz(10000) 5 days ago [-]

There's almost no reason payment couldn't be handled with a reputable, totally decentralized stablecoin like Dai. The solutions exist, but they need to be socialized and streamlined.

EDIT: Would love some reasoning around downvotes.

hychoi99(10000) 5 days ago [-]

Good. Porn is poison to society

Jamwinner(10000) 5 days ago [-]

This is unsubsatntive flamebait.

simonswords82(2829) 5 days ago [-]


leonlag(10000) 5 days ago [-]

Even if you agree with this premise. It's not on you to decide what people poison themselves with.

cabaalis(10000) 5 days ago [-]

IANAL, why isn't a payment processor refusing to service a payment a form of tortious interference?

mrosett(10000) 5 days ago [-]

Refusing to facilitate isn't interference.

justboxing(3601) 5 days ago [-]

Calling Sex Workers 'Models' is kinda rich.

rationalfaith(10000) 5 days ago [-]

One need only look at the STDs impacting each profession to clearly see models != sex slaves (uh I mean 'workers' uh i mean human trafficking)

lonelappde(10000) 5 days ago [-]

They so stuff for people to look at at sometimes copy. That's modelling.

jimnotgym(3758) 5 days ago [-]

A contributor once wrote on hn about the huge market for a porn friendly payment system. Key feature, must not look suspicious on credit card bill...

uberduber(10000) 5 days ago [-]

Have worked in men's and women's porn. Women have this solved, all their porn is from Amazon. Historically don't chargeback so that hurdle never existed.

I wouldn't be surprised if more than half of Kindle revenue and certainly more than half of Kindle profits were just women's porn. Amazon occasionally has half-hearted crackdowns because some conservative finds it and complains.

TulliusCicero(10000) 5 days ago [-]

There's a good sized market, but IIRC porn payments have an outsized portion of problems, like lots of chargebacks.

__jal(10000) 5 days ago [-]

> Key feature, must not look suspicious on credit card bill

That's called a hotel room.

emidln(2881) 5 days ago [-]

These have existed for quite awhile. CCBill was the big player around 10 years ago. They take a slightly larger cut for the service relative to something like a Braintree or a Paypal.

gnahckire(10000) 5 days ago [-]

That's essentially what the movie Middle Men was about: https://en.wikipedia.org/wiki/Middle_Men_(film)

bproctor(10000) 5 days ago [-]

There are a lot of porn friendly payment systems. The big issue is chargebacks and fraud.

hindsightbias(4174) 5 days ago [-]

How about a giftcard system?

spookthesunset(10000) 5 days ago [-]

There are plenty of porn friendly payment providers. They just cost a lot of money relative to 'normal' merchants because of a few simple reasons:

1) Chargebacks -- people chargeback the shit out of these kinds of transactions. Significant other sees the bill, gets pissed / confused about the charge / whatever and gets it charged back. 2) Fraud -- people use stolen CC's to pay for this kind of thing. 3) Porn operators are, as a whole, rather shady themselves and present their own set of risks to a payment provider. A lot of those chargebacks in #1 are actually valid chargebacks.

You could say 'use bitcoin', which would solve the chargeback problem but do nothing for fraud and shady porn sites. If anything it would exacerbate the shady behavior because consumers would have no recourse against fraud.

In short, collecting payment for porn sites a tough, costly business.

noxer(10000) 5 days ago [-]

Crypto? Credit card bill would show whatever exchange people use to buy crypto.

WrtCdEvrydy(3985) 5 days ago [-]

The issue here is that it needs to be easy for a person viewing their credit card bill to know that they paid for something... and not be super weird.

I had a buddy who nearly did a chargeback against a club that put 'E 11 EVN' on his bill... name is Eleven Club.

bena(10000) 5 days ago [-]

The problem is that even if there is an innocuous sounding name, once it gets known as 'the porn clearing house', it doesn't matter.

mrandish(10000) 5 days ago [-]

Perhaps an opportunity for Pornhub themselves to get into the payments business? I assume they have enough transactional volume and vendors to make it happen.

toomuchtodo(2025) 5 days ago [-]

My hunch is that would incur unwanted liability from FOSTA-SESTA [1]. Producing content and paying talent out directly is fairly straightforward for adult entertainment. There be dragons if you're attempting to build a marketplace to do something similar.

'The Stop Enabling Sex Traffickers Act (SESTA) and Allow States and Victims to Fight Online Sex Trafficking Act (FOSTA) are the U.S. Senate and House bills that as the FOSTA-SESTA package became law on April 11, 2018. They clarify the country's sex trafficking law to make it illegal to knowingly assist, facilitate, or support sex trafficking, and amend the Section 230 safe harbors of the Communications Decency Act (which make online services immune from civil liability for the actions of their users) to exclude enforcement of federal or state sex trafficking laws from its immunity. Senate sponsor Rob Portman had previously led an investigation into the online classifieds service Backpage (which had been accused of facilitating child sex trafficking), and argued that Section 230 was protecting its 'unscrupulous business practices' and was not designed to provide immunity to websites that facilitate sex trafficking.'

[1] https://en.wikipedia.org/wiki/Stop_Enabling_Sex_Traffickers_...

dextralt(10000) 5 days ago [-]

>'... is a private company and it is their right to refuse service to anyone'

>'Freedom of speech does not mean freedom from consequences'


Nothing is out of the order.

gnulinux(3486) 5 days ago [-]

Just because something is legal, doesn't mean it's necessarily neutral. We can still argue this development being negative, for reasons unrelated to 'freedom of speech'.

scarface74(3825) 5 days ago [-]

I'm the last person to jump on the bitcoin bandwagon, but this is one case where I think it's warranted.

BinaryIdiot(1798) 5 days ago [-]

Bitcoin isn't good for transactions. It's too slow, too cumbersome for he average consumer and its value fluctuates so much that you'd have to have dynamic pricing.

I love the idea of crypto currency and I really do think, one day, some form of it will be out future. But Bitcoin today just doesn't fit that bill at all.

spookthesunset(10000) 5 days ago [-]

> this is one case where I think it's warranted

How will you deal with fraudulent / sketchy site operators scamming people into sending them money via irreversible payment channels?

egdod(10000) 5 days ago [-]

Nothing better than receiving your salary in a token with huge minute-to-minute volatility.

yunesj(10000) 5 days ago [-]

I'm surprised that people suddenly think that banks aren't greedy enough to profit from the porn industry!

Maybe the reason PayPal and so many other companies won't service adult industries is that there are laws that put them within arms reach of criminal liability?

On a related note, this reminds me of Operation Choke Point (2013-2017), where the DOJ harassed banks that did business with customers in legal, but 'undesirable', industries [1].

[1] https://cei.org/blog/operation-choke-point-targets-porn-and-...

[2] https://reason.com/2014/04/28/doj-operation-chokepoint-and-p...

rpmisms(10000) 4 days ago [-]

What a legendary name for an operation.

aazaa(4112) 5 days ago [-]

Financial deplatforming is a monster, long-term trend to watch over the next decades. Deplatforming happens with sex workers. It also happens in the US marijuana industry. It happens with perfectly legal political content.

The number of ways in which someone, somewhere will be offended by products or services for sale is growing, driven by a rapidly improving worldwide communication network and delivery system, and by people terrified of the change that brings.

The response by PayPal and many others has been to cut service. With each case, the circle of people finding themselves cut off from the traditional economy grows.

Bitcoin is often criticized for its lack of use cases. I find a very high correlation between the strength of this belief and lack of awareness of financial deplatforming in its many guises.

Spooky23(3656) 5 days ago [-]

PayPal is an accessible platform that works is a wide variety of scenarios — it's almost unique in that respect.

Most forms of sex work are illegal in the US and other jurisdictions, and sites like PornHub are compliance nightmares where all manner of sleazy or illegal actions are likely happening. PayPal doesn't want to be the financier for the next scandalous "amateur" producer.

1337biz(867) 5 days ago [-]

To be honest I appreciate PayPal doing this. The more we see prominent cases of deplatforming, the more people are going to take it as a serious problem. One can always hide behind some pr bullshit when it comes to political free speech and so on. But porn it is finally something enough people care about to take it seriously.

qxga(10000) 4 days ago [-]

Bitcoin would be great for this in theory, but the problem is that customers will not pay you in Bitcoin. People want their porn and they want it now, that's why they're paying. If you don't make it practically seamless for a layman to pay you for your porn, they're either pirating it or looking elsewhere. Sure, a few people might pay, but you're looking at probably a 50-75% revenue loss, best case.

paulddraper(4026) 4 days ago [-]

PayPal has also deplatformed gun dealers.

No firearms, firearm parts, or ammunition. [1]

[1] https://www.paypal.com/us/smarthelp/article/What-is-PayPal%E...

harry8(4201) 5 days ago [-]

Ask Julian Assange [1] how this works with political speech and journalism. Though I've probably provoked 'the hate' by naming the major example rather than thought about the underlying principles and what's at stake.

[1] https://wikileaks.org/Banking-Blockade.html

bcheung(10000) 5 days ago [-]

I've had several friends in the adult industry have their checking accounts closed at a several major banks as well. Traditional banking should definitely not be discriminating against fully legal activities.

epicureanideal(4165) 5 days ago [-]

Does anyone know of any solutions to this?

Are there any payment options that have so far been immune to this issue?

Maybe it's worth setting up alternative payment methods, even if the percent they charge per transaction is much higher, just to ensure stability of access to funds.

samstave(3779) 5 days ago [-]

It would be terrific if as much energy would be spent on shutting down human sex trafficking finances for children and black and international markets.

Certainly there is some % of pornhub sex workers that are there against their will, but the dark web child porn networks are what our primary focus should be going against.

dwighttk(3012) 5 days ago [-]

> It also happens in the US marijuana industry.

That's because it is still federally illegal

dragonwriter(4218) 5 days ago [-]

> It also happens in the US marijuana industry

The US marijuana industry is, under federal law, entirely criminal, so that's kind of understandable, even if the underlying law is arguably undesirable.

octocop(10000) 4 days ago [-]

I totally agree with you and I also believe money(currency) should exist in it's own layer, beyond the control of corporations and governments.

basch(4222) 5 days ago [-]

wheres the disruptive fintech for these underbanked? Sounds like a gold mine for the right marketing team.

asjw(10000) 3 days ago [-]

Bitcoin is useless for this use case

At a certain point they will start to ask you 'where did you get this coins?' because sooner or later you'll have to convert them to real currencies

Cash is still a better option, there are plenty others: rechargeable CC, gift cards, vouchers, etc etc that work better than bitcoin, require no setup, no knowledge and can be used without hassles.

na85(3879) 5 days ago [-]

But Bitcoin won't solve deplatforming, because the existence of Bitcoin doesn't preclude a bank from deplatforming you.

Until Bitcoin is ubiquitous (doubtful this will occur in my life time) we are still vulnerable to deplatforming when we go to exchange btc for fiat.

Daishiman(4049) 5 days ago [-]

Bitcoin isn't criticized for this being an invalid use case, because it's totally valid.

It's criticized because it's useless for this use case, people involved in this scene in practice don't care about it to any serious degree beyond it being a talking point, and because anything for legally working around financial deplatforming still needs strong interfacing with the rest of the system as well as compliance with legal regimes, which BTC has nothing to say on that.

In fact, you can make your own payments platform denominated in USD and that's as useful as BTC for legal activities. So why go through the extra steps?

Amezarak(4060) 5 days ago [-]

Bitcoin's use case for this is limited. If I were to purchase a product or service that might be subject to deplatforming, it'd better be anonymous, or I risk facing the social sanctions instead. "Amezarak's bitcoin address sent 10BTC to a sex worker, fire him" is not a much better situation.

badrabbit(3537) 4 days ago [-]

How do people get bitcoin? Pay with a credit card? If so, btc exchanges can ban you still based on the name on the card.

I tried other means such as money transfer and cash gift cards, but the exchange rate can mean to pay for $100 I have to spend $3-400!!

A proposal: sell btc giftcards. Each card will have a qr code which is a url that allows download of a wallet (with priv key) where the URL is active after payment. You can transfer the funds to your permanent wallet or spend it directly. Maybe you can even use it to pay for IRL goods somehow. Either way, the participation barrier is lowered. I heard of btc ATMs as well but I can imagine carrying one more giftcard is easier for stores than adding an ATM.

LoSboccacc(4201) 5 days ago [-]

some of the greatest damage the progressives have done to our society came from convincing people that the slippery slope is a fallacy

pluma(3884) 4 days ago [-]

I'm with you except for conflating right wing propagandists with sex workers and the cannabis industry.

Right wing propagandists are rarely deplatformed compared to the other two groups and when they are it's generally for spreading conspiracy theories (remember that people have already died because of Pizzagate and QAnon 'truthers') or inciting (racial/religious/homophobic) hatred.

The difference is that right wing propagandists are also a lot more vocal than the other group (or left wingers for that matter), skewing the perception. There's a widely known joke about how many comedians have shows on mainstream platforms with titles like 'Cancelled' or 'Triggered', complaining about 'things you can't say anymore'. Also see the 'Intellectual Dark Web' whose whole shtick is being 'deplatformed' by academia and the mainstream while having a massive mainstream media presence.

raxxorrax(10000) 4 days ago [-]

Printable cash for internet transaction would be nice...

Honestly, the behavior of paypal is abysmal and I wish there to be a real alternative. I despise credit cards and would like to have more options.

I doubt I will ever pay anything for porn, but there are a lot of content creators that can only receive money via paypal/credit card.

I do actually think a government should have some control over its currency, but if the institutions implement their own arbitrary rules, solutions like Bitcoin are the best alternative.

Nasrudith(10000) 5 days ago [-]

Marijuana is separate from the deplatforming in the traditional sense because federal banking laws and the DEA's stubborn Schedule 1 tautology when it clearly has medical uses are to blame as opposed to uncooerced choices.

briandear(1366) 5 days ago [-]

> It also happens in the US marijuana industry

Most of the marijuana industry is illegal in the US. States making it legal doesn't change the Controlled Substances Act. As banking is federally regulated, banks don't have the freedom to knowingly allow illegal businesses.

Porn however, is legal, and deplatforming a legal product/business is wrong in my opinion.

To be clear, I am not suggesting marijuana should be illegal at the federal level at all, only that it's not inconsistent that a bank not serve those customers — after all they would then become criminals themselves by facilitating the moment of funds for a product that is illegal.

xtracto(10000) 5 days ago [-]

It will be funny that, as with other technologies from the past, pornography will be the use case that will finally bring cryptocurrencies to widespread adoption in society

steelframe(10000) 5 days ago [-]

As a few others have already pointed out, marijuana is a Schedule 1 drug, and so no bank that operates at a federal level can touch it.

That said, I've heard that the ATMs in recreational marijuana shops are themselves significant sources revenue for the shops.

cable2600(10000) 5 days ago [-]

This is Big Brother type censorship. Anyone they disagree with will get shut out of Paypal, Banks, Credit Cards, etc.

There was Hatreon but Visa canceled their services for them.

I agree the porn industry is legal and nobody forces people to watch Pornhub, now I doubt the future of Pornhub. I don't use it but I support those who do under the freedom of speech.

cft(1043) 5 days ago [-]

IRS treats Bitcoin as property. Accordingly, the recipients need to report every bitcoin transaction on Schedule D and calculate its cost basis ( appreciation or depreciation of Bitcoin). I am not talking about the reporting of the cumulative income, but the additional requirement of reporting of each Bitcoin payment or disposal. Say you received 0.0001 BTC held it for 5 days and then converted to USD. These are two taxable events, not one, with the requirement of calculating BTC appreciation between these two events. Now multiply this burden by every transaction.

In my opinion, IRS successfully killed BTC use in the US by treating it as property rather than currency (for currency FX, transactions under $200 are exempt from cost basis reporting on schedule D)

ericb(3161) 5 days ago [-]

Bitcoin isn't a currency that can fix this. Bitcoin is either censorable (lightning network) or unaffordable when used heavily with slow confirmations and ridiculous fees.

The lightning network is a joke--a frankenstein 'Engineering UI' layed over a system to route around bad development decisions re:block size.

bigpoppa(4222) 5 days ago [-]

Screw BTC, they could use XRP, it's virtually a stable coin.

throwaway857384(10000) 4 days ago [-]

There is this guy I watch on YouTube, he had a Patreon account and a BTC address. I don't want to sign up for Patreon, so I sent him BTC.

The great thing about BTC is you can send it so easily without going through all the hoops other 'systems' need. Sure, it isn't so easy to get started with BTC, but once you are in, it is a breeze.

stiray(4105) 4 days ago [-]

This practice is getting seriously annoying in scope of current events like google going into banking bussiness. One part are privacy applications but even worse is that suddenly pulling the application from play store using aurora (3rd part app for those that dont want google mobile services spyware on the phone) could cost you bank account which google will surely tie to google account. This is not good, we are beeing pulled to distopia at progressing rate.

sieabahlpark(10000) 5 days ago [-]

I like how this is spun in a way as if PayPal has a vendetta against specific forms of income alone.

Nevermind the sex industry is probably the biggest attractor for fraudulent charges and overall sketchy business.

justinzollars(2649) 5 days ago [-]

Bitcoin will solve this problem.

EpicEng(3973) 5 days ago [-]

Not until it's near as stable as the dollar it won't.

jlgaddis(2877) 5 days ago [-]

It's had a decade and it hasn't yet.

nafizh(694) 5 days ago [-]

Thank you Paypal. Porn is a disease on the society exploiting on vulnerable people. Any company concerned with mental health of the society shouldn't let these porn streaming companies use their infrastructure. Also, a private company is not compelled to let anyone use their infrastructure, if anything, it violates that company's first amendment rights.

rationalfaith(10000) 5 days ago [-]

As much a yc attracts 'smart' people, they are usually plagued with a lack of wisdom.

You only need to venture to yourbrainonporn.com to meet your rationalization's end. At least if you can read english.

tamaharbor(10000) 5 days ago [-]

Have you ever been with a girl?

craze3(4068) 5 days ago [-]

I'm working on a decentralized app to remedy these type of moves by big payment processors.

The new flow I suggest is: BTC -> Coinbase -> PayPal.

Withdrawals are instant & only subject to standard fees. The can funds can also be instantly converted to cash, via the PayPal Business Debit Card.

References: [1] https://blog.coinbase.com/instant-paypal-withdrawals-now-ava... [2] https://support.coinbase.com/customer/portal/articles/295995... [3] https://support.coinbase.com/customer/en/portal/articles/210... [4] https://www.paypal.com/us/webapps/mpp/business-debit-card

ur-whale(4149) 5 days ago [-]

> The new flow I suggest is: BTC -> Coinbase -> PayPal

The flow still can be censored. As a matter of fact, you now have two opportunities for your payment to get nixed.

makomk(3691) 5 days ago [-]

If I remember rightly. Coinbase bans porn-related payments just like PayPal.

mirekrusin(4216) 5 days ago [-]

Why not just exchange btc/eth and leave the headache away? Pornhub should move to btc - at least they'll educate some population about cryptocurrency/how to use it. There are many ways to move money in/out, I think one of no fee paths is via bybit (just network fees).

rationalfaith(10000) 5 days ago [-]

Porn is ripe with human trafficking and its consumption is violently destructive to individuals and relationships.

Source: http://yourbrainonporn.com/

In the future, when people stop acting so ignorant, it'll be categorized as a threat to society in corrupting objective, rational and clear thinking.

Everything in extremes is bad, even freedom.

Jamwinner(10000) 5 days ago [-]

Are you here just to troll?

wufufufu(10000) 5 days ago [-]

Square and Cash App now is your time. You talk about economic empowerment but are you really bout that life?

nickstinemates(3865) 5 days ago [-]

Doubt. Porn is one thing, but, to take a moral public stand in the same year of the Chef/Github/ICE controversy seems risky.

Historical Discussions: Writing userspace USB drivers for abandoned devices (November 17, 2019: 833 points)

(861) Writing userspace USB drivers for abandoned devices

861 points 2 days ago by signa11 in 21st position

blog.benjojo.co.uk | Estimated reading time – 13 minutes | comments | anchor

Nov 17 2019

Writing userspace USB drivers for abandoned devices

I recently found some USB devices on eBay (Epiphan VGA2USB LR) that could take VGA as input and present the output as a webcam. Given that I was keen on the idea of not needing to lug out a VGA monitor ever again and there was claimed Linux support I took the risk and bought the whole job lot for about £20 (25 USD).

When they arrived, I plugged one in under the expectation that it would come up as USB UVC Devices but they did not. Was I missing something?

After looking through the vendors site I discovered there was a custom driver required for them to work. As I normally live the easy life on Linux of never needing to pull in drivers because the distribution kernel I am using has them already, this was a reasonably novel concept.

Sadly, it seems like driver support for the devices in question ended at Linux 4.9. Meaning none of my systems would run this device anymore (Debian 10 [Linux 4.19] or latest LTS Ubuntu [Linux 5.0])

But surely this was something I could patch myself right? Surely the package files where actually just a DKMS package that built the driver from source code on demand like a lot of the out of tree drivers out there...

Sadly. This was not the case.

Inside the package is just a pre-compiled binary called vga2usb.o. I started doing some basic investigations on how hard it might be to reverse engineer and found some interesting string table entries:

$ strings vga2usb.ko | grep 'v2uco' | sort | uniq

Is this device actually an FPGA-on-a-stick? What would the process be to get something like that running even look like?

Another both amusing and mildly alarming find was the strings for DSA private key parameters. This made me wonder if there was private key material inside this driver and what could be protected with it:

$ strings vga2usb.ko | grep 'epiphan' | sort | uniq

To observe the driver in its normal operating environment, I made a Debian 9 (the last supported release) VM, and did a KVM USB Passthrough to give it direct access to the device. I then installed the driver and confirmed that it worked.

After that, I wanted to see what the wire protocol looked like. I was hoping that the device sent raw (or close to raw frames) over the wire as this would make the task of writing a user space version of the driver easier.

To do this, I loaded the usbmon module on the VM's host machine and used Wireshark to take a packet capture of the USB traffic to and from the device during startup and whilst capturing video.

I found that on device startup there was a large number of small packets to the device before the device could capture data. I assumed that this meant that the device was in fact as described above an FPGA based platform that had no persistent storage. Every time the device was plugged in the devices firmware would have to be "bitstreamed" from the driver itself.

I confirmed this by opening one of the units up:

ISL98002CRZ-170 - Acting as an Analog to Digital Converter for the VGA signals

XC6SLX16 - Xilinx Spartan 6 FPGA

64 MB of DDR3 RAM

CY7C68013A - USB Controller / Frontend for the device

Given that to "boot" this device I needed the bitstream to send to it, I got to work on the pre-compiled binaries to try to extract the bitstream/firmware. After running binwalk -x and watching it find a few (zlib) compressed objects. I wrote a script that would search them for a known hex sequence and picked 3 bytes from the pcap that I knew were from the bitstreaming process to search for

$ bash scan.sh '03 3f 55'
trying 0.elf
trying 30020
trying 30020.zlib
trying 30020.zlib.decompressed
trying 84BB0
trying 84BB0.zlib
trying 84BB0.zlib.decompressed
trying AA240
trying AA240.zlib
trying AA240.zlib.decompressed
000288d0  07 2f 03 3f 55 50 7d 7c  00 00 00 00 00 00 00 00  |./.?UP}|........|
trying C6860
trying C6860.zlib

After decompressing the AA240.zlib file. I found that there was not enough data there to be the full bitstream. So I instead went down the route of extracting the firmware out of the USB packet capture.

I found that while both tshark and tcpdump can read USB packets inside pcap files, they both would only dump bits of information in the capture. Since that each program had different parts of the puzzle, I wrote a small program that would unify the output of both programs into go structs so they could be replayed back to the device.

At this point I noticed that bootstrapping comes in two stages, first the USB controller and then for the FPGA itself.

For at least a few days I was stuck on an issue where it would appear the whole bitstream would upload. But the device would not start up, despite it seeming like the packet captures between the real driver and userspace one looking identical.

This was eventually solved by combing through the pcap paying attention to the time it took to respond to each packet and noticing a large difference in one particular packet's timing:

It turned out a manually entered typo caused an USB control write to go to the wrong area of a device. Serves me right for manually entering a value in...

Regardless, I now had a green blinking led on the device! A massive achievement!

Your browser does not support H.264 HTML5 video.

Since it was relatively trivial to replicate the same packets that seemed to start the data streaming, I was able to write up a USB Bulk transfer endpoint and have data being dumped to disk in no time!

This is where the real challenge started. Because after analysis it appeared that the data was not obviously encoded in any way.

To start with, I used perf to get a general view of what the driver stack traces looked like while it was running:

Whilst I made progress with being able to hook functions that had frame data in them, I still didn't get any closer to figuring out the encoding of the image data itself.

I did try the NSA's Ghidra to get a better idea of what was going on inside of the real driver:

While Ghidra is incredible (this was my first time using it compared to IDA Pro) it still wasn't quite good enough to reasonably help me understand the driver. I needed another path of investigation if I was going to reverse engineer this.

I decided to provision a Windows 7 VM and check if the Windows driver was doing anything different, I also noticed during that time that there was a SDK for the devices. One of the tools ended up being of particular interest:

PS> ls
    Directory: epiphan_sdk-\epiphan\bin
Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        10/26/2019  10:57 AM     528384 frmgrab.dll
-a---        10/27/2019   5:41 PM    1449548 out.aw
-a---        10/26/2019  10:57 AM     245760 v2u.exe
-a---        10/26/2019  10:57 AM      94208 v2u_avi.exe
-a---        10/26/2019  10:57 AM     102400 v2u_dec.exe
-a---        10/26/2019  10:57 AM     106496 v2u_dshow.exe
-a---        10/26/2019  10:57 AM     176128 v2u_ds_decoder.ax
-a---        10/26/2019  10:57 AM      90112 v2u_edid.exe
-a---        10/26/2019  10:57 AM      73728 v2u_kvm.exe
-a---        10/26/2019  10:57 AM      77824 v2u_libdec.dll
PS> .\v2u_dec.exe
      v2u_dec <number of frames> [format] [compression level] <filename>
               - sets compression level [1..5],
               - captures and saves compressed frames to a file
      v2u_dec x [format] <filename>
               - decompresses frames from the file to separate BMP files

This tool let you fire "one shot" captures, noting that in the source it didn't apply compression to the frames so that the output can be processed on a faster machine later. This was practically perfect, and I replicated the USB packet sequence to obtain these uncompressed blobs, and looking at the byte counts, it matched with getting around 3 bytes (RGB) per pixel!

Initial processing of these images (just taking the output and writing it as RGB pixels) resulted in something roughly inspired by the input I was giving to the device over VGA:

After some more debugging with a hex editor, I discovered there was some kind of marker every 1028 bytes, it took a slightly embarrassing amount of time to write a watertight filter for that, On the other hand I ended up producing some modern art in the process.

After realising that the tilt/sheer in the image was caused by me skipping and carrying over a pixel on every line (x=799 != x=800), I finally ended up with an image that was almost spot on apart from the colour:

Initially I thought this might have been a calibration thing, caused because I took some sample data when the VGA input was stuck on a solid colour, in order to fix this I built a new test image that would try to smoke these issues out, in hindsight I should have used something like a Philips PM5544 test card

After loading this image on to the VGA producing laptop, I ended up with an output of:

At this point I had a flashback to some 3d rendering/shader work I did long ago. This looked a lot like YUV colour.

I ended up reading up on YUV and remembered during my reverse engineering of the official kernel driver I'd found if I set a breakpoint on a function called v2ucom_convertI420toBGR24 the system would hang without the ability to resume. So maybe the input was in I420 encoding (of -pix_fmt yuv420p fame) and the expected output was Blue Green and Red as 8 bit bytes?

After using Go's built in YCbCrToRGB the image suddenly looked much closer to the original.

We did it! Despite the "WIP" quality we were able to do 7 FPS. Honestly, for me that was good enough, since my use for these are as an emergency VGA screen rather than anything else.

So now we know this device well enough to explain how to operate it from a cold boot:

  1. You need to initialise the USB controller. I assume based on the size of information, this is actually uploading code to the USB controller
  2. When you've done the USB upload, the device will disconnect from the USB bus, and come back a moment later, with only a single USB endpoint.
  3. You can then start sending the FPGA bitstream, one 64 byte USB control packet at a time.
  4. Once you are finished, the LED on the device will start blinking green. At this point you can send what appears to be a parameter sequence (overscan and other properties)
  5. You can then fire a single control packet to get a frame, the control packet has the resolution embedded in it. If you use a 4:3 request packet on a widescreen input you will often end up with a corrupted input.

To make use as easy as possible, I ended up rigging up a small web server inside the driver to make it super easy to use in a rush. Thanks to the MediaRecorder API in browsers, it also allows for an easy way to record the output of the screen to a video file.

As I'm sure a lot of people can relate regarding experimental code; I can't say I'm proud of the code quality. But it's likely in the state where it works well enough for me to use.

You can find the code for this (and pre-built versions for Linux and OSX) on github: https://github.com/benjojo/userspace-vga2usb/

Even if this is never used by anyone else, this was a hell of a roller coaster in USB protocol details, kernel debugging/module reverse engineering, and general video format decoding! If you liked this kind of stuff, you may like the rest of the blog. If you want to stay up to date with what I do next you can use my blog's RSS Feed or you can follow me on twitter

Until next time!

All Comments: [-] | anchor

aportnoy(2431) 2 days ago [-]

How does one learn how to write device drivers under Linux?

megous(4179) 2 days ago [-]

By trying and immitating. There's awful lot of examples out there in the kernel source code.

raverbashing(3723) 2 days ago [-]

There used to be a Linux Device Drivers book https://lwn.net/Kernel/LDD3/

While it's still based on the 2.6 kernel, it's probably good enough for the basics. Not sure there is a more modern source.

(Robert Love's Linux Kernel Programming was also a nice introduction to how the Kernel works, though it might be outdated in some areas now as well)

stevekemp(880) 2 days ago [-]

Studying existing drivers, following tutorials, carrying out research, and lots of trial and error.

The same way people learn to do many other similar things.

jsharf(10000) 2 days ago [-]

Google search 'lkmpg pdf'

It's part of tldp (also google searchable)

matheusmoreira(10000) 1 day ago [-]

I recently wrote a user space driver for my Clevo laptop's backlit keyboard. I'm currently trying to convert it into a Linux kernel driver. I'll describe my approach.

First of all, I had to realize that the keyboard was in fact controlled via USB. It seems obvious but other Clevo laptops did it via ACPI:



I assumed my laptop worked the same way and wasted a lot of time dumping my laptop's ACPI tables and trying to figure them out. I even installed some Windows tools to decompile the WMI buffers. Didn't find anything. I was about to give up when someone on IRC set me on the right path and I was able to make progress. Even though I saw my keyboard on lsusb output, I never thought to look for a USB protocol. Learned this one the hard way.

Before I started, I emailed Clevo and asked for technical documentation on the keyboard. Their marketing department replied: 'Ubuntu was not supported'. I emailed Tuxedo Computers (mentioned above) and they were much more helpful: their developers shared a fan control application they wrote with me! It was great but the keyboard drivers were missing.

So I decided to reverse engineer it.

While not as complex as the VGA-to-USB device, the process of reverse engineering the keyboard was similar:

  1. Boot Windows 10 with the proprietary driver
  2. Start Wireshark
  3. Capture USB traffic
  4. Use the manufacturer's keyboard control program
  5. Correlate the data sent to the keyboard with what I did
  6. Document the structure of the messages
For example, to set the color of a specific RGB LED on my keyboard, the following bytes must be sent to the device through a USB control transfer:

  01 = some kind of operation code
  kk = target LED
  rr = red
  gg = green
  bb = blue
  cc, 7f always surround all bytes (part of USB protocol?)
I used hidapi-libusb to create a small program that sends those bytes to the keyboard:


This user space driver gave me access to most of my keyboard's functionality on Linux. I used to have to boot into Windows in order to configure the keyboard lights -- not anymore!

The keyboard also generates some custom key codes that are used by the proprietary Windows driver to implement some Fn hotkeys. I know this because the kernel logs these unknown key events. To implement this, I'll have to make a real kernel module that maps them into something user space can make use of.

To this end, I've been studying the Linux kernel's documentation:




The input_mapping, input_mapped, event and raw_event hooks seem especially relevant to my case: the custom driver just needs to process the custom key codes and let the generic Linux keyboard driver handle the rest. Might as well add sysfs entries for the LEDs but I don't know how to do that yet.

I expect to end up with something like the open razer driver:


I used this site to gain a better understanding of USB itself:


joewee(3176) 2 days ago [-]

I have a few devices with no public drivers of any kind. Thinking about how to develop drivers on Linux, I'm guessing I should lookup into the chipsets used on the board and see if those companies have drivers I can use as a starting point?

nwallin(10000) 1 day ago [-]

Does it have Windows drivers? If it does, you can set up a VM, set the device as passthrough, and sniff the traffic to it. Then it's not really any different than reverse engineering a network protocol or file format.

What are they, out of curiosity?

Yajirobe(10000) 2 days ago [-]

How do you learn how to do something like this?

qiqitori(10000) 1 day ago [-]

There are many ways to get somewhere, but here's a general list of things you have to do:

1) Think for a little bit about what you have got and what you want to do

2) Do research. Has this been done before? Have similar things been done before? (Usually the answer is yes to either question)

2) Think whether it's theoretically possible to do what you want to do with the information and parts you have (in this case: there is a working driver for Linux and Windows, but they're closed-source)

3) Fixing computer things usually involves a lot of debugging (tracing). Will tracing help us out here? What tracing tools/skills do you have, what tracing tools/skills would you like to have? Often you'll need to extend the tracing skills/tools you already have, but normally you don't want to spend too much time on this. (In this case, we can trace without any specialized equipment -- we just need a virtual machine.)

4) Once you have the appropriate traces, take a good look at them (you'll probably want some scripting abilities here) and see if you notice any patterns (e.g. 'lots of data after this', 'just a little data after this', 'this bit of data is always the same', etc.)

5) If you don't know what to do at this point, maybe go back to the tracing step

xg15(2434) 2 days ago [-]

Hang on, did he just buy essentially a freely reprogrammable FPGA with built-in RAM, A/D converter and USB support - for $20? Seems to me, you could do a lot more with those devices, now that the wire protocol is known.

(Edit) It doesn't appear that he decoded the bitstream format though, so some more work would be needed before you could use it as a general-purpose FPGA.

benjojo12(1298) 2 days ago [-]

It was actually 20 pounds for 10 ish, so it comes out even cheaper.

Sadly it seems that was just a stroke of luck, since I can't find anyone else selling them anywhere near that cheap now

jack12(10000) 2 days ago [-]

Yeah, a 16K Spartan 6, 64MBytes of DDR3 RAM, and an FX2LP USB High-Speed controller is a pretty terrific repurposed FPGA dev board in the vein of the chubby75 or panologic, even if the high-speed ADC turns out to have too many limitations / too weird signal conditioning (i.e. is too VGA-focused) to be very useful (but VGA probably at least means a pair of I2C pins is exposed on the DSUB-15, right?). Considering the FX2LP is already wired up to load the bitstream to the FPGA, and expects to on every power-up, that's a huge bonus for a dev board. Even if the binary blob of FX2LP firmware provided in the manufacturer's drivers does checksums, signature checks, etc. to make changing the bitstream data difficult, it would be pretty simple to write a new firmware to allow uploading of any custom bitstream (see: fx2lib), since it's such a well-known and widely-hacked chip -- maybe you can even manage to trick Xilin's ISE into believing it's a real Xilinx 'USB Platform Cable' (FPGA JTAG programmer) since at least some of them use an FX2 for their USB interface. With a generic bitstream uploader utility for the FX2 you wouldn't even need to deal with any hardware modifications or even need to buy and connect a JTAG programmer if you were just starting out. And with everything being uploaded on every reset of the devboard, you don't have to worry about bricking anything either, since you just power-cycle the devboard to bring the programmer AND FPGA back to their default waiting-for-firmware/'rescue' mode.

But while the article says he got a lot of them (at least 3 from the photo) for $25 total, it looks like they're more typically ~$50+S/H each on ebay, which is less exciting.

The same company seems to have moved on to 'av.io'-branded USB3 devices now, which from a quick glance/binwalk of the downloadable firmware package (inspired by this article) appears to have FX3 USB Super-Speed controllers, and Spartan FPGAs. But it's designed to store the FX3 firmware and FPGA bitstream on the device itself (to allow it to boot up as a UVC webcam with no drivers required on the host, like this article was hoping would be the case for the older device), so it would require a bit more reverse engineering to figure out how to kick the FX3 into firmware update mode to replace the FX3 firmware with a custom bitstream uploader. But it's a ton more expensive, and who knows what generation and size of FPGA or how much and what sort of memory it would have.

CamperBob2(10000) 2 days ago [-]

Correct, that's actually a pretty nifty piece of general-purpose hardware for the price. It's essentially an SDR (software-defined radio) for baseband, meaning it provides simultaneous sampling on three channels from DC to whatever a suitable Nyquist rate for VGA decoding would be. Several MHz at least.

The bitstream format isn't anything special. It's whatever Xilinx says it is for the Spartan 6. Any bitfile generated by ISE for the XC6SLX16 could be uploaded via the (generic) Cypress FX2LP chip.

This is all relatively old hardware, very well-documented and well-understood but still quite capable. It has a lot in common with the earlier-generation Ettus USRP receivers that used the FX2.

codedokode(3806) 1 day ago [-]

If Linux had stable kernel-to-driver interface, that driver could work today.

Iolaum(10000) 1 day ago [-]

If the driver had been mainlined it would still work today.


Nzen(10000) 2 days ago [-]

tl;dr Ben Cox bought a vga in => webcam out device that its parent company hadn't updated the linux drivers in a while. This details his journey of using tcpdump, et al, to analyze the sent packets. Eventually, he simulates the usb firmware and fpga download such that he can use the device. A quick, engaging read that was an appropriate level for this non C programmer.

Looking through his archive, I recognize his bgp battleship post from last year on HN.

horsecranium(10000) 2 days ago [-]


jsharf(10000) 2 days ago [-]

This seems pretty formal/professional for an HN comment. Is there some reason you're writing summaries of HN posts with this degree of effort?

Also, thanks.

mLuby(4011) 2 days ago [-]

This TL;DR made me actually read the post, so thanks for the summary.

sulam(10000) 2 days ago [-]

BGR instead of RGB seems to be very common in hardware land.

Anyone know why?

spease(10000) 2 days ago [-]

Best answer I found is this: https://retrocomputing.stackexchange.com/questions/3023/why-...

There was apparently an HN thread as well https://news.ycombinator.com/item?id=17808191

danmg(10000) 2 days ago [-]

Basically, little endian vs big endian but for frame buffer devices instead of CPUs.

jason0597(4050) 2 days ago [-]

It's things like these that sometimes make me think of all those businesses that rely on some ancient Windows XP machine because it's the only thing that supports the proprietary drivers for the engineering machine they bought nearly 2 decades ago for hundreds of thousands of pounds. If instead the drivers were open source (and the relevant controller software for that machine), they could more easily move over to a more recent operating system that doesn't run the risk of being infected with some virus (think Wannacry)

I imagine also how more difficult it would be for someone to reverse engineer the drivers for something a lot more complex than a VGA-to-webcam box.

WaylonKenning(10000) 1 day ago [-]

I used to work for a Power Company, that had a hydroelectric dam. There were sensors in the dam that needed Windows 98 to read data from. I remember we ended up having a fleet of old Asus Eee PCs specifically for that, that obviously never ever ever connected to the Internet, or really any network.

ShabbyDoo(4186) 1 day ago [-]

I'm surprised a manufacturer of such engineering machines hasn't differentiated itself by offering a SLA on driver (and general software) availability for future operating system versions. It would be reasonable to require users to pay a subscription fee for extended support. I'd think the resale value of machines with drivers still available would be much higher than for those without and that this eventually would allow the manufacturer to charge a premium price for the machine at initial sale.

I own a Fujitsu Scansnap s1500. It's out of support, and the existing drivers are incompatible with Catalina. I now must pay a 3rd party $100 for drivers or fiddle around with a Linux scanner server or similar. Never again will I pay $400 for a Fujitsu scanner, that's for sure.

codedokode(3806) 1 day ago [-]

If the drivers were open source, they would need good programmers to maintain them. And if they had good programmers, they could have reverse-engineered proprietary drivers. But I guess that buying an old PC might be cheaper than hiring a good programmer.

ChuckNorris89(4216) 2 days ago [-]

Well yes, open source firmware and drivers would be cool but for most companies that's part of their proprietary IP and they think that making it open would make their product easier to copy/exploit by competitors. Since it cost them lots of time and money to develop there's no way your gonna convince management to just make it open unless you're some stinking rich FAANG Co. and those HW product sales are not your main source of income.

Just look how locked down Broadcom documentation and even datasheets are. Everything is under NDA.

HW business make money by selling more HW so making you buy their latest product just for the latest updates even though the 10 year old one still does the job is a viable business model for them.

Source: I worked in the hw industry and it's how they think.

rburhum(3197) 2 days ago [-]

The device itself is not that interesting, but I just love the level of detail that describes the reverse engineering process. Excellent blog post!

moftz(10000) about 12 hours ago [-]

It seems like the hardware could be repurposed to be a multichannel DAQ or even a VHF SDR. It's a neat device considering how cheap it is.

tombert(4137) 1 day ago [-]

I know virtually nothing of low-level programming, so bare with me; to a lay person, can someone explain why drivers are every closed source? Isn't a driver inherently a loss-leader for some kind of hardware product, and wouldn't it be better for the manufacturer to have it open-source so that more people could contribute to it?

I'm not asking this passive-aggressively, if someone knows the answer to this I'd really like to hear it.

hatsubai(10000) 1 day ago [-]

Sometimes, it just doesn't make sense. For example, I write and maintain the Linux kernel modules for one of the largest, most popular, and most powerful land vehicles in the world. It wouldn't make sense for us to release these to the open source community because it requires specific hardware that we make, specific FPGA firmware that we interface with, and it will only work with specific software we make. So there's at least one case where it wouldn't make sense to release this, and that's not even getting into the security aspect of things.

HeyLaughingBoy(4094) about 21 hours ago [-]

In addition to the very good replies you already got, sometimes it just wouldn't make a difference.

In some environments, if the code was not developed using a particular process (my experience is with regulated medical devices), it may as well not exist, because putting it into production would be in violation of Federal law.

There are also small niche products where most of the IP is in the driver: lose control of that and you're on your way to being out of business.

malnourish(4201) 1 day ago [-]

A simple answer is that, from a business perspective, closed source is simply easier. It's likely the default way they handle their code.

And at least for GPUs, drivers are pitched as a feature and may contain secrets worth protecting in the eyes of the business.

phendrenad2(10000) 2 days ago [-]

Reminds me of this: https://www.theinquirer.net/inquirer/news/1047633/one-writes... ('One man writes Linux drivers for 235 USB webcams')

ChuckNorris89(4216) 2 days ago [-]

Wow, cool thanks for sharing. Very interesting read.

Historical Discussions: (November 18, 2019: points)
(November 13, 2019: points)
(May 21, 2019: points)
(May 14, 2019: points)
(May 14, 2019: points)
(May 14, 2019: points)
(May 14, 2019: points)
(May 14, 2019: points)
(May 14, 2019: points)
More Intel speculative execution vulnerabilities (November 13, 2019: 640 points)
MDS: Microarchitectural Data Sampling side-channel vulnerabilities in Intel CPUs (May 14, 2019: 256 points)
TSX Asynchronous Abort (TAA): Speculation-Based Vulnerability in Intel CPUs (November 12, 2019: 6 points)
RIDL: Rogue In-Flight Data Load [pdf] (May 15, 2019: 4 points)

(640) More Intel speculative execution vulnerabilities

640 points 6 days ago by wolf550e in 4198th position

mdsattacks.com | Estimated reading time – 5 minutes | comments | anchor

On Nov 12, 2019, we (VUSec) disclose TSX Asynchronous Abort (TAA), a new speculation-based vulnerability in Intel CPUs as well as other MDS-related issues, as described in our new RIDL addendum. In reality, this is no new vulnerability. We disclosed TAA (and other issues) as part of our original RIDL submission to Intel in Sep 2018. Unfortunately, the Intel PSIRT team missed our submitted proof-of-concept exploits (PoCs), and as a result, the original MDS mitigations released in May 2019 only partially addressed RIDL. You can read the full story below.

In particular, at the request of Intel, we withheld the following details on the original RIDL/MDS disclosure date:

  • TSX Asynchronous Abort (TAA). Intel's TSX hardware feature can be used to efficiently mount a RIDL attack even on allegedly non-vulnerable CPUs (with hardware mitigations).
  • Alignment faults. These can be used to trigger an exception, giving an attacker yet another way of leaking data. This attack vector seems to be fixed in the latest generation of Intel CPUs.
  • Flawed MDS mitigation. The initial mitigations against MDS clear the buffers by writing stale, potentially sensitive, data into these buffers, allowing an attacker to leak information despite mitigations being enabled.
  • The RIDL test suite. We can now release the RIDL test suite at https://github.com/vusec/ridl.


TL;DR: an attacker can mount a RIDL attack despite the in-silicon mitigations/microcode patches published in May 2019 being in place.

In particular, TSX Asynchronous Abort (TAA) allowed us to mount a practical RIDL exploit (as already shown in /etc/shadow leak presented in the RIDL paper) even on systems with in-silicon/microcode MDS mitigations enabled. Moreover, one of our brilliant Master students (Jonas Theis) used TAA to optimize our initial password-disclosing exploit to run in just 30 seconds (!!), rather than the initial 24 hours (as seen in the video below).

We leak the full root password hash from the /etc/shadow file on Linux in around 30 seconds using a TAA-optimized RIDL exploit.

Why only release this now?

In short, this was part of our original RIDL submission but was missed by Intel during the first embargo and a new last-minute embargo followed. Check out the updated timeline at the end of the page or keep reading for the full story.

On Sep 29, 2018, we submitted several proof-of-concept exploits (PoCs) for a number of RIDL variants to Intel. Despite our many attempts, we received no technical feedback/questions on our submission except that Intel was working on the mitigations.

In fact, due to a lack of transparency on Intel's part, we only got a complete picture on Intel's MDS disclosure plan on May 10, 2019, just 4 days before public disclosure. We were able to find the microcode updates published by Intel online and tested them on May 11. We quickly found that Intel's fixes did not fully mitigate the vulnerabilities we had reported in Sep 2018 and immediately informed Intel.

On May 13, 2019, just one day from the RIDL/MDS public disclosure date, Intel requested TAA and any other RIDL issues that were not mentioned in the MDS whitepaper to be placed under a new last-minute embargo until Nov 12, 2019. At the request of Intel, and to protect users, we complied to the new embargo, withheld several details from the RIDL paper (leaving only some traces of our results in Table I), and did not release our now public RIDL test suite.

On July 3, 2019, we finally learned that, to our surprise, the Intel PSIRT team had missed the PoCs from our Sep 29 submission, despite having awarded a bounty for it, explaining why Intel had failed to address - or even publicly acknowledge - many RIDL-class vulnerabilities on May 14, 2019.

On Oct 15, 2019, we learned that Intel had not found this issue internally and the only other independent finder was the Zombieload team, which disclosed TAA to Intel in April, 2019.

On Oct 25, 2019, we tested Intel's latest microcode update, and still saw leaks with the VERW mitigation enabled, using the RIDL PoCs we shared with Intel in May 2019. We notified Intel and shared a polished PoC to make the issue clear. Intel requested a new embargo and yet suggested adding the following to our RIDL addendum: 'A new microcode update release by Intel in November is required to adequately address the issue'.

On Nov 12, 2019, TAA and the other scheduled RIDL issues are disclosed. Unfortunately, we believe that, given the piecemeal (variant-by-variant) mitigation approach pursued by Intel, RIDL-class vulnerabilities won't disappear any time soon.

We are particularly worried about Intel's mitigation plan being PoC-oriented with a complete lack of security engineering and underlying root cause analysis, with minor variations in PoCs leading to new embargoes, and these 'new' vulnerabilities remaining unfixed for lengthy periods. Unfortunately, until there is sufficient public / industry pressure, there seems to be little incentive for Intel to change course, leaving the public with a false sense of security. Slapping a year-long embargo after another (many news cycles apart) and keeping vulnerabilities (too) many people are aware of from the public for a long time is still a viable strategy.

All Comments: [-] | anchor

wyldfire(600) 6 days ago [-]

> On July 3, 2019, we finally learned that, to our surprise, the Intel PSIRT team had missed the PoCs from our Sep 29 submission, despite having awarded a bounty for it, explaining why Intel had failed to address - or even publicly acknowledge - many RIDL-class vulnerabilities on May 14, 2019.

What does this usage of the word 'missed' mean in this context? That they lost it / failed to deliver the PoC to the relevant team? Or that they released a 'fix' knowing that it didn't defeat the PoC?

nolok(4169) 6 days ago [-]

From the way the phrase is turned, I believe they released a fix that covered all previously known PoCs but not those from that submission.

Generally speaking, that really illustrate the dumb way Intel is going about it, fixing on a PoC basis rather than going after the strong underlying problem. It basically screams 'there will always be issues, the question is can you find them !'.

jacquesm(43) 6 days ago [-]

Is there a list of the exploits found 'in the wild' that depend on speculative execution?

wnevets(10000) 6 days ago [-]

for regular consumers, its doubtful there are any.

voidmain(10000) 6 days ago [-]

Can we please have an architectural MSR to disable TSX?

hansendc(10000) 6 days ago [-]

The good news: There's a new MSR which lets you do this:


The less good news: as far as I can tell, Intel did not commit to how architectural this will be going forward. Considering the role TSX has played in speculation-based attacks, it appears to me to be a generic mitigation that would be great to accompany TSX wherever it is available in the future. Now that MSR_IA32_TSX_CTRL is defined, it should be easier to implement going forward.

Disclaimer: I work on Linux at Intel.

ageofwant(3853) 6 days ago [-]

Running the below over my machines gives me back the 8-30% cycles I originally paid for, depending on load type. This will have to do until everything is swapped to AMD. Note you only need 'mitigations=off' in later kernels.

    - name: Disable CPU-sapping security mitigations
      become: yes
        path: /etc/default/grub
        line: GRUB_CMDLINE_LINUX_DEFAULT='noresume noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off mitigations=off'
    - name: Update grub
      become: yes
      command: /usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg
No I don't give a fuck about the 'risk' this introduces, but I expect my bank to.
yrro(10000) 6 days ago [-]

FYI, you can do the same thing with drop-in config files and avoid the annoying dpkg-conffile prompt whenever you upgrade grub2-common:

  $ cat /etc/default/grub.d/mitigations.cfg
coldtea(1239) 6 days ago [-]

>No I don't give a fuck about the 'risk' this introduces

Well, let's hope you're not powned because of this and get dragged into giving some fucks, for a 5-10% of performance hit you wouldn't notice anyway...

uwydr(10000) 6 days ago [-]

You can do the same thing on Windows by using InSpectre: https://www.grc.com/inspectre.htm

wolf550e(4198) 6 days ago [-]

Your web browser runs untrusted code. On a computer that never runs untrusted code you can do that.

m_eiman(3510) 6 days ago [-]

Is it possible to apply the mitigations on an per-application level in Windows? IMHO it'd be pretty useful to be able have them on by default, but disable them for specific applications where you care about maximum performance and know that you won't be running untrusted code.

systemdtrigger(10000) 6 days ago [-]

Tagging on with a similar query. I am on Windows 10. If I were to run Firefox in sandboxie, would be the attacker have to deal with an extra layer of security or does it offer no help?

alkonaut(10000) 6 days ago [-]

Surely if you emulate a processor without speculative execution with good fidelity (Such as a good NES emulator) then the program running ON that emulator can't deduce anything from speculative execution?

Is the answer (for home users) to just sandbox some processes under an emulator layer? I'd be happy to just sandbox some sensitive processes like my browser even if it took a huge performance hit, so long as some other apps like games did not take the same hit.

annoyingnoob(10000) 6 days ago [-]

Is disabling hyper-threading a via work-around for these vulnerabilities?

OrangeMango(10000) 6 days ago [-]

My understanding is that this works for many (but not all) of the vulnerabilities. Interestingly, in the past few years Intel has dramatically increased the number of non-SMT processors on their product list.

If that is going to be your personal way of mitigating the issue, you've got a choice of 4, 6, and 8 core parts at a significant discount compared to their HyperThreaded variants.

systemdtrigger(10000) 6 days ago [-]

One question- do these vulnerabilities , including spectre and meltdown only help in stealing information or can they also hijack your computer to do arbitrary things?

chii(3664) 6 days ago [-]

depends on the information you steal.

If you steal passwords, then you can use said password to hijack whatever the passwords are protecting.

If you steal private keys, you may be able to use said keys to impersonate the victim (like via ssh into their remote machines).

But if you're asking if speculative vulns could directly lead to remote code execution, then no (since you already have given the attacker a measure of control, as they are able to execute code already).

rocqua(4221) 6 days ago [-]

It can be used to defeat ASLR, which is a way to make exploiting code harder. However, defeating ASLR just makes it easier to deploy an exploit against a program, but you still need the exploit.

It doesn't immediately give code exec, but generally it wouldn't be very hard to turn arbitrary memory read capabilities into privilege escalation. As long as you know what the system is running.

paulddraper(4026) 6 days ago [-]

To exploit these vulnerabilities, you already need (unprivileged, sandboxed) RCE.

These vulnerabilities 'only' steal information; however that information could of course be leveraged into privilege escalation or anything else.

aeiou1234(10000) 6 days ago [-]

another 0-4% performance hit for skylake

soulofmischief(4222) 6 days ago [-]

I'm glad I started off not overclocking my 6700k with the intent of cranking up the clock over time in order to maintain the same performance with regard to software bloat and mitigations like these. I got lucky and can hit 5Ghz safely, with a base of 4Ghz.

Definitely jumping to AMD next time around though. My next upgrade was originally going to be dual Xeons but those Ryzen Pro 3000s are looking nice.

stock_toaster(2351) 6 days ago [-]

What are we down to now? 70%? 80%? of original performance with all the mitigations added up by now?

If a car was advertised as emitting 20% less pollution than it actually did, people would be pissed[1]!

[1]: https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal

atq2119(10000) 6 days ago [-]

The really damning part is that it applies even for processors that are supposedly fixed in silicon because Intel dropped the ball by playing wack-a-mole with proof of concept exploits instead of thoroughly building their chips with security in mind.

If the history of Microsoft and Windows security is any indication, it'll take Intel many many years to turn that ship around.

There's a question of whether AMD has been mostly unaffected only because their chips haven't received as much scrutiny, but for the time being it does seem that if you care about security, you'd better go with Epyc.

foxes(3734) 6 days ago [-]

Another nail in the casket lake. Is the solution just to throw everything out and start again? Do we just abandon speculative execution?

wolf550e(4198) 6 days ago [-]

AMD and others are fine. Just have security people on the team during microarchitecture design.

Spectre variant 1 is probably unavoidable so security inside a single virtual address space is kinda dead. Use separate processes and mmu.

userbinator(737) 6 days ago [-]

Or we abandon sharing the same hardware for trusted and untrusted code. (Yes, that includes things like whitelisting JS.)

Symmetry(1952) 6 days ago [-]

Abandoning speculation entirely is too much. If you just go to in order with a reasonable pipeline length nobody has yet figured a way to smuggle out speculative state on something like an ARM A53. That means giving up performance but only a factor of 4 or so. Giving up speculation entirely would mean an order of magnitude larger performance loss.

nolok(4169) 6 days ago [-]

The problem for most of these is not speculative, it's not doing proper acl during speculation. Just because the cpu is speculating doesn't mean it shouldn't check if you are allowed to access this or that, but that's what Intel did, and they only did the security check at the end before giving back the result, except at this point its too late you've already accessed it.

Other manufacturers AMD included didn't get affected by those variants.

Jonnax(10000) 6 days ago [-]

So Intel failed to mitigate the vulnerability when it was first reported. Then they extended the embargo from May until November.

And they still didn't fix it.

What's going on with Intel? Like they're going all in with lying in benchmarks against AMD and straight up forgetting what has been reported as security issues.

dd36(4052) 6 days ago [-]

That seems to be the new corporate PR move.

coldtea(1239) 6 days ago [-]

>So Intel failed to mitigate the vulnerability when it was first reported. Then they extended the embargo from May until November. And they still didn't fix it.

The assumption being that they could in the time given, but are sitting on their hands?

chooseaname(4220) 6 days ago [-]

A focus on managing by spreadsheet instead of focusing on good solid engineering, maybe?

nootropicat(10000) 6 days ago [-]


>Company is dying and has no way to turn itself around.

>Culture is a 'go along to get along' / don't rock the boat. Most workers are very passive. Inclusive culture focused on internal 'networking' rather than winning. A lot of make work going on, probably 25% extra headcount

>Middle and upper management are in direct revolt against CEO and his plans. During my orientation (2nd quarter 2013) my orientation meeting was about how the CEO is wrong on his plans.

>Advice to Management

>Not much you can do. These problems are the result of near monopoly on PC CPUs for 20 years. This place is what Hewlett-Packard was probably like in 2000 (with the printer monopoly), the collapse is coming, but without starting over there is no way to fix it.

krageon(10000) 6 days ago [-]

Intel has become big and rich and has stopped (or perhaps never was) being very responsible. Given their fairly entrenched position in the industry, it's doubtful this strategy will impact their profit all that much and therefore we can expect to see more of this behaviour in the future.

rrss(4161) 6 days ago [-]

> going all in with lying in benchmarks against AMD

I'm assuming this is referencing the 'intentionally misleading benchmarks' piece in servethehome. It's worth reading the follow up, in which the author discovers their biggest complaint (older version doesn't have avx2 enabled by default on zen2) was not an issue because Intel manually enabled avx2.


nolok(4169) 6 days ago [-]

Zen and Zen+ may have been strong competitors that Intel was unprepared for, but at least they were still competing, notably in per core performance. Now Zen 2 is out and Intel is taking a beating in every category, from price to power usage to multi core performance to single core performance and everything in between.

Meanwhile it became clear that a decent part of Intel's per core advantage was because of massive shortcuts they took, notably security wise, and so fixing those security holes takes away the performance.

In that situation, I doubt management at Intel is eager to take out yet another one of their shortcut. Intel needs a win, but they are not at all in the state of mind necessary to achieve one...

IntelThrowaway(10000) 6 days ago [-]

One of the problems with Intel culture - especially under BK was that the philosophy was 'Focus on our key goals to the exclusion of everything else'. It was meant to keep focus and ensure we moved quickly. The problem with that is it meant we were entirely unresponsive. It doesn't matter if something important has come up because you've already agreed what the priority is, you've already committed to what you're going to do. So even if something does come up, communicating that problem to the team that needs to fix it is impossible because you'll get ZBB'd (if we do this, we will drop that). Then once you've got engineering to commit, the bureaucracy won't let you just release anything so you need to line up into a release process.

I'm sure no one intended to mislead, but organisationally Intel just isn't designed to fix bugs. It doesn't have a process to respond to issues.

zaphirplane(10000) 6 days ago [-]

Ok this has been bugging me for a while. How does speculative execution roll back side effect like write to disk or send a packet on the network, when the speculation is wrong. At a guess there are safe instructions that can be run when doing branch prediction?

jfnixon(10000) 6 days ago [-]

I/O takes seven eternities in microcode-ville. It doesn't hurt too much to wait for the speculation to be resolved before changing state visible outside the microarchitecture of the CPU.

wolf550e(4198) 6 days ago [-]

Nothing like that happens. Writes to dram or pcie devices or data bus cannot be rolled back. You have a misunderstanding of what speculative execution entails. The things that get rolled back are writes to general purpose registers.

cwzwarich(10000) 6 days ago [-]

Accesses to memory that is configured for device I/O (UnCacheable in x86 terms) are never speculated, so there's nothing to roll back.

xucheng(3399) 6 days ago [-]

> We are particularly worried about Intel's mitigation plan being PoC-oriented with a complete lack of security engineering and underlying root cause analysis, with minor variations in PoCs leading to new embargoes, and these 'new' vulnerabilities remaining unfixed for lengthy periods. Unfortunately, until there is sufficient public / industry pressure, there seems to be little incentive for Intel to change course, leaving the public with a false sense of security. Slapping a year-long embargo after another (many news cycles apart) and keeping vulnerabilities (too) many people are aware of from the public for a long time is still a viable strategy.

This is troubling.

Mirioron(10000) 6 days ago [-]

I wonder if it might not come to bite them though. Doing it this way means that the media will constantly keep talking about yet another vulnerability in Intel CPUs for years. There's a chance that Intel might and up picking a reputation like Flash did for being insecure.

segfaultbuserr(2584) 6 days ago [-]

Perhaps researchers can use the tested-and-proven 'Full Disclosure' tactic to exert public pressure on Intel. It doesn't need to disclose everything, just two or three additional unpatched PoCs with full source code would be enough.

However, unlike buffer overflow exploits, most researches on CPUs are conducted within academic institutions, doing this certainly breaches the code of conduct. Also, CPUs are the most critical components of all computers and their vulnerabilities are difficult to fix, doing this would put a lot of users under immediate risks, unlike a root exploit, which is less risky and can be fixed within a week. Doing Full Disclosure of hardware exploits that users can not fix is much more ethically problematic than software exploits.

But leaving the users in the dark and allowing Intel to delay its fixes by not to exert pressure is obviously irresponsible, which is the original argument for full disclosure.

So I'm not sure. Perhaps Google's Project Zero is a good model and a good compromise between resposible/full disclosure - embargo for 90 days, full disclosure later, all information becomes public after 90 days and no extension is allowed, period. For CPUs, perhaps we can use 180 days.

simion314(4216) 6 days ago [-]

Delaying the fixes is probably making them more money, if they lose a few more percent of performance they lose the number one position on all possible use cases/workflows , the PR hit will be huge, people could demand refunds or compensation (I mean you just sold them an expensive product without disclosing it is insecure and it will lose it's performance)

mtgx(152) 6 days ago [-]

It certainly is, and that much was obvious as Intel 'committed' to fixing Spectre V1 in software as opposed to hardware. Worse yet, it doesn't think a hardware solution is necessary. Except, that pretty much all recent Spectre-related bugs are variants of Spectre V1.

josteink(3602) 6 days ago [-]

> This is troubling.

Only for people who for whatever reason still continue to buy Intel-based systems.

spookydookeedoo(10000) 6 days ago [-]

The Deep State works in mysterious ways in plain sight. Freedom is an illusion. ;-)

iforgotpassword(10000) 6 days ago [-]

AMD is suffering much less from these flaws. Seems they didn't ignore as many security boundaries with their implementation.

Craighead(10000) 6 days ago [-]

or there are just as many vulnerabilities on amd but researched less

wolf550e(4198) 6 days ago [-]

AMD (and ARM OoO chips) are vulnerable to Spectre variant 1 (bypass in-process array bounds checking) but not to the vast majority (any?) of the other issues which are Intel-only.

AMD chips don't have the feature that speculation failure is determined at instruction commit time when it is already too late, so most issues just can't happen.

justapassenger(10000) 6 days ago [-]

There are some architectural differences that help amd with this specific issues, but I'm pretty sure main reason is just scale - amd is still niche nowadays and receives much less scrutiny and attention from security researchers

Historical Discussions: The Efficiency-Destroying Magic of Tidying Up (November 17, 2019: 559 points)
The Efficiency-Destroying Magic of Tidying Up (September 04, 2019: 2 points)

(581) The Efficiency-Destroying Magic of Tidying Up

581 points 2 days ago by cryptozeus in 3590th position

florentcrivello.com | Estimated reading time – 7 minutes | comments | anchor

In his seminal book Seeing Like a State, James Scott describes what he calls "high modernists:" lovers of orders who mistake complexity for chaos, and rush to rearrange it from the ground up in a more centralized, orderly fashion. Scott argues that high modernists end up optimizing for a system's legibility from their perspective, at the expense of its performance from that of the user.

Indeed, that love of order is above all else about appearances. Streets arranged in grids, people waiting in clean lines, cars running at the same speed... But everything that looks good doesn't necessarily work well. In fact, those two traits are opposed more often than not: efficiency tends to look messy, and good looks tend to be inefficient.

Efficiency ain't pretty

This is because complex systems — like laws, cities, or corporate processes — are the products of a thousand factors, each pulling in a different direction. And even if each factor is tidy taken separately, things quickly get messy when they all merge together.

The chaotic look of structural orderliness shouldn't be so surprising. Intellectually, we do understand that appearances are misleading — things don't have to look as they are, nor be as they look. But intuitively, we all remain hopeless slaves of appearances, no matter how often we were misled by them.

This natural messiness of efficiency is demonstrated by recent advances in industrial design. When a God-level AI takes over in a science fiction book, it often remakes the world in its image: full of straight lines, smooth acceleration rates, and lots of chrome (AIs love that stuff). But as we start using algorithms to design things, we get results that look a lot more chaotic than that, confirming that our intuitive preference for "straight line" designs has nothing to do with performance — it just comes from our limited ability to reason about more complex solutions. Ironically, it's us humans who think like robots.

The Wendelstein 7-X fusion device looks asymmetrical and messy, like it's out of a bad sci-fi set. pic.twitter.com/Bx1jjsWYwS

— Jo Liss (@jo_liss)

December 8, 2015

London's tube map only uses 45° angles to aid its human readers.

Now can you see the humanness in mainboard design? pic.twitter.com/7giQJjI8iI

— Jo Liss (@jo_liss)

December 9, 2015

An evolved antenna design, Hornby et al 2006. This is orderly — it's straight antennae that really are messy.

This is a messiness similar to that you'd find in nature — which makes sense, since both these algorithms and nature are optimizing for efficiency.

I submit that we should look with suspicion at simple-looking systems. The physical world is like a river in which a thousand streams come rushing — it is supposed to look messy.

Again, this insight applies to any complex system. For example, a city can look as messy as an anthill. But really, it's a beautiful equilibrium that evolved to satisfy a thousand competing constraints: topology, weather, people's traditions, skills, wealth, preferences... Planners may make their maps look better when they use zoning to separate the city into business, residential, and commercial neighborhoods, but they also destroy a subtle, efficient balance. They forget that the only activity that goes on in any city is that of people living their lives, which requires all the activities above — preferably in close proximity. Splitting a city into residential, commercial and business zones is like throwing dough, cheese and pepperoni into the different compartments of a bento box and calling it a pizza.

An urban planner's dream pizza

Speaking of pizza. One good example of an attempt to avoid the high modernist fallacy is Amazon's "two pizza teams," set up to run like small companies and encouraged to use the tools and processes that make sense at their level. The outcome can be messy — new Amazonians who come from more centralized companies often complain about the chaos there. But I, for one, would choose chaotic success over tidy failure any day.

Interestingly, other managers often pick the latter. They'll see a dozen teams, each using different project management software, and lament: "what a mess! No one even knows what everybody else is doing!" They'll then resolve to "put some order in here," by mandating every team uses a single company-wide tool. But teams had been using different tools for a reason. Maybe they're in different businesses — for example, hardware and software groups have different needs. Or their members could just have different preferences, which matter too — people are more productive when they use tools they like! It can be okay to sacrifice a company's productivity so as to increase the coordination between its parts — so long as one makes this compromise with their eyes wide open. Unfortunately, managers all too often look only at the upside of such changes, and ignore their downside.

"Please clean up your room," asks the mother. "Fool," retorts the three-year-old with an eerily deep voice. "Can't you see the beauty in my glorious chaos?"

I'm not suggesting all chaos is good. But when you hear someone suggest we put some order into a system, there should at least be a red light flashing in your head. Before you touch anything, you should make sure that mess is not concealing a deep order, by answering the following questions:

  • How much information is contained in the system's current state? What constraints are expressing themselves through it? For example, when throwing dice, where they land depends on how you threw them. You do lose this information when you take the dice and put them back into a box — and that's probably fine. But in the case of a city, its structure is the function of all the factors mentioned above — most importantly, its people's skills, means and preferences. When you use zoning to impose your arbitrary aesthetic upon people, you block them from shaping their environment so it fits their lives.

  • How old is the system? How malleable is it? How strong are the forces put on it? The older a system, the more malleable or subject to strong forces, the least likely it is to be truly chaotic — and the more careful you should be when messing with it. That's because its components had the time, ability and stimuli to order themselves into a stable configuration. Crystals are a good example of this. Take some randomly arranged molecules, keep them warm enough so that they remain mobile, and put them under high pressure for long enough, and they'll spontaneously self-organize into one of the most orderly structures in the Universe.

  • Finally: who is complaining about the chaos? If outsiders complain, but people living inside the system seem happy with it, it probably means that the chaos is serving them right, and that it's just foreign eyes who are unable to perceive its underlying order.

This is a special case of Chesterton's Fence, which states you should never take down a fence before knowing why it was put up. Here, I propose Scott's Law: never put order in a system before you understand the structure underneath its chaos.

All Comments: [-] | anchor

TeMPOraL(2647) 2 days ago [-]

Incidentally, I think the producers of Star Trek nailed this with design language of the Borg, perhaps accidentally.

For those who don't know, the Borg are a machine intelligence hive mind species that roams the galaxy in their cubes, spheres, diamonds and other ships shaped like basic solids, destroying or assimilating anything interesting in their path, and speaking a lot about 'bringing order to chaos'. Yet seemingly despite their focus on order and perfection, the individual drones and the microstructure of the ships both look like a haphazard bundle of lights and wires. This is in line with the article - machine intelligence optimizing for extreme efficiency across multiple dimensions isn't going to create sleek-looking constructs.

yellowapple(4183) about 20 hours ago [-]

I suspect it ain't all that accidental. Each Borgship behaves (ostensibly) as a single giant organism (and the Borg within it its cells), and multicellular organisms are indeed pretty messy inside, even if they look externally simple and uniform.

lonelappde(10000) about 19 hours ago [-]

The drones are messy because they have ugly messy human bodies. The ships are highly patterned and repetitive.

daguar(10000) 2 days ago [-]

I think I have to quibble with this:

> Here, I propose Scott's Law: never put order in a system before you understand the structure underneath its chaos.

James C. Scott wouldn't probably never underwrite re-ordering of systems from the top down.

Central to his argument is that viewing complex systems from any singular position requires a process of simplification (legibility) that prevents a complete understanding.

The presumption that one has gotten to a place of 'understand[ing] the structure underneath [the] chaos' is in fact the false confidence he attributes to most of these ordering projects.

I think if you want to wrangle a suggestion from Scott's book, it's more about making lots of small pokes at a system and seeing how it reacts, and slowly building on positive reactions from the system.

(Also, messiness and complexity are not intrinsically linked to the efficiency of a system — systems can optimize for lots of variables and it's really context-specific. So as much as you shouldn't take order to be innately good, don't take messy to be innately efficient!)

mtts(4110) 2 days ago [-]

Have an upvote.

By coincidence I just read Scott's book not too long ago and can attest that it contains not a single suggestion that imposing any sort of order on a system is ever a good idea. Quite the opposite, in fact.

There may be lessons for city planners and agriculturalists in the book, but I'm pretty sure there isn't a single one for software developers.

FrankyHollywood(3778) 1 day ago [-]

> 'I think if you want to wrangle a suggestion from Scott's book, it's more about making lots of small pokes at a system and seeing how it reacts, and slowly building on positive reactions from the system.'

I can feel a bit for it. I have worked on a complex multi-million line codebase. There was this predictible behavior when a new guy came to join us. Many times they had never worked on such a large project.

First they rant about how terrible the system is, than tell us how wrong the system is, followed by advice about how we should use this an this method or design patterns to make things right.

Only after months of working with the code the guy would cool down. When a (software) system is really complex, you have to get familiar with it. No matter how you structure it, a minimum number of logic and dependencies will always exist. Restructuring will not take away the complexity. It's just replacing them.

Offcourse a clear structure is better for you're understanding than a messy one, but even in the most clearly written code a lot of complexity can exist. I think Scott is talking about this, the point beyond clear structure.

thats complexity :)

cookiecaper(3181) 1 day ago [-]

> Here, I propose Scott's Law: never put order in a system before you understand the structure underneath its chaos.

Previously formulated as Chesterton's Fence [0], among others.

There's definitely a blind spot in software for the general principle that you should understand a thing well before you decide to remove it. Anyone want to propose some theories on why disregard for an extant body of work tends to plague software so extensively?

[0] https://www.chesterton.org/taking-a-fence-down

PeterStuer(10000) 1 day ago [-]

I'm a 'lover of chaos'. My working desks always gather strata of documents, more often than not disheveling onto the surrounding floor.

This article however rubs me the wrong way right from the start. 'Cars running at the same speed' aren't 'efficiency-destroying' nor about spurious 'appearance'. A controlled laminar flow is incredibly more efficient than chaotic turbulence.

It does not get any better later on, when the author waxes lyrically about 'beautiful equilibrium that evolved to satisfy a thousand competing constraint' in the absence of planning, casually glossing over the part where communal zoning regulation prevents not just externality dumping races to the bottom and the basis for non-speculative investment to those that can't afford to gamble or force their own 'manu militari' regulation.

While self proclaiming 'not suggesting all chaos is good', it is exactly what the rest of the article's suggestive language tries to convey. The deregulation agenda, while not explicitly spelled out, is omnipresent in the tenure of the writing.

P.S. I'm not surprised to learn that the author works for Uber.

jazzdev(3710) about 24 hours ago [-]

Isn't the trend in cities for more mixed-use space?

And wouldn't Uber be more profitable if people lived and worked in different parts of the city?

Crashbat(10000) 1 day ago [-]

Very well said, and I think you've hit the nail on the head with regards to the 'deregulation agenda'. The subtext of articles like this is always that interference, in the form of regulations, is a negative action, and by stripping away the messy human meddling we'll reap the rewards. It's never considered that the human meddling is so often done to protect the humans themselves, most often the very weakest or most powerless.

rsync(3656) 2 days ago [-]

Clutter and tidying is simply a time debit/credit ledger.

If you had unlimited time, you would tidy and organize everything in real time. However, we don't have unlimited time and so we borrow time from the future by leaving things slightly messy.

The behavior to avoid, as in money ledgers, is not using the ledger but never paying it back.

vlasev(10000) about 13 hours ago [-]

You gain back some efficiency by cleaning up all in one go at a later time! Think of trips to the sink with the dirty dishes.

m0zg(10000) 2 days ago [-]

I strongly disagree with this. Software should not be written to suit the machine. Software should be written to accommodate the weakest link: humans who are going to need to figure it out 6 months from now when its current programmer departs for the greener pastures. If there's no rhyme or reason to it, it fossilizes immediately and costs a lot of money to rewrite (often into another fossil). So whatever effort you spend to reduce complexity and cognitive load, will pay for itself several times over. As a rule of thumb, the hallmark of a good system is when your guesses about how something would be done are right almost 100% of the time. You don't arrive in this state by accident.

FooHentai(10000) 2 days ago [-]

Yep. What this article skips over is maintainability in all of the things it discusses, and in most cases also flexibility.

The optimization it praises only has single focus (operational efficiency), whereas if you take into account all of the aspects of the things in question, suddenly you see that the way things are are usually a solid compromise between the aspects it's really optimizing for. That includes human interactions such as analysis, repair, modification, and so on.

carapace(2884) 2 days ago [-]

I once cleaned up my sister's room. I thought I was doing her a favor but she was mad at me for a week! In hindsight I should have known better. Spacial memory and all that, she knew where everything was even though it looked messy.

The article is not talking about the kind of tidiness that is the hallmark of a well-run workshop, IMO. That kind of tidiness is part of what makes more efficiency in that context.

TeMPOraL(2647) 2 days ago [-]

I've been mad at my parents plenty of times for cleaning up my room as a kid for the same reason - even though it might have looked like a mess, everything naturally ended up in the most obvious place for me; a kind of mind-environment feedback loop setting on an equilibrium.

FooHentai(10000) 2 days ago [-]

>Spacial memory and all that, she knew where everything was even though it looked messy.

This is something my wife and I have figured out over time - If it's your own space and nobody else uses it, it's fine to descend to 'chaos' because you can just remember where you last put something.

That model breaks if the space is occupied by more than one person, because where one person put something isn't witnessed by the other person. So, having designated places for things and returning them to their 'home' makes the space most usable by multiple people.

Personally, even for solo spaces, I still favor everything having a home because I don't have a good memory for where things were placed.

BlueTemplar(10000) 2 days ago [-]

Hmm, I thought that Amazon was famous for how they standardized their cross-company software? (With AWS as a side effect.) Or is this not the case anymore?

foobiekr(10000) 2 days ago [-]

If standardized means APIs, yes. If standardized means anything to you about implementation, even at the level of "don't reimplement what another team - or even another part of your own team - already implemented" then no.

ReptileMan(10000) 2 days ago [-]

They didn't standardize the software, they standardized that all software should be accessible by API and that all software should be designed in such way to be openable to third party users if need arises.

quicklime(10000) 2 days ago [-]

That's what I was thinking when I read the article.

When I worked at Amazon, everyone worked out of the same repo(s), used the same build system (Brazil), deployment system (Apollo), deployed to the same place (EC2), most APIs were written using the same framework (Coral), etc. This is not that unlike the other large tech companies, and now that I'm back at a large non-tech company where everyone builds their own slightly-incompatible tools and deployment pipelines, I do miss that standardization.

Sure, some people chose to use different tools here and there, but most people wouldn't choose to reconstruct the universe just to stand up a microservice. It's not like all the 'two-pizza teams' were all doing completely different things with incompatible tooling.

It's been a while since I worked there, so things might've changed completely since then. But as a shareholder, I hope not.

nickbauman(4216) 2 days ago [-]

IIRC, they only standardized the interfaces, not the implementations. You could write your team's code any which way you wanted, but you had to expose a consistent REST api in front of it all and publish its specification to the rest of the company.

dugditches(4214) 2 days ago [-]

I mean that image of the 'before/after 'topological optimization' seems like a poor analogy.

>confirming that our intuitive preference for "straight line" designs has nothing to do with performance

As the first is made to be made because of its 'straight lines'. You can see the welds and logic in it, as something easily made.

While the second is something only achievable via 3D-Metal printing or possibly very complex 5 axis CNC.

Ensorceled(10000) 2 days ago [-]

That's the first thing that struck me as well; a person like my dad (a retired blacksmith), could make the first one ...

pizzaparty2(10000) 2 days ago [-]

Sorry but there's no excuse for crap code. If it's hard to understand then time is lost every time you have to work with it.

You shouldn't defend it by trying to assign positive attributes to it (its effecient. ...yeah). Maybe there's nothing good about bad code and we should use being called out on it as an opportunity for improvement.

No, lets double down on our delusions of superiority by telling ourselves that crap we wrote is somehow effecient in some way.

Im tired of it. Its not effecient. Its bad and you should feel bad.

cookiecaper(3181) 1 day ago [-]

Code, at least the way we currently write it, is a fundamentally lossy transfer. The coder has the context in their head; they understand what's happening up to that point, what they want to happen now, and what needs to happen later. Writing code is translating those concepts and ideas into a specialized step-by-step list of machine instructions.

In the course of writing the concepts in a dumbed-down format for an instruction machine, context is inevitably lost. You can say 'write comments' until you're blue in the face, but it doesn't solve the problem.

Then, someone new comes along, and they don't understand the context, they never knew anything about it before, and you quit 8 months ago. This person must infer the context and determine an accurate conception of the context from the pieces left behind.

The approach taken when one finds themselves in that position is one of the key tells of their skill and experience IMO. Regardless, it seems that if the software got out of the loose demo/messing around phase, it deserves at least some consideration before it's dismissed out of hand as 'bad code'.

matthewaveryusa(10000) 2 days ago [-]

I disagree with such a blanket statement. Interfaces must remain simple, while implementations are free to be as complex as needed. Take a look at this article -- clearly the SIMD code is dense to get through, but it's very-much-so worth it for the performance gains.


mewpmewp2(10000) 2 days ago [-]

What about a mvp scenario? Many successful businesses have crap legacy code, but maybe they would not have been able to become successful ever had they not rushed with the code? Are you saying taking tech debt is never good?

To me it seems like a failure to take a look at the bigger picture. In the end it is all about the value provided. You are creating a start up that has 10 percent chance of succeeding and only if it gets to market fast.

Also it is never binary. It is always time invested vs quality of the code. It is a spectrum. From what point can you call code crap?

The closer you get to perfect code the more time it takes to get it better as you are closing near perfect.

cryptozeus(3590) 2 days ago [-]

Agree but this can also become subjective very quickly. Oh this code is crap according to "me" so lets re write it the way "I" think is better without understanding why it was written that way to begin with. If you apply this article to code then I would say it argues that find out the reason before just removing it.

MaxBarraclough(10000) 2 days ago [-]

> its effecient. ...yeah

> Im tired of it. Its not effecient. Its bad and you should feel bad.

You're ignoring that sometimes, measurably more efficient code is less readable, possibly much less readable. If this weren't the case, there'd be no such thing as sophisticated data-structures and algorithms.

This doesn't mean anyone should make an uncommented ball of mud with no comments, of course.

> No, lets double down on our delusions of superiority by telling ourselves that crap we wrote is somehow effecient in some way.

The usual wisdom about efficient code, answers this: if you aren't measuring performance, that means you don't really care about performance.

If you're going to great efforts, and writing less readable code, on the hunch that this is more efficient, then sure, you're doing it wrong.

andy_ppp(4121) 2 days ago [-]

I actually think having business people who have control over what programmers do is the mistake very often. It leads to very bad decisions for the codebase as a whole and the programmers often don't know the best way to fit in the random business needs because they are never told why the feature should exist. I think we need a better method to be able to make the right decisions for the code while still achieving the desired outcomes without making quite such a mess. Maybe something like sessions where the business has to persuade the developers to build the features in detail rather than throwing random unclear solutions over the fence and saying we need this.

dionian(10000) 2 days ago [-]

Often true, but often 'beautiful code' does not impact the bottom line - even in the long term.

mc3(10000) 2 days ago [-]

Yes using the 'city' analogy - the 'city' might be the entire of the AWS stack, where as the 'building' might be what is produced by the 2-pizza team. And the code produce by that team (like the building) should be tidy as much as possible, except where there is a genuine need to get performance and that requires making things a bit messy.

Forge36(10000) 1 day ago [-]

I'm a big fan of design two systems. If timelines permit, ship the second, else ship the first. The first code will almost always be crap code because it takes designing the system to better see how the system should have been designed.

Problems arise when the first system gets left in place for too long. It starts to grow developers sometimes whole teams arise, and many people ask why, but are too afraid to touch it

uulu(10000) 2 days ago [-]

Strange, how did you come to the conclusion that this is a defense of 'crap code'? For one thing - crappy code is most often less efficient than elegant code.

TeMPOraL(2647) 2 days ago [-]

When applying this article to code, I don't think it has anything to do with spaghetti being good. It could be viewed instead as an argument against 'architecture astronautics', the '15 layers of abstraction to print 'hello world'' school of software design.

detaro(2090) 2 days ago [-]

There is no globally applicable standard for what's crap code, so there's plenty of reason to excuse code some people might label 'crap'.

'Don't write crap code' is a terribly empty statement. Trying to 'tidy up' 'crap code' is also a great way to screw things up further if you turn out to be wrong about it.

jariel(10000) 1 day ago [-]

The article does not imply that chaos is inherently good or efficient.

When I look at big, 'good', codebases, I'm always amazed at how much complexity there is at every level, and wonder why it can't all just be clean and orderly.

The trade off between performance, bureaucratic code and leaky abstractions seems to be ever-present.

I can never make an API 'fit' that elusive ideal of what it should be, there are always bits of dangling weirdness.

Try for example doing anything in Unicode; it's as though there are permanent grey areas and ambiguities. There's always the corner case that someone, somewhere in some corner of the world will either break it, or be without the ability to 'spell their name' or whatever.

I don't think this is about arbitrarily crap code.

jerf(3378) 1 day ago [-]

Most of the reasons why code is hard to understand will be on a fairly small scale. It is possible to yank that out and replace it, while honoring and respecting any underlying chaotic organization that may have occurred despite the crap quality of the code. (As the article says, this sort of chaotic adaptation requires flexibility and crap code only inhibits that.)

That's pretty much what I've done in this last calendar year, taken two messy systems chaotically (in this article's sense) interfaced with the rest of the company, and upgraded them. I gave them new, hopefully-non-crap code, certainly better documented code, documentation on the system structure, better operational deployment, massive upgrades to security, general speed improvements, and in one case, a fundamental architectural change at the most foundational level even though the surface that the users interacted with hardly appeared to change.

And yet... despite all that, I would not say I 'rewrote them from scratch', because I did not simply start with a blank sheet of paper and start scribbling what I think the ideal solution would be. I took the underlying adaptations, respected them, assumed that they likely had a reason even if I didn't know what they were, and built systems that largely dropped into place on top of the old ones rather than being totally foreign bodies that cause cascading requirements for other systems to also be significantly rewritten.

If, in the future, those other systems do get rewritten, I even have some paths prepared (but not yet written) in the code for true architectural upgrades to occur in the future. But in the meantime, I have improved systems that work now.

It's a much better approach to replacing a system that denying the chaotic adaptations. Even 'crap code' can be mined for them, and they are not generally the crappiness itself.

GuB-42(10000) about 16 hours ago [-]

The more I code, the more I like messiness. By that I don't mean bad code. I mean code where all techniques in and out of books are used together.

For example, a mix of exception and error returns, direct access to class members combined with accessors, etc... I like to see best practice rules being broken when there is a good reason for that.

Rule of the thumb: good code is short code. If your 'tidying up' makes the code longer, then it is probably better to leave it as it is. What you are going to do is most likely add useless layers of abstraction or try make it abide by some made-up rule that shouldn't apply here.

As for 'efficient' code. Most efficient code I see is actually quite good and readable. The worst code I see usually gives no fucks to efficiency. At least premature optimizers show some love to their code.

pmoriarty(51) 2 days ago [-]

'When a God-level AI takes over in a science fiction book, it often remakes the world in its image: full of straight lines, smooth acceleration rates, and lots of chrome (AIs love that stuff). But as we start using algorithms to design things, we get results that look a lot more chaotic than that...'

It would be interesting to see a movie where an AI takes over and turns reality in to a Rube Goldberg machine.

CPLX(3320) 2 days ago [-]

> It would be interesting to see a movie where an AI takes over and turns reality in to a Rube Goldberg machine.

How would that be distinct from actual reality?

api(1184) 2 days ago [-]

Huge corporations and governments are kind of like hybrid silicon/meat AIs and that's exactly what they've done. I think this is the most likely outcome.

Mindless2112(4047) 2 days ago [-]

Having seen AI play Mario, I'd expect to see it try to achieve its goals by exploiting bugs in reality based on whatever simulated reality it was trained in.

koboll(4221) 2 days ago [-]

The movie Annihilation kind of evokes this, albeit without the AI (and a lot more slowly)

Mirioron(10000) 2 days ago [-]

I don't think AI would turn reality into a Rube Goldberg machine, but rather that it would create so much complexity that we can't see through it.

Mathnerd314(3648) 2 days ago [-]

Joel has a post about working in a bakery: https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...

His idea is that there is a pretty high standard of cleanliness in a bakery, just not the obvious one. A similar 'hidden rule' presumably applies to cities etc.

NoodleIncident(10000) about 23 hours ago [-]

I'm almost certain I've read this article before, but I think I've read a lot more about types since then. In particular, this line is hilarious:

> I'm using the word kind on purpose, there, because Simonyi mistakenly used the word type in his paper, and generations of programmers misunderstood what he meant.

It sure doesn't sound like Simonyi was the one making a mistake here! It sounds like Hungarian notation is what you use to compensate for missing types; it got its bad reputation from people using it for types that the language already provides, because that's easier.

ratel(10000) 1 day ago [-]

This article is wrong on so many levels:

1. Theoretical: chaotic and complex are not the same thing. Organized and tidy are neither the opposite from or excluding complex systems. Some very tidy systems can still be mind-boggling complex. Furthermore Chaotic systems have a single underlying order (it might just be impossible to discern), but they are not complex. Complex systems might have multiple orders, none at all or everything in between.

See for an in-depth discussion: The Collapse of Chaos; Discovering Simplicity in a Complex World, Jack Cohen & Ian Stewart, Penguin, 2000.

2. Practical: The amount of tidiness in a complex system does not say anything about its efficiency, but neither does its untidiness. Complex systems might be untidy through sunken costs, like most cities are, revolution, evolution, historical accident, or even attempts to make an ordered system (like a system of law) always leaving some gap. The fact that a complex system exists does not say anything about it being efficient or its fitness for purpose. Like the three year old who thinks the mess is beautiful, but cannot find her favorite plush toy.

Changing complex systems is indeed hard, because different subsystems will interact, have feedback loops, create externalities, exclude external information or are near impossible through sunken costs. But that still leaves the calculation of how much the new system or altered system might be more efficient or better than the current system and what the risks involved are.

Tiding up complex systems has risks, but also benefits even if you don't believe the second law of thermodynamics applies to human created systems. Tidied up systems are more easy to reason about and thus can be more easily fixed, adapted or expanded upon. The act of tiding up has the additional benefit of adding to our knowledge about how a system actually works.

3. This guy works for Uber. The cab company with a computer that wants to uproot the current (complex) systems and replace it with something simpler. Yet fails to turn a profit. Do we need to read something into this article, or did he just not realize he contradicts his employers mission?

stjohnswarts(10000) about 23 hours ago [-]

3. I don't think Uber is less complex, just complex in different areas. Uber's sole purpose is to make its owners more money, currently that means providing a upheaval in the industry and providing easy ride access to humans. When those humans are no long the best (or most obvious way) to make money the capital and investors will move on.

rossdavidh(4125) 1 day ago [-]

All true, and I would also add a lesson from software design. Spaghetti code might, in some limited sense, be more efficient in some cases, but it is difficult to understand, therefore difficult to fix when something is wrong. Making things orderly, even from a top-down perspective, may be more important than being as efficient as possible, if you need to be able to understand what's happening in order to use (or fix) whatever it is.

kioieiure(10000) 2 days ago [-]

> London's tube map only uses 45° angles to aid its human readers. Now can you see the humanness in mainboard design?

What was that russian electronic board design software where nothing is vertically/horizontally aligned, no trace is straight?

hazeii(10000) 2 days ago [-]

Don't about the russian software, but for any high-speed PCB design the trace layout matters and 90-degrees bends are to be avoided. On motherboards you'll often see lines that have an extra squiggle or two to equalise the length (and thus delay) because they happened to have a shorter distance to cover.

rcxdude(10000) 2 days ago [-]

Interestingly, I contend that the prevelance of 45 degree angles is in large part due to the capabilities of ECAD software. If you look at old manually-routed PCBs, 45 degree bends are rare. If you try to make a design with non-45 degree bends in most ECAD software, prepare for pain as much of the features of the layout engine fail to work properly (I had experience of this after being inspired by some images of TopoR-designed boards).

jacobush(10000) 2 days ago [-]

DOS based?

nroets(10000) 1 day ago [-]

Take a moment and consider the quest to find the fastest multiplication algorithm. For millennia, we used the classical method. (Shift and add. 3 lines of pseudo code)

Then, along came Fast Fourier Transform based algorithms.

pfdietz(10000) 1 day ago [-]

The Karatsuba algorithm was found before Schönhage-Strassen.

ken(3749) 2 days ago [-]

> When computers design things, they look very different.

Yes, because the computer assumes they're not going to change. The 'tensile structure' looks cool now, but throw it in the back of a truck for 3 months and see if it's still algorithmically perfect. Parts get beat up. Tabs get bent a little. Maybe we'll want to grind off one of those tabs that we're not using because it's in the way, or weld on a new one. With the old structure, I can see which parts are relevant to maintaining the integrity of the system.

It looks like an un-optimized binary, which is very close to its source code. That's a feature. I can reason about the parts on their own.

The 'topologically optimized' one is like an optimized binary. It's great for saving a few grams as long as you never have to change anything. The downside is that it's impossible to reason about. If one tab gets a little bent, that may be harmless, or it may cause the entire structure to lose its strength. You don't know.

Similarly, the truss on the catwalk over the crazy-looking Wendelstein 7-X is made of nice even rectangles and right triangles, and I guarantee that's not because it's a topologically optimal shape for this truss.

TeMPOraL(2647) 2 days ago [-]

There are ways to make such 'organic' structures more resillient. Life is one huge proof of that. One of the reason biologists and medical researchers have so much trouble figuring how anything works in living organisms is because in biology, there are very few clear boundaries; every process is mixed up with a lot of other ones. And yet the final result is incredibly resilient.

d0mine(2870) 2 days ago [-]

I see it more like a rigid grid architecture vs. organically build/evolved structures like in the book 'A Pattern Language: Towns, Buildings, Construction' by Christopher Alexander et al.

Here's some pictures https://medium.com/design-matters-4/a-new-approach-to-design...

xondono(10000) 1 day ago [-]

The other point I find missing is that this kind algorithmic optimization tends to miss solutions that are quite obvious to 'system designers' due to local maxima (evolutionary roadblocks).

For instance, nature has not figure out how to evolve wheels even in flat regions.

madengr(10000) 2 days ago [-]

In our tour at work, they show off a 3D printed, computer optimized part. It looks very similar to that tensile structure; ver organic looking. The customer didn't like the look, and they had to fill in some areas to make it look less alien.

Uhuhreally(10000) 2 days ago [-]

this reminds me of that time blowhard Richard Dawkins disected an animal (can't remember which) and proceeded to bang on about how stupid the vagus nerve is, wandering all over the place as it does for no apparent reason. People used to say the same of the appendix and adenoids; that they're pointless vestiges, the blindness of evolution, contingent structures. Dawkins said obviously the nerve should simply connect directly instead of following its ancient evolutionary program up and down and around the organs.

A classic case of Chesterton's Fence

madengr(10000) 2 days ago [-]

It was a giraffe, for which that nerve just seems out of place.

mtts(4110) 2 days ago [-]

He said it would have been stupid had someone designed it that way but given that it evolved it made perfect sense why it was the way it was.

z3t4(3937) 2 days ago [-]

Are there any interesting theories? I think a lot can be attributed to just randomness. Some time in the evolutionary process we got infected by a bacteria, the mitochondria is now a vital part in our bodys energy system...

friendlybus(10000) 2 days ago [-]

Author mistakes order for purity. The end of excessive purity is the same storyline as aquaman. The dichotomy of chaos and order is still helpful.

cryptozeus(3590) 2 days ago [-]

I think the point he is trying to make is more about "...mistake complexity for chaos, and rush to rearrange it "

dgreensp(10000) 1 day ago [-]

I'm loving these intelligent criticisms of the article.

My own biggest beef with the article is it entirely misses the deep point of Marie Kondo's work, which is about efficiency and joy. If the article is not actually a response to her philosophy and definition of tidiness, and just needed a cute title, well, that's confusing at the very least. The commenters who point out that codebases, for example, need to be reasonable for humans to work in, or that it sure is easier to change source code than a compiled binary, are on the right track.

Kondo's method is all about identifying the clutter _that is actually clutter_, and deleting it. Would you rather maintain a program that's 1000 lines, or 100? If the 1000-line program was written by you, for you, over the course of your life, and by your own admission is extremely messy and moderately unpleasant to deal with, containing lots of unused stuff, yet you run it, and modify it, every day, no one is going to force you to clean it up, but you can find joy in doing so.

si1entstill(10000) 1 day ago [-]

Based on the content, it seems to be primarily a clever title. I don't think the author is trying to refute a hypothetical group of engineers who are trying to apply Marie Kondo's philosophy to software. (Not that there aren't devs who are doing that, but that doesn't seem to be the author's goal.)

lonelappde(10000) about 19 hours ago [-]

Kondo is much more than just throwing away trash. https://www.rd.com/home/cleaning-organizing/marie-kondo-fold...

nickysielicki(3851) 2 days ago [-]

I wrote the following:

> Symmetry underlies almost everything in mathematics and nature.

> It's much more reasonable to assume that our computer programs are not yet good enough to recover that symmetry, than to take the output of current programs as some sort of evidence that asymmetry itself is some sort of ideal.

And then I looked out my window at a tree without leaves, between two buildings, and I'm looking at how unorganized the branches look, and I'm not so sure anymore. It's like the 'spherical chickens' joke.

(Sidenote: it's incredible how angry and volatile this comment section is --- my original comment included. Seems this subject can really strike a personal nerve in many of us. Why?!)

btbuildem(10000) 2 days ago [-]

Unorganized at first glance, right? And yet a tree (on average) is perfectly balanced in terms of weight distribution around the trunk.

Historical Discussions: Tesla's European gigafactory will be built in Berlin (November 12, 2019: 570 points)

(570) Tesla's European gigafactory will be built in Berlin

570 points 7 days ago by etoykan in 10000th position

techcrunch.com | Estimated reading time – 1 minutes | comments | anchor

Elon Musk said Tuesday during an awards ceremony in Germany that Tesla's European gigafactory will be built in the Berlin area.

Musk was on stage to receive a Golden Steering Wheel Award given by BILD.

"There's not enough time tonight to tell all the details," Musk said during an on stage interview with Volkswagen Group CEO Herbert Diess. "But it's in the Berlin area, and it's near the new airport."

Tesla is also going to create an engineering and design center in Berlin because "I think Berlin has some of the best art in the world," Musk said.

Musk took to Twitter after the ceremony and provided a bit more detail, including that this factory will build batteries, powertrains and vehicles, beginning with the Model Y.

Diess thanked Musk while on stage for "pushing us" towards electrification. Diess later said that Musk and Telsa is demonstrating that moving towards electrification works.

"I don't think Germany is that far behind," Musk said when asked about why German automakers were behind in electric vehicles. He later added that some of the best cars in the world are made in Germany.

"Everyone knows that German engineering is outstanding and that's part of the reason we're locating our gigafactory Europe in Germany," Musk said.

All Comments: [-] | anchor

Zooper(10000) 7 days ago [-]

Nice! I hear a bunch of Lithium was just democratized for the freedom market.

woodandsteel(3502) 7 days ago [-]

I don't even understand what you are saying.

sidcool(227) 7 days ago [-]

How in the world does Tesla execute so fast? The Tesla team seems relentless in its ambitions.

grecy(2178) 7 days ago [-]

Checkout this internal memo from Elon just as the Model 3 was going into ramp-up for production:

– Excessive meetings are the blight of big companies and almost always get worse over time. Please get of all large meetings, unless you're certain they are providing value to the whole audience, in which case keep them very short.

– Also get rid of frequent meetings, unless you are dealing with an extremely urgent matter. Meeting frequency should drop rapidly once the urgent matter is resolved.

– Walk out of a meeting or drop off a call as soon as it is obvious you aren't adding value. It is not rude to leave, it is rude to make someone stay and waste their time.

– Don't use acronyms or nonsense words for objects, software or processes at Tesla. In general, anything that requires an explanation inhibits communication. We don't want people to have to memorize a glossary just to function at Tesla.

– Communication should travel via the shortest path necessary to get the job done, not through the "chain of command". Any manager who attempts to enforce chain of command communication will soon find themselves working elsewhere.

– A major source of issues is poor communication between depts. The way to solve this is allow free flow of information between all levels. If, in order to get something done between depts, an individual contributor has to talk to their manager, who talks to a director, who talks to a VP, who talks to another VP, who talks to a director, who talks to a manager, who talks to someone doing the actual work, then super dumb things will happen. It must be ok for people to talk directly and just make the right thing happen.

– In general, always pick common sense as your guide. If following a "company rule" is obviously ridiculous in a particular situation, such that it would make for a great Dilbert cartoon, then the rule should change.

If there is something you think should be done to make Tesla execute better or allow you to look forward to coming to work more (same thing in the long term), please send a note to [redacted]

[1] https://medium.com/@StartupJourney/elon-musks-6-productivity...

ptaipale(3717) 6 days ago [-]

Headline should be corrected: the site is near Berlin, but not in Berlin.

Reported site is Grünheide, some 40 km drive from Brandenburger Tor, and a bit shorter distance from the under-construction BER airport.


What makes me curious is that this village is actually within a nature protection area, Naturschutzgebiet Löcknitztal. I could imagine that building such a symbol of international market economy there might attract anti-capitalist demonstrators and involve protesting and riots. But perhaps I'm wrong.


I also find it somewhat interesting that in the political composition of this village of 8000 residents, the Social Democrats and Linke (Left) have 3 seats, and two seats are held by CDU, AfD, and.... Freiwillige Feuerwehr.

Yes, the voluntary fire brigade is apparently popular enough to feature in local politics. I think this is a somewhat sympathetic phenomenon in German local politics.


oska(470) 6 days ago [-]

Looking at that map I'd assume they're planning on building the factory in the cleared area at the top, just off the Lichtenower Weg, with good close access to Highway 1.

vertex-four(10000) 6 days ago [-]

Watching a SV company suddenly have to deal with German workers' rights (think: unions, significant protection from being fired without cause, rights to paid holiday and enforced maternity leave, even workers' councils which have representation at a management level) is going to be hilarious.

mikojan(10000) 6 days ago [-]

Germany has been spearheading the neoliberal assault on democracy alongside the US and Great Britain as early as the 1980s. In the 2000s we created 'Europe's best low-income sector' (Gerhard Schroeder) and destroyed the pension scheme. This dramatically decreased social security among low- to middle-income workers.

German labor institutions are strategically oriented towards collaboration (much like in the US), they are extremely centralized and anti-democratic and if you're in the workers' council and you're a member of the supervisory board, socio-economically you belong to the upper class. This facilitates the same corporate conspiracies as everywhere else.

Furthermore, East-Germany's labor market as a whole is much more like Eastern European labor markets. It is super-exploited. However, it is right in Western Europe where you've got access to decent infrastructure and massive subsidies.

avar(3290) 6 days ago [-]

You seem to be unaware that Tesla already has a sizable presence in Europe, including a final assembly plant in the Netherlands. Some of the details are different, but I don't see employing a significant amount of Germans being novel for them.

short_sells_poo(10000) 6 days ago [-]

I'm all for some cultural drama, but I'd wager that they have done basic due diligence and they are aware of these factors :)

Krasnol(10000) 6 days ago [-]

Maybe this is part of the reason why he does it.

'Look he made it work in Germany with their 'strict' workers rights. It can't be so bad'

Also going to Germany, right in the face of the traditional car industry is a great show off too.

reitzensteinm(2496) 6 days ago [-]

They already own Grohmann, so I'm guessing they have a fair idea. I remember a flap at the time about Tesla forcing them to ditch their other clients as quickly as possible, which was a bit of a culture shock, but why else would they have bought them in the first place?


roomey(10000) 6 days ago [-]

I get you are joking, but there are two points that I would like to raise:

There are many, many SV companies with large offices in parts of the world with very strong workers rights. I work in one of these. There is no problem, other then I'm not expected to work weekends and be on-call at unreasonable hours. If this is required I would need to be compensated for it.

Secondly; perhaps I am reading into this, but I get the sense you are implying that 'less work' will be done due to these protections. You may be surprised to find out that German factories are actually quite efficient ;) , and that treating workers well gets them to work more, not less. Having a healthcare system that means you can bring your children to A&E at the weekend and not get caught for $1000s means you can come to work on Monday with a clear head.

Having protection from 'right to work' means you can voice an opposing opinion without fear.

Having a paid holiday means you can come back to work refreshed with a clear mind, and not 'burn out' as fast.

Having maternity, and paternity leave, means happier parents (like, in a abstract way, not like in a oh I haven't slept in 8 months I'm soooo happy way). Let's call it out here. Giving children a good start in life sets them up for success.

On a global team you can quite clearly see the results of the different work environments, and the various impacts each environment has. From a tech point of view, I see this most critically in the ability to voice controversial opinions, or push back against your manager for example.

radicalbyte(4118) 7 days ago [-]

German labour laws are very well suited plus there labour pool of qualified people is also very large.

lone_haxx0r(10000) 7 days ago [-]

I would have guessed the opposite. Germany is known for having all sorts of restrictions

arez(10000) 7 days ago [-]

why are german labour laws very well suited? Tesla had several incidents with injured workers and safety violations

IOT_Apprentice(10000) 7 days ago [-]

This is all good news. Just wondering about repairs globally as Tesla's start aging and they need to be repaired, or for whatever the current % of their vehicles require service year over year. They must have that info given the amount of telematics they generate AND that they are sole source for maintenance and repair.

natch(4088) 7 days ago [-]

With no carburetors, alternators, starter motors, fuel pumps, fuel filters, oil filters, oil leaks, gas leaks, exhaust buildup, spark plugs, timing belts, radiators, head gaskets, mufflers, rings, manifolds, belts... and with brake pads that last forever... it's significantly less maintenance.

And Teslas are not designed to milk the consumer for repairs over the years; they are designed to convince consumers that electric cars are a better choice.

The Model 3 is designed to last 1 million miles. So sure, it will be fascinating to see how they do... but that's like 25 or more years from now.

iagooar(4152) 6 days ago [-]

It is great news for Berlin (and Brandenburg) as it will bring (big) money and attract talent. It might even attract people from Poland which is only an hour drive away and has a large pool of very talented and experienced engineers. So far besides maybe Siemens there wasn't too much industry here, which is more present in the south of the country.

Being outside of the city center it won't contribute too much to gentrification and probably it will help Brandenburg getting more people to live there, pay taxes, buy property, shop, etc.

So far reactions have been neutral to good, which by Berlin standards is an amazing result.

I kind of wonder if the decision of bringing the factory to Berlin wasn't based on the amount of software engineering talent that is available, more than the more traditional engineers.

In any case, I'm happy about Tesla coming to Germany, interesting times!

navigatesol(10000) 6 days ago [-]

>It is great news for Berlin (and Brandenburg) as it will bring (big) money and attract talent.

Like the Gigafactory 2 in Buffalo did for New York state?:

But according to SolarCity's financial statements, state officials "quietly issued a series of 10 amendments" to Tesla, allowing the high-tech jobs to become regular positions and requiring just 500 hires within two years -- down from 900. Tesla told Vanity Fair it was still responsible for creating 5,000 jobs, but the timing for the additional jobs has been extended to 10 years after the factory's completion.

SolarCity was the center of the Buffalo Billion corruption probe by federal prosecutors, which last year led to high-profile bribery convictions of several people, including Louis Ciminelli, whose firm won the solar plant construction deal.


lm28469(4222) 6 days ago [-]

> which by Berlin standards is an amazing result.

They only, and rightfully, get pissed when mega corps like google/amazon come in the city center, buy a 40 storey building and starts paying everyone 100% more than the average for the same position.

jillesvangurp(10000) 6 days ago [-]

I think it makes sense from that angle but also from the angle that Germany is rich grounds for poaching people that are currently working for one of the other big car manufacturers. E.g. VW just opened a huge plant for producing their EVs in Zwickau which would about 300km away.

Brandenburg as a state is an interesting. Crucially, it's not the state of Berlin (Berlin is s city state within the German federation). That means access to lots of grants/subsidies and also that they are somewhat shielded from the chaotic governance in Berlin.

blablabla123(10000) 6 days ago [-]

It's definitely positive, although I'm also slightly disappointed. I think the original plan was to build it near Dresden. That would have been quite a signal, but on the other hand it might be good for it to be successful to have it connected with the metropolitan region Berlin/Brandenburg.

mrmonkeyman(10000) 6 days ago [-]

> It might even attract people from Poland

I bet the Berlin people are thrilled.

odiroot(10000) 6 days ago [-]

> So far besides maybe Siemens there wasn't too much industry here, which is more present in the south of the country.

Siemens is big in Niedersachsen (e.g. Braunschweig) so it's more about East/West divide.

romanovcode(10000) 6 days ago [-]

Brandenburg is one of the poorest regions in Germany. Berlin as a city brings negative GDP to country.

I think they can just get it for (relatively) cheap there and that would be the main reason.

lnsru(10000) 6 days ago [-]

It's interesting what kind of talent one needs for a factory. Engineers? I doubt... These will be simple workers, couple equipment operators with some education. All the blueprints are coming from Tesla's HQ anyway, so not much engineering needed. There is also Tesla in Germany that does all the engineering: https://teslagrohmannautomation.de/de/

rf15(10000) 6 days ago [-]

No, this is bad news; it's a smaller version of the economic bomb/distortion of creating a google campus somewhere. This will be bad for everything from housing up to the general health of startup culture in Berlin. Desperate move by the economically starved Brandenburg. (and of course they won't suffer the consequences)

nonsince(3896) 6 days ago [-]

I absolutely do not want more tech companies gentrifying Berlin and continuing the encroachment of foreign corporate Neubauten. The amount of berlin for the working class is shrinking by the day and the culture that draws people to this city is being scrubbed out as a result. When the protests inevitably happen, as they did for the Google campus before it, I'll be standing on the picket line.

propercoil(4109) 7 days ago [-]

Good strategy from Treelon overall. You'll have Tesla producing cars and employing thousands of workers in the U.S, China & Germany so that no one would stifle the company's path towards autonomous EV's.

This will produce pressure on Europe to ease for instance the AI regulations and basically removed Customs tax from many other countries because Tesla can send the cars from three different locations.

I just don't understand why Tesla is slow in expanding to other countries. Here in Israel we don't have Teslas and it's an extremely innovative place with very good conditions for an EV as Israel stretches 424 km (263 mi) from north to south. like two or three charging locations and you're done.

Traster(10000) 7 days ago [-]

How are the temperatures in Israel? I could imagine the summers get too hot for Tesla to guarantee battery performance?

pbreit(2538) 6 days ago [-]

Slow to expand to other countries? I would say they are operating at lightspeed! Think for a moment the colossal effort involved in expanding a car company around the world. And not just a car company, but a direct-to-customer model with a completely new fueling method.

woodandsteel(3502) 7 days ago [-]

>I just don't understand why Tesla is slow in expanding to other countries

Tesla is expanding as fast as they can. After huge struggles they finally got their Fremont plant working right. Then they immediately set to work building a plant in China, and got done remarkably fast.

Now they are going to build a plant in Europe. Probably next will come India, and then maybe Latin America or the Middle East, and there I suppose Israel would be the best location.

jayalpha(4209) 7 days ago [-]

I don't understand your post at all.

'so that no one would stifle the company's path towards autonomous EV's.'

Who is 'stifling'? And many companies work on autonomous cars.

'This will produce pressure on Europe to ease for instance the AI regulations'

What AI regulations in Europe? And what is Europe? The EU?

'and basically removed Customs tax'

This is not how custom taxes work. The EU is one of the biggest free trade zone in the world. If Mercosul joins, it is the biggest.

'I just don't understand why Tesla is slow in expanding to other countries. Here in Israel we don't have Teslas and it's an extremely innovative place'

Because Tesla is burning through cash like a motherfucker. And Israel is a tiny place. This is also why, to my knowledge, nobody build cars in Israel. I mean, why not Dubai?

Tuna-Fish(10000) 6 days ago [-]

> I just don't understand why Tesla is slow in expanding to other countries.

They don't have the capacity. Last quarter, they made as many M3s they have the capacity to make, and sold all of them. Opening new markets just means underserving the current ones more. They will start operations in new countries as they increase capacity and start saturating sales in their current ones.

fyfy18(4182) 7 days ago [-]

Israel is probably just too small of a market at the moment. In most European countries there still isn't a official Tesla store or service center.

I live in the Baltic's and a friend of a freind has a Model X which was involved in a minor accident (other drivers fault) last year. To get it repaired without affecting the warranty it had to be sent to Tesla, which meant it had to be taken to Finland (700km) or Germany (1000km).

oska(470) 7 days ago [-]

> Here in Israel we don't have Teslas

You might have missed this very recent news:

> International car company Tesla published an ad seeking an operations manager in Israel, thus confirming that it plans to launch significant operations here. [1]

[1] https://www.haaretz.com/israel-news/business/tesla-to-launch...

xxpor(4076) 7 days ago [-]

Isn't the issue in Israel that you must have a local importer and dealer?

coldfire(10000) 7 days ago [-]

ROI may be? the market size in Israel is not as big in other countries, so it makes sense to target bigger audience first. Also does Israel have any good local EV manufacturers that can give Tesla any competition?

jsight(4119) 7 days ago [-]

It sounds likey they've been hiring in Israel. Maybe that will change soon.

SEJeff(2123) 7 days ago [-]

I will say that Tel Aviv and Haifa both have excellent engineers. As Tesla grows I'd be a bit surprised if they didn't have engineers in Israel. That said, most of the engineering is still based out of Silicon Valley, so that might be why?

As a tesla fan and owner, I hope you get your superchargers. I've heard rumors that they've just started seriously producing the supercharger v3 hardware at the Buffalo, NY factory (Gigafactory 2), so maybe as that ramps up you'll be good.

Consider tweeting @elonmusk and asking. I did that for slow parts delivery and had the issue resolved in two days.

k__(3307) 7 days ago [-]

Could build it on the old airport.

sneak(3056) 7 days ago [-]


There are a lot of Berliners who love Tegel and don't want it to close even when BBI is eventually open (whenever that may be). TXL is truly a fantastic airport, and would become even better when not overcrowded.

zepearl(4200) 7 days ago [-]


It was originally intended to replace both Schönefeld and Berlin Tegel Airport and become the single commercial airport serving Berlin and the surrounding State of Brandenburg, an area with a combined 6 million inhabitants. However, it is now planned that it will not replace any, with Schönefeld Airport currently being expanded due to rising passenger numbers, and Tegel which will remain open after a referendum.

After almost 15 years of planning, construction began in 2006. Originally planned to open in October 2011, the airport has encountered a series of delays and cost overruns.

Since German reunification, air traffic in Berlin has grown greatly. In 1991, the combined passenger volume of the city's airports was at 7.9 million per year. By 2014, this number had risen to 28 million. When Berlin Brandenburg opens, it will have a capacity of 27 million passengers per year.

Summarized: when/if BER will open, if it will manage to handle immediately its max capacity (best-case scenario), it won't have enough capacity => the other airports will still be needed (until BER is expanded) :|

NGC404(10000) 7 days ago [-]

The number sold electric vehicles in germany is still exponential not saturated, the government recently announced to build additional charging stations across the country in the order of 10 thousands and to increase the (tax) incentives for hybrid and fully electric cars. It is a good time to build a factory for electric cars in germany. Berlin is a sweet spot when it comes to talent in high tech, suppliers and transport connection.

tannhaeuser(2480) 6 days ago [-]

Except Germany has the highest energy prices in Europe, and as long as that doesn't change drastically, e-mobility is for a minority of enthusiasts and people who can afford to be bad at math. Just look at actual numbers of vehicles sold where Diesel still rules. Well, government could raise taxes on mineral oil (already at 63% [1]), but the German state is already drowning in cash and could issue bonds with negative yield. Charging stations are also an area where it goes completely wrong since these are owned and operated by municipalities and monopolist energy providers. Housing also goes wrong especially in Berlin where socialist government doesn't seem able to attract investors and plays ideological games for winning a left-wing electorate instead despite massive influx of 40000 people per year, in addition to energy-efficiency laws and lack of capacity making construction unprofitable (btw. solar industry also goes south).

[1]: https://www.bundesfinanzministerium.de/Content/DE/Standardar...

choeger(10000) 7 days ago [-]

Please drop your inevitable BER jokes below so we can recylce them properly later.

PopeDotNinja(4184) 7 days ago [-]

I loved flying in and out your new airport!

NGC404(10000) 7 days ago [-]

I bet elon will not be able to land in BER for the celebation of the millionth car rolled out there.

wbillingsley(10000) 7 days ago [-]

Get in quick, there's only 49 days left of BER months this year!

ptaipale(3717) 6 days ago [-]

Der Postillon:

Musk: 'We chose Berlin as the location because it has the only CO2-free airport in the world'


robertkrahn01(3884) 7 days ago [-]

Nah, BER will actually be turned into the Tesla factory. This way they can at least stop spending millions just to keep the lights on.

notJim(10000) 7 days ago [-]

Placing it near that airport may be the biggest risk Musk has ever taken.

odiroot(10000) 6 days ago [-]

Hey, there's always Stuttgart 21 and Elbphilharmonie!

jcranmer(10000) 7 days ago [-]

Wait, so if I have this correct, the airport took ~6 years to construct and 9 years and counting to correct problems with its construction? Meaning that it would have been faster to tear the entire building down and rebuild it from scratch than it is taking to remedy the issues?

How does that happen‽

dmix(1368) 7 days ago [-]

For anyone confused like me they mean the airport https://en.m.wikipedia.org/wiki/Berlin_Brandenburg_Airport?w...

jayalpha(4209) 7 days ago [-]

Best place to store the Boeing 737 MAX airplanes.

xky(4159) 6 days ago [-]

This provides a decent and entertaining overview of why the Berlin airport has still not been completed.


xvx(10000) 7 days ago [-]

Well, the Germans ARE world famous for their factories...

Edit: I was definitely referencing the Shoah where Germany and its collaborators systematically murdered six million Jews with brutally inhumane and genocidal efficiency.

pfundstein(10000) 7 days ago [-]

Edit: Ahh my bad, I jumped to the conclusion of this being a concentration camp reference. It was a bit of a stretch admittedly.

dang(179) 6 days ago [-]

Would you please stop posting flamebait to HN? You've done it repeatedly and we ban accounts that do that here.


agumonkey(877) 7 days ago [-]

And their greenness

tnolet(2499) 7 days ago [-]

I'm curious about how the massive left/anti-fa movement in Berlin will react to this one.

They kept out Google after massive protests. Not sure this is the same thing though.


p.s. I'm based in Berlin. Have no horse in the race except that the Google office would have been right down the street of where I work.

zeristor(782) 7 days ago [-]

Anti-fa, isn't that just a term used by fascists? Fas?

Wasn't the Allied War effort in WW2 "Anti-Fa"? Or am I missing something?

Perhaps this could be added to the reminders list for the next Remembrance Day?

watt(10000) 6 days ago [-]

It's curious to be sure, but I think they will react positively to Tesla.

1) Berlin is suffering from pollution, especially the diesel. For city dwellers the switch to electric car can't happen fast enough, and German auto makers have been absolutely dragging their heels. Tesla has become a symbol, a fuck-you to BMW, Mercedes, VW and their gas guzzling, diesel pushing ways. It's unbelievable how out of touch the german automakers have been.

2) The factory means some working-class jobs. The left won't see anything wrong with that. It's the coming gentrification that caused the upset against Google.

3) Factory is just outside Berlin, technically in Brandenburg. Near the (forever in construction) BER airport means less concerns about noise, etc.

erikpukinskis(3077) 7 days ago [-]

Tesla average salary is way below google so their main concern is moot here.

robert_foss(3622) 6 days ago [-]

Being a Berliner, on a personal level it would have been nice for me to have a Google office where they would have been.

But I'm very happy about people standing up for their neighborhood and actually protesting. Having a Google office with >400 decidedly well-compensated people move in would very much change the neighborhood and not for the better.

People fighting for their own local interests is what makes Berlin great, and it is _so_ refreshing after having lived in North America where no-one protests anything and letting corporate interests dominate yours is the norm.

mqus(4202) 7 days ago [-]

As they want to open their factory next to the new Airport, it will basically be in Brandenburg and not in the center of Berlin (not that it would fit there). They probably will change the neighbourhood there, but automobile engineers are perceived as worker class while google techies generally are not. I also think that Google would have had no problems If they would have opened their office in Adlershof.

So to summarize:


- builds factory in a practically greenfield area

- employs middle-class workers


- buys old building in the middle of the 'hip' leftist city district

- employs high-wage workers which will drive gentrification

So I think this will be received at least with mixed voices, but mostly positively.

jakobmi(10000) 7 days ago [-]

As a reaction, Google doubled down on the Munich office (+1500 employees) and Zurich, Switzerland (+4500 employees).

pjc50(1404) 7 days ago [-]

Tesla isn't particularly political and there's no Fascist angle?

troymc(4220) 7 days ago [-]

Google has offices in Berlin, just not in Kreuzberg.

trianglem(10000) 6 days ago [-]

I think tesla is going to be fine because it promises massive environmental improvements.

trianglem(10000) 6 days ago [-]

Isn't anti fascist just normal?

DanielleMolloy(2578) 7 days ago [-]

This was a pure territorial war. I've seen the German campaign material, there was no sign of good reasons, at most a lot of misunderstanding of the idea of a startup incubator. I remember how I had to explain the idea to a very vocal campaigner who clearly did not get that Google was not planning to open a big dev center in their kiez after Google gave up. The underlying reason was the fear of rents rising in Kreuzberg.

They didn't protest against the new Spandau Siemens campus either. Siemens being in the arms industry, while Googlers openly protested against any military-related research during the same year. It's just that no nobody wants to live in Spandau.

The area around BER is even further away, and the close-by Adlershof is pretty much Berlins center for STEM research (many suppliers are there too). Could be a good move.

abstractbarista(10000) 6 days ago [-]

Seeing a lot of comments saying their choice is good because it won't contribute to gentrification.

Doesn't sound right to me. Why is that the first~third thing you're mentioning? It's almost like we scorn the concept of more people working in an area towards a positive goal (renewable transport). Yikes.

Nasrudith(10000) 6 days ago [-]

I think it highlights an issue with housing in general - if you judge by actions instead of words the logical but uncomfortable conclusion is that /everyone/, poor, middle class, or rich is horribly selfish and unwilling to agree to any cost which doesn't benefit them directly. From rent control to property values everyone has a 'screw everyone else I have mine' attitude.

I actually start to feel sorry for politicians for a change when they have people who want mathematical impossibilities.

jansan(4151) 6 days ago [-]

Near Berlin means the state of Brandenburg, which is famous for failed mega projects:

- Airport BER (still not open)

- Cargolifter airship factory (now used as a waterpark)

- Lausitzring (was supposed to host the F1, now hugely unprofitable)

- Semiconductor factory in Frankfurt/Oder (abandoned, only the building got finished)

I am not aware of one single project that materialized as planned. The odds are against Elon on this one...

Kutta(3947) 6 days ago [-]

You can always put a project into a reference class consisting only of failed projects. You can also put this project into the class of Tesla factories, none of which failed so far.

mellosouls(10000) 6 days ago [-]

The Brexiteers scoffing at 'Project Fear' and crowing about the uppity Europeans begging us for our (UK) car business have been strangely quiet on this one.

agent008t(10000) 6 days ago [-]

To be fair:

1. Brexit hasn't happened yet. I.e. UK has not been able to introduce policies that would make it a lot more competitive and friendly for business than the EU/Germany, make it friendlier for international talent, which is one of the opportunities created by Brexit.

2. We do not know what would have happened had Parliament and civil service been more positive and cooperative on Brexit. It could be a self-fulfilling prophecy - remainers in Parliament inadvertently creating greater uncertainty, which in turn drives business away as the leavers are unable to realistically guarantee a more competitive / friendlier for business Britain in the future.

mdorazio(10000) 7 days ago [-]

They obviously needed to get one built in Europe sooner rather than later to boost marketshare there and stay ahead of demand without needing to ship from China or the US, but why Berlin? Does anyone know if they got subsidies out of it?

detaro(2090) 7 days ago [-]

Could be partially political. Close to federal power, far-ish (in a different state than) from other car companies. No idea if there's supplier or other partner concerns that align (some R&D stuff in Berlin maybe, if they plan on doing anything like that there? I know they work with some Berlin companies, but that doesn't seem so important). Would expect Brandenburg to give some kind of subsidy, but other states would too.

dzhiurgis(4126) 7 days ago [-]

East Germany still has lower wages.

jariel(10000) 7 days ago [-]

Without knowing the backroom deals - Berlin is a nice choice for things because it's like 'East Europe wages' with 'West Europe capabilities'.

Germany is a very stable robust economy that might be their #1 'export source' - there's a lot of related knowledge - and in a 'cramped Europe' - the Berlin area is wide open.

FYI Berlin has 'wide avenue streets' which are totally unlike most other European places, sometimes it feels like Los Angeles and not Europe.

It might have been possible to go to Czech/Poland but there are just a large number of 'little things' that can go wrong over there, issues best left to those who know and operate the market well.

Compound this with the probable politics of the game and it makes sense.

In a way it's the most obvious choice.

odiroot(10000) 6 days ago [-]

Cheap real estate. Cheap workforce from Brandenburg and possibly Sachsen.

On top of that Brandenburg government will definitely sprinkle some tax breaks.

jakobmi(10000) 7 days ago [-]

I think Berlin is a great choice, because it excels at the metrics that matter most in constructing a new gigafactory. I believe the most critical metrics are: - cost of construction, including cost of land: extremely low in Berlin - cost of labour/wages: extremely low in Berlin - standard of living: very high, healthy - reachability: near Airport, highways, railroad - location within Europe: Germany/Berlin is perfectly in the middle of Europe with great railroad/truck connections to all other - regulation/friendliness to car industry: exceptional in Germany, as auto industry is the world's biggest - talent: world-class in Germany - law and order/rule of law: one of the best in the world

webkike(4020) 7 days ago [-]

I think it makes sense. Berlin is very spacious and connected to a lot of cities via rail.

doener(50) 7 days ago [-]
zwieback(1176) 7 days ago [-]

You were cheated, Mr Kebab, upvoted yours.

Koremat6666(10000) 7 days ago [-]

Why should a GIGA factory be located near the airport and in a city ? A factory meant to mass produce something should be somewhere the land is cheap should not it be ?

yorwba(3662) 7 days ago [-]

The area just south of Berlin (it's not in the city) is suitable for the gigafactory for the same reason it was chosen as the site for the new airport: there's not much else there, so you can build big.

Of course that applies to many other places in Brandenburg with similar distance to Berlin, but the airport means that transportation to and from the city has been (or will be) built out for higher capacity.

TomMarius(10000) 7 days ago [-]

Why not both when you can have it? Clearly there is some just right place in Berlin

danhak(3743) 7 days ago [-]

The area immediately around major airports is usually not terribly desirable. I don't know about Berlin, specifically.

Also it has to be close enough to the city to make commuting from the population center reasonable, no?

AngryData(10000) 7 days ago [-]

If it is directly under the flight path the noise would make it less desirable and lower costs while the easy access to labor could possibly make up for the rest of the land costs.

_ph_(10000) 7 days ago [-]

Berlin is in a special situation due to the history. As only west Berlin belonged to west Germany and east Berlin and all the surrounding areas were part of the DDR, there is still a steep fall-off in industrialization as soon as you leave the direct Berlin area. Also in the DDR the industrial centers were in other cities than Berlin. So there should be plenty of cheap real estate not too far from Berlin.

Also in Bavaria, once you leave Munich (BMW), Augsburg or Ingoldstadt (Audi), there is quite a steep drop-off in density. The traditional high-density industrial regions of Germany are rather in the west and south-west.

pbreit(2538) 6 days ago [-]

As the poster above you wrote: 'East Europe wages' with 'West Europe capabilities'

That is about as good as it gets.

They probably got the land cheap and commitment to build at another record breaking pace.

hinkley(4121) 7 days ago [-]

supply chain, labor?

I mean we used to build factories on the river for that reason. Made a royal mess, but easy to ship things in and out.

ravedave5(10000) 7 days ago [-]

Airports are normally located where land is cheap (example - 'Denver' air port)

Historical Discussions: I Created a $60K/Month App That Collects In-Person Payments Through Stripe (November 15, 2019: 531 points)

(532) I Created a $60K/Month App That Collects In-Person Payments Through Stripe

532 points 4 days ago by patwalls in 1574th position

www.starterstory.com | Estimated reading time – 12 minutes | comments | anchor

Hello! Who are you and what business did you start?

I'm Ryan Scherf, the founder of payment.co (@payment), an app built on top of the Stripe payment gateway for creating card-present charges. Payment is available on iOS and Android, and allows customers who have created online stores to easily collect payments in person, in the same account, with no barriers or card readers required.

The app launched in January 2015, and since has grown to process over $70M in volume annually. The app collects a 1% service fee on every single charge.

What's your backstory and how did you come up with the idea?

I went to school for Computer Science and was gated into being a developer. I mostly hated everything about development environments at the time (Java was the most popular at the time), and thus transitioned into a career of design. After working at several healthcare startups (these are abundant in Minneapolis, due to Unitedhealth Group being headquartered in the area), I quickly realized that working for people wasn't a long term career path that I was willing to commit to.

In 2013, I was approached by someone I had worked with previously on an invoicing app, and he had an idea for building an analytics app on top of a company in its infancy: Stripe. This was pre-huge valuations and funding rounds.

At the time, I knew nothing about Stripe, nor payment gateways. I had 4 & 2-year-old boys and had just built a new house in the suburbs.

One of my biggest regrets is not focusing on my business earlier. We were making money, and I always treated it as just a hobby business.

Our first app, Paid, launched in late 2013 and was the only analytics app for Stripe revenue. I won't get into too many specifics about what happened afterward, but I did an interview with Mixergy in 2015 and explained the process.

We were seeing significant traction, but we were unable to come up with a great way to monetize. As a designer and iOS dev combination, our one requirement for building apps is that we didn't want to manage a backend server. Especially one calculating resource-intensive analytics data.

Therefore, we decided to launch Payment; an app for collecting card-present payments. This was a gaping hole in the Stripe offering and didn't require much validation because it was so obvious that it was missing. Stripe offered the ability to create direct charges via their API, however, they did not have an app. And due to the simplicity of their API, this feature would have been incredibly easy to include in their Dashboard app, yet they deliberately chose to leave it out.

Take us through the process of designing, prototyping, and manufacturing your first product.

Having worked with my partner on several projects and companies in the past, our working relationship was very natural. I focused on product design, and he focused on development.

The first version of Payment was designed, built and tested over a span of 2 months. Development began in November 2014 and launched in January 2015.

The only thing we needed the MVP to do was to allow for someone to create a charge which enabled us to collect a fee. This simple functionality wasn't possible via an app (built on Stripe) at this time. No card readers, nothing fancy. Just processing a manual input credit card number, similar to what you'd do in an eCommerce experience.

Describe the process of launching the business.

We didn't have any special tactics or tricks we used to launch. There wasn't any hype, and nobody saw us coming. We chose the "build it and they will come" route because the hole was so immense, and the project was so small. We only focused on iOS to begin as the search results when searching for "Stripe" in the App Store had nothing to do with finance. We figured it would be relatively easy to own that keyword, and we did.

From our previous app (Paid), we did have a user list of several thousand account emails that might be interested in charging via an app, so we sent campaigns to them.

In our first 15 days after launch, we made $184 from fees. I can remember how incredibly exciting this was, and how all of a sudden this app idea felt validated. Our previous app (Paid) relied on IAP's. This app was different. All we needed to do was get more people using.

Because we bootstrapped everything and worked in our spare time, we had no costs, only time. We both had families, stable finances and an interest in working for ourselves. We let the app sit while we worked on some features, but essentially, it was all organic growth for 2015. We ended up making $38,000 that year, split 50/50.

Since launch, what has worked to attract and retain customers?

I've found that the best way to keep customers is by building what they want. That sounds cliche, but I've been super diligent in figuring out exactly what to build. I don't keep a product roadmap, nor do I have an aspirational product I want to get to. My customers are people that run real businesses. And they run many types of businesses: from private jets to selling surfboards to donation collection at a church. There is no one size fits all for the product. The only exception to that is they want to be able to accept credit cards, in a variety of ways, very quickly. By focusing on just that user experience, we've been able to keep the app small and nimble, and dead simple to get started.

You may be wondering how this is possible? Well, I do all of the customer support. I don't have a support team, nor do I have automation for answering the requests. I've added a few pieces to the app for the very common questions (like Where is my money?!), but other than that, I personally respond to every single support request. This allows me to keep my pulse on what's working and what isn't, and get ahead of the features that are being requested often.

In terms of growth, we've mostly focused on app stores (Apple and Google Play) to distribute and optimize the app. We're in the process of launching an initiative to start funneling leads from the web as well but haven't traditionally focused here.

We position the app strongly as a preferred, verified Stripe Partner. In fact, our partnership with the Stripe team is deep enough that their account managers and support teams will refer their users to our apps to collect payments. Since we were first to the market, we mostly saw the competitors copy our descriptions verbatim. The most important term we can rank for at the moment is "stripe", so we try to pack that term as much as possible (even though some competitors did it more).

Since our app relies so heavily on Stripe accounts, we essentially buy paid ads for the keyword "stripe" on all of the relevant stores. Interestingly, we also rank #1 organically for the keyword "payment", which is ahead of some really popular services like Venmo, Google Pay, PayPal, Zelle, etc. Although this gives us some downloads, most abandon as they have no idea what a Stripe account is or why they would need it.

How are you doing today and what does the future look like?

In 2018, my partner and I parted ways as he was looking to focus on other things, and I was looking to go all-in on Payment. The app has been profitable since day 1, so there has never really been cash flow problems. At the time of writing this, the app is processing $6M per month in volume, which equates to $60,000 per month in revenue. Roughly 80% of that is profit after advertising, hosting, card reader inventory and other miscellaneous expenses. Charges are made all over the world.

I'm still the only true employee, though I have several contractors that assist in various capacities, to the tune of about 1 day per week. I still do the design, most of the smaller development, marketing, and business development.

I've set a goal to be processing $10M/month in Stripe volume by Spring, which is a pretty big stretch goal. There's product-market fit, a great organic and retention strategy, so focusing on growth seemed like the proper next step. Roughly speaking, the CMGR equates to 25% YoY growth.

Through starting the business, have you learned anything particularly helpful or advantageous?

One of my biggest regrets is not focusing on my business earlier. We were making money, and I always treated it as just a hobby business. My partner also did the same. Had we taken 6 months and just focused on building it out, we could have drastically accelerated the path to where we are now. In addition, I would have parted ways with my unengaged partner sooner. It wasn't his fault, we just wanted different things out of the business. When we finally realized it, it was too late.

In addition, I'm not so naive that I don't realize that there was a serious amount of luck involved with our success. As mentioned previously, Stripe was in its infancy, and we latched on and rode the wave for 5+ years.

What platform/tools do you use for your business?

The tools I check every day ironically have nothing to do with how much money I'm making. They're all development and customer-focused:

What have been the most influential books, podcasts, or other resources?

I've never been a huge fan of the "this is how I did it" books. I don't think that formula is repeatable for most situations. For instance, the just build it and ship it mentality is something you won't find in any book -- as it doesn't discuss testing, finding product-market fit, etc.

I do however believe in the lean startup, and taking that even further, bootstrapping. People like Ruben Gamez and Rob Walling have been friends of mine for years, and they've launched several successful bootstrapped startups. Rob's podcast Startups for the rest of us is especially great.

Advice for other entrepreneurs who want to get started or are just starting out?

I think the best thing anyone can do is to learn how to build it themselves. With the abundance of frameworks, it's easier than ever to get something scrappy up and running. Don't wait until you save up $10k to hire an engineer. It will never be enough money if you can't do some things on your own.

In addition, there isn't any perfect timing. You don't need to wait to quit your job, as something else will naturally come up. Jumping in and shipping something is the only way to get yourself motivated to free yourself from the constraints of working for people.

Are you looking to hire for certain positions right now?

I'm in the process of looking for someone to assist with growth. I've realized over the last few years that I can't do everything, even though I want to, so I've been trying to identify my weak spots and find consultants for that.

This person would handle iOS and Android store campaigns, SEO, Adwords and any other ways they think they could grow the user base while targeting Stripe users.

Where can we go to learn more?

Want to start your own business?

Hey! 👋I'm Pat Walls, the founder of Starter Story.

We interview successful business owners and share the stories behind their business. By sharing these stories, we want to help others get started.

If you liked this story, join our mailing list for new interviews every Tuesday.

Interested in sharing your own story? Find out how!


Are you ready to boost your revenue?

Using Klaviyo will open up a massive, untapped sales channel and bring you closer to your customers!

We've interviewed many impressive businesses who swear by the results of the product, including Brumate, Beardbrand, and many more.

Level up your email marketing with Klaviyo!

All Comments: [-] | anchor

NoblePublius(10000) 4 days ago [-]

What does this do that Square doesn't?

tempestn(1477) 3 days ago [-]

It's convenient for businesses that already use Stripe.

Scoundreller(4218) 4 days ago [-]

Diff their ToSs and then focus on whatever equals (Stripe minus Square).

Or maybe he ranks #1 in Google for 2% of the search terms that relate to processing credit cards.

jagira(2951) 4 days ago [-]

Why does USA not have a system like UPI (https://en.wikipedia.org/wiki/Unified_Payments_Interface)? It simpler, easier and cheaper (mostly free).

krtkush(4024) 4 days ago [-]

UPI must be the single greatest thing that have come out of India in recent years; digital payments made absolutely simple.

paulddraper(4026) 3 days ago [-]

Most countries, let alone the USA, do not have a system like UPI.

That UPI was agreed upon by enough people was a happy accident for India.

homero(4084) 3 days ago [-]

I haven't been able to connect to https://payment.co/ yesterday or today

rscherf(4221) 3 days ago [-]

Looks like the redirect is broken. Try https://paymentforstripe.com

alibaba_x(10000) 4 days ago [-]

How can such an app compete against Stripe's very own Terminal? Don't this guy's customers know that an official option exists?


giarc(10000) 3 days ago [-]

That's just one more thing to carry. If you have the option of just using your phone, what are you going to pick?

GhostVII(10000) 4 days ago [-]

I think terminal is largely targeted at bigger businesses, where as this product is more like Square, targeted at small independent businesses and shops.

muzani(3698) 3 days ago [-]

Honestly, it sounds like the OP is doing a great job at sales, and he's just making money as an underpaid affiliate.

kohanz(3925) 4 days ago [-]

At first glance, Terminal looks like it still needs some level of integration and development work. It's not a ready-to-go, install-the-app-and-process-payments solution.

Dunedan(10000) 3 days ago [-]

> I've set a goal to be processing $10M/month in Stripe volume by Spring, which is a pretty big stretch goal.

As the app is already highly profitable, why suddenly such an ambitious goal?

rscherf(4221) 3 days ago [-]

Great question. It's not about the money, but more about how I was passive for too long just letting it be. I've waited to invest in several things for growth until I knew had the market cornered. I'm not too concerned about adding more features now, so the only next logical step is growth.

Growing from $6M to $10M didn't seem huge at the time, but now it does...

prostanac(10000) 3 days ago [-]

How do they handle physical receipts? In my country you are required to provide a paper receipt for every transaction.

rscherf(4221) 3 days ago [-]

Not required in the US. Email receipts are built into Stripe if you provide emails.

unnouinceput(10000) 3 days ago [-]

In our country you can also just get the transaction itself printed on your own and use that as a receipt for tax man. Sanatate!

plinkplonk(721) 3 days ago [-]

Hey Ryan,

In the interview you say that you transitioned from dev to design.

How did you go about doing this? Any advice for someone planning a similar shift?

rscherf(4221) 3 days ago [-]

I always had an interest in design, designing a lot of websites before and during college. My first job out of college was a Java Support Dev, for a health care company, which was soul-sucking. Troubleshooting production bugs and fixing them was not something I wanted to do.

I volunteered to build the interface of a product we were working on and never looked back.

homero(4084) 3 days ago [-]

How'd you manufacture a card reader?

rscherf(4221) 3 days ago [-]

Didn't. First integrated with Magtek (insanely painful). Then I wished that another company would make it simpler. Along cane Stripe's Terminal and here we are.

heyalexej(3782) 4 days ago [-]

Slightly off-topic and not doubting this particular story. Posts from this blog get submitted every few minutes it seems. Is anyone checking and verifying the numbers these companies and 'startups' claim to make? If so, how?

rscherf(4221) 3 days ago [-]

I'm the founder, and although I can't speak for the other posts, I can assure you the numbers are real. Several posts in my personal and business Twitter feeds can confirm.

jimnotgym(3758) 3 days ago [-]

Does this have a market outside of the US, without chip&pin card readers? You are taught in the UK never to take face to face payments without chip & pin or chargebacks are indefensible.

rscherf(4221) 3 days ago [-]

We have an integration with Payworks for non-US, but mostly hoping Terminal rolls out globally soon.

But, no card reader is required.

echelon(4041) 4 days ago [-]

Isn't this just Square, but with cheaper processing fees (since Stripe is eating the bill)?

PunksATawnyFill(10000) 4 days ago [-]

That's funny: I read 'Stripe,' but was thinking of Square the whole time and asking, WTF, how did this functionality not exist? Wasn't that the whole point?

Now I think how the hell did Stripe even work without being able to accept cards? What's the point of it, then?

gruez(3724) 4 days ago [-]

>but with cheaper processing fees

What? The product's page says it charges 1% + whatever Stripe charges ('generally' 2.9% + $0.3). A quick search says that Square charges 2.6% + $0.10 per in-person transaction. Seems like that this product is more expensive than Square.

ratsimihah(4005) 3 days ago [-]

> For instance, the just build it and ship it mentality is something you won't find in any book -- as it doesn't discuss testing, finding product-market fit, etc.

I think that's essentially what This is marketing by Seth Godin covers.

K0SM0S(10000) 3 days ago [-]

Iirc also Lean Startup by Eric Ries —this is basically part of that— as well as The War of Art by Steven Pressfield (to some degree). It's also generally the motivation behind agile from a management standpoint.

Definitely tons of literature covering this approach.

jliptzin(4020) 4 days ago [-]

Couldn't stripe just add this feature in their mobile app? Why haven't they done so?

scottmcf(10000) 4 days ago [-]

That would be my concern for this business. It'll disappear instantly when they do (and it's got to be a when, right?) but he's getting good value for moving quickly. Or maybe they get acquired.

ianai(4218) 4 days ago [-]

After the news about pornhub and PayPal yesterday, any chances you'd add their use case?

rscherf(4221) 4 days ago [-]


giarc(10000) 3 days ago [-]

OPs app appears to be for physical, in person transactions. Pornhubs problem was online transactions.

claviska(4042) 4 days ago [-]

I believe that would be classified as a restricted business and disallowed by Stripe's TOS. [1]

1. https://stripe.com/restricted-businesses

ttraub(10000) 4 days ago [-]

He's getting 80% of the gross = 48,000/month. That's impressive income, half a million a year for what amounts to a good idea implemented at the right time. I love reading stories like this, and it demonstrates that there are still many opportunities out there. I don't understand some of these comments which seem to resent his success or belittle his company as 'not a unicorn'. That's good money, he works for himself, not beholden to investors or creditors, and he's raising a young family in an affordable region. Congratulations to him for a job well done.

rscherf(4221) 4 days ago [-]

Thank you! I'll be first to admit it was perfect timing (which wasn't planned at all). The freedom is what I was looking for; if nobody buys it, I'll happy just keep running it. If Stripe launches the same features, I'll figure out how to pivot. Or, build something completely new. Who knows what the future holds.

cryptica(10000) 3 days ago [-]

It seems that almost all the successful startup ideas these days involve integrating with some major platform and reinforcing their dominance whilst being at their mercy.

wallace_f(1030) 4 days ago [-]

>I love reading stories like this, and it demonstrates that there are still many opportunities out there.

It's a white male taking advantage of capitalism. He has no regulatory oversight and no degree or license pertaining to this work.

dvt(1708) 4 days ago [-]

It's interesting how a common tenet of startups is 'build products, not features' -- and yet here's a very clear example of a 'feature' that can be very successful.

xwdv(10000) 4 days ago [-]

It's still good advice, if you simply tell people "build whatever you want" they are more likely to fail and usually it's because of having a featureable business.

phodo(921) 4 days ago [-]

Wrong. Solve a problem... via a product or a feature.

wpietri(3351) 3 days ago [-]

How is 'mobile point of sale app' not a product?

Apocryphon(2794) 4 days ago [-]

It's probably to get away with building a successful feature startup when it's just a hobby business.

dimino(10000) 4 days ago [-]

It seems like startup advice is based on the repeatable observations that people who've worked with a lot of startups have discovered. That means there are a bunch of situations that don't get covered because they don't come up a lot.

When playing an odds game like startups, you want to do all the maximally viable things possible, but that doesn't mean other decisions aren't also viable, they're just not observed to be as successful, so they're not mentioned. Doesn't mean they won't work.

There's also opportunity cost. What could this team have done if they were working on something more... disruptive? The execution here is incredible, applying that to a new problem would have probably also done very well.

filleokus(3718) 4 days ago [-]

Well, isn't that advice also mostly given to founders aiming to create a potential ≈unicorn and not a lifestyle business like this?

Don't get me wrong, what this guy has done is impressive. Growing a company while staying profitable all the way, and being able to support himself is great. I would switch places with him (ex post facto) in a heart beat.

But still, this company will probably never be bought by anyone (besides maybe Stripe), and could very well exist for many years to come with 1-2 employees including the founder.

(526) Sweden Drops Julian Assange Rape Investigation

526 points about 4 hours ago by schappim in 1630th position

www.bbc.com | Estimated reading time – 3 minutes | comments | anchor

Image copyright Getty Images
Image caption Assange was arrested in London after Ecuador abruptly withdrew its protection in April

Prosecutors in Sweden have dropped an investigation into a rape allegation made against Wikileaks co-founder Julian Assange in 2010.

Assange, who denies the accusation, has avoided extradition to Sweden for seven years after seeking refuge at the Ecuadorean embassy in London in 2012.

The 48-year-old Australian was evicted in April and sentenced to 50 weeks in jail for breaching his bail conditions.

He is currently being held at Belmarsh prison in London.

The Swedish investigation had been shelved in 2017 but was re-opened earlier this year following his eviction from the embassy.

What did the prosecutors say?

Deputy Director of Public Prosecution Eva-Marie Persson took the decision to 'discontinue the investigation regarding Julian Assange', the Swedish Prosecution Authority said.

'The reason for this decision is that the evidence has weakened considerably due to the long period of time that has elapsed since the events in question,' it added.

Image copyright EPA
Image caption Eva-Marie Persson said the decision had been taken after interviews with seven witnesses

Ms Persson said: 'I would like to emphasise that the injured party has submitted a credible and reliable version of events.

'Her statements have been coherent, extensive and detailed; however, my overall assessment is that the evidential situation has been weakened to such an extent that that there is no longer any reason to continue the investigation.'

Media playback is unsupported on your device

Media captionJulian Assange being dragged from the Ecuadorean embassy in London

Separately, the prosecutors held a news briefing in the Swedish capital Stockholm, saying that the decision to drop the inquiry had been taken after interviews with seven witnesses in the case.

What was the Swedish investigation about?

Assange was accused of rape by a woman and sexual assault by another one following a Wikileaks conference in Stockholm in 2010. He has always denied the allegations, saying the sex was consensual.

Media playback is unsupported on your device

Media captionWho is Julian Assange?

He also faced investigations for molestation and unlawful coercion, but these cases were dropped in 2015 because time had run out.

What charges does Assange face in the US?

The US is seeking Assange's extradition from the UK over his alleged role in the release of classified military and diplomatic material by Wikileaks in 2010.

Australian-born Assange faces a charge of conspiracy to commit computer intrusion in the US. He is accused of participating in one of the largest ever leaks of government secrets, which could result in a prison term of up to five years.

In June, the then UK Home Secretary, Sajid Javid, formally approved an extradition request from the US.

All Comments: [-] | anchor

criley2(10000) about 3 hours ago [-]

Sad that powerful people can avoid regular people justice by complaining loudly enough and throwing big enough fits. The Donald Trump model of avoiding prosecution: make it too painful to follow through with the rule of law.

ceejayoz(2116) about 3 hours ago [-]

Assange isn't a great example of someone escaping justice (or 'the arm of the law', if you'd prefer).

He spent years in a prison-like environment, and is now in prison for skipping out on bail - he's been in jail-like conditions far longer than the original crime would've incurred. He's likely to wind up in US hands later on, too.

Slashbot(10000) about 3 hours ago [-]

Hitlery Criminalton still isn't behind fucking bars.. along with scumbama and the rest of demonrats... guess knocking people off works or when you work for the biggest cesspit for kikes and liars in mainstream media... you get away with murder and all crimes are covered up... they should all be behind bars or better yet dead ya little bitch.

Trump is a saint in comparison to all that fucking filthy so called elite trash.

auslander(3694) about 3 hours ago [-]

So sick seeing how UK judges listen to US.

u10(10000) about 3 hours ago [-]


They were responding to a valid extradition request by Sweden. How does the US factor in here?

turtlesdown11(10000) about 3 hours ago [-]

He is an odious figure who has been improperly deified. I'm not sure why folks love his self-aggrandizing, reckless behavior. The guy helped Russia destabilize the globe, resulting in our current geopolitical situation, for his own selfish goals rather than any other.

Don't get me wrong, I appreciate released information exposing bad behavior by governments (Snowden, etc). Just not this guys selfishness and cooperation with Russia.

proffeature8(10000) about 3 hours ago [-]

I agree with what you say and still support him. This has nothing to do with his personality and everything to do with the fact that a free press is more important than any one of the particular things you stated.

DiogenesKynikos(10000) about 3 hours ago [-]

A lot of people liked Assange back when he had only yet published information about corrupt politicians in the Third World. Some people abandoned Assange when he published information about US war crimes in Iraq and Afghanistan. Some people abandoned him when he published information about the back-room dealing of US diplomacy. Some people abandoned him when he published embarrassing information about a Democratic Presidential candidate.

As for destabilizing the globe: some would say the people Assange has exposed are the ones destabilizing the world.

rcMgD2BwE72F(4209) about 2 hours ago [-]

>The guy helped Russia destabilize the globe, resulting in our current geopolitical situation

Who still thinks Russia is most responsible for destabilizing the globe? Did you miss some history classes or haven't your heard yet of the USA?

geofft(3275) about 3 hours ago [-]

By the way, Chelsea Manning (the actual leaker) has been in jail for just over 6 months - again - for refusing to testify in the Assange extradition case.

If you're so inclined, send something to her legal defense fund: https://actionnetwork.org/fundraising/chelsea-manning-needs-...

Bostonian(3311) about 3 hours ago [-]

Chelsea Manning was sentenced to 35 years, but her sentence was commuted by Obama in part because she was transgender. She deserves to be in prison.

https://www.cnn.com/2017/01/17/politics/chelsea-manning-sent... President Barack Obama on Tuesday overruled his secretary of defense to commute the sentence of former Army soldier Chelsea Manning, who was convicted of stealing and disseminating 750,000 pages of documents and videos to WikiLeaks.

The decision -- which a senior defense official told CNN was made over the objections of Secretary of Defense Ash Carter -- immediately touched off a controversy in the closing days of the Obama administration. A former intelligence official described being 'shocked' to learn of Obama's decision, adding that the 'entire intelligence community is deflated by this inexplicable use of executive power.' The official said the move was 'deeply hypocritical given Obama's denunciation of WikiLeaks' role in the hacking of the (Democratic National Committee).'

k1m(2250) about 3 hours ago [-]

Worth mentioning that Nils Melzer, the UN Special Rapporteur on Torture, visited Assange and described his treatment as psychological torture:

> "It was obvious that Mr. Assange's health has been seriously affected by the extremely hostile and arbitrary environment he has been exposed to for many years," the expert said. "Most importantly, in addition to physical ailments, Mr. Assange showed all symptoms typical for prolonged exposure to psychological torture, including extreme stress, chronic anxiety and intense psychological trauma.

> "The evidence is overwhelming and clear," the expert said. "Mr. Assange has been deliberately exposed, for a period of several years, to progressively severe forms of cruel, inhuman or degrading treatment or punishment, the cumulative effects of which can only be described as psychological torture.


This article by the UN expert is also well worth reading:

Demasking the Torture of Julian Assange - https://medium.com/@njmelzer/demasking-the-torture-of-julian...

burtonator(2084) about 2 hours ago [-]

> typical for prolonged exposure to psychological torture, including extreme stress, chronic anxiety and intense psychological trauma.

Due to recent imprisonment or is this due to living for so long in a tiny 'dorm room' in the Ecuadorian embassy?

No joke I think that could easily drive someone insane given enough time.

jpz(10000) about 2 hours ago [-]

He confined himself to a room. He skipped bail and was on the lam.

Ridiculous for the UN to qualify this as the UK state torturing him - it denigrates the reputation of the UN.

goerz(3655) about 3 hours ago [-]

> Mr. Assange's health has been seriously affected by the extremely hostile and arbitrary environment he has been exposed to for many years

Let me correct that: Mr Assange's health has been seriously affected by the environment that he exposed himself to

(Edit: I'm talking about the embassy episode here, not his more recent incarceration)

t0ddbonzalez(10000) about 1 hour ago [-]

Nope, sorry. He could have left the embassy any time he wanted. Any 'torture' is purely self-inflicted.

pjc50(1404) about 3 hours ago [-]

It's not torture if you can leave at any time. He wasn't being held in the embassy, he was hiding there.

duxup(3967) about 3 hours ago [-]

I find the idea of the threat of prosecution in a free country as a form of 'torture' a bit questionable.

It's something, but i'm not sure I agree with calling that torture.

ckastner(4176) about 2 hours ago [-]

You quote:

> "Mr. Assange has been deliberately exposed, for a period of several years, to progressively severe forms of cruel, inhuman or degrading treatment or punishment, the cumulative effects of which can only be described as psychological torture."

That quote lacked information on who was exposing him and what that exposure actually was, so I followed your source. That source elaborates as follows:

> "In the course of the past nine years, Mr. Assange has been exposed to persistent, progressively severe abuse ranging from systematic judicial persecution and arbitrary confinement in the Ecuadorian embassy, to his oppressive isolation, harassment and surveillance inside the embassy, and from deliberate collective ridicule, insults and humiliation, to open instigation of violence and even repeated calls for his assassination."

Apart from the last item, that all sounds either like hyberbole ('systematic judicial persecution' -- unless he is above the justice system, that's the system at work), or self-inflicted.

shadowgovt(10000) about 2 hours ago [-]

I would believe he's exhibiting symptoms correlated with torture victims, but most torture victims don't choose to stay in the torture chamber voluntarily. Assange sought and was granted asylum and could have ended the torturous circumstances at any time by rendering himself to UK custody.

buboard(3489) about 3 hours ago [-]

7 years confined in a room, no matter how nice the embassy, common sense says it is obviously psychological torture.

jaimex2(4136) about 3 hours ago [-]

Title should really be 'US takes hand out of Swedish puppet now that its free to extradite Assange'

The take away here is stay anonymous when whistle blowing or handling leaks. Don't talk to anyone, don't say anything just take a USB stick to an Internet cafe while disguised and host a torrent.

Then email the press the magnet link using a throw away email service.

Iv(10000) about 2 hours ago [-]

Thing is, someone, at one point, needs to check the source. Also, most potential whistleblower are not privacy experts and need help. Assange wanted to be in that role, that is necessarily public.

He did many bad things (especially uncovering the name of sources, not redacting the cables in the cablegate) but like he said, it is interesting to see how censorship works in a democracy.

I also like how no one is talking about the (probably externally provoked) internal drama at wikileaks that led to the destruction of the Bank of America files that they had.

plesner(3955) about 3 hours ago [-]

It's possible to be both unfairly targeted by a superpower and commit sexual assault, one doesn't cancel out the other. I'm not saying he did, I don't know how any of us could know. You seem quite confident he didn't though, I'm curious what you base that on.

geofft(3275) about 3 hours ago [-]

Er, Assange himself was the press, here. Chelsea Manning was the anonymous whistleblower who tried to keep her identity safe until Adrian Lamo (may God have mercy on his soul, because I sure won't) reported her to US authorities.

eigenvalue(4179) about 3 hours ago [-]

Even that approach seems very scary to me from an opsec perspective. You could be traced by CCTV cameras outside the internet cafe for example.

wolco(3683) about 3 hours ago [-]

That's nice of them to drop the trumped up charges 10 years later.

whamlastxmas(4132) 9 minutes ago [-]

There were never any charges. He was only wanted for questioning.

dominotw(2050) about 2 hours ago [-]

What a weird statment. How can a country decide to drop a crime against its citizen. Shouldn't the person who got raped be deciding that.

whamlastxmas(4132) 7 minutes ago [-]

The alleged victims never pressed for Assange to be charged for sexual assault.

mikl(3691) 44 minutes ago [-]

In most European countries, the state prosecutor decides who to charge with a crime, regardless what purported victims think.

It might be a bit foreign, but it does help with some cases, like partner violence, where the victim might not want to press charges against a violent partner. Under this model, if the police learns of the crime, they are obliged to prosecute it if possible.

ivl(10000) about 4 hours ago [-]

> 'The reason for this decision is that the evidence has weakened considerably due to the long period of time that has elapsed since the events in question.'

I'm not sure I understand what's meant by this. Did the victims testimony change? How does evidence weaken over time?

geofft(3275) about 3 hours ago [-]

Sweden already announced in 2017 they were giving up prosecuting the case because of the futility of proceeding.


kkarakk(10000) about 3 hours ago [-]

He got picked up in a country that the US can extradite from and they would actually have to look at the case now. This thing has been so transparent.

Stay safe Snowden and all you other possible whistleblowers/leakers out there!

To people downvoting me - check out the vox article on this earlier this year.


The women were 'vehement' about restarting the case. Reportedly were 'willing to do whatever it takes to force case to be re-opened' The statute of limitations is August,2020.

But suddenly it doesn't matter according to sweden

gutnor(10000) about 3 hours ago [-]

Maybe the victim have moved on.

Rape without physical violence like this rely entirely on the victim testimony and even if there is a record and everything that will weaken the case terribly if the victim is no longer on board.

philwelch(3841) about 2 hours ago [-]

Sometimes a rape victim will just want to move on with her life instead of being unfairly smeared as a CIA puppet by a bunch of neckbeards on the internet.

ceejayoz(2116) about 4 hours ago [-]

Memory weakens over time, and the case would've largely hinged on memory-based testimony.

jakelazaroff(3770) about 4 hours ago [-]

Eyewitnesses' memories get worse, records get lost, etc. This is one of the reasons that statutes of limitations exist.

lettergram(1410) about 4 hours ago [-]

If I recall the evidence was always pretty weak. I don't have time to find the news article from back then, but it was "sexual assault" because he didn't wear a condom and at the time the woman (perhaps plural) didn't want to really participate.

Could be wrong though...

Does anyone have some news from back then?

mikl(3691) about 1 hour ago [-]

There was no evidence, just two women accusing him of sexual misconduct (if memory serves, it wasn't even rape in the classical sense, but Assange not wearing a condom when he'd said he would).

trentnix(10000) about 4 hours ago [-]

The charge had served its purpose and the evidence was eventually going to actually be scrutinized.

It was just lawfare all along.

zaroth(2931) about 3 hours ago [-]

I don't think there's any reason to believe the prosecutor is going to tell the BBC the full story behind the decision making for dropping the case, versus a sound bite that seems reasonable.

If they had obtained internal first-person memos discussion the reasons for dropping the case, then you could probably assume those would be credible.

injb(10000) about 3 hours ago [-]

He's now safely in police custody and the US has already revealed its plans to extradite him, so there's no longer any need for the charges in Sweden. That's what's meant by this, imo.

trpc(10000) about 3 hours ago [-]

There were, not long ago, times when Assange was considered a saint of freedom after he exposed the corrupt Bush administration, the liberals everywhere were defending him and his case everywhere, they even made a Hollywood film about him. But he then made the mistake of his life, he exposed the corruption of the Clintons and got implicated in the Russia gate meme because Trump somehow won. Now he is totally on his own. Nobody will defend him even if these corrupt governments burnt him alive and broadcasted it live. Makes you think if these liberals actually give a single shit about freedom anyway.

I know that this is an ultra liberal community and I will get downvoted and flagged within a few minutes. But if someone is too scared to even lose a worthless website account to say what he sees as the right opinion, he should not live anyway.

alwaysanagenda(10000) about 2 hours ago [-]

Good to see this voice among the masses who have a memory long enough to understand the big picture.

I think Assange will be taken care of when all is said and done. This is from 2016, Dr. Steve Pieczenik explains it all -- Assange was part of a 'counter-coup' from within the US Government to take down the Clinton cabal.


homonculus1(10000) about 3 hours ago [-]

In fairness to the ultra liberals of HN, there is still plenty of support for Assange around here so your brush may be a touch broad.

ur-whale(4149) about 1 hour ago [-]

>I know that this is an ultra liberal community

It isn't. HN is actually very far from even a mildly liberal community in the correct sense of the word.

Unless you use the word in the same way people from rural Kansas do when what they actually mean is a leftist.

deiznof(10000) about 2 hours ago [-]

If you were on reddit this would be true, but here people aren't on 'I hate Assange' autopilot.

kuu(4222) about 4 hours ago [-]

Does it mean he is free again?

Miner49er(10000) about 3 hours ago [-]

Nah, he's gonna be extradited to the U.S. where we'll stick him in prison for life to set an example.

djsumdog(1073) about 4 hours ago [-]

No, he still has to serve 50 weeks for jumping bail in the UK.

djsumdog(1073) about 4 hours ago [-]

So he still gets almost another full year in a UK jail. After that, where can he go?

Let's just say he's somehow successful at avoiding US extradition (unlikely), he's not going to be able to stay in the UK (unless he has dual citizenship or can somehow claim dual citizenship through a relative). He'd have to request asylum from another European nation that he can safety get to without flying.

Australia did little to help him in his situation, and it would be likely America could extradite him from there. There's a good chance it may never be safe for him to return home.

buboard(3489) about 3 hours ago [-]

why doesn't australia protect its citizens?

Apparently, 'The Commonwealth will not extradite or deport a prisoner to another jurisdiction if they might face the death penalty'

pjc50(1404) about 3 hours ago [-]

If not extradited to the US, he would most likely end up being deported to his country of origin (Australia) directly.

Roark66(10000) about 3 hours ago [-]

If he is released from UK jail he can hop onto a ferry and go to mainland EU. There he could travel to anywhere within the Shengen zone. I'm not sure about western countries, but Poland(as well as other eastern countries I believe) will not extradite anyone anywhere unless the crime is also considered a crime here, there is no possibility of death penalty, and there is strong evidence submitted that makes a conviction likely. By its definition treason against a country can be committed, by a citizen of that country. Assange is not a US citizen so he can't commit a crime of treason against US - end of story. US could try to get him on some other charges - espionage perhaps, but there would have to be strong evidence which allegedly is severely lacking. This is why it is very important for US to have Assange extradited from UK. UK has a 'special' extradition deal with US that simplifies the procedure.

lucozade(4218) about 3 hours ago [-]

I wouldn't imagine that he'll be offered a choice. He'll be shipped back to Aus. I can't see any reasonable defence against the deportation that would stand a chance in a UK court.

Historical Discussions: Eigenvectors from eigenvalues (November 15, 2019: 524 points)
Eigenvectors from eigenvalues (November 13, 2019: 4 points)

(525) Eigenvectors from eigenvalues

525 points 5 days ago by bigpumpkin in 10000th position

terrytao.wordpress.com | Estimated reading time – 6 minutes | comments | anchor

Peter Denton, Stephen Parke, Xining Zhang, and I have just uploaded to the arXiv the short unpublished note "Eigenvectors from eigenvalues". This note gives two proofs of a general eigenvector identity observed recently by Denton, Parke and Zhang in the course of some quantum mechanical calculations. The identity is as follows:

Theorem 1 Let be an Hermitian matrix, with eigenvalues . Let be a unit eigenvector corresponding to the eigenvalue , and let be the component of . Then

where is the Hermitian matrix formed by deleting the row and column from .

For instance, if we have

for some real number , -dimensional vector , and Hermitian matrix , then we have

assuming that the denominator is non-zero.

Once one is aware of the identity, it is not so difficult to prove it; we give two proofs, each about half a page long, one of which is based on a variant of the Cauchy-Binet formula, and the other based on properties of the adjugate matrix. But perhaps it is surprising that such a formula exists at all; one does not normally expect to learn much information about eigenvectors purely from knowledge of eigenvalues. In the random matrix theory literature, for instance in this paper of Erdos, Schlein, and Yau, or this later paper of Van Vu and myself, a related identity has been used, namely

but it is not immediately obvious that one can derive the former identity from the latter. (I do so below the fold; we ended up not putting this proof in the note as it was longer than the two other proofs we found. I also give two other proofs below the fold, one from a more geometric perspective and one proceeding via Cramer's rule.) It was certainly something of a surprise to me that there is no explicit appearance of the components of in the formula (1) (though they do indirectly appear through their effect on the eigenvalues ; for instance from taking traces one sees that ).

One can get some feeling of the identity (1) by considering some special cases. Suppose for instance that is a diagonal matrix with all distinct entries. The upper left entry of is one of the eigenvalues of . If it is equal to , then the eigenvalues of are the other eigenvalues of , and now the left and right-hand sides of (1) are equal to . At the other extreme, if is equal to a different eigenvalue of , then now appears as an eigenvalue of , and both sides of (1) now vanish. More generally, if we order the eigenvalues and , then the Cauchy interlacing inequalities tell us that

for , and

for , so that the right-hand side of (1) lies between and , which is of course consistent with (1) as is a unit vector. Thus the identity relates the coefficient sizes of an eigenvector with the extent to which the Cauchy interlacing inequalities are sharp.

— 1. Relating the two identities —

We now show how (1) can be deduced from (2). By a limiting argument, it suffices to prove (1) in the case when is not an eigenvalue of . Without loss of generality we may take . By subtracting the matrix from (and from , thus shifting all the eigenvalues down by , we may also assume without loss of generality that . So now we wish to show that

The right-hand side is just . If one differentiates the characteristic polynomial

at , one sees that

Finally, (2) can be rewritten as

so our task is now to show that

By Schur complement, we have

Since is an eigenvalue of , but not of (by hypothesis), the factor vanishes when . If we then differentiate (4) in and set we obtain (3) as desired.

— 2. A geometric proof —

Here is a more geometric way to think about the identity. One can view as a linear operator on (mapping to for any vector ); it then also acts on all the exterior powers by mapping to for all vectors . In particular, if one evaluates on the basis of induced by the orthogonal eigenbasis , we see that the action of on is rank one, with

for any , where is the inner product on induced by the standard inner product on . If we now apply this to the -form , we have , while is equal to plus some terms orthogonal to . Since , Theorem 1 follows.

— 3. A proof using Cramer's rule —

By a limiting argument we can assume that all the eigenvalues of are simple. From the spectral theorem we can compute the resolvent for as

Extracting the component of both sides and using Cramer's rule, we conclude that

or in terms of eigenvalues

Both sides are rational functions with a simple pole at the eigenvalues . Extracting the residue at we conclude that

and Theorem 1 follows. (Note that this approach also gives a formula for for , although the formula becomes messier when because the relevant minor of is no longer a scalar multiple of the identity .)

All Comments: [-] | anchor

banachtarski(10000) 5 days ago [-]

For those curious about applications:

Suppose you have a learning algorithm that requires the eigenvectors/eigenvalues for a large matrix. Suppose new data is streamed in that increase the size of the matrix. well, now you have an algorithm for updating your existing solution instead of needing to compute the entire thing from scratch. (consider updating principal components over time for example).

Suppose you want to accelerate the process of computing eigenvectors for a large matrix. Well, you can now do parallel reduction relatively trivially given the theorem.

*edit: wording.

yxhuvud(4192) 4 days ago [-]

Won't you need a way to figure out the sign of the vector, though? Am I wrong in that this method only output the norms?

mxcrossb(4015) 4 days ago [-]

Can you go into more detail? I don't see how that would work. In this algorithm, you need the eigenvalues of all of the minors. So if I'm adding rows/columns to the matrix, yes I have one of the minors from my last solution, but I have to compute N more. And I also need the eigenvalues of the new full matrix.

I think this result has some intriguing opportunities, and have been toying with them since yesterday's post, but I don't see something so straightforward.

suslik(10000) 4 days ago [-]

But you would still need to know the eigenvalue λi(A) of the full matrix, no? How would you do it in this streaming setting?

ArtWomb(736) 4 days ago [-]

Not to downplay the pure and applied mathematics implications. But the immediate discovery was made in the context of calculating neutrino oscillation probabilities. Yielding insight into matter / anti-matter imbalance in the known universe ;)

Eigenvalues: the Rosetta Stone for Neutrino Oscillations in Matter


Deep Underground Neutrino Experiment


auggierose(3545) 4 days ago [-]

If the matrix is symmetric (or hermitian in the complex case).

acd(4132) 4 days ago [-]

When I see people who look similar I think of them as having similar Eigenfaces!

jjgreen(3841) 4 days ago [-]

Eigenfaces are a thing! https://en.wikipedia.org/wiki/Eigenface

rossmohax(10000) 4 days ago [-]

Will it boost FPS in 3d games?

mkl(4126) 4 days ago [-]

No, it doesn't speed anything relevant up. I'm curious as to what made you ask, though.

bhl(3982) 5 days ago [-]

There's a backstory to this paper from the r/math subreddit [1]. One of the authors initially posted the question there, and stumbling upon a similar question on math overflow emailed Terry Tao. And surprisingly, he responded.

[1] https://www.reddit.com/r/math/comments/ci665j/linear_algebra...

leereeves(10000) 4 days ago [-]

Not only responded, but responded in under 2 hours, which is amazing from a busy professor like Tao.

dmix(1368) 4 days ago [-]

Someone replied:

> Apparently this relationship was discovered in 2014 by a dutch mathematician or physician. https://arxiv.org/abs/1401.4580


I don't know anything about the subject, I'm curious what they mean by that.

Ceezy(4084) 4 days ago [-]

That s great, but any good bachelor students should be able to give you a similar answer.

james_s_tayler(10000) 5 days ago [-]

Wow. This is so great.

j1vms(3995) 4 days ago [-]

Somewhat related to some of the questions that have been raised here:

- for the set of matrices that possess them ('transformation matrices that only perform stretch/contract'), eigenvectors (with their associated eigenvalues) play a role quite analogous to the role primes play in the integer set. They provide a unique, identifying 'spectrum' for said matrix. This is made explicit by eigendecomposition (spectral decomposition).

- with extension via singular value decomposition (SVD method) to any square matrix (e.g. 'transformation matrices that might also shear, rotate'), certain operations such as exponentiation of the square matrix can performed very quickly once eigenvectors/eigenvalues have been obtained via the SVD method.

creative-coder(10000) 4 days ago [-]

Wow... your comment is infact more useful pearl of wisdom for getting the intuition of Eigen-values/vectors. Can you suggest some book/reference that best conveys this intuition ? Somehow, Gilbert Strang's Linear Algebra does put me to sleep :/

MontyCarloHall(10000) 4 days ago [-]

>for the set of matrices that possess them ('transformation matrices that only perform stretch/contract'), eigenvectors

Minor nitpick: this should end with "real eigenvectors." Rotation matrices certainly have eigenvectors, they're just complex.

ur-whale(4149) 4 days ago [-]

Before you excitedly jump to coding a new algorithm to produce eigenvectors from eigenvalues, it's worth noting that:

    . This only applies to hermitian matrices and is therefore of limited scope.
    . The identity only produces the magnitudes of the coefficients of the eigenvectors. The signs of the coefficient sare still unknown.
    . Even if knowing only the magnitude of the coefficients is somehow useful, this is still very expensive to compute - just like Cramer rule which is of great theoretical interest, but has very little practical use when actually computing a matrix inverse or a determinant.
None of the above diminishes the importance of the result, btw.
gajomi(4191) 4 days ago [-]

Came here to pretty much write these same notes. The fact that you need to know the spectrum of all the minors really limits the usefulness for solving general eigen* problems. But I can imagine that this could make a great starting point for all kinds of approximate calculations/asymptotic analyses.

Regarding the first caveat that you bring up though, whereas the problem statement says you need a Hermitian matrix I think the results should generalize to non-hermitian matrices. In particular take a look at the third proof of the theorem at the end of the article. The only assumption required here is that the eigenvalues are simple (which does not preclude them being complex/coming in complex pairs).

Protip: I had to read the second step in the proof a few times before I could see what was going on. Explicitly what you do here is to (i) multiple from the right by the elementary unit vector e_j (ii) set up the matrix vector inverse using Cramer's rule (iii) notice that this matrix can be permuted to have a block diagonal element equal to the minor M_i with the other block diagonal entry equal to 1 and the corresponding column equal filled with zeros (iv) Write the determinant in the block form, which simplifies to the expression involving M_i by itself then finally (v) multiply by e_j from the left to get scalar equation.

MattSayar(10000) 4 days ago [-]

Despite the limited scope, this information is 'out there' now, and could potentially be crucial for some future 'useful' application. Not that I think you're trying to downplay it or anything, I'm just seeing this as another example of how the internet is still a good thing.

angel_j(4195) 4 days ago [-]

Does this mean you can test large neural networks with smaller ones?

c1b(10000) 4 days ago [-]

Are your neural networks parameterized as Hermitian matrices? (No.)

xxpor(4076) 5 days ago [-]

So it's been checks notes 9 years since I've had to deal with eigenvectors and eigenvalues. So the post went way over my head, but folks on twitter were making quite a big deal over the post last night when it went up on arXiv. Could someone explain this like I have a computer engineering bachelors degree? :)

FabHK(3966) 4 days ago [-]

So, you have some infinite space, and now you transform it with a linear transformation A - that could include mirroring, rotation, stretching (but not translation - one of the properties of linear transformations is that the origin, zero, stays at the origin).

Now, there could be lines through the origin that map to themselves under this transformation. These lines are called eigenvectors of A, and they characterise the transformation A. (If you mirror something, a line in the mirror plane maps to itself. If you rotate something, the rotation axis maps to itself. If you stretch something, a line in the direction you stretch in might map to itself. And so on.)

Now, these lines map onto themselves, but they might be stretched or shrunk in the process - that's given by the Eigenvalue.

The famous equation is A x = lambda x: You transform eigenvector x by A, and you get back x itself, but stretched by a factor lambda (the Eigenvalue).

So, if you have some mapping of 5d space to itself, for example, it can be characterised by 5 eigenvectors (lines that map unto themselves), and 5 eigenvalues (the stretch factor for each of these lines).

Now, 'traditionally', you compute the eigenvector by solving a 5d linear system for each of the eigenvalues. (5 eigenvalues, 5 slightly modified linear systems, each gives you an eigenvector of 5 coefficients, for a grand total of 25 eigenvector coefficients).

Here, it is shown that you can compute the eigenvectors of the full system without even solving that traditional linear system, but by instead taking the eigenvalues of a slightly modified smaller system. (5 eigenvalues, 5 slightly modified smaller systems M (that you get from A by striking out the i'th column and row). Each of those has 4 Eigenvalues, so you have a total of 5 + 4x5 = 25 Eigenvalues. Now, you compute some differences and products and ratios of those numbers, and hey presto, you get your 25 Eigenvector coefficients (thus the title, 'Eigenvectors from Eigenvalues').

Pretty neat, though it's unclear to me how the computational complexity really compares.

EDIT: formatting, reference to title

te_platt(3474) 4 days ago [-]

It's something like an economist finding a $20 dollar bill on the ground and it's actually a real $20 bill that no one has picked up yet.

randomsearch(4204) 4 days ago [-]

Don't know how simple you want me to go, so apologies if required.

You've got a matrix A, ie a 2D array of numbers.

You've got a vector v, that is a 1D array of numbers.

You can multiply A by v, written Av, to get a new vector.

Matrices can therefore be viewed as devices for taking a vector and giving a new one, ie as linear transformations. You can design a matrix to rotate vectors, stretch them etc.

An eigenvector of a matrix is a vector that changes only by a constant factor c, when you apply (multiply by) the matrix. So Av = kv where k is a number. For example, if k=2 then applying A just doubles every number in v. Eigenvectors are stretched or squashed by the matrix, no funny business like shearing or rotating.

An eigenvalue is the "k" mentioned above, ie there's at least one eigenvector such that Av=kv. There could be multiple eigenvectors all with the same k, ie all changed in the same simple way by applying the matrix.

Ok? Slight complication: matrix A may contain complex numbers, not just your common garden variety real numbers. But the above all still applies.

"Symmetric matrices" are matrices that have a symmetry across the top left to bottom right diagonal. So the number at A[i][j] is the same number as that at A[j][i].

"Hermitian" matrices are the equivalent of symmetric matrices when we're dealing in complex numbers. Rather than A[i][j] being equal to A[j][i], if one entry is a complex number then the other entry is the "complex conjugate" of the first entry.

(In case you've forgotten complex numbers: we extend the real numbers to including numbers of the form r + zi, where r is the "real part" (just a real number) and "zi" is a real number z multiplied by i, where i = sqrt(-1). Imaginary numbers were introduced to allow us to solve equations we couldn't otherwise solve, but have turned out to be super useful everywhere. The "complex conjugate" of an imaginary number is obtained by flipping the sign of z, eg (2 + 3i) -> (2 - 3i).)

Anyway, so Hermitian matrices are "symmetric" in this way and turn out to be super useful in physics. They also have the property that when you multiply a Hermitian matrix H by a vector v, the possible eigenvalues are real numbers, despite the fact that H may contain complex numbers.

What the paper states is a relationship between the eigenvalues and eigenvectors of a Hermitian matrix H and the eigenvalues of a smaller "submatrix" matrix H', where H' is just the same as H but with a column and row removed, where the index of the row and column is the same. This is neat, because it can be used to speed up the calculation of eigenvectors (as important as say array sorting in CS) in some specific situations. Commenting further is beyond me.

klodolph(4214) 5 days ago [-]

You might prefer the earlier post, https://news.ycombinator.com/item?id=21528425

Basically, this allows you to compute eigenvectors from eigenvalues of minor matrices. It is an interesting identity, but speculating on its applications is beyond my expertise.

auggierose(3545) 5 days ago [-]

This is a good example of math being presented in an unnecessarily confusing way.

Values lambda_i(A) are introduced, but then in the formula also values lambda_i(M) are referenced for some M which is not A.

Basically, we have no idea what lambda_i(M) might mean from the statement of the theorem. Is the order lambda_1(M), lambda_2(M) ... related to the order lambda_1(A), lambda_2(A), ... How many lambda_i(M) exist? And so on.

Maybe this is all clear if you deal with minor matrices all the time, but when I see a statement like that I am not even interested in its proof anymore.

banachtarski(10000) 5 days ago [-]

Terry is being abundantly clear. Why are you trying to correlate indices in two separate product series?

_hardwaregeek(10000) 4 days ago [-]

Actually...yeah the two orderings are related by the Cauchy Interlace Theorem: https://www.semanticscholar.org/paper/Cauchy%27s-Interlace-T....

debbiedowner(10000) 4 days ago [-]

In that formula the order of the eigenvalues doesn't matter...

liuyao(10000) 4 days ago [-]

Come on, if I defined f(x)=x^2, and later used f(y) for y=x/2 or something, do you have problem with that?

tomrod(2257) 4 days ago [-]

Yes, it is written for people who deal with minor matricies all the time. Academic papers are incremental materializations -- it is up to the reader to bring or build the requisite base understanding.

aj7(3918) 4 days ago [-]

Since I've always been terrible at abstract linear algebra, I would request that someone do a simple problem, say finding the eigenvalues and eigenvectors of a 3 x 3 Hermitian matrix with three distinct eigenvalues. I'd also be curious under what circumstances the above method leads to a computational advantage.

aj7(3918) 4 days ago [-]


Historical Discussions: GitHub Archive Program (November 13, 2019: 522 points)

(522) GitHub Archive Program

522 points 6 days ago by tosh in 6th position

archiveprogram.github.com | Estimated reading time – 2 minutes | comments | anchor

We're convening a GitHub Archive Program advisory panel, including experts in anthropology, archaeology, history, linguistics, archival science, futurism, and more, to advise us on what content should be included in the archive and how to best communicate with its inheritors.

A thousand years is a very long time. Ancient ruins such as Angkor Wat, Great Zimbabwe, and Macchu Picchu had not yet been built a thousand years ago. Nevertheless, we can consider and plan for a broad range of possibilities over the next 1,000 years. This program builds on the best ideas we have today.

The introduction to the archive will include technical guides to QR decoding, file formats, character encodings, and other critical metadata so that the raw data can be converted back into source code for use by others in the future. The archive will also include a Tech Tree—a roadmap and Rosetta Stone for future curious minds inheriting the archive's data.

An overview of the archive and how to use it, the Tech Tree will serve as a quickstart manual on software development and computing, bundled with a user guide for the archive. It will describe how to work backwards from raw data to source code and extract projects, directories, files, and data formats.

Inspired by (and including elements of) the Long Now's Manual for Civilization, the archive will also include information and guidance for applying open source, with context for how we use it today, in case future readers need to rebuild technologies from scratch. Like the golden records of Voyager 1 and 2, it will help to communicate the story of our world to the future.

In the range of possible futures in which humanity has working modern computers, but no software to run on them, the archive and its Tech Tree could be extremely valuable. However, the value is more likely to be historical, perhaps ensuring that today's technology is not lost by a tomorrow that carelessly considers it irrelevant—until an unexpected use for our software is discovered.

All Comments: [-] | anchor

vortico(3674) 6 days ago [-]

An interesting thought: The license of your open-source software will become irrelevant if this is cracked open in 1000 years because everything will be public domain, assuming copyright laws aren't changed drastically, but it's interesting to wonder about what future humans will think about the open-source license movement of the 1980's-to-present.

jobigoud(10000) 6 days ago [-]

This has me wondering what is the first free or open-source software that will 'fall' into the public domain, and when that will be?

t0astbread(10000) 6 days ago [-]

In a thousand years someone will inherit the ultimate legacy codebase.

falcor84(3842) 6 days ago [-]

At least they wouldn't be expected to maintain it running in production on some old mainframe

michaelmior(3855) 6 days ago [-]

> For greater data density and integrity, most of the data will be stored QR-encoded

At face value, this seems like a really odd choice. I don't understand why you would choose QR encoding unless this was being printed. I feel like I'm missing something here.

zuck9(3017) 6 days ago [-]

In addition, they provide excellent error recovery.

333c(4011) 6 days ago [-]

It's being stored on film, according to the page.

munificent(1794) 6 days ago [-]

> unless this was being printed.

It is. QR codes printed onto microfilm, essentially.

darkwater(4167) 6 days ago [-]

Almost totally unrelated but:

> As today's vital code becomes yesterday's historical curiosity

shouldn't be 'tomorrow's historical curiosity'?

K0SM0S(10000) 6 days ago [-]

They probably mean that today will be referred to as 'yesterday' in the future. The sentence structure is ambivalent, though, it would work both ways. I think your interpretation is actually better, I'd have written 'tomorrow' as well too.

ksec(1729) 6 days ago [-]

Same question. If it is not wrong, could someone please explain?

Nuzzerino(4134) 6 days ago [-]

Nice to see the Long Now Foundation involved with this. A friend of mine is a member. The work that they do is a pretty big deal given that not much of it is being done in today's culture of short term thinking.


julianmetcalf(10000) 6 days ago [-]

We are thrilled to be partnering with the Long Now Foundation on this effort. They were a huge inspiration to us!

jedieaston(10000) 6 days ago [-]

will there be a computer in there that can survive 10,000 years so that you can compile the code?

julianmetcalf(10000) 6 days ago [-]

I'm the PM at GitHub for the Archive Program. We aren't making the assumption that you will have computer. The archive will include a Tech Tree that explains in plain language the fundamentals of computer programming and how to use the material in the Arctic Code Vault. We will also include technical guides to QR decoding, file formats, character encodings, and other critical metadata so that the raw data can be converted back into source code for use by others in the future.

soheil(3439) 6 days ago [-]

Github has blocked access to my account with 10s of popular projects because one day they randomly sent me an email to click on a link to enable 2fa auth. I was coerced to enable it. A while later I lost access to the phone where the 2fa auth app was installed not having back up codes since I was pushed to enable 2fa in a rush now I'm completely locked out. I contacted support no fewer than 15 times with them saying I need to create a new account since I did not link my cell phone nor have back up codes. I had that account for over a decade and now I cannot even control the projects I was working on nor access any of my private repos. I have been communicating with them using the email I have on my account, but this is not sufficient for them to restore access to my account.

beart(10000) 6 days ago [-]

I'm sorry you lost your account (really, that sucks) but I don't understand framing this like Github is doing something wrong. It isn't like they banned you for spamming emojis.

You enabled 2fa, you lost your 2fa, and you did not have any recovery codes. Now you are asking for them to bypass the 2fa, and they are refusing.

Again, that sucks, but when I compare this to what the cell phone companies are doing with sim swapping, it increases my respect for Github.

alkonaut(10000) 6 days ago [-]

This is why I'd rather have the phone number/email 2fa than a device 2fa even with the risk of sim swap.

If a human can't give me my account back through tech support I'm not very keen on trusting my account a gadget that can break or get lost.

The risk of losing a phone and the backup codes is probably several orders of magnitude larger than the risk of being the target of a sim swap attack for the vast majority of users.

mythz(2901) 6 days ago [-]

It's unfortunate, but I'd consider it a feature that you're not able to sign in without access to the physical devices linked to your 2FA account, i.e. it shouldn't be possible for someone with access to your Email account to be able to 'phish' their way passed 2FA access.

Nevertheless the anxiety of losing the physical device with all my 2FA logins is what prevented me from enabling 2FA on most of my accounts until I was referred to Authy (authy.com) where you can sync your 2FA across multiple devices including your PC, which other than being very convenient, the effortless syncing + redundancy gave me confidence to enable 2FA on all my accounts as the redundancy ensures I'll still be able to access my accounts if one of my devices is broken/lost.

Latty(10000) 6 days ago [-]

If you can talk your way past two factor auth, it is useless.

I can see the argument for being forceful about prompting you to write down backup codes or whatever, but fixing that after-the-fact is something they should absolutely not been doing.

I do think that sites should offer better options for recovery. NearlyFreeSpeech do a really good job of this, offering seven methods of recovery and letting you decide how many you need to fulfil to be given access and which you want to configure. However, things like checking photo ID and more 'offline' options are expensive to support, so I get why that is rare.

usmannk(10000) 6 days ago [-]

I have restored access to a lost 2FA github account by presenting support with an artifact signed by my SSH keys I had linked to the account.

If you still have the keys, try:

ssh -i ~/.ssh/your_linked_privkey -T [email protected] verify

Edit: I previously wrote to pass your public key to the ssh client rather than your private key. Of course, that was incorrect.

morpheuskafka(10000) 6 days ago [-]

If you were 'coerced' to enable 2FA, that was a decision made by one of the organizations/teams (company accounts) you were a member of, not GitHub. You had every option to leave it disabled, but, per that team's policies, you would have lost access to their repos.

rmrfrmrf(10000) 6 days ago [-]

Luckily it's git and you can just change the remote config of your local copies.

buildbot(4215) 6 days ago [-]

If github did their 2FA correctly/very securely, it may literally be impossible for them to give you access again.

The entire point of 2FA is to avoid someone taking over your email and then being able to access _anything_ tied to that email.

itsrajju(10000) 5 days ago [-]

This happened to me a couple of years ago. Lost my phone, no recovery codes, so no access to github. Contacted support, they told me that they wouldn't be able to restore access even when I was communicating with the email associated with my account.

Luckily I only had public repos, so I created a new account and forked them all. Support had told me that if there isn't any login activity in the blocked account for a period of six months, they would delete the account and release the username. And yes, I had to follow up after six months were up to make that happen.

est31(3909) 5 days ago [-]

So that's what happens if you enable 2fA. I have disabled it after I left a Github org where 2fA was required, and haven't enabled it since. However, Github now regularly sends me e-mails with auth codes, basically forcing mail-based 2fA on me. It's annoying as hell and I can't disable it, nor does Support want to do anything about this. Very sad as I'm using a password manager and my password should be safe.

jeffwilcox(10000) 6 days ago [-]

There's a reason that it's so important to save those recovery codes... that's your opportunity of last resort to recover access.

tssva(10000) 6 days ago [-]

An email suggesting enabling 2FA isn't coercion. You voluntarily enabled 2FA and then choose not to protect the backup codes as they repeatedly warn to do because loss of them along with loss of your 2FA device will result in exactly this situation.

Now instead of accepting the result is the consequence of your own poor choices you are trying to shift the blame to GitHub.

twobat(10000) 6 days ago [-]

They probably have enough tracking info they could spot you just by you blinking. But they don't care.

urda(4202) 6 days ago [-]

E-mail newsletters announcing new features or options are not 'coercing' you to do anything.

You clicked it, failed to know the ramifications of 2FA (which GitHub does spell out), and didn't secure your backup codes.

Take some responsibility instead of blame shifting.

dmix(1368) 6 days ago [-]

> On February 2, 2020, GitHub will capture a snapshot of every active public repository, to be preserved in the GitHub Arctic Code Vault. This data will be stored on 3,500-foot film reels, provided and encoded by Piql, a Norwegian company that specializes in very-long-term data storage. The film technology relies on silver halides on polyester. This medium has a lifespan of 500 years as measured by the ISO; simulated aging tests indicate Piql's film will last twice as long.

That just sounds like a fun project.

breck(365) 6 days ago [-]

Right? Sounds so fun.

But also a great engineering exercise. I wouldn't be surprised if this exercise leads to lots of valuable improvements for GitHub users in the here and now. Trying to solve such a grand challenge forces you to develop a vocabulary and understanding of your current systems that can lead to more immediate improvements. I think it unlikely any of these archives will actually be accessed, but simply building them could lead to great side effects.

It's also great marketing, as I now believe that Microsoft/GitHub takes the job of not losing user data extremely seriously, more so than if they had spent an equivalent some of money buying an ad that says 'We take not losing data seriously'.

rusini(4220) 6 days ago [-]

Wow, are we preparing for a global catastrophe?

dredmorbius(199) 6 days ago [-]

We are a global catastrophe.

sneak(3056) 6 days ago [-]

We always have been.

bloopernova(10000) 6 days ago [-]

> are we preparing for a global catastrophe?

It seems prudent. We may be entering a period of global upheaval and panic where large parts of human civilization will not make it past the next 100 years. It makes sense to attempt to preserve what we've accomplished in a way that will last for 1000s of years.

oscargrouch(4086) 6 days ago [-]

I think this is more in the line of

'Si vis pacem, para bellum'

Its very important to protect the civilizatory process from fallout, as severe or black-swan conditions might swipe what we have acomplished so far, and history is full of examples of advanced civilizations that were vanished and we had to dig out from the mud slowly without a chance to learn, starting all over again from scratch.

The civilizatory process is fragile and is always menaced by all sides, all the time. Its good to be always vigilant and prepared to anything.

beokop(10000) 6 days ago [-]

Is there a way to opt out of this? I'm not sure I want my code to be stored forever in a glacier with no way of deleting it.

chrisseaton(3077) 6 days ago [-]

If it's already public then how do you know people aren't already archiving it?

thrownaway954(4219) 6 days ago [-]

You do know that if anyone has forked your project, they have a complete copy of your work. That said, if they don't opt out... well then I guess your work IS going to be archived.

ris(3593) 6 days ago [-]

Sure - just let me take a fork first...

julianmetcalf(10000) 6 days ago [-]

Hi, Julia here, the PM for the Archive Program at GitHub. Yes, you will be able to opt out of the program. One option is to make your repo private, as only active public repos will be archived. If you don't wish to make your repo private please contact support at support.github.com.

marmada(10000) 6 days ago [-]

Out of curiosity, what would you want to opt out?

Exuma(3829) 6 days ago [-]

Yes because 9000 bots in China haven't already done that to your public repo...

toomuchtodo(2025) 6 days ago [-]

Make your repo private might be one way.

tjr(882) 6 days ago [-]

I'm thinking I might want to opt in. What should I add to GitHub before February, to make sure it gets in the archive?

jedieaston(10000) 6 days ago [-]

If you have public GitHub repos, people crawl them all of the time. I've seen at least three top-of-HN articles that say something like: 'I scripted through 10 billion LOC on GitHub, here's a bunch of passwords', or something to that effect. Just set your repo to private if you don't want it in the vault, and if it's open source anyway, who cares?

tosh(6) 6 days ago [-]

> The GitHub Arctic Code Vault is a data repository preserved in the Arctic World Archive (AWA), a very-long-term archival facility 250 meters deep in the permafrost of an Arctic mountain. The archive is located in a decommissioned coal mine in the Svalbard archipelago, closer to the North Pole than the Arctic Circle. GitHub will capture a snapshot of every active public repository on 02/02/2020 and preserve that data in the Arctic Code Vault.

Already__Taken(10000) 6 days ago [-]

Hope it's water tight

cayblood(4073) 6 days ago [-]

The Arctic World Archive description says: 'The film technology relies on silver halides on polyester. This medium has a lifespan of 500 years as measured by the ISO; simulated aging tests indicate Piql's film will last twice as long.'

Why not use a much more accessible medium like M-DISC, which claims a lifespan of at least 1,000 years? https://en.wikipedia.org/wiki/M-DISC

jerrysievert(4017) 6 days ago [-]

> Why not use a much more accessible medium like M-DISC

just wondering how an encoded digital format is more accessible than something printed on a transparent film that you can see by looking through it.

CameronBanga(3585) 6 days ago [-]

10,000 years from now, society has been destroyed. And people come across a glass plate with laser etchings. How bummed will they be when they go through all of the effort of decoding it, only to learn it's like some python library for managing drivers on a 2014 Dell laptop running linux?

Laforet(10000) 5 days ago [-]

Well, the Rosetta Stone was just a mundane tax decree and that does not take anything away from it's significance.

osener(4067) 5 days ago [-]

Look on the bright side: in 10,000 years Python dependency management will be in a better shape and they'll have a fighting chance of running your software.

dev_dull(10000) 6 days ago [-]

'Python 2.7 only??!'

jlgaddis(2877) 6 days ago [-]

You mean once get the past the ads and the popup form asking for their email address?

gwd(10000) 5 days ago [-]

It's surprising how apparently-useless information can give people a lot of insight. In 'Guns, Germs, and Steel' and 'Collapse' for instance, there were people who dug through the latrines of ancient societies to determine that the early Easter Islanders' diet consisted of 25% porpoises; and that the Greenland Norse colony didn't eat any fish. Looking at the receipts of medicines bought in the court of Henry VIII allowed people to conclude fairly conclusively that he did not have syphilis.

The thing is, it's really impossible to predict what information will and will not be a critical key to unlocking understanding of future generations. Just keeping it all, history, comments, and all, will be a huge boon to future historiographers trying to figure out what developing in the early 21st century was like.

saagarjha(10000) 6 days ago [-]

I'd consider Linux to be the culmination of some of humanity's greatest achievements. Certainly I'd prefer they found that instead of the average website on the internet.

sebazzz(3283) 6 days ago [-]

Only to find a packages.json file without a packages-lock.json file checked in, to make it impossible to restore the node_modules directory.

nickjj(2568) 6 days ago [-]

Maybe using tabs instead of spaces will be punishable by death in 10,000 years and a number of people will be held accountable for accidentally having git repos on their computing devices that contain primarily tabs (Makefiles get an exception of course!).

znpy(1634) 5 days ago [-]

future generation that will rebuild civilization will find enough code to enjoy teledildonics once again

wewake(3637) 5 days ago [-]

Software/code rots if not touched for months. This is such a fruitless effort but does get Github some good PR. They should instead preserve ideas that make all software happen.

dgl(4133) 5 days ago [-]

I find the assumption that software rots after months a sad thing. Most of that is poorly versioned libraries that change quickly. Something like the Go 1 compatibility promise means code written just against core Go should be fine if someone has the latest (maybe last) version of Go 1.

Also archeologists study rot.

krschultz(2330) 6 days ago [-]

I was really hoping this would simply mean enabling a 'deprecated' flag for a repo.

julianmetcalf(10000) 6 days ago [-]

Hey there, we have functionality in GitHub where you can archive a repo. Go here to learn how: https://help.github.com/en/github/creating-cloning-and-archi...

saghm(4181) 6 days ago [-]

I might be wrong, but isn't there already a way to archive a repo in the sense of marking it as deprecated and read-only?

stblack(10000) 6 days ago [-]

Conceptual acid test: Go to https://archiveprogram.github.com, then disable all CSS.

Observe: resultant web page is unusable.

MauranKilom(10000) 6 days ago [-]

See the above comment tree: https://news.ycombinator.com/item?id=21528446

I would consider it pretty obvious that the permanence of usability/readability for archiveprogram.github.com is orthogonal to the actual goal of this project. Do you think future generations will learn about the archive from archiveprogram.github.com?

liquidise(3651) 6 days ago [-]

Not sure if OP meant JS instead. When i disable JS in Firefox i get a dark grey page visibly lacking any content.

hollerith(3328) 6 days ago [-]

How would I disable CSS in Chrome?

sdinsn(10000) 6 days ago [-]

Solution: just don't disable CSS.

euske(4051) 6 days ago [-]

I'm skeptical of the idea of static archives, especially when it comes to code. I think software is pretty much a living creature - with all of its environments and contexts and baggage. The only feasible way to survive it is to keep changing it. When it's no longer updated, we should pay a respect and bury them.

Plus, people are going to reinvent the wheel no matter what.

reikonomusha(2428) 6 days ago [-]

Common Lisp code begs to differ. It's not perfect but code that's half a century old (Lisp that's not even Common Lisp!) can run nearly untouched. It's a huge reason I write it.

gwd(10000) 5 days ago [-]

> I'm skeptical of the idea of static archives, especially when it comes to code.

I don't think anyone is going to be hoping to use this code for their own purposes, but rather to study programmer culture and development. For instance, when did the idea of 'immutable by default' start, and how did it spread from esoteric academic languages into mainstream languages like Rust? How did the decades-long specter of Perl 6 affect the course of Perl 5, and how did the rename to Raku lead to its resurgence (or lack thereof)?

I'm no expert in history, but I have read a reasonable amount; and what's often fascinating is when there are two groups who think that they are in complete disagreement, but in fact share a common assumption which you, from the perspective hundreds of years in the future, do not share.

There are things that are so obvious to us now that we don't bother writing them down or explaining them, which will be a complete mystery to people 1000 years hence; and it's basically impossible to imagine what those might be.

chx(812) 5 days ago [-]

How awfully convenient the news that github is literally putting an archive on ice breaks swallowing search traffic about how Github workers are resigning because they want the company to break their contract with the ICE...

dwoozle(10000) 5 days ago [-]

Oh please, if you want to change border policy use the political process, not harangue a version control software company.

snak(10000) 5 days ago [-]

Icee what you did there.

Historical Discussions: How Containers Work: Overlayfs (November 18, 2019: 232 points)

(519) How Containers Work: Overlayfs

519 points about 22 hours ago by saranshk in 2523rd position

jvns.ca | Estimated reading time – 6 minutes | comments | anchor

I wrote a comic about overlay filesystems for a potential future container zine this morning, and then I got excited about the topic and wanted to write a blog post with more details. Here's the comic, to start out:

container images are big

Container images can be pretty big (though some are really small, like alpine linux is 2.5MB). Ubuntu 16.04 is about 27MB, and the Anaconda Python distribution is 800MB to 1.5GB.

Every container you start with an image starts out with the same blank slate, as if it made a copy of the image just for that container to use. But for big container images, like that 800MB Anaconda image, making a copy would be both a waste of disk space and pretty slow. So Docker doesn't make copies – instead it uses an overlay.

how overlays work

Overlay filesystems, also known as "union filesystems" or "union mounts" let you mount a filesystem using 2 directories: a "lower" directory, and an "upper" directory.


  • the lower directory of the filesystem is read-only
  • the upper directory of the filesystem can be both read to and written from

When a process reads a file, the overlayfs filesystem driver looks in the upper directory and reads the file from there if it's present. Otherwise, it looks in the lower directory.

When a process writes a file, overlayfs will just write it to the upper directory.

let's make an overlay with mount!

That was all a little abstract, so let's make an overlay filesystem and try it out! This is just going to have a few files in it: I'll make upper and lower directories, and a merged directory to mount the combined filesystem into:

$ mkdir upper lower merged work
$ echo 'I'm from lower!' > lower/in_lower.txt 
$ echo 'I'm from upper!' > upper/in_upper.txt
$ # `in_both` is in both directories
$ echo 'I'm from lower!' > lower/in_both.txt 
$ echo 'I'm from upper!' > upper/in_both.txt 

Combining the upper and lower directories is pretty easy: we can just do it with mount!

$ sudo mount -t overlay overlay 
    -o lowerdir=/home/bork/test/lower,upperdir=/home/bork/test/upper,workdir=/home/bork/test/work 

There's was an extremely annoying error message I kept getting while doing this, that said mount: /home/bork/test/merged: special device overlay does not exist.. This message is a lie, and actually just means that one of the directories I specified was missing (I'd written ~/test/merged but it wasn't being expanded).

Okay, let's try to read one of the files from the overlay filesystem! The file in_both.txt exists in both lower/ and upper/, so it should read the file from the upper/ directory.

$ cat merged/in_both.txt 
'I'm from upper!

It worked!

And the contents of our directories are what we'd expect:

find lower/ upper/ merged/

what happens when you create a new file?

$ echo 'new file' > merged/new_file
$ ls -l */new_file 
-rw-r--r-- 1 bork bork 9 Nov 18 14:24 merged/new_file
-rw-r--r-- 1 bork bork 9 Nov 18 14:24 upper/new_file

That makes sense, the new file gets created in the upper directory.

what happens when you delete a file?

Reads and writes seem pretty straightforward. But what happens with deletes? Let's do it!

$ rm merged/in_both.txt

What happened? Let's look with ls:

ls -l upper/in_both.txt  lower/lower1.txt  merged/lower1.txt
ls: cannot access 'merged/in_both.txt': No such file or directory
-rw-r--r-- 1 bork bork    6 Nov 18 14:09 lower/in_both.txt
c--------- 1 root root 0, 0 Nov 18 14:19 upper/in_both.txt


  • in_both.txt is still in the lower directory, and it's unchanged
  • it's not in the merged directory. So far this is all what we expected.
  • But what happened in upper is a little strange: there's a file called upper/in_both.txt, but it's a... character device? I guess this is how the overlayfs driver represents a file being deleted.

What happens if we try to copy this weird character device file?

$ sudo cp upper/in_both.txt upper/in_lower.txt
cp: cannot open 'upper/in_both.txt' for reading: No such device or address

Okay, that seems reasonable, being able to copy this weird deletion signal file doesn't really make sense.

you can mount multiple "lower" directories

Docker images are often composed of like 25 "layers". Overlayfs supports having multiple lower directories, so you can run

mount -t overlay overlay
      -o lowerdir:/dir1:/dir2:/dir3:...:/dir25,upperdir=...

So I assume that's how containers with many Docker layers work, it just unpacks each layer into a separate directory and then asks overlayfs to combine them all together together with an empty upper directory that the container will write its changes to it.

docker can also use btrfs snapshots

Right now I'm using ext4, and Docker uses overlayfs snapshots to run containers. But I used to use btrfs, and then Docker would use btrfs copy-on-write snapshots instead. (Here's a list of when Docker uses which storage drivers)

Using btrfs snapshots this way had some interesting consequences – at some point last year I was running hundreds of short-lived Docker containers on my laptop, and this resulted in me running out of btrfs metadata space (like this person). This was really confusing because I'd never heard of btrfs metadata before and it was tricky to figure out how to clean up my filesystem so I could run Docker containers again. (this docker github issue describes a similar problem with Docker and btrfs)

it's fun to try out container features in a simple way!

I think containers often seem like they're doing "complicated" things and I think it's fun to break them down like this – you can just run one mount incantation without actually doing anything else related to containers at all and see how overlays work!

All Comments: [-] | anchor

marmaduke(4208) about 21 hours ago [-]

A lot of utility in Docker comes from incremental (cached) builds based on overlay but in fact you can get it from any CoW system such as LVM/ZFS/BtrFS snapshots.

flas9sd(10000) about 20 hours ago [-]

fyi, btrfs as /var/lib/docker/btrfs has a disadvantage under certain circumstances: running commands (du,rsync,..) on the subvolume folders altering the access time will incur a storage cost if the filesystem is mounted without noatime option: https://github.com/moby/moby/issues/39815 - not sure if it applies to ZFS as well (see https://lwn.net/Articles/499648/)

seabrookmx(10000) about 20 hours ago [-]

Docker defaults to overlayfs but you don't have to use it. If you use ZFS or another storage driver it will leverage their capabilities to provide the same functionality: https://docs.docker.com/storage/storagedriver/select-storage...

curt15(3963) about 19 hours ago [-]

Facebook, for one, uses Btrfs for its containers: https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebo...

jzl(10000) about 13 hours ago [-]

In fact, this is explained in the article which also has an interesting anecdote about btrfs.

dfox(3229) about 19 hours ago [-]

I somehow feel compelled to point out that this idea of union/overlay FS layers has nothing to do with containers per se. But on the other hand is somehow critical for why containers got popular as that is the way to make the whole thing somehow efficient both in terms of hardware resources and developer time.

mikepurvis(4220) about 16 hours ago [-]

They really don't, and it was funny that period where you'd see Dockerfiles with all the commands in a single invocation to avoid 'bloating' the resulting image with unnecessary intermediate products that ended up deleted.

Maybe it's out there and I've just missed it, but I really wish there were richer ways to build up a container FS than just the usual layers approach with sets of invocations to get from one layer to the next, especially when it's common to see invocations that mean totally different things depending on when they're run (eg 'apt update') and then special cache-busting steps like printing the current time.

I know part of this is just using a better package manager like nix, but I feel like on the docker side you could do interesting stuff like independently run steps A, B, and C against some base image X, and then create a container that's the FS of X+A+B+C, even though the original source containers were X+A, X+B, and X+C.

djsumdog(1073) about 16 hours ago [-]

True, you can use other stacking filesystems with Docker (I believe it had/has a ZFS driver at one time?) The example she shows in the comic are just about the filesystem and leave out the Docker pieces, so I'm wondering if this is just one part of a series.

seminatl(10000) about 16 hours ago [-]

Yeah the title should be 'unionfs: a kernel feature having nothing whatever to do with containers, and some ways to use it' but I guess that's too long :-) . Problem is there is not some central marketing department for Linux that can even tell us what 'containers' means. There are lots of people who think they are 'using containers' who do not use this style of mount, and there are lots of people using this style of mount who do not consider themselves container users.

fulafel(3480) about 11 hours ago [-]

In common usage it seems containers is synonymous with 'what Docker does'. Because meanwhile Docker has blurred in meaning since various other things were called Docker. Such as calling the non-native product 'Native Docker' or whatever 'Docker Enterprise' is (impossible to tell by the landing page description).

krab(4221) about 18 hours ago [-]

As I understand it, containers are just a set of concepts and kernel features put together to provide an abstraction that's not that different from virtual machines for common use cases.

nwellinghoff(10000) about 20 hours ago [-]

I recently made a system that uses overlays to provide work spaces for a complex build process. However, there is significant overhead paid for unmounting an deleting the files in the overlay after all the work is done. I was thinking about changing the system such that I allocate a partition ahead of time, write all the overlays there, and on success just blow away the partition and with it the overlays. This is kind of a pain in the ass. Can anyone suggest a method for rapidly deleting all the data generated by using 100's of overlays? Maybe BtrFS snapshots would be better? What are the pros and cons? Thank you so much and I apologize for 'anything' up front :)

rzzzt(10000) about 20 hours ago [-]

Tangentially related: are there any good solutions for shipping data with containers (for which the CoW mechanism is not suited particularly well)? Is there a 'hub' for volumes?

tlb(1201) about 19 hours ago [-]

Can you use a Linux tmpfs as the overlay? It's much faster to create/delete files, and you can simply unmount it at the end and its memory is immediately reclaimed.

viraptor(1897) about 20 hours ago [-]

I'm not sure I understand your use case well, but have you tried lvm's snapshots? That could be the simplest solution.

If you're going to try btrfs, check if your system/tools handle the space overcommit correctly. Some ways to check the available space don't really play well with snapshots. (As in, they report less space available)

the8472(4169) about 7 hours ago [-]

With recent kernels you can combine overlayfs and btrfs.

btrfs subvols/snapshosts have their own costs, they can get slow once you accumulate thousands of them (it's fine if you just use a few at a time). But you can create a single btrfs subvolume, store your overlays in there and then delete the subvolume when you're done.

Cedricgc(10000) about 21 hours ago [-]

I enjoyed this blog post. Julia does a great job of distilling an idea down with examples.

I am fairly comfortable with Linux as a user for things like understanding processes, ports, key files and utilities, etc. The way I understand how to model abstractions like containers is to know the various OS primitives like cgroups, changing root, network isolation. Once one sees how those pieces come together to create the container abstraction, they can be mapped to the system calls provided by the OS. Usually they also have utilities bundled (like `chroot`) to interface with those primitives as an operator.

whytaka(10000) about 8 hours ago [-]

I have been confused about containers for so long but having read your comment and looking up the terms you mentioned allowed me to finally find the right articles that explained containers to me. Thanks!

hackerm0nkey(4166) about 17 hours ago [-]

Great article and nice style distilling all this into a bite size chunks.

Is it me or just the title is a little bit inaccurate in the sense that there's more to 'How containers work?' than overlays, e.g. it made me think that it covers more than it actually does, e.g. cgroups, namespaces, etc...

Anyone knows of a more in depth coverage of containers building block type of article that allows one to build a rudimentary container from scratch to appreciate what goes into building one ?

skywhopper(10000) about 16 hours ago [-]

The title just means it's one piece of the puzzle. She's thinking about making a comic about how containers work, and one important piece of that is overlays. So this is that piece.

djsumdog(1073) about 16 hours ago [-]

Yea, it did a great job of covering overlays, but didn't get into how Docker uses a hash value for each overlay piece. Maybe this will be part of a series where she does more of that?

This was posted a few months back on here and it a cool little tools for seeing how Docker fits layers together:


2019119(10000) about 6 hours ago [-]

I wrote this script[1] a while ago which creates an overlay and chroots into the overlays workdir. It's pretty useful, with it, you can do something like

> overlay-here

> make install (or ./wierd-application or 'npm install' or whatever)

> exit

and all changes made to the filesystem in that shell is written to the upperdir instead. So for example in the above example, the upperdir would contain files such as upperdir/usr/bin/app and upperdir/usr/include/header.h.

It's useful when

* You want to create a deb/rpm/tgz package, but a makefile does not have support for DESTDIR

* An application writes configuration files somewhere, but you don't know where

* An application would pollute your filesystem by writing files in random places, and you want to keep this application local to 'this directory'

* In general when you want to know 'which files does this application write to' without resorting to strace

* or when you want to isolate writes, but not reads, that an application does

[1]: https://gist.github.com/dbeecham/183c122059f7ba288397e8c3320...

ChrisSD(4082) about 4 hours ago [-]

I'd be wary of that last point depending on what you mean by 'isolate'. Chroot is not a security feature so the isolation is not perfect. This shouldn't matter if you trust the application but if it could be malicious (or manipulated by something malicious) then you'd want a harder boundary. `pivot_root` perhaps?

xyzzy_plugh(3980) about 5 hours ago [-]

Debian's schroot was made to do pretty much this, though largely obviated by modern container runtimes.

ZoomZoomZoom(10000) about 8 hours ago [-]

There was a practice of using MergerFS/OverlayFS for pooling multiple drives (often by SnapRAID users), but what's still missing (to my knowledge) is some sort of a balancing layer, that could distribute writes.

I got this idea many years ago, when first personal cloud storages appeared and offered some limited space for free. I thought it would be nice if I could pool them and fill them taking their different capacities into account. And if I could also stripe and EC them for higher availability...

I still wonder if there's something that can do this and if there isn't I would like to know why, since it looks like a useful and obvious idea.

_trapexit(10000) about 4 hours ago [-]

What do you mean by 'distribute writes'? One of mergerfs' primary features has always been it's policies which provide different algorithms for choosing a branch to apply a particular function to.


asdfaoeu(10000) about 7 hours ago [-]

aufs sort of could do that at the file layer. The issue is you run into a bunch of incompatibilities with how applications can expect it to work. As soon as you want to start looking at striping and EC then you really need to just go with something like ZFS or btrfs.

pcr910303(969) about 18 hours ago [-]

Hmm... how is Overlayfs and Unionfs different? From the explanation I can't find any differences...

Unionfs: A Stackable Unification File System[0]:

> This project builds a stackable unification file system, which can appear to merge the contents of several directories (branches), while keeping their physical content separate.

> Unionfs allows any mix of read-only and read-write branches, as well as insertion and deletion of branches anywhere in the fan-out.

> To maintain unix semantics, Unionfs handles elimination of duplicates, partial-error conditions, and more.

If it is the same thing (but maybe more maintained or has more features...) , can we implement something like trip[1][2] package manager on top of Overlayfs?

(Porg is a package manager where all files that are installed by make install is tracked and mounted on a Unionfs layer.)

[0] http://unionfs.filesystems.org

[1] https://github.com/grencez/trip

[2] http://www.linuxfromscratch.org/hints/downloads/files/packag...

loeg(3808) about 16 hours ago [-]

Note that there is also a BSD UnionFS filesystem doing much the same thing. I don't know what relation it has to the Linux OverlayFS, only that they (superficially) do very similar things.

tyingq(4179) about 18 hours ago [-]

They are very similar. Overlayfs is in the default kernel tree, which would be the biggest difference.

Jasper_(3452) about 18 hours ago [-]

They're mostly the same, with different answers to tricky questions. e.g. if I stack filesystems A, B, C, and have the same file /foo/bar in all of the layers, and then do rm /foo/bar, what happens:

1. Does /foo/bar get removed from the topmost layer, exposing the one below?

2. Does /foo/bar get removed from all three layers?

3. Does /foo/bar get replaced with a 'tombstone' record to pretend that it was deleted, while still appearing in some of A, B, or C on its own?

These semantics are tricky to get right, and during the process of upstreaming unionfs to the kernel, they made some incompatible changes to the model and chose different answers for these questions, and as a result, renamed it overlayfs.

Historical Discussions: Supreme Court to Hear Google-Oracle Copyright Fight (November 15, 2019: 518 points)

(518) Supreme Court to Hear Google-Oracle Copyright Fight

518 points 4 days ago by nwrk in 628th position

www.axios.com | Estimated reading time – 3 minutes | comments | anchor

  • Police told protesters to put down their weapons and surrender or face a bombardment of tear gas and rubber bullets, the New York Times reports.
  • 'At least' 116 people were wounded in the campus clashes, medical officials said, per the NYT.

What they're saying: Police threatened to use live bullets against dissidents on Sunday, after protesters shot arrows and threw petrol bombs at them outside the university. The police said an officer was shot in the leg with an arrow.

Why it matters: The past week has seen some of the bloodiest clashes between police and protesters since the massive pro-democracy demonstrations began in June. Schools 'have become a driver of the city's uprising against China's ruling party,' the Wall Street Journal notes.

  • More than a third of the 4,000 protesters arrested are ages 21 and younger, per police records, the youngest of whom is 11, per the WSJ.
  • Beijing blames the school system for failing to impart a strong sense of Chinese national identity to Hong Kong's young people. It's written new new education guidelines with 'the goal to build a stronger national identity for students in Hong Kong and Macau,' WSJ notes.
  • More than 390 high schools, about 80% of which are secondary schools, established "concern groups" to organize protests. Educators fear a curriculum push for Communist Party ideology could be re-upped in classrooms, per WSJ.

The big picture: Five months of unrest intensified during rush hour last Monday morning when police opened fire on protesters, injuring a 21-year-old man. There have been daily daily demonstrations lasting from the morning into the night since then.

  • Chinese army troops stationed in the semiautonomous territory cleared streets on Saturday, which protesters clogged with debris to slow down police. An official said the Chinese army operation was a 'voluntary community activity,' per Reuters.
  • The protest focus has since shifted to the university.
  • Students threw petrol bombs to stop police from storming the campus over the weekend, per the Times.

Background: Authorities hoped the October withdrawal of an extradition bill that triggered the city's protests would quell the unrest.

  • However, protesters are concerned China may suppress the high degree of autonomy they've enjoyed since the former British colony was returned to the country in 1997.

Go deeper:

Editor's note: This article has been updated with new details throughout.

All Comments: [-] | anchor

gpm(10000) 4 days ago [-]

Notes on scheduling for anyone who wants to follow along:

Google now has 45 days to file a brief on the merits that explains their position.

Once that is filed, Oracle has 30 days to file a brief on the merits that explains their position.

Google then has 30 more days to file a reply to Oracle's brief. That brings us to the 28th of February, assuming everyone uses all their time (and no more).

The court can extend all those deadlines.

Once all the briefs have been filed the case will (probably) be scheduled for oral argument. It looks like oral argument is usually scheduled several months out, and the last day for oral argument this term is April 29. It might meet that deadline, otherwise it will be pushed to next October. If the oral argument is heard this term then we can expect a ruling by the time the court goes into recess for the year (end of june).

Of course a ruling doesn't mean the case is over, it may well then return to lower courts for more argument. (It almost certainly will for various details, like attorney's fees and/or damages).

This case started August 13, 2010. It's been almost a decade. Something is very wrong with how our court system functions.

zucker42(10000) 3 days ago [-]

> This case started August 13, 2010. It's been almost a decade. Something is very wrong with how our court system functions.

For civil cases between large companies, this speed doesn't seem entirely terrible. Criminal cases and civil cases between individuals are another story.

kijin(4052) 3 days ago [-]

A decade is obviously too long, but there is some value in taking a long time to decide important matters. If you rush everything in a short time span, you risk getting carried away by the public opinion and political environment of the moment. This can be especially problematic when it comes to the kind of deep constitutional issues that we expect the Supreme Court to grapple with; the court needs to keep a certain distance from the propaganda du jour. Besides, it's not uncommon for people on the death row to dig up evidence that exonerates them many years after the case was deemed closed. If we sped up the whole legal system and carried out sentences asap, they might not have been given enough time to do so.

Meanwhile, the system can be surprisingly agile if it needs to be. New York Times Co. v. United states only took 12 days from the first hearing to the Supreme Court ruling! Of course that was because Nixon wanted to rush the case, but in the end he lost hard and the heat of public opinion probably didn't help, either. In the case of Google v. Oracle, nobody seems to be particularly in a hurry. Both sides can afford to drag out the dispute as long as they want to.

anon1m0us(4213) 4 days ago [-]

It is property law and the ability to protect one's rights to property that has granted the creators of property a return on their investment of resources into property.

If creators of property cannot protect their property, there will be less return on that property and thus, less incentive to create it.

Oracle bought Sun, and with that purchase, the ownership of intellectual property in the form of the Java API. Google then proceeded to copy that property into their own property without abiding by the usage restrictions Oracle, the property owner, specified. That was theft of Oracle's property.

If the Supreme Court does not uphold protections of property, I believe, we will see less investment in such property.

I am not sure if that is good or bad for the future of the world, but I do believe had Sun not had property rights to Java, they wouldn't have created it.

nwallin(10000) 4 days ago [-]

> Oracle bought Sun, and with that purchase, the ownership of intellectual property in the form of the Java API. Google then proceeded to copy that property into their own property without abiding by the usage restrictions Oracle, the property owner, specified.

You have the order backward. Google made Android with the full consent from Sun. Their CEOs had a handshake deal, and neither of them believed there was any need for a legal contact.

Fourish years later Oracle bought Sun and promptly sued Google. The handshake deal wasn't legally binding, apparently.

I know this doesn't necessarily change the legal picture, but it certainly changes the ethical one. Your timeline makes Google out to be a predator, maliciously stealing Oracle's precious IP. The actual timeline makes Oracle out to be the predator, buying Sun for little other purpose than to sue people.

I suspect Google wouldn't have gone with Java if it were owned by Oracle at the time. Which would have made for an interesting alternate timeline. Maybe they'd have gone with D? Who knows. C++03 was kind of ass, but if they'd have gone with c++ we'd have new c++ as a first class citizen which would also be rad.

pvelagal(10000) 4 days ago [-]

How about similarities of C++ and Java ? Who copies whom ?

int, float, char key words ? if - then - else, while loop, for loop, curly braces { } comments // and /* main() method ? socket API : socket, connect, listen, accept ? and So many keywords similar to C++ ? (break, continue, char, double, new, public, private, return, static etc.)

Java benefitted from Collective knowledge of Computer Science and design patterns that were invented outside of Sun microsystems.

The JVM is written in C++ ? Can we cay JVM APIs are wrappers around C++ API ? Do they pay Bjarne Stroustrop ?

erikpukinskis(3077) 4 days ago [-]

What property has Google destroyed that Oracle is trying to protect?

Owning a property doesn't give you the right to sue other people who develop a property that offers the same services.

wtallis(4124) 4 days ago [-]

You seem to be trying really hard to give the impression that copyright law is a special case of common law property rights. The US Constitution plainly disagrees with you on that.

vessenes(3890) 4 days ago [-]

I'd say almost every software engineer should be rooting for Google here. The implications of being able to claim copyright infringement on anyone implementing an Application Programming Interface are staggering - it would impact every open source project that tried to interoperate with any company's services, for instance.

The world is not going to be better for extending the protections Oracle wants here.

Alsup's ruling is sane, shows his clear understanding of coding and the history of Java and how it was licensed out to the world under Sun, and is really very simple to understand: http://www.groklaw.net/articlebasic.php?story=20120531172522...

The appeals court fucked this up, hard. I would like to think Alsup's ruling will be upheld -- groklaw has fantastic quotes from him during the trial. At one point, he himself notes he had coded from the spec some of the functions Oracle complained about and disagreed with Oracle counsel statements. Refreshing from our judicial branch, to say the least.

MrGman(10000) 3 days ago [-]

Can you explain why you think 'the appeals court fucked this up, hard.'?

Isn't the way the law is written the real root cause here? Judges and courts don't write laws...

Basic questions pop up (hey look 'pop' that's an API) in my mind like: is copyright on code a thing that should exist?

yarg(10000) 4 days ago [-]

Seriously people - read the comment before down-voting it. A flood of down-votes coming in in less time than it takes to read what I wrote implies that you aren't even considering the argument I'm presenting.

I hate to say it, but I'm with Oracle on this - but not for any reason that I've seen mentioned (so I guess I want neither party to win).

(Notwithstanding Sun's internal excitement at Google's use of Java prior to the Oracle acquisition.)

There is a justification on the grounds of obviousness for certain method headers to be uncopyrightable.

But for inobvious and complicated method headers I do not believe that the restriction should prevent the API from being subject to copyright, but rather to limit the applicability of the copyright for fair use purposes.

Specifically I believe the copyrights on such interfaces should not be applicable where interoperability is the justification for the reuse of the interface.

Google broke compatibility with Java. I do not believe that Google is using the API under fair use constraints.

I'm not sure how the law operates here - but I believe that the precedent set by a Google victory would be more damaging to my position. I'd like to see Google lose, and for someone later - operating with cross-compatibility in mind - to present and win a fair use case for reuse of an API.

rasz(10000) 3 days ago [-]

IBM should be warming up their lawyers to sue Oracle for SQL.

pjmlp(298) 3 days ago [-]

The current state of Java support on Android proves that Google has achieved their own variant of J++, forcing Java library authors to write Android Java, if they wish to support the platform.

'James Gosling Triangulation's Interview on Google vs. Sun'


Google should face the same penalty as Microsoft did, plain and simple.

stopadvertising(10000) 3 days ago [-]

Why does it matter? The API is as dead as the telegraph, everyone is just building garden walls higher instead.

I worked on and with lots of APIs back in the day. They are all gone. All of them. Eliminated for business reasons. The internet has become a tool for carving out fiefdoms of engagement to be monetized, it's not about connecting people or systems anymore.

kpU8efre7r(4158) 4 days ago [-]

Would this open the door for some schmucko to copyright push, pop, enqueue, dequeue, etc?

psaux(4213) 3 days ago [-]

Did you read the content? Google did copy the private API's. I am a legacy Java person and it is blatant theft. Not rooting for Oracle in any sense, but it was a total copy.

smokinman(10000) 2 days ago [-]

Yeah, fancy a scumbag using the legal system to hold people to ransom with their worthless and bogus claim.

Sounds a bit like what Peter Vessenes is doing to 25,000 creditors of MtGox, even after being offered an incredibly generous $10 million dollar kiss off.

snarf21(10000) 3 days ago [-]

Exactly this. Oracle is nothing but a common patent troll here. They bought SUN for one and only one purpose..... to sue Google. I know someone involved in the preliminary investigations into this. It is not a secret at all. It would be awesome if Google would take the Cloudflare approach and try to crowdsource prior art for all of Oracle's patents. (Best case, 1M software patents go away on all sides)

Hopefully as things continue to the cloud with private options and their loss of the JEDI contract puts them right out of business.

abvdasker(4218) 4 days ago [-]

Also, Oracle just sucks in general. Larry Ellison is a notorious jerk and philanderer. Oracle's entire business is based on the success of one expensive database product from the 70s, which is mostly still in use due to decades-long vendor lock-in of stagnant enterprise giants like SAP, IBM, most banks and a good chunk of the DoD.

lettergram(1410) 4 days ago [-]

On the contrary, I'm actually not 100% sure who's side I should be in on...

Look at it this way, if oracle loses, then what's the implications of the software licenses in general. The whole point is that they were protected by copyright. If that's shot down, then what's at stake?

Not saying I side one way or the other, simply that I don't think it's clear. Depending how the court rules one way or the other could have a huge impact (as you pointed out).

choppaface(4108) 4 days ago [-]

Rooting for fair use? Yes. Rooting for a big company spending big money to pay for the precedent? Sure.

Rooting for Google itself, who is simply pursuing the most effective solution for their own business (and brand)? That's something entirely different.

ecopoesis(3422) 4 days ago [-]

But this case isn't about consuming or interacting with an API, it's about implementing an API. So less writing a compiler targeting a CPU instruction set and more building a CPU based on someone else's instruction set.

I've always found it strange that other programmers are so dismissive of APIs as copyrightable work. They're they hard part! Building APIs requires creativity and careful thought.

If you support the idea that APIs can't be copyrighted, and ignoring the evilness of Oracle and Google, what's your reasoning?

snagglegaggle(10000) 4 days ago [-]

Issue exists with instruction sets as well. If Oracle wins everyone who has written a compiler for an instruction set they did not have explicit permission to write one for is in violation.

innagadadavida(10000) 3 days ago [-]

Google screwed over Sun Microsystems. Let's not forget that all this started before Oracle.

The original Java license mandates that mobile usage is differently licensed. Google basically writhed their own JVM just to bypass this - not for any other technical reasons.

LeftHandPath(10000) 3 days ago [-]

Honestly, they need to put it into layman's terms.

The exposed function calls of an API are meant to be implementation-blind.

Let's say car-company A builds the first ever car that's operated with a steering wheel, gas pedal, brake pedal, clutch pedal, and shifter.

Then, Company B comes along and - both to create a complete control system for the car AND to potentially benefit consumers who are already familiar with those controls - also uses a steering wheel, gas pedal, brake pedal, clutch pedal, and shifter.

The steering column and its inner workings are completely different. The gas pedal activates a V8 combustion engine instead of an I4. The clutch pedal triggers a dual clutch for faster shifting between gears. The brake pedal activates disc brakes instead of drum-brakes.

The appeals court is, in effect, saying that Company B has violated copyright because, when faced with the same problem, they came up with a solution that looks very similar on the outside. They turn a blind eye - willingly or unwillingly - to all the new engineering that went on to improve the product and create a new expression of an automobile.

pron(440) 4 days ago [-]

> it would impact every open source project that tried to interoperate with any company's services, for instance.

This is categorically untrue. The discussion is not about anything called 'API' but only certain kinds of APIs (code APIs, as opposed to protocols like REST 'APIs'), and even then there is fair use.

pacala(4193) 3 days ago [-]

Not copyrightable APIs is a fairly dangerous precedent.

1. Good APIs are hard.

2. Under Alsup's ruling, good luck competing with AWS sales org when they decide to offer your own APIs as a service.

3. Nothing prevents Google from inventing their own APIs, and licensing them under whatever terms they like.

As of Google vs. Oracle, it's pretty clear that Google appropriated Java to benefit from its penetration in the market place, instead of putting in the hard work to build their own. Apple has built their own ecosystem, ObjectiveC/Cocoa/Swift/SwiftUI. Microsoft has built their own ecosystem, C#/DotNet, or adopted and contributed to open standards, HTML5/Javascript/Typescript. Heck, even HP has built WebOS around open standards.

tinco(4097) 3 days ago [-]

This guy Peter Vessenes is holding thousands of peoples' trading accounts hostage in a frivolous lawsuit against a now-defunct exchange.

Multiple people have committed suicide because their life savings are locked up in this lawsuit. All because Vessenes here thinks he should be awarded half a billion dollars for making a half hearted attempt at opening a US based branch of it.

It makes me sick just to see your name on here. You should be ashamed of yourself, at least own it and make your comments under a pseudonym. Or do you actually feel you are somehow justified in robbing us?

martin1975(4169) 4 days ago [-]

I'm just hoping the judges see this for what it really is - Oracle's unabashed greed. Google's pales in comparison.

anon1m0us(4213) 3 days ago [-]

Google is not greedy? Oracle paid money for Sun and the property Sun owned at the time. Google said, nope, we'll just copy that property for ourselves and keep our money. That seems greedier.

gorgoiler(10000) 4 days ago [-]

Thinking from first principles, and forgive me for going into hippie philosophical mode here but I can't see how an API can have copyright that is owned.

Creating an API is of course a creative process. Getting the API right is, for me, the most creative part of programming. When well written, it describes to a human how the software truly works, with the implementation really being the computer version of what the function name is already telling you. A browser rendering engineer could be defined just as much by the model that the DOM API describes as it is defined by the actual source code. They both describe the outside and inside of the same thing, a system which, when it's interface is shared, is shared between us all and not just under the sole ownership of the first person to describe it.

It's hard to see an API therefore, particularly a published one, as a traditional piece of intellectual property that can be subject to copyright. One cannot copyright F=ma or e=mc^2. Once these object models about software are discovered they are like descriptions of the natural world and their formulae are open and shared for all to use.

Yes: how you build your specific machine that makes use of and conforms to the API — the actual source code for function implementations — can be your intellectual property, but the underlying description of the natural world and its API are discoveries that are part of the commons, for all to interact with, use, and re-use.

The only difference between natural laws, in my analogy to physics, and software APIs is that there is only one physical world with a limited set of natural laws that describe it. With software engineering we create our own new universes everyday, but they are common universes for us all to share.

m1sta_(10000) 4 days ago [-]

It needs to be its own unique class of IP. The intersection between industrial design, spare parts, and patents deal with some similar issues.

jfasi(3947) 4 days ago [-]

I hate to say it, but I'm not optimistic about Google's case here. From a purely technical point of view, APIs being free to reuse is an awesome thing that makes for a more vibrant and competitive software ecosystem.

At the same time, Oracle's characterization of their API as "original software" is not entirely off base, as anyone who has spent time and energy creating and API would know. The amount of design and work required to create an elegant and useful API is huge, and while it would irreparably harm software as a field to call it copyrightable, calling it anything other than an "original" work is a weak position.

Personally, I'm dreading the outcome of this case.

monocasa(2855) 4 days ago [-]

Taking time and energy doesn't define copyright. There's tons of acts that take time and energy, and don't grant you a century long government backed monopoly on anything almost like it.

For instance, recipes, and tables of contents aren't copyrightable.

Going into case law, Sony v Bleem made it pretty clear that clean room reimplementations of APIs are on the table.

Going into US code, the otherwise crappy DMCA explicitly allows reverse engineering for interoperability. ie. interoperability even when the original vendor won't even tell you what the API is.

Going into practicalities, who owns SQL? Who owns POSIX?

The entire idea that APIs can have copyright is blatantly in contrast to decades of law, and is only happening because the CAFC is going off on it's own and ignoring 9th circuit precedent.

burtonator(2084) 3 days ago [-]

If Oracle wins the Open Source community should adopt a new license to punitively punish companies like Oracle who abuse this system.

We can adapt our license to allow anyone to use the API except if they're using API licensing themselves, at which point they would be in violation.

eqvinox(10000) 4 days ago [-]

> From a purely technical point of view,

The technical point of view needs to be combined with the legal one here, and —

> At the same time, Oracle's characterization of their API as "original software" is not entirely off base, as anyone who has spent time and energy creating and API would know. The amount of design and work required to create an elegant and useful API is huge, and while it would irreparably harm software as a field to call it copyrightable, calling it anything other than an "original" work is a weak position.

just because something takes a lot of work, that doesn't automatically mean it is copyrightable.

An API doesn't implement anything, it describes a function. Even if it describes a lot of functions and they mesh together really well, there's a distinction between 'what' a program does and 'how' a program does something.

To compare with literature copyright, there are a ton of romance novels out there — pretty sure you can find a lot of common patterns on 'what' story they tell. However, only the 'how' is copyrighted, you can't prevent anyone from writing a story with the same outline as an existing one.

(For literature, the problem becomes really tricky since the crossing over from 'what' to 'how' is kinda fluid; e.g. you can't just swap out character names. Software is actually easier there.)

thesausageking(3977) 4 days ago [-]

Google's argument is that it falls under fair use to do a clean room implementation of the APIs. District Judge Alsup agreed with this legal idea:

'So long as the specific code used to implement a method is different, anyone is free under the Copyright Act to write his or her own code to carry out exactly the same function or specification of any methods used in the Java API. It does not matter that the declaration or method header lines are identical.'

ocdtrekkie(2741) 4 days ago [-]

My understanding is that this case has produced emails inside Google about essentially how to avoid licensing Java. In the light of the incredible profit machine Google is, and how large a monopoly Android is, its hard to imagine any judge looking favorably on a plan to avoid paying licensing for something to build a multi-billion dollar industry.

That being said, I feel ruling against Oracle would be also very perilous for open software from for profit entities, as it would have a harmful chilling effect on companies trying to dual license or keep their technology open. Arguments Google had made in earlier stages used the GPL-licensed OpenJDK to justify using their non-GPL implementation.

abledon(3862) 4 days ago [-]

Wasn't there some judge a couple years back who , in order to understand a case properly, taught himself java ?

wyldfire(600) 4 days ago [-]

> From a purely technical point of view, APIs being free to reuse is an awesome thing that makes for a more vibrant and competitive software ecosystem

This is not merely a technical benefit, it's a practical one and represents the status quo. The fact that there's no precedent isn't because it's new, it's because no one ever thought it was infringing before.

jrochkind1(1999) 3 days ago [-]

In general, the amount of time/energy/cost invested in something is not a prime indicator of whether it is copyrightable, under the law. This is a misconception.

Facts are not copyrightable no matter how much time/energy/cost went into compiling them, and this is clearly established law.

Google's case is that an API specification or implementation is more like a set of facts. Which as a software engineer, seems pretty plausible to me, they do seem like a set of facts, a description of fact about how software works. On the other hand, unlike facts, they were not purely observed, but were indeed invented by humans -- but recipes aren't copyrightable either, even though they are not observed but invented too. Neither are the rules of a game. These are all considered more like 'facts' than creative works. Doesn't matter if eg you spent years and millions of dollars researching food chemistry to make your recipe.

I don't think its entirely clear who will win, but I don't think Google's case is as weak as you think, although i agree that Oracle's contention isn't entirely off base. . In particular, in general, how much time and energy went into making something is not generally one of the most significant factors in determining copyright or fair use. (Additionally, the well-established law around reverse engineering and creating clones -- that it is allowed -- is in Google's favor, as that analogy seems pretty strong too). Copyright law is -- has always been, or at least for 100 years -- about a bunch of competing factors balanced against each other.

And when it comes to technology advances, has always relied on analogies to previous technologies and industries, and who has the most persuasive analogy. And then we have the fact that the people deciding what analogy applies best may not entirely understand the technology as a social fact...

jcranmer(10000) 4 days ago [-]

If you read the amici briefs, some of the people who actually wrote that API are explicitly disagreeing with Oracle's position.

drallison(1663) 2 days ago [-]

The possibility that Oracle might prevail keeps me awake some nights.

Oracle's 'original software' is not. It is derivative of other work, for example, Smalltalk and its libraries. The idea of including explicit interface declarations with each module was, I think, introduced by the programming language SUE designed by Rick Holt.

Copyright does not protect ideas; copyright can only protect the expression. And when the two are intrinsically combined, there cannot be copyright.

danShumway(4077) 4 days ago [-]

We have precedent that game mechanics can't be copyrighted -- they get classified as 'inventions' and have to be patented instead.

Obviously IANAL, but to me as a game designer, mechanics aren't any less creative work than narrative. In fact, I'm spending more of my creative energy on mechanics than I am on story. So the lines to me just seem incredibly arbitrary, or at least I don't understand the legal differences well enough to figure out intuitively where they lie. I am incredibly grateful that game mechanics can't be copyrighted, but game mechanics don't feel like inventions to me. A game mechanic is how I express an idea.

I tried to make a prediction about which way this would go, and I genuinely don't know -- not even that my prediction is uncertain, I don't feel like I know enough to even make a prediction at all.

It does make me nervous. I think it's important that the Supreme Court hear it, and I'm glad they agreed to, but it would be utterly disastrous if this got decided in Oracle's favor. My (perhaps incorrect) impression is that the Supreme Court is not particularly fond of the 9th, and have something of a history of slapping down attempts at copyright expansion. A ruling against Oracle would be fantastic, and would maybe even open the door for talking about blocking copyright on grounds of compatibility.

I guess I'm just nervous because it feels like the stakes are really high.

At this point, there's nothing really that people like me can do, right? It's just up to Oracle and Google's lawyers?

bborud(3862) 3 days ago [-]

I don't agree with your argument.

I agree that designing good APIs requires a great deal of work, but Sun/Oracle were not alone in doing this work. In fact, they had considerable help from community, competitors and customers.

Java APIs are to a great degree a collaborative effort and it isn't right that Oracle should be the sole benefactor of this uncompensated work that has only made their product more valuable.

If this is true for these particular APIs isn't very relevant in my view. What is relevant is that the Java platform as a whole has gained much from its community. Without the Java platform the APIs would hardly have any relevance at all.

zmmmmm(10000) 3 days ago [-]

I don't think it's actually about that.

APIs are facts, and facts are not copyrightable. How is an API a fact? You have a system, in the real world. It is a 'thing'. There are facts about this thing: if you send it particular bytes, it does X, if you send other bytes, it does Y. The fact that it does this is empirically derived. There are not two or more options for it, it's like gravity or electromagnetism. APIs are facts about the systems they apply to, so while there is creativity in designing them, there is no creativity in building a system that interacts with them. There is exactly one way derived from the empirical fact of how it works.

Where the case IS weak is that there IS creativity in how you document and organise the API. And I'm pretty sure Google's re-implementation was very similarly organised to the real Java. That will be the crack Oracle will be trying to exploit here. But its not actually about the original work being 'original' or even 'creative'.

jhanschoo(10000) 3 days ago [-]

In the case that Oracle wins solely on Structure, Sequence, and Organization, I don't see how such a ruling immediately spells doom for competitors trying to interoperate.

Competitors can still publish a functionally identical but differently organized API, then provide a competitor2us.sh script to statically change references to their own API. Sure, the friction is higher, but not unreasonable.

alok-g(2731) 2 days ago [-]

Wow. This makes sense.

tehjoker(10000) 4 days ago [-]

This case clearly shows how taking something from the public and making is private is essentially a greedy and harmful activity. Unfortunately, our economy is based on this idea.... it doesn't have to be though.

anon1m0us(4213) 3 days ago [-]

They didn't take it from the public any more than Led Zeppelin took their songs from the public when they sued those who violated their IP rights.

Software has a real and tangible value. Songs ... well, you decide their value, but I don't understand why the value songs create should be protectable, but software not.

That's really what this is about. This is about granting to those who write software the same rights to that property as anyone else who has rights to what they write.

AnimalMuppet(3739) 4 days ago [-]

Good. In my view, the existing ruling desperately needs to be overturned.

And I'm cautiously optimistic. The Supreme Court has shown itself to be far more sane on IP than the Federal Circuit.

belorn(4179) 4 days ago [-]

In heard a good explanation for that in a talk by law professor Eben Moglen.

Congress passed in 1982 a statue called the Federal Courts Improvement Act, where instead of having a random judge make decision, you now instead have selected judges specialized in IP. Those specialized judges comes from lawyers who have worked for companies protection their IP, and thus a bias in favor of IP in the court of appeal. When the case goes to the supreme court this bias goes away.

NotMelNoGuitars(10000) 4 days ago [-]

If you don't mind expanding, interested to hear why you believe the current ruling should be overturned.

m463(10000) 4 days ago [-]

I think APIs open for interoperability is better for society.

As an example, the Z80 and the 8080 were interoperable and it was better for everyone.

meddlepal(10000) 4 days ago [-]

What's better for society doesn't matter in the face of the Constitution which is what the SCOTUS will judge this on.

runn1ng(2176) 3 days ago [-]

I am mostly surprised this is still ongoing... isn't it going for around 10 years already?

aiCeivi9(10000) 3 days ago [-]

SCO took even longer: https://en.wikipedia.org/wiki/SCO%E2%80%93Linux_disputes . It stops only when one side can no longer pay the lawyers :/.

crazygringo(3841) 4 days ago [-]

It's ridiculous this is being decided by the courts instead of legislatively.

Fair use doctrine was obviously never intended to apply to reimplementing API's either way because it didn't exist yet.

Rather than have a court make up some kind of ultimately arbitrary precedent ruling either way, Congress should be debating the ramifications of whether reimplementing API's is explicitly fair use or not, considering both pros and cons to the economy, with opportunity for all tech companies to weigh in -- and then pass a good law.

Courts interpret law, they aren't supposed to make it, and the Supreme Court certainly isn't even remotely qualified to determine what's the best policy for a healthy dynamic tech economy here. The law is so ambiguous here that Congress is shirking its duties by not establishing relevant law here.

CSMastermind(3655) 3 days ago [-]

Congress can still pass a law after the court's decision that would change the outcome.

skybrian(1703) 4 days ago [-]

This isn't at all unusual. You might have been taught in school about the courts 'interpreting' the law, but a lot of American law was inherited from English common law which does basically come from court decisions, and a lot was created by court decisions since then.

In times like these where Congress is often deadlocked, someone needs to make decisions. Congress can pass a new law if they can get their act together.

legulere(4128) 3 days ago [-]

You can still have laws preceding APIs that are abstract enough to also apply for it well enough. In Germany for instance we have a special clause that specifically allows reverse engineering for making something compatible with another product.

byuu(10000) 4 days ago [-]

> 'We are confident the Supreme Court will ... reject Google's continuing efforts to avoid responsibility for copying Oracle's innovations.'

Serious question: which part of Oracle buying Java from Sun was innovative?

Spivak(10000) 4 days ago [-]

I feel like that's being super rude to the devs at Oracle that have been constantly innovating and improving the language. Java has changed a lot in the last 9 years.

Oracle's upcoming design of continuations I think is genuinely novel and will inspire a lot of other languages' implementations.

Stupulous(10000) 4 days ago [-]

I imagine the correct read is 'the innovations belonging to oracle', rather than 'the innovations performed by oracle'.

Historical Discussions: Pointless work meetings 'really a form of therapy' (November 15, 2019: 503 points)

(504) Pointless work meetings 'really a form of therapy'

504 points 4 days ago by DarkContinent in 1965th position

www.bbc.com | Estimated reading time – 4 minutes | comments | anchor

By Sean Coughlan BBC News family and education correspondent

Image copyright Getty Images

Meetings at work should be seen as a form of 'therapy' rather than about decision-making, say researchers.

Academics from the University of Malmo in Sweden say meetings provide an outlet for people at work to show off their status or to express frustration.

Professor Patrik Hall says they are becoming increasingly frequent - as more managerial and 'strategy' jobs generate more meetings.

But he says despite there being more meetings 'few decisions are made'.

Prof Hall has investigated an apparent contradiction in how people can have a low opinion of work meetings, yet their numbers keep increasing.

Looking for a purpose

The political scientist says the rise in meetings reflects changes in the workforce - with fewer people doing and making things and an increase in those involved in 'meetings-intense' roles such as strategists, advisers, consultants and managers.

'People don't do concrete things any more,' he says.

Instead he says there has been a rise of managerial roles, which are often not very well defined, and where 'the hierarchy is not that clear'.

'Many managers don't know what to do,' he says, and when they are 'unsure of their role', they respond by generating more meetings.

'People like to talk and it helps them find a role,' says the professor.

Many of these people can spend half of their working hours in meetings, he says.

Image copyright Getty Images
Image caption Managers uncertain about their purpose will generate more meetings

These can spill over into pre- and post-meetings, to such an extent that people might begin to 'disguise' how much time they spend attending them.

Prof Hall, who has co-authored a book on meetings, gives the example of the Swedish border police, who describe their overseas meetings as 'power weeks'.

'Opportunity to complain'

Meetings can 'arouse feelings of meaninglessness', he says. But he argues that is often missing their point.

Once in a meeting - particularly long ones - their function can become 'almost therapeutic'.

Image copyright Getty Images
Image caption Meetings - a chance to catch up on messages on the mobile

Regardless of what they are meant to be discussing, they serve a purpose as an 'opportunity to complain and be acknowledged by colleagues'.

But people going to many meetings can lose patience - and can spend much of the time playing with their mobile phones, say the researchers.

Prof Hall says as a result, meetings can become 'maligned somewhat unnecessarily'.

'Some people find this frustrating and question why they must endure them.'

But he argues that negativity towards meetings can be because their real purposes are misunderstood.

Many regular, internal meetings might seem entirely 'pointless' to those taking part, says Prof Hall.

But he says the real purpose of such meetings might be to assert the authority of an organisation, so that employees are reminded that they are part of it.

Such meetings are not really about making any decisions, he says.

'Power struggle'

Prof Hall suggests booking rooms for shorter periods, as he says meetings will expand to fill whatever time is given to them.

He also says that 'equality' of participants is important.

'When you have meetings with colleagues at the same level, as a professional, you get to discuss different issues that interest you,' he says.

When the meetings are dominated by different levels of status, they become a 'power struggle' and leave participants feeling frustrated.

He also says that meetings can unfairly become the focus of other dissatisfactions.

'People often feel marginalised. They feel that they have no influence or position. In these cases, the perception is that meetings do not improve anything, but actually cause even more frustrat