Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

July 06, 2020 01:07



Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: The End of the Redis Adventure (June 30, 2020: 2274 points)

(2276) The End of the Redis Adventure

2276 points 5 days ago by kristoff_it in 1740th position

antirez.com | Estimated reading time – 5 minutes | comments | anchor

antirez 5 days ago. 326964 views. When I started the Redis project more than ten years ago I was in one of the most exciting moments of my career. My co-founder and I had successfully launched two of the major web 2.0 services of the Italian web. In order to make them scalable we had to invent many new concepts, that were already known in the field most of the times, but we didn't know, nor we cared to check. Problem? Let's figure out a solution. We wanted to solve problems but we wanted, even more, to have fun. This was the playful environment where Redis was born. But now Redis is, incredibly, one of the main parts of so many things. And year after year my work changed from building this thing to making sure that it was also as useful as possible, as reliable as possible. And in recent years, what I do every day changed so much that most of my attention is spent in checking what other developers tell me about the Redis code, how to improve it, the changes it requires to be more correct or faster or more secure. However I never wanted to be a software maintainer. I write code in order to express myself, and I consider what I code an artifact, rather than just something useful to get things done. I would say that what I write is useful just as a side effect, but my first goal is to make something that is, in some way, beautiful. In essence, I would rather be remembered as a bad artist than a good programmer. Now I'm asked more and more, by the circumstances created by a project that became so important, to express myself less and to maintain the project more. And this is indeed exactly what Redis needs right now. But this is not what I want to do, and I stretched myself enough during the past years. So, dear Redis community, today I'm stepping back as the Redis maintainer. My new position will be, on one side, an "ideas" person at Redis Labs, in order to provide inputs for new Redis possibilities: I'll continue to be part of the Redis Labs advisory board. On the other hand however my hands will be free, and I'll do something else, that could be writing code or not, who knows, I don't want to make plans for now. However I'm very skeptical about me not writing more code in the future. It's just too much fun :D I leave Redis in the hands of the Redis community. I asked my colleagues Yossi Gottlieb and Oran Agra to continue to maintain the project starting from today: these are the people that helped me the most in recent years, and that tried hard, even when it was not "linear" to follow me in my very subjective point of views, to understand what my vision on Redis was. Since I don't want to be part of how the new Redis development setup will be shaped (that is the most meta of the maintenance tasks, exactly what I want to avoid), I'll just leave Yossi and Oran the task of understanding how to interface with the rest of the Redis developers to find a sustainable development model, you can hear directly from Yossi and Oran in this blog post: https://redislabs.com/blog/new-governance-for-redis/ I believe I'm not just leaving Redis in the hands of a community of expert programmers, but also in the hands of people who care about the legacy of the community spirit of Redis. In eleven years I hope I was able to provide a point of view that certain persons understood, about an alternative way to write software. I hope that such point of view will be taken into consideration in the evolution of Redis. Redis was the most stressful thing I did in my career, and probably also the most important. I don't like much what the underground programming world became in recent years, but even if it was not an easy journey, I had the privilege to work and interact with many great individuals. Thank you for your humanity and your help, and for what you taught me. You know who you are! I want to also say thank you to the companies and individuals inside such companies that allowed me to write open source every day for so many years, with the freedom to do what I believed to be correct for the user base. Redis Labs, VMware and Pivotal, thank you for your great help and generosity. As I said, I don't really know what there is for me in my future, other than the involvement with the Redis advisory board. I guess that for some time, just look around is a good idea, without doing too many things. I would like to explore more a few hobbies of mine. Writing blog posts is also a major thing that I wanted to do but did less and less because of time concerns. Recently I published videos in Italian language explaining technological concepts to the general public, I had fun doing that and received good feedbacks, maybe I'll do more of that as well. Anyway I guess some of you know that I'm active on Twitter as @antirez. If you are interested in what an old, strange programmer will do next, see you there. Please enable JavaScript to view the comments powered by Disqus.
rss feed | twitter | google group | old site:




All Comments: [-] | anchor

slim(4224) 5 days ago [-]

  Recently I published videos in Italian language explaining technological concepts to the general public
where are the videos ? I can't find them
blfr(2466) 5 days ago [-]

My co-founder and I had successfully launched two of the major web 2.0 services of the Italian web.

What are these services?

angott(10000) 5 days ago [-]

One of them was LLOOGG: https://github.com/antirez/lloogg, which used to be at lloogg.com but it seems like the domain was not renewed.

I remember using it on my personal website long before I had even heard of Redis, it was a nice analytics tool for small sites with little traffic.

antirez(983) 5 days ago [-]

Segnalo and Oknotizie. Segnalo was similar to delicious, Oknotizie similar to Reddit/Digg. Later in order to create the first service, LLOOGG, I had to write Redis.

antirez(983) 5 days ago [-]

Ok, so many thank you here, thanks! It's very nice to read the comments here. But I hope to interact more on HN, since basically the idea is to write more blog posts, write more OSS software too. Just totally random :D I'll just do whatever every morning I want to do for a long time. Then maybe I'll find a new long term interest.

ChrisMarshallNY(4332) 5 days ago [-]

Thanks so much!

I did have a post that basically said 'I relate,' but it wasn't popular (I guess it came across as self-promoting. My apologies).

In any case, I can relate, and completely support you in your future endeavors. I am not a redis user, but I have seen nothing but good said about it.

siculars(2441) 5 days ago [-]

Thank you antirez! I've used your remote dictionary server from the earliest days and it has been a staple of my tool chest ever since. Three cheers and enjoy your time doing whatever it is you want to do!

psankar(4235) 5 days ago [-]

You are a hero sir and thanks for all the effort that you have contributed for redis. Someday, I too hope to create a project that would touch at least a fraction of as many people as redis did. You are a big inspiration. All the best for whatever you want to do next.

WJW(4373) 5 days ago [-]

Thank you for Redis and some of the other stuff too! I had great fun reading through the small string compression code and learned a lot! Hope you have many decades of happy hacking ahead of you.

tasuki(10000) 5 days ago [-]

Thank you! And this is a very good idea:

> I'll just do whatever every morning I want to do for a long time.

sytse(1865) 5 days ago [-]

Thanks for everything you have given the community. And I look forward to more blog posts, they are always fun to read.

I also want to thank you for making this decision while you were still active, Redis didn't skip a beat. Right up to announcing this you made sure that the community made progress. As we at GitLab detail in https://about.gitlab.com/blog/2020/06/24/scaling-our-use-of-... in April you personally responded to an issue https://github.com/antirez/redis/issues/7071 within an hour.

I glad you are choosing for what makes you happy, you more than earned it.

jacquesm(45) 5 days ago [-]

Hello Salvatore,

I normally don't write about what I come across during my work, but in aggregate I can tell you that Redis, Linux and MySQL are the most common recurring elements across 150+ jobs looking at different companies, and using it rarely if ever leads to trouble.

So even if I don't use it myself directly quite a few of the companies we have invested in do, and an indirect 'thank you so much' is well deserved. I am very curious what it is that you will do next besides blogging. Linus had 'git' as his second major project, arguably it has had just as much effect on the world of free software as Linux did, you've definitely raised the bar for yourself :)

Much good luck!

xsace(4332) 5 days ago [-]

Thank you for your dedication. You will not be forgotten

rglover(2793) 5 days ago [-]

Thank you for creating Redis :) Best wishes with whatever comes next.

switz(1992) 5 days ago [-]

+1s are frowned upon at HN, but I can't think of a more warranted post than this one for offering up an underlined 'thank you'

bborud(4091) 5 days ago [-]

I haven't used Redis before, but I have been considering it for a project I'm working on so I've been having a look.

Just wanted to offer you my thanks. I've seen the project a bit from the sidelines over the years out of interest, and I think you should be proud of it.

Hope you will do something that excites you next. My only tip is : don't worry about what other people think is exciting - follow your gut.

simonebrunozzi(310) 5 days ago [-]

Ciao Salvatore!! Hope to see you again soon in San Francisco! Onward and upward.

I am glad to read that you made a choice, as difficult as it might have been, and I hope it will make you happy (or happier).

stephenitis(4341) 5 days ago [-]

Beautifully written end to a personal journey. I wish you the best!

jorangreef(2282) 5 days ago [-]

Thank you antirez!

I started using Redis around 2010 and I learned to appreciate so many things from you along the way:

* Data structures are fun.

* In-memory is fast and can be safe!

* Append-only logs are awesome.

* Databases can be more than MySQL.

* Complexity analysis is worth making clear in the documentation.

* Hybrid L1 cache-friendly data structures beat complexity analysis for small data.

* There are only so many hours you can work in a day.

* It's cool to sit by the pool.

* It's cool to have a screen name.

* Above all, code is poetry, not dependencies.

bbeausej(4433) 5 days ago [-]

@antirez Thank you for all your amazing work! Redis and your overall approach has changed the way I approach _my_ work daily.

samgranieri(10000) 5 days ago [-]

Thank you very much for creating Redis! It's an outstanding piece of software that I've enjoyed using over the past decade.

ibrault(10000) 5 days ago [-]

Thank you Salvatore! You have been an inspiration to me as a young programmer and have opened my eyes to the wonders of OSS. Excited to see whatever you do next!

gallamine(4430) 5 days ago [-]

Thank you for your work!

portroyal(10000) 5 days ago [-]

Thank you! As people have commented, your ability to stick to your convictions, even under fire, is admirable. I look forward to your blog posts.

bigiain(2667) 5 days ago [-]

Hey man, like a lot of people here, I wanna say 'Thanks fort Redis, it's solved a bunch for problems for me at work', but also 'Thanks for dump1090, I've had heaps of fun with that, and learned a lot from digging into your code there.'

Best wishes for whatever it is you choose to do next.

krp(10000) 5 days ago [-]

@antirez Thanks man! Maximum respect for everything so far, and for your honesty over the years. I rarely login or post on HN but I have just for this post.

I've been using redis since I think 2.2 and have learned so much from your posts over the years! More than just about redis. Are you able to post links on your site to the videos you're making? I don't speak Italian but I'd love to learn from subtitles.

I'm really excited for you and look forward to whatever fun things you decide to do next! Greets from NZ

unixhero(4006) 5 days ago [-]

Basically you changed the world, in your own way. Redis is here to stay now. Nothing short of sheer brilliance.

swivelmaster(4048) 5 days ago [-]

Your initial post about Redis is one of the first things I remember seeing when I started reading Hacker News. The great thing about Redis to me was that it was extremely clear what it did and what the use cases were, and I always appreciated how you talked about it in such a simple, practical way.

Thank you for Redis, and I hope you can relax and have some fun with whatever is next!

mathieubordere(10000) 4 days ago [-]

I really enjoyed your Youtube 'Writing System Software' episodes. Would be awesome to see some more of that in the future too ... !

j4pe(3907) 5 days ago [-]

I've always admired your work, and read your source code to learn how to structure a great project.

Looking forward to reading whatever you do next!

dvirsky(3774) 5 days ago [-]

Thanks so much for everything, Salvatore. Working with Redis, working on Redis and working with you have all been a blast.

ksec(1105) 5 days ago [-]

Thank You so much for Redis. Looking forward to your next project or hacking around existing OSS project. May be CRuby or Lua given you like both languages?

drol3(10000) 5 days ago [-]

Thank you so much for all that you have done. Redis is a fantastic piece of work. It is everything software is supposed to be: it's simple and it just works :)

Good luck exploring your other options

mbillie1(10000) 5 days ago [-]

Thanks for putting so much effort into Redis! The software itself is awesome (I've been using it one way or another over the span of nearly a decade now), and in my opinion anyway you've handled the project and yourself in an exemplary way. You're the only programmer I can really say I think of as a personal hero. Best of luck in the future!

robohoe(10000) 5 days ago [-]

Thank you! Redis is what brought back the spark of fun in developing in me.

divtiwari(10000) 5 days ago [-]

Thank you so much for your contributions!!!!! I might have not used Redis, but I learned a lot from your text editor tutorial.

adeeshaek(10000) 5 days ago [-]

Huge thanks!

Redis is simple and straightforward and a joy to use. Best of luck :)

ilaksh(3451) 5 days ago [-]

Congratulations on completing such a very successful project!

Also, good decision, and happy to see that you now can finally take a real vacation if you want.

Also, it is good news for everyone, because it means you will be able to concentrate your energy on new creative pursuits if you want. Which, for example, leads to beneficial things like video tutorials, etc.

It feels like people may still kind of be on your back a little bit but this time they may be asking you what the next big project is. I hope you can shake them off and just follow your passion without needing to make a commitment to something new (especially not anytime soon). This will be best for everyone in my opinion.

atonse(10000) 5 days ago [-]

Thank you for all your work on Redis. As you already know, it powers so much of the internet.

Wishing you the best in whatever you do next.

vsmhn(4360) 5 days ago [-]

> I write code in order to express myself, and I consider what I code an artifact, rather than just something useful to get things done. I would say that what I write is useful just as a side effect, but my first goal is to make something that is, in some way, beautiful. In essence, I would rather be remembered as a bad artist than a good programmer.

This entered my list of favorite quotes! For this, if not for your huge contribution to OSS, grazie!

seemslegit(10000) 5 days ago [-]

You are a unique kind of hero of a unique age.

craked5(10000) 5 days ago [-]

Thank you so much antirez for all your work! Looking forward to those blog posts!

abmobi(10000) 5 days ago [-]

Thank you for all your work these years. Your blog was/is inspirational and I especially learnt a lot from your 'writing an editor in 1000 lines' article, when every other example on the net used to use a library.

MR4D(4223) 5 days ago [-]

Thank you, and kudos for keeping at it for so long!

munificent(1982) 5 days ago [-]

I want you to know that your code is a real inspiration to me. Redis is a beautiful C artifact and an excellent reminder of how joyful it can be to create and read code.

Also, congratulations to moving off of Redis. In open source, there is a thing where people look at retiring from a project as some sort of failure of either the project or the maintainer, but I think the opposite is true.

We all have a finite amount of time on Earth, and everything has its natural beginning and end. Moving on to something else means creating the opportunity for yourself to find the next big project for you. And for the project itself, it means bringing in new eyes and perspectives.

Change is scary, but healthy.

matlo(10000) 5 days ago [-]

Grande

adverbly(10000) 5 days ago [-]

You've got at least one eagerly awaiting reader right here!

Staying tuned :)

ed25519FUUU(4410) 5 days ago [-]

I really hope you take time to do some non-software projects. Just like we rest and rotate crop fields, we must rest and rotate our minds!

olingern(3787) 5 days ago [-]

I became a sponsor of yours on Github today just to say, 'thanks.' I know it's not much, but I really do appreciate your work. Hopefully, my small donation will add up some worthwhile coffee overtime!

aidos(3975) 5 days ago [-]

Massive hat tip for all your work over the years. Redis is a lovely bit of work.

I first used it as the datastore for a flash sale about 8 years ago now. Even though close to 130k people were trying to book rooms at the same time, it performed just as I hoped it would.

I was going to say that I haven't used it in a few years, but really I've just forgotten that it's running a critical part of our system day in day out because it never causes me any trouble.

Good luck with your next adventure, here's hoping it's as fruitful for the community as your last one.

jpittis(4393) 5 days ago [-]

Hey antirez,

Your work that's had the most impact on me isn't Redis but kilo! It taught me how to have fun hacking on C back in university. Me and a couple friends cloned the repo and started adding fun features.

Here's to more 1000 line whatevers! <3

thatoneuser(10000) 5 days ago [-]

Wow I guess I don't have nearly the knowledge of redis as many others here, but either way redis has been one of my favorite coding experiences! It works well, it's reliable, and it's very easy. Thanks so much for your efforts man!

wolco(3904) 5 days ago [-]

I would love a story on

'I don't like much what the underground programming world became in recent years'

What could he mean? What underground programming world is he talking about?

tuyguntn(2825) 5 days ago [-]

I am assuming low level systems in commercial software systems, would like to hear from @antirez :)

WJW(4373) 5 days ago [-]

Underground as in programming for fun or as art, because you enjoy it as a hobby/subculture. Opposed to the 'above ground' programming world of programming for other people (possibly also for money, but OSS is often just for recognition) where people have demands about unimportant features, terminology, coding styles, codes of conduct, etc. Sometimes they get quite aggressive about their demands, too.

lazyjones(4368) 5 days ago [-]

Perhaps the fact that a recent (2018, but posted here recently) blog post of his got flagged on HN after 141 comments has something to do with it...

dilandau(10000) 5 days ago [-]

Probably the roving mobs on Twitter who occasionally brigade project issue trackers and mailing lists. Check out the master/slave post he wrote a while back.

Or just the toxic nature of some open source communities, especially on Reddīt for whatever reason.

elemant(10000) 5 days ago [-]

If you don't know it's because we don't want you to know ;)

kissgyorgy(1724) 5 days ago [-]

Probably the same which was the reason Guido left Python as BDFL.

koolba(609) 5 days ago [-]

@antirez - Thank you for Redis! It's been a joy to use across so many projects.

> However I never wanted to be a software maintainer.

And nothing say that you have to be. There's this perverted view that anytime someone creates a popular FOSS project, they need to dedicate every waking minute to maintaining it. That's neither economically feasible nor psychologically reasonable.

> Redis was the most stressful thing I did in my career, and probably also the most important. I don't like much what the underground programming world became in recent years, but even if it was not an easy journey, I had the privilege to work and interact with many great individuals.

What is 'underground programming world'?

rconti(10000) 5 days ago [-]

There's absolutely nothing wrong with him not wanting to be a software maintainer. I don't want to be either, and I'm not even a programmer.

But there is DEFINITELY something wrong with the fact that, seemingly, NOBODY wants to be a software maintainer.

chii(3891) 5 days ago [-]

> 'underground programming world'

it's where you program in a basement.

cies(4062) 5 days ago [-]

I was also intrigued by that sentence. I expect it means (open source) programming that happens outside of of big corps.

But I'm unsure.

amelius(804) 5 days ago [-]

> What is 'underground programming world'?

The programming world that nobody else sees.

Everybody uses shiny FOSS tools, and some even make big $ using them, but nobody wonders where they come from and how much pain went into building them.

conjectures(10000) 5 days ago [-]

> What is 'underground programming world'?

If you have to ask... ;)

justaguyhere(2409) 5 days ago [-]

I would rather be remembered as a bad artist than a good programmer

Takes a rare person to say this, made me smile :)

tedk-42(10000) 5 days ago [-]

When I read that line my heart broke a little.

I think he's done a great job with Redis. It goes to show sometimes that you need to let go of the thing you created for the good of the community. If I were in his position, I would have made the same move

ilaksh(3451) 5 days ago [-]

Personally I think it's mainly the result of having people constantly being hypercritical of your code.

And I think he's proven himself to be a superior programmer. In that in almost every category of engineering skill and knowledge, he is better than 90% of the people criticizing him. If he didn't start out that way, over the years his knowledge and skills grew to make it so.

But also maybe it reflects a bit that there is always some new 'best practice' or something that some months ago no one heard about, but now many people are unfortunately using the adoption of as a proxy for programming skill because they are unwilling or unable to actually judge something or someone on actual merit.

raverbashing(3952) 5 days ago [-]

Sounds like a bit of Italian culture leaking into that phrase

But don't worry Antirez, Redis is far from a bad work of art, quite the contrary.

avmich(10000) 5 days ago [-]

It's long past due that we should move to software which isn't maintained - because there's nothing to do. Adding features could be done either by writing a new one - because the requests don't really fit the meaning of old one, the creators saw that and rejected; fixing bugs - yes, fixing bugs may remain, but are there many bugs in, say, TEX code?

We don't yet know how to write a completed software well - the one which isn't updated on Github for years yet nobody calls it stale or outdated.

shurane(10000) 5 days ago [-]

I think video games fit in the area of 'completed software'. Most video games today rarely see updates a few years after they're released. And older video games were on read only mediums, so you would hope whatever you release is very stable and feature complete.

akrain(4223) 5 days ago [-]

I know at least one software that was complete - hasn'tbeen updated in a long time and I still useit. It's Winamp.

jpxw(4367) 5 days ago [-]

The fact that one person can (mostly on their own) create a tool as widely-used and respected as Redis is very inspirational to me. Good luck for the future!

JosefAssad(10000) 5 days ago [-]

If you flip that around, it's possible that being driven by one man was a contributing factor to redis' success.

Design by committee and all that.

poooogles(10000) 5 days ago [-]

First and foremost thanks and good luck @antirez, your comments and pointers here have been nothing less than amazing.

On a more selfish note, how is this going to affect Redis development?

antirez(983) 5 days ago [-]

Hey! I'm not part of how the new setup will be. I don't liked to maintain, and selecting a new development setup is the most mantainer-ing thing ever :D

Aeolun(10000) 5 days ago [-]

Looks like they're already well covered with community contributions and two new (or pre-existing) maintainers?

eric_b(10000) 5 days ago [-]

So they're moving to a new 'community based', 'light governance' model. [1]

There are plenty of problems with BDFL-style projects, but I think there are a lot of advantages too. Redis is unique in my experience in that it works the way you'd expect - it doesn't cause outages, it is fast, and it has a vanishingly small number of gotchas. The feature set is well curated and for the most part fits together cohesively.

The most important thing Antirez did, in my opinion, was to say 'No' to things. No to new features that didn't make sense. No to PRs that didn't fit the vision. Saying no is one of the most important jobs of a project maintainer. It's a thankless task that upsets a lot of people. But it's critical for a project to stay successful and achieve the leader's vision.

Maybe I'm a pessimist, but I predict after a few years of this new model we'll see weird features, more stability issues, and performance regressions as more cooks enter the kitchen. Time will tell.

[1] https://redislabs.com/blog/new-governance-for-redis/

aboringusername(10000) 5 days ago [-]

It's an interesting proposition. Whether it's Python, Redis, Laravel or Linux itself, a lot of software projects have been driven primarily by singular people and their decisions which shape the project over a period of time.

One example that was mentioned in this thread was the whole 'master' thing going on.

Is Torvalds really going to get involved in that? Most likely no, and that's a decision he's free to make.

I worry in the future, after he's gone, depending on the model being used, we'll see more politics in software projects where Linux itself gets dragged into these things and doesn't have a single guy to say NO!

Or maybe you see an old white guy in charge of Linux and that can be considered a problem in itself, I'm sure many would love to see some change there and finally overrule Torvalds' decisions.

So in reality, we're seeing a slow shift away from BDFL type situations, and in the near term future, very few (what I consider influential) projects will remain with that setup.

cletus(3309) 5 days ago [-]

I 100% agree with you and I think it explains why most OSS UI projects are, well, terrible. Questionable features, inconsistent UX, millions of 'options'... these are the hallmarks of a community model lacking strong leadership and a coherent vision.

As for the future of Redis I choose to be more optimistic. I think the Redis philosophy (thanks to Antirez) has become ingrained. The benefits of a minimal feature set are hopefully well-established.

In the very least I suspect this will last much longer than many expect before the bureaucrats take over.

umvi(4446) 5 days ago [-]

> The most important thing Antirez did, in my opinion, was to say 'No' to things.

This is hard. It makes you feel bad inside to tell people 'No', especially when they open PR and clearly did a lot of work adding some feature.

1_player(4005) 5 days ago [-]

> There are plenty of problems with BDFL-style projects, but I think there are a lot of advantages too.

BDFL governance is extremely underrated in the opensource world. People attribute the success of the Linux kernel to its open contribution style, but I will argue that Linux is successful because there's a dictator at the top that enforces a direction, a long term goal, and most importantly, says NO.

Community based governance is a direct cause of that jwz's Cascade of Attention-Deficit problem one can find all over the open source world, especially Linux desktop related.

jffhn(10000) 5 days ago [-]

>Saying no is one of the most important jobs of a project maintainer. It's a thankless task that upsets a lot of people. But it's critical for a project to stay successful and achieve the leader's vision.

Reminds me of Brian Goetz: 'We have to say no to almost anything. If we didn't, things would pretty quickly degenerate to the point where the system collapses on its own weight (...) if we did this, which of these eighteen possible features might we foreclose on, and might we want to leave those possibilities open, at the cost of saying no to this thing here.' (https://www.infoq.com/presentations/java-history-present-fut..., 24m+)

zebraflask(10000) 5 days ago [-]

Do you have any tips for explaining to stakeholders that things in a Redis cache expire on async schedules?

derefr(3913) 5 days ago [-]

I feel like Antirez has already done everything needed to ensure this doesn't happen, by 1. building the module system into Redis, and then 2. working with companies like Redis Labs to establish that Redis is enhanced through these modules, and to ensure that modules offer everything required to enhance Redis in all the ways that matter.

In essence, Antirez has protected the core of Redis from needing to bloat, by making it possible for anyone who wants to build on top of Redis to literally do so—build on top, rather than inside. As such, I expect future PRs to Redis Core to look much like current PRs to Redis Core: just fixing bugs, adding stability, and resolving infrastructure-level interoperability concerns.

albertzeyer(547) 5 days ago [-]

Instead of a single BDFL, you could also have a board of e.g. 3 BDFLs. Those 3 people would have the final word, but they rank equal, and have to come to a common conclusion, e.g. by discussing internally. That only works if the board is not too big, and the people are somewhat compatible to each other. But in that case, I think it works much better, because there is less pressure on a single person, also less work (they can distribute it somewhat), and probably more reasonable and consistent conclusions.

It seems that this model is not so common? At least the list on Wikipedia (https://en.wikipedia.org/wiki/Benevolent_dictator_for_life) only lists Django as an example with 2 BDFLs.

I was active in a smaller open source project (http://www.openlierox.net) where it mostly was like that, i.e. we were 3 main developers, and all major design questions were discussed among us. But this example is probably not so representative, as there rarely were further contributions by other people. But anyway, I was very happy with this model, that there is not a single leader, but 3 people.

neogodless(3081) 5 days ago [-]

BDFL: Benevolent dictator for life https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

(This is also referenced and linked in the article.)

calpaterson(3140) 5 days ago [-]

> The most important thing Antirez did, in my opinion, was to say 'No' to things. No to new features that didn't make sense.

Redis has had tremendous mission creep over the years. It started of course mostly as a volatile cache but I've now seen it also used as a message bus (in three different ways: BLPOP, PUB/SUB and Streams), for service discovery and as a general purpose database - something that Redis Labs (in my opinion: wrongly) encourages.

Memcache existed in 2009 when Redis was first released, also as a volatile cache...and still is just a volatile cache. Memcache is what 'saying no' looks like. Redis is what 'saying yes' looks like - and there are a lot of gotchas.

dvirsky(3774) 5 days ago [-]

The one drawback of the BDFL model is that you need a person, and a very specific kind of person at that, which tends to be irreplaceable - to sustain it.

Redis has succeeded because of antirez. I really wish it can continue doing so for the long run without him. More than anything it reminds me of Steve Jobs' death (although gladly this is in much happier circumstances and I wish antirez many more decades of hacking). It's true that a decade later Apple is still doing great, but you just know things would have been different if he was still around.

binarymax(2417) 5 days ago [-]

HN hugged to death. Here's the snapshot on archive: https://web.archive.org/web/20200630130517/http://antirez.co...

ignoramous(2361) 5 days ago [-]
entropie(10000) 5 days ago [-]

I actually wonder how many request/s one gets from being on the top of HN.

bactisme(10000) 5 days ago [-]

I think its website need a fast cache.

rubyfan(4376) 5 days ago [-]

> In essence, I would rather be remembered as a bad artist than a good programmer.

words of wisdom right there

eplanit(165) 5 days ago [-]

It's egotistical humblebrag.

JamesSwift(10000) 5 days ago [-]

Dang.

I say this frequently both online and when discussing system design with newer devs, but will repeat here: of all the production issues I've debugged, the culprit has has never been redis. In fact, redis has been a critical piece of achieving cost-effective scaling. It is one of only two pieces of software (along with postgres) that I blindly recommend without any caveats. From following along here and on your blog about how you approach things and think about the software, I think its clear that you and your vision for the project are a large factor of why it has been so reliable.

Thank you antirez!

wokwokwok(10000) 5 days ago [-]

Redis is simple. Good. Has a nice api. Has good libraries. Single threadsed. Extremely hard to scale. Impossibly difficult to cluster in containers because it uses hard coded ips to address nodes. Performs poorly with large payloads. Doesn't run on windows properly. Is extremely expensive as a hosted service (orders of magnitude in some cases, eg. azure).

You'll love it until you don't.

The scaling and clustering story is not nearly as nice as the quick start.

It's definitely worth recommending... with caveats.

smoyer(3399) 5 days ago [-]

I read it twice trying to figure out why you were invoking @dang.

In any case, I've had the same experience with only two programs - CouchDB and Redis.





Historical Discussions: Oppose the Earn IT Act (July 01, 2020: 2227 points)

(2227) Oppose the Earn IT Act

2227 points 4 days ago by joeyespo in 66th position

foundation.mozilla.org | | comments | anchor

The U.S. Senate is advancing legislation that creates a threat to strong encryption, the bedrock of digital security.

While the senators championing the bill, which they've named the Eliminating Abusive and Rampant Neglect of Interactive Technologies Act (EARN IT Act), may have good intentions, they are seriously misguided about the impact of their proposal.

Encryption ensures our information, from our sensitive financial and medical details to emails and text messages, is protected. But the EARN IT Act will create a broad path for government actors to seriously undermine strong encryption, putting our information at risk. That's why Mozilla is joining dozens of other internet health and civil society organizations in calling on the U.S. Congress to vote no on the EARN IT Act.




All Comments: [-] | anchor

justinyan(10000) 4 days ago [-]

The act itself seems to revolve around limiting section 230 protections (https://uscode.house.gov/view.xhtml?req=(title:47%20section:...) to those who pass a checklist of TBD requirements determined by a TBD bureaucracy.

I've struggled a bit to find resources helping to describe what section 230 protections are actually for, however. Many of the examples explaining section 230 protections seem to revolve around things like defamation in Youtube comments or product reviews, but I'm having trouble making the leap from that to why messengers are so concerned about this bill. Why do tools like messengers need section 230 protections to begin with? AFAICT they still have to submit to things like national security letters, so what does section 230 buy, say, Whatsapp or Signal?

jmole(10000) 4 days ago [-]

not to mention that the TBD requirements have to be approved and voted on by congress to become law.

that said, there is also some text that changes specific wording in other existing laws that may have an impact on future court cases, e.g.: "(2) CIVIL REMEDY FOR CERTAIN ACTIVITIES RELATING TO MATERIAL INVOLVING THE SEXUAL EXPLOITATION OF MINORS.—Conduct by a provider of an interactive computer service (as defined in section 230 of the Communications Act of 1934 (47 U.S.C. 230)) that would violate section 2252 or section 2252A if that section were applied by substituting 'recklessly' for 'knowingly' each place that term appears shall be considered a violation of section 2252 or section 2252A for purposes of paragraph (1) of this subsection.".

We could be charitable about what 'recklessly' means, but given the tendency of pre-digital judges to misunderstand technology - I can understand why there is a pushback against this kind of language.

'What do you mean you don't keep a history of your clients' internet activity? I would construe that as reckless behavior.'

sailfast(10000) 4 days ago [-]

I reached out to my representatives on this when the Act was first proposed. I'm confused about the changes that have happened since that time and the post does not say anything about what has changed. What does 'Advancing' mean in this case? If I have already contacted all of my legislators about strong encryption and opposition to this bill, what is the latest change that requires re-engaging?

suizi(10000) about 21 hours ago [-]

The latest version now involves State law so if one State out of 50 decides encryption is bad for children they can go after any company in the U.S.

https://www.eff.org/deeplinks/2020/07/new-earn-it-bill-still...

pfranz(10000) 4 days ago [-]

I haven't followed the specifics, either, but it looks like it was introduced back in March. When I reached out to my Senators and Reps back then I got a reply from my House Rep (who I believe was supportive of the bill). The bill is currently in the Senate and only this week is getting 'marked up' in the Judiciary Committee--I imagine if your Senator is on that committee it's more important to reach out.

Sadly, I think issues need constant re-engaging since they pop up under different names and in different forms.

programmarchy(4426) 4 days ago [-]

Thankful for Mozilla and EFF, but have any tech elites spoken out against this or put money into defeating it? Thiel, Musk, Altman, Graham... hello? Should we assume SV is complicit?

Klonoar(3691) 4 days ago [-]

Not even touching on those individuals, it's been bothering me that it feels like mostly silence from Google, Apple, Facebook (who this would presumably really hit at), etc.

I'd really love to be missing something. It feels like this is just fading into the background... and while I can acknowledge the idea that you might just try to fight this as unconstitutional after it passes, that feels inherently risky given this administration (if not downright stupid).

google234123(10000) 4 days ago [-]

Warrant-proof encryption is not compatible with a society that accepts the principle that we are accountable to the law. There is nothing sacred about your cell phone.

snazz(3516) 4 days ago [-]

Encryption that can only be unlocked by the 'good guys' is fundamentally not how encryption works.

'JoshTriplett explained it better than I did: https://news.ycombinator.com/item?id=23704928

darkarmani(10000) 4 days ago [-]

That's a very bold claim. We already have warrant-proof toilets for flushing drugs down. Why do you think crypto should be illegal because old-technology warrants don't work very easily with it?

We've had warrant-proof encryption for a long time now in our society (at least 20 years?).

chejazi(2436) 4 days ago [-]

Obviously this sucks as a proposal but... how do we stop the child porn?

gspr(10000) 4 days ago [-]

I don't know. But I do know that it's not a technology problem.

orthecreedence(3407) 4 days ago [-]

Just a thought, but maybe we should arrest people who make child porn.

analyte123(10000) 4 days ago [-]

Serious answer:

- Undercover police work and informants, which is how most large busts already happen now. The worst offenders are not swapping child porn on Facebook or Dropbox

- Encourage marriage and remove benefits penalties for two-parent households, as children are substantially more likely to be abused when their biological father is not in the household

- Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly

manuelabeledo(10000) 4 days ago [-]

This sounds like the start of a false dichotomy.

antepodius(10000) 4 days ago [-]

Have a mandatory bodycam implanted in everyone's right eyesocket, that livestreams to government servers 24/7.

This is an obvious short-term mitigation tactic until they develop brain implants that can pre-emptively moderate harmful thoughts.

suizi(10000) about 20 hours ago [-]

I don't think it is the real problem. It is a problem in the sense that it is disgusting but it isn't worth destroying a free society over and I don't think it is possible to stop.

What you can crackdown on are the producers.

You can infiltrate the places where it is originally disseminated.

You can run face and voice recognition on the content.

You can flip lower level mass distributors into informants to gather evidence on the producers.

For rapists.

You can train teachers and counsellors to identify children who have been abused and to handle other psychological problems like depression which are endemic in our society.

You can train police officers to better spot criminals on a basis other than race or status.

You can better fund the CPS.

You can conduct assessments on parents who appear to be mentally unstable.

zelly(10000) 4 days ago [-]

Make possession carry the death penalty.

Before you think that's too harsh, consider that the other options are to ban encryption or to not try to stop it at all (quasi-decriminalization).

I don't think anyone would oppose a bill to make CP carry the death penalty. The problem is they won't ever make this law because then they won't be able to milk the cow anymore.

darkarmani(10000) 4 days ago [-]

A modest proposal would suggest killing all of the children. What did you have in mind?

einpoklum(10000) 4 days ago [-]

After years of ever-deepening privacy violations in the US, I'm surprised that this hasn't passed yet.

I mean, the government already legally demands that all the large corporations make them a copy of all of your communications that pass through them. So why should it be legal for you to encrypt any of it without them being able to decrypt it? How are they supposed to spy on you effectively with just the meta-data, you know?

suizi(10000) about 20 hours ago [-]

The CIA kills people over metadata.

annoyingnoob(10000) 4 days ago [-]

We need to vote out the people pushing for backdoors. They seem to want backdoors at any cost, without any technical understanding of why its a terrible idea. Vote them out!

linuxftw(10000) 4 days ago [-]

Just like we needed to vote out the people that voted for the bailouts and the Iraq war.

Politics keeps moving on. I'm still on the Iraq war. I'm not interested on any other issue until we solve that one.

Edit: s/bailouts of/bailouts and/

djsumdog(669) 4 days ago [-]

In American politics, voting really doesn't make a difference as much as just having opinions alighted with those of the top 10% of income earners[0]. Making noise and protests are more effective in some ways, but the sad reality is that most of senate/congress is not beholden to the selectorate of all voters, but rather the smaller selectorate of influences that control the advertising and narrative that allows them to continue to be re-elected.

[0]: https://battlepenguin.com/politics/video/does-voting-make-a-...

cevn(3900) 4 days ago [-]

Lindsey Graham (R-South Carolina) - vote for Jamie Harrison on Nov 3 in SC

Richard Blumenthal (D-Connecticut) - term ends in 2022*

Josh Hawley (R-Missouri) - 2024

Dianne Feinstein (D-California) - 2024 as well... good luck with this one

blintz(10000) 4 days ago [-]

I cannot believe that the one thing we have bipartisan consensus on is destroying strong encryption.

I guess I'm biased since this is essentially my whole livelihood, but this is crazy, right?

ocdtrekkie(2962) 4 days ago [-]

We have bipartisan consensus[1] that tech companies have acted badly and that Section 230 should be repealed or significantly amended. I think EARN IT will probably kill Section 230, not strong encryption, so I think EARN IT is pretty great. Tech companies probably should be coming to the realization by now that it's only a matter of time (probably 4-8 years) before we close the 230 loophole: So if they're given a choice, they should choose to protect strong encryption, and forego Section 230 protection, since it's going away anyhow.

[1]Kind of, Democrats think tech companies got Trump elected, Republicans think tech companies are suppressing conservative viewpoints, but either way, they agree on the problem.

xwdv(10000) 4 days ago [-]

I can. I have one administration in particular that I think can be blamed for this massive expansion of the surveillance state in the past 10 years.

jmspring(3990) 4 days ago [-]

There was a huge push for backdoors and weakened crypto during the Clinton years. It's not a right/left thing, it's government control thing.

didip(4239) 4 days ago [-]

It's not crazy. The class war between the super rich and everybody else has been happening forever. It just becomes more blatant now.

Not sure what is the solution here, the popular people's choice couldn't even get elected.

asdkhadsj(4403) 4 days ago [-]

I'm most curious on when they'll be knocking on the door of open source projects next. Notably, anyone who uses any crypto.

As much as I hate it, I can at least understand the back door argument from a [ignorant] lawmaker perspective. If I pretend and say their intentions are noble, I understand.

What concerns me though, beyond the obvious backdoor problems, is the who is next? Because I doubt big corporations will satisfy their greed for power and information. Especially since anyone who has anything to hide or cares about security will move into open source.

As a developer with a passion for developing distributed, encrypted software - when are they going to threaten me? Worse yet, the software I write I purposefully do not have control over. So am I going to be held liable for the fact that I literally cannot help them?

No matter what they threaten me with, the best I could do is break the application for future users. So what are they going to do to control these distributed systems? Especially ones who truly aim to be distributed, P2P & self hosted by every user?

As terrifying as the current anti-encryption behavior is, I'm oddly more concerned about the move after this.

treis(10000) 4 days ago [-]

>I guess I'm biased since this is essentially my whole livelihood, but this is crazy, right?

There's clearly a valid argument from the other side. For example:

>Facebook announced in March plans to encrypt Messenger, which last year was responsible for nearly 12 million of the 18.4 million worldwide reports of child sexual abuse material, according to people familiar with the reports.

https://www.nytimes.com/interactive/2019/09/28/us/child-sex-...

It's not clear how many of those lead to convictions but even a tiny fraction of a percent represents a significant number of children being rescued. Encrypting Messenger, as an example, will stop 3/4s of abuse reports and make it much safer and easier for paedophiles to exchange images. There's a pretty direct line from that decision to an increase in abuses like:

>"inserting an ice cube into the vagina" of a young girl, the documents said, before tying her ankles together, taping her mouth shut and suspending her upside down. As the video continued, the girl was beaten, slapped and burned with a match or candle.

>"The predominant sound is the child screaming and crying," according to a federal agent quoted in the documents.

phendrenad2(10000) 4 days ago [-]

And wars, don't forget wars.

Aeolun(10000) 4 days ago [-]

It's one of those things that people think is a grand idea, right up until it's been in force for a few weeks and it turns out that China is reading all the internal US message traffic.

Then everyone will be like "Whoa?! Who would have thought something like this could happen!". I feel like this is a tale as old as time.

TheSpiceIsLife(4288) 4 days ago [-]

Your government is an Advanced Persistent Threat.

seph-reed(4432) 4 days ago [-]

> we have bipartisan consensus

> we

'It's a big club and you ain't in it' - https://www.youtube.com/watch?v=i5dBZDSSky0

throw_m239339(10000) 4 days ago [-]

> I cannot believe that the one thing we have bipartisan consensus on is destroying strong encryption.

Most differences between republicans and democrats are superficial. Republicans don't really believe in small government no more than Democrats really believe in social justice.

snarf21(10000) 4 days ago [-]

Fear-mongering is strong. Look at all the freedoms we lost after 9/11. Pedophiles and terrorists and porn are always the scapegoats.

It is also a very electable sound bite. Everyone is against the 'bad people'. Re-election at any cost is the goal. We have government of the people and by the people but we stopped being for the people long long ago.

cntblvntbtr(10000) 4 days ago [-]

I can't really believe the shock.

Have you noticed routine protest and informed debate occurring throughout the US for oh say 30 years to be generous?

Political reality isn't logical, it's on the nose. There's no engagement to prevent this, and decades of flag waving as enforcement funding is cut.

That has an impact.

The generation that still holds the most seats in government has no idea how any of this works.

Giving them the benefit of the doubt it's ignorance not informed malice their intuition is nevertheless "expand police state."

I'm sure we'll all take a break from HN and debating flat or skeuomorphic to tell them piss off

AsyncAwait(4445) 4 days ago [-]

> I cannot believe that the one thing we have bipartisan consensus

Hawkish foreign policy would be another one.

cloudier(10000) 4 days ago [-]

Do you remember the bipartisan consensus on SOPA/PIPA?

'This bill, COICA, was introduced on September 20, 2010, a Monday. And in the press release heralding the introduction of this bill, way at the bottom, it said it was scheduled for a vote on September 23 — just three days later.

And while of course there had to be a vote — you can't pass a bill without a vote — the results of that vote were a foregone conclusion. Because if you looked at the introduction of the law, it wasn't just introduced by one, rogue, eccentric member of Congress. It was introduced by the chair of the committee — and co-sponsored by nearly all the other members — Republicans and Democrats. So there would be a vote, but it wouldn't be much of a surprise, because nearly everyone who was voting had signed their name to the bill — before it was even introduced.

I can't stress enough how unusual this is. This is emphatically not how Congress works. I'm not talking about how Congress should work, the way you see on Schoolhouse Rock. I mean the way it really works. I think we all know that Congress is a dead zone of deadlock and dysfunction. There are months of debates and horse-trading and hearings and stall tactics.

...

Whoever was behind this was good.'

https://www.buzzfeednews.com/article/jwherrman/how-aaron-swa... (excerpt from talk at https://www.youtube.com/watch?v=OqriFtr2k-k -- FWIW the lobbyist behind sopa/pipa was Chris Dodd, https://en.wikipedia.org/wiki/Chris_Dodd#Motion_Picture_Asso...)

arminiusreturns(10000) 4 days ago [-]

I can. Our representative government represent corporations and the deep state, in the original Peter Dale Scott sense of the term (as opposed to the more recent abuse of it), and not the people.

They have continually made moves like this, in a boiling frog fashion. Until we the people wake up from the two party divide and conquer system, very little will change, excepting very rare circumstances of pressure exerted en masse (SOPA/PIPA is a good example outlier).

I've thought about solutions to this problem for a long time, and my conclusion is the things needed are the following (in order to prevent a long term overton window shift as has been happening):

1. Renewed participation in local politics and elections, especially at the state level, but also county, city, etc.

2. Once that is achieved, voting in ranked choice voting initiatives. This will enable the next step.

3. Stop voting for anyone based on party, in particular the main power structure is based on the majority rule in the house and senate... the end goal would be to take away the majority from both parties. This is a huge undertaking, so I'm not saying it would be easy, or even possible, but I think it is what is required. I think those paying attention enough and not enured to the tribalism of the parties understands they cannot be reformed from the inside. They are simply too entrenched, and have too many mechanisms to get rid of those who seek to do so. The point is that we don't need to gain a majority with this new coalition party of independents and third parties, we simply need to take the majority away from the two main parties. To get this done though, [1] must be done, because many of the state laws have been manipulated by the duopoly to prevent third parties and independents.

3.1 To avoid fracturing of the coalition, there should be some very rudimentary and base document that all persons running can agree on. This is also difficult but, since all congresspeople swear an oath to the constitution (5 U.S. Code § 3331), something that reaffirms the principles of the constitution would be the best start. If they violate this, the coalition should work to remove them immediately. (something similar to Gingrich's 'Contract with America')

4. Currently, and many don't know this, congresspeople have to sign an affidavit (5 U.S. Code § 3333) confirming their oath. I think we should then work towards enforcement of the laws against violation thereof, the primary one being perjury, or something similar (18 U.S. Code § 1918, 5 U.S. Code § 7311). For example, how many people have straight up lied to congress and had no action taken against them? (Clapper is one of the more egregious ones that comes to mind, but the point is congress is in dereliction of duty in enforcing perjury laws)

This would give the people a real opportunity to start passing laws that represent the people (such as term limits), but are still constitutionally sound.

The legislative is the branch closest to the people, and this is why it should be the focus. From a cleaned up legislative we can begin to resurrect the separation of power between the branches that has been egregiously eroded, primarily by the executive. The current checks and balances system is crumbling. A huge part of this erosion is the surveillance system, which enables compromise of congresspeople by the executive and the MICC. We must fight for privacy, both encryption and anonymity, as a fundamental part of protecting the checks and balances system.

sudosysgen(10000) 4 days ago [-]

There is always bipartisan consensus in whatever gives more actual power to powerful people. It's just that some would rather the power be for billionaires and others for the government.

raxxorrax(10000) 4 days ago [-]

Doesn't surprise me at all and it was painted on the wall pretty clearly in my opinion. It is of course a sad state of affairs. I think it shouldn't need to be mentioned but apparently it does.

BiteCode_dev(4319) 4 days ago [-]

Well, nsa mass surveillance and legalized torture were never repelled by any side.

In fact, the patriot act has been prolongued 5 times, no matter the administration, and is still in effect.

As a european, I don't see any party that actually have american people's interest in mind.

Fighting for either seems so futile from here.

Of course, we have similar problems as well so I'm not going to pretend I have a solution.

wnevets(10000) 4 days ago [-]

Its being framed as a way to prevent terrorism and child sex abuse, it's really hard for a politician to come out in against preventing those things.

ypcx(10000) 4 days ago [-]

It is worse than just crazy. Government is made of people, and people are corruptible. If the encryption backdoor key leaks from the government to a bad actor, it will create a 'national security' issue of magnitude never seen before. If this line of reasoning is correct, then proposing an encryption backdoor is akin to committing an act of treason in itself, because it is purposely weakening the technological infrastructure of the businesses and people in the country and thus the country itself. Attempts like these are either doomed to fail, or they will doom the country to fail.

apeace(4225) 4 days ago [-]

If you want to convince your friends they should support encryption, here's how I like to get past the 'nothing to hide' argument.

Imagine we're sitting at a bar, chatting. None of us have anything to hide. Then the government passes a law that all conversations must be streamed on Youtube Live, so an agent comes in, sets up a camera at our table, and starts streaming.

We still don't 'have anything to hide'. We're just having a conversation. But the conversation used to be private--that's normal. Now it's not private, which is not normal.

Whether or not you feel like you have to 'hide' anything during a bar conversation is not the point. It's whether you think we should make changes to our society where having a private conversation is never allowed.

This kind of analogy, in my experience, helps people understand that the 'nothing to hide' argument assumes that privacy is only for evil people, when in reality it's the very normal default of daily life. The parable posted in another top-level comment is also great.

cgrealy(10000) 4 days ago [-]

If someone says they have nothing to hide, just say "cool, how much do you earn?"

Or if you can do it with a straight face "when was the last time you masturbated?" :)

marta_morena_23(10000) 2 days ago [-]

This analogy is quite weak though. I mean in all honesty: Who in the world would be okay with streaming is drunken bar ramblings on youtube? Of course you have something to hide.

There was a Movie 'Circle', which kinda was spot on. Privacy is about being able to be yourself. Exploring things. Doing things other people might find creepy. Things that might get you fired, if your co-workers or employer knew about it. By removing privacy, you remove the ability to be yourself. You need to be exactly as society demands it, at all times.

The effects of that are catastrophic on humanity. China being a world leader in demonstrating what this does to people and societies, but even China allows for some measure of privacy, as long as you don't do it online and do it at home. Take that away, and society is done. There is only the borg collective left.

goldenManatee(10000) 3 days ago [-]

I've had those encounters, but I do not understand how many people don't view privacy as a default to being human.

rapnie(2127) 4 days ago [-]

I've used most of the arguments against 'nothing to hide' mentality mentioned in this thread, and guess what: most of the people still don't care enough to change a single one of their bad privacy habits. Their eyes glaze over, or they think you are just overly dramatic or a conspiracy thinker, or 'I'm not that important' or 'Small chance it happens to me', 'I trust the government' (when it comes to encryption), etc. Some friends and family I educated in length, and still they are not even willing to e.g. switch their Samsung browser to FF, or install an ad-blocker.

The concept of privacy and the consequences of not having it are too abstract for them. It is like saying that climate change will lead to disaster unless we all act now. They will only act after something bad happens to them personally.

cortic(10000) 4 days ago [-]

My nothing to hide argument is a little different;

Nothing to hide is an incomplete sentence. Nothing to hide from who? Surly you want to hide your children from abusers and predators? Don't you want to hide your banking details from con artists and fraudsters? Your identity from identity thieves.. Your location from burglars, your car keys from car thieves or your blood type from some rich mobsters with kidney problems..

we don't know who are any of these things. So we should protect ourselves from all of them, in effect we have everything to hide from someone, and no idea who someone is.

sameerds(10000) 4 days ago [-]

You could move that a notch up and say that only the owner of the bar and their friends get to watch all these conversations and they keep copies. That should really freak out some people.

kulahan(10000) 4 days ago [-]

The best argument I ever heard was:

'Would you give up your right to free speech just because you have nothing important to say? No? Then why would you give up your right to privacy just because you have nothing to hide?'

nopeNopeNooope(10000) 4 days ago [-]

This is better: 'Cool, call your girlfriend while I listen to you two having phone sex. Right now. You have nothing to hide right? Let's hear it. I can turn around if that makes it easier, so you don't have to see my face. Come on then, you've got nothing to hide right? Should I call her and see if she's game since you're chicken?'

addHocker(10000) 4 days ago [-]

The problem is with the monopol on violence the government is supossed to have. Today, anyone rich enough can have violence or the threat of violence, at a industrial scale like government used to have them. Bezos or Musk could have a rod of god in orbit, and you wouldnt even know.

And to prevent this from escalating, as artefacts are getting ever denser, in information, energy and potential - you get a panopticon. Those liberal ideas, where forged in a world, where the peak of energy density and danger was a musket.

This world is gone. To prevent it from coming back, we get the panopticon.

sixhobbits(3724) 4 days ago [-]

not sure who to attribute but I like the 'privacy is not secrecy. You close the door to the toilet even though we all know what you're doing in there'

create5(10000) 4 days ago [-]

why not just straight forward, 'I do have something to hide, I do want to hide them from everyone, so what'.

thelittleone(10000) 4 days ago [-]

Another argument is that a conversation we are having today (nothing to hide and between friends) may be used against us in the future. It could also be used against your children.

jrumbut(10000) 4 days ago [-]

I think the key here isn't broadcasting, it's storage. In the future (?) we might think we can detect Parkinson's or MS very early through voice/video/typing characteristics.

Your conversations were recorded, the data leaked to or purchased by a fly by night company hired by your bank, and now suddenly despite your excellent credit history you can't get a 30 year mortgage, you try to find a new job but companies are only offering 6 month contracts, etc.

It's hard to prove a database doesn't exist, better to encrypt the content beforehand and make it harder to create.

davidd8(4352) 4 days ago [-]

My take: The "nothing to hide" argument is a fallacy for two reasons: (1) The connotation that things discussed in private are bad. True, criminals may discuss illegal activity in private, but that does not mean everyone's private conversations are illegal. (2) The notion of what needs "hiding" is subject to opinion, and changes over time. You may not be breaking any laws today, but if the law changes and governments have access to your past data, you are now vulnerable and have nowhere to hide.

From my post about privacy not too long ago, https://www.davidexmachina.com/2020/02/privacy-matters.html

DarkWiiPlayer(4446) 4 days ago [-]

I find it much easier to argue from the moral perspective.

It's my right to hide whatever I want, regardless of whether somebody else considers it 'something to hide' or not; just as it's your right to buy a 3m tall chocolate statue of yourself, regardless of whether I think it's something you need.

The question is not 'is it really necessary?' but 'who the #£$! gave you the right to decide whether I consider it to be necessary for myself?'

Then again, here in (east) Germany, somebody argues 'I have nothnig to hide', you just tell them 'Neither did people 40 years ago' and the conversation is pretty much over.

shay_ker(4428) 4 days ago [-]

I like the speeding analogy.

'I have nothing to hide'

'Have you ever sped on the highway?'

'Well, yes'

'Imagine if the government reveals it's been tracking everyone's speed, and has decided to write everyone tickets for speeding.'

'That's insane, the government can't do that!'

'Exactly'

Nition(3905) 4 days ago [-]

I saved a link to this post a few years ago: https://www.reddit.com/r/changemyview/comments/1fv4r6/i_beli...

I won't try and summarise it because I think I'd do a disservice to it, but it's another really good argument, ostensibly from experience, re corrupt countries and surveillance.

wirespot(10000) 4 days ago [-]

My go-to analogy for understanding privacy in the 'nothing to hide' context is simple: 'Do you have window curtains?'

manfredo(10000) 4 days ago [-]

While this is on the right track, one would point out that these conversations wouldn't be public they would only be visible to government authorities.

I think the due process argument is more compelling. Part of preventing abuses and corruption by law enforcement is ensuring they can only surveil people with reasonable suspicion. Yes there are people who should be under surveillance. But it's crucial that this gets approval from the courts to prevent agencies from snooping on people not for law enforcement but to gain leverage and power. Your privacy helps us prevent the next J Edgar Hoover even if you have nothing to hide.

jay_kyburz(4300) 4 days ago [-]

I think this misses the point. When people say 'nothing to hide' they mean nothing to hide _from Law Enforcement_.

I think the only counter argument is to demonstrate how close many of our western democracies are to becoming fascist dictatorships.

We need privacy because if ever laws are passed that we don't agree with, we need to be able to break them until we can have the law changed.

For example, if it was against the law to be a pro-democracy protestor.

mzakharo1(10000) 2 days ago [-]

Strong encryption is another tool of the rich and powerful to help evade public oversight and scrutiny. Unpopular thought here, cuz, well, most people commenting here are the top wealthy 5-10%, the rich and powerful. It would be a cognitive dissonance to blieve otherwise.

newfeatureok(10000) 4 days ago [-]

I'm not really sure this is a good argument at all. In your scenario, if the two people have nothing to hide why would they care?

You're better of appealing to things that they normally might want to hide, like an embarrassing act or things that are said that are questionable out of context, etc.

A person who truly has nothing to hide, by definition, would not care about privacy. The entire point of privacy is that there are things that we do not want to share, for whatever reason.

nickthemagicman(4421) 4 days ago [-]

Moxie Marlinspike made a great point in one of his essays on his website about this.

There's 10 to a hundred thousand laws on the books already.

How do you know you don't have something to hide?

teddyh(2321) 4 days ago [-]

That argument is good, and is an example of the type which Bruce Schneier makes in The Eternal Value of Privacy:

https://www.wired.com/2006/05/the-eternal-value-of-privacy/

lr4444lr(4405) 4 days ago [-]

Most people implicitly mean, 'nothing unlawful to hide' when they say that, I think.

_pmf_(10000) 4 days ago [-]

Woke cancel culture is a bigger threat to honest people than most governments.

quadrifoliate(10000) 4 days ago [-]

In the US, this argument is even easier to debunk, especially when it comes from middle-class adults who've been working for a while.

'I have nothing to hide'

'What's your salary and your spouse's salary?'

The huge amount of secrecy around salaries in the US is proof that a lot of people have perfectly innocuous secrets that they very much like to hide.

Rapzid(4404) 4 days ago [-]

Having a conversation in a bar thinking it's private..

Anyway, it might at well be Pleasantville 1950s for a lot of people. They are both hiding everything and being suspicious AF of anybody with something to hide.

EGreg(1708) 4 days ago [-]

The police should wear body cams all the time, and we can end qualified immunity

Got nothing to hide and there will be no problem, right?

standardUser(10000) 4 days ago [-]

The 'nothing to hide' argument is so old and easily debunk-able, I don't take seriously anyone who offers it up at this point.

_AzMoo(10000) 4 days ago [-]

For those who aren't convinced by this argument, you can take it a little further by introducing a hypothetical situation in which their political opponents obtain power and criminalise criticism of government or other powerful 'common-good' entities. Now your recorded conversations could become evidence of sedition, and even if it wasn't criminal at the time it could certainly be used to show evidence of previous behaviour.

metta2uall(10000) 4 days ago [-]

But YouTube Live is publicly accessible, it's not the same as a law enforcement warrant. I think a better analogy is whether it should be okay for a judge to allow police officers to storm into your private residence with only a few seconds warning, and then proceed to examine and even take away anything in your room.

arafa(10000) 4 days ago [-]

I like to tell people about programs like 'LOVEINT' (https://en.wikipedia.org/wiki/LOVEINT). Where there is personal information available to others, it can and will be abused in ways you won't approve of. Privacy might seem like an abstract concern until you or someone you know is being stalked, etc.

treis(10000) 4 days ago [-]

This is a terrible analogy.

What's actually happening is that Facebook, Google, et.al. can read your messages and the Government can do so with a warrant. Nobody is planning on broadcasting everything to everyone. We don't have to ask people if they would do that or not. The vast majority of them do that today.

grawprog(3767) 4 days ago [-]

>here's how I like to get past the 'nothing to hide' argument.

Personally I like to just remind people if we had nothing to hide, nobody would wear clothing, own curtains, or care about peeping toms.

mindfulhack(4329) 4 days ago [-]

Humans need privacy because of the selfishness of our survival biology. There simply are bad people in the world, and I wish it were different, but it's not. 'Just trust us' are words often told to us by people who only want power - at our expense.

The best sentiment I've ever heard about privacy is from Shoshana Zuboff, and it's now influenced my thoughts. It's at the end of her amazing 2018 book 'Surveillance Capitalism', the best book on tech I have ever read. She shares a conclusion along this line: To be private and free from surveillance is to be human. If we have no privacy, we lose our humanity.

She explains that there is a direct connection between being watched, and being controlled. We need freedom of diversity - to will - to decide for ourselves. Otherwise, and now I conclude, bad actors with power lust will only bend the world to their vision.

The guilt-tripping words of 'nothing to hide' come from a place of privilege - or worse, hypocrisy. It is an insult to anyone who is different from the norm, and who wants to be themselves. What you have every right to hide does not even have to be illegal, but often it is, like LGBTQ people last century. Illegal ≠ immoral.

Privacy is a core freedom to protect human diversity. It must not be shamed. It must be loved. Cherished. Treasured.

Privacy begets diversity. Surveillance begets conformity. Which one is more important for the development of life in the universe?

As a child, I instinctively knew that privacy was something natural and important in life. Now as an adult, with education and intellect I can articulate why - and resist psychological manipulation and intellectual dishonesty from others trying to tell us that privacy should die.

I'm able to see the insanity of my own age. And I will guard privacy with my life on the street, if I have to.

jancsika(10000) 2 days ago [-]

The correct response to, 'I've got nothing to hide,' is a smug, 'You definitely have something to hide.'

Then the person responds, 'What are you talking about?' you say:

'I can post it on Reddit tomorrow if you're serious about this.'

Wait a few seconds, then change the subject.

lubesGordi(10000) 3 days ago [-]

You don't feel you have anything to hide, now. But if ever the government started doing something you disagreed with, like started getting too powerful, or using that power unjustly, your ability to check that power would be significantly diminished.

RcouF1uZ4gsC(4241) 4 days ago [-]

> Imagine we're sitting at a bar, chatting. None of us have anything to hide. Then the government passes a law that all conversations must be streamed on Youtube Live, so an agent comes in, sets up a camera at our table, and starts streaming.

If we are doing imaginary situations, the proponents of this bill can also do something like this:

Imagine there is a bar, where the police are unable to go inside. Every day, people go into the bar to plan terror attacks or to spread hate and racism. They can have whoever from across the world join them in the private bar without the police being able to find out. Sometimes they also bring young children into the bar to sexually exploit them. Other times, they bring vulnerable young men and indoctrinate them with their racist hate. The owner of the bar knows that those kinds of things are happening, but they say, a lot of people just come into the bar to talk with friends, so we never allow the police to come. In addition, before serving drinks, the owner puts on a blindfold and stuffs their ears with cotton so they won't be able to see or hear what is going on. They live by the principle: See no evil, hear no evil. The owner of the bar is making a lot of money from the people using the bar.

Don't you think we should keep the terrorists, and neo-Nazis, and child molesters from having a bar that they can go to to do and plan their evil stuff that police cannot get to? Don't you think that the bar owners, instead of stuffing their ears and covering their eyes so they are unable to see the bad stuff, should make sure that bad people aren't using their bar for evil stuff. Sure, just like in the real world, you will have your privacy just like you have in your home or at the local bar. But, you would want a real life bar owner to not close their eyes to people using their bar as a planning ground for terrorism, or who are bringing young kids with fake id's to the bar to exploit them, or who are filling young men's minds with hateful, racist lies? Also, just like the police can get a search warrant for a bad person's home or a bar where a lot of bad people are congregating and planning and doing bad things, we think they should also be allowed to do that in the online world.

Now tell me, to the average non-tech voter, which will pull the most emotional strings?

pwdisswordfish2(10000) 4 days ago [-]

Unless this bill's critics are mistaken, if your friends are using 'file encryption', i.e, encrypting the contents of the communications themselves before sending, then this bill has no effect on that.

IMO, your friends, unless they are nerds, are never going to place much value on encryption when they never actually use it themselves. When some third party tech company is doing the 'end-to-end encryption' for them then your friends are really not in control of the situation, let alone having any reason to understand it. It could be a sham and they might not know it.

This bill stands to hurt tech companies but according to its critics (surprise: it's the tech companies) it does not interfere with any individual's use of encryption for communications.

No one wants third parties to have potential access their communications. However there is no law preventing a tech company that provides 'encrypted messaging' from decrypting people's communications. There is not even a contract with users. If they do this, their liability is zero. There are only 'promises', something like, 'We promise we won't do it, unless we receive a valid subpoena' or 'We have made it technically impossible to see your messages.' (Nevermind metadata.)

Well, what if some other third party, like the people pushing this bill, made the same promises? Would that be 'good enough'?

Of course not.

For the record, I am not taking a for/against position here. I am not a tech company nor someone trying to get access to encrypted messaging services provided by tech companies. However if something came along -- not suggesting this is it -- that created the necessary evolutionary pressure to make every person, not just 'bad guys', have a basic understanding of how to encrypt a file and send it over an insecure network, without relying on a third party for the encryption, then I would think that would be a good thing.

2OEH8eoCRo0(4403) 4 days ago [-]

I'm stealing your analogy but I'm also ready for people to smugly proclaim that they would have no issue with livestreaming their entire life since they have nothing to hide.

lovelyviking(10000) 4 days ago [-]

May be a good argument would be that it's not about 'nothing to hide' it's about your authority to decide wether you wish something to hide or not.

godelski(4409) 4 days ago [-]

I mentioned something about this yesterday in another thread: https://news.ycombinator.com/item?id=23693298

Basically it isn't about having anything to hide but the instability of democracy and what power bad actors have.

Do I have anything to hide from Google? Not really. But do I want that same data that Google has to be in the hands of someone like Putin? No. I don't even want it in the hands of the NSA. The issue is that if we say that ads can manipulate people to buy things, why can't they manipulate people to do things like vote or divide. Russia's strategy since the Cold War has been to divide and prod The West, to sow disruption. That disruption has caused consolidation of power but also makes it difficult for coalitions to get things done.

It is clear that anger generates more clicks, so why is it unrealistic to think bad actors can use that data to better divide us and sow discontent?

The next factor is that democracy relies on a distribution of power. Data collection is a means to consolidate power. There's the term 'turnkey tyranny' that's thrown around. The reason isn't because we think a tyrant is going to come to power and destroy our way of life, but rather that we recognize that such a thing is possible and want to ensure that such actions would be infeasible if a malicious actor gained power. In democracy power is distributed. This has pros and cons. But the point of distribution is so that consolidation is difficult and we can never have a monarch or tyrant.

So it has never been about having something to hide (which btw, do people know they are referencing Goebbels?), but about stability in democracy. Distribution of power that was inherent to the system in the past is no longer built in. Technology has changed and enabled things we never previously imagined.

andrepd(3845) 4 days ago [-]

There's other ways. Ask them for their passwords, ask them to hand their phone and riffle through their conversations and photos. If I know they won't be offended them I'll ask them to send me nudes or if they would let me put a camera in their shower.

After all, if you're have nothing wrong you have nothing to hide, so why should you close your shower curtain?

DudeInBasement(10000) 4 days ago [-]

Why does this matter? The US government can force you to give them your keys, and gag you so you can't say anything...

gspr(10000) 4 days ago [-]

Yes, but at least when they do that they don't have to fundamentally break crypto. Not that it's OK. I'm just saying.

knodi123(10000) 4 days ago [-]

Because what you're describing is a method for violating the rights of one person, and they know it's happening. What's described in the article is a method for violating the rights of anyone, and they won't even know.

ipnon(4405) 4 days ago [-]

The primary impediment to combating child sexual abuse material is the scarcity of law enforcement agents relative to the scale of offenders.[0] Law enforcement agencies are already at the limit of how many offenders they can prosecute because offenders are primarily detected by agents who are posing as offenders or by agents who are posing as victims.

The solution proposed by the researchers in the referenced documentary is to create AI agents that can automate the work of posing as offenders or posing as victims. This solves the scaling problem of the law enforcement agencies.

[0] https://www.youtube.com/watch?v=xcCj1zNpKoM

darkarmani(10000) 4 days ago [-]

So, create some dog-and-ponies AI start-up for this in the districts of the most influential lawmakers, with stated job counts (for AI-mechanical turk work that hires lots of people).

Tie the 'correct' solution to their real political motivations and it will get done in a heartbeat. It has to be something they can sell to their base though.

ajzinsbwbs(10000) 4 days ago [-]

That scares me. If that entrapment-at-scale is legal, it could be deployed to throw huge numbers of people in jail for buying drugs or supporting terrorism.

zelly(10000) 4 days ago [-]

This is the consequence of companies like Discord letting child predators run rampant on their platform and doing nothing about it. There are two sides to every story. You can imagine how the government would react after seeing hundreds and hundreds of cases coming out of Discord, Snap, even FB.

Reading the actual bill[1] rather than sensational headlines, you can see this is mostly about creating a tip line and bureauracy for reporting CP. Ctrl+F 'encrypt' 0 results. Me no care. Deal with it Discord. Pass it.

[1] https://www.congress.gov/116/bills/s3398/BILLS-116s3398is.pd...

ascotan(10000) 4 days ago [-]

>>>Not later than 1 year after the date on which a bill that contains recommended best practices submitted to Congress ... and annually thereafter, an officer of a provider of an interactive computer service may submit a written certification to the Attorney General stating that the provider (1) has conducted a thorough review of the implementation and operation of the best practices; and (2) has a reasonable basis to conclude that review does not reveal any material non-compliance with the requirements of the best practices

>>>Whenever the Attorney General has reason to believe that an officer of a provider of an interactive computer service has filed a false certification ... the Attorney General may issue ... a civil investigative demand requiring the provider to (i) produce any documentary material ... answer in writing written interrogatories ... give oral testimony ... furnish any combination of such material, answers, or testimony.

>>>The Attorney General shall maintain on the website of the Department of Justice a public list of each provider of an interactive computer service for which a certification has been submitted ...

>>>The best practices required to be developed and submitted ... shall include alternatives that take into consideration ... the transmission and storage of information on behalf of other interactive computer services ... provides the capability to transmit data to and receive data from all or substantially all internet endpoints on behalf of a consumer

For the last 10 years, encryption/privacy has become the watchword for people around the globe. I remember back when gmail was the first to use HTTPS for email and people freaked out. Today it's an expectation for everything and every service.

I think there are competing concerns here. The need for privacy and the need for safety. In order to keep everyone safe, we need to monitor what your doing. Some people will buy this, others wont. I think it depends on if you trust the monitor.

Its back to the 'do you have anything to hide' conversation, which I think misses the point. The point, I think, is that there is a fundamental right to privacy, that in some sense, is frustrating and annoying. It's the battles over WhatsApp, Apple IPhones, etc.

Safety and Freedom don't always go hand-in-hand. I remember watching TV shows/movies back in the day where there Soviets with AK-47s asking people for their 'papers' as they get on a train. Today there are armed police walking the hallways of my kids schools, at concert venues, TSA screeners at the airport, etc. There is a price for safety imho. There's a price for freedom too.

I'm not sure what the right answer is, but i wholly support petitions like this because they are the voice of the people.

markkanof(4439) 4 days ago [-]

I initially thought you couldn't possibly be right about this, but what you stated does seem to be the case. After skimming through the full text, the main point of this seems to be about setting up a committee that will come up with 'best practices' that online service providers can follow to reduce/eliminate the exploitation of children on their platform. Can someone explain how this 'creates a threat to strong encryption'.

qppo(10000) 4 days ago [-]

To Californians, Dianne Feinstein is one of the cosponsors of the bill in the Senate. It really grinds my gears seeing our senior leadership from California take actions that directly harm our interests, both towards our peoples' freedom and our businesses.

I feel vindicated for voting against her in 2018, because I thought she'd be out of touch with our interests as a state.

choppaface(4029) 4 days ago [-]

Send her an e-mail in 2 minutes telling her to stop supporting EARN-IT https://www.feinstein.senate.gov/public/index.cfm/e-mail-me

coldpie(2406) 4 days ago [-]

Feinstein is easily the worst Democratic Senator. It's absolutely bonkers to me that a state where tech is as important as it is keeps re-electing her. Surely there's someone out there who wants to primary her?

hedora(3777) 4 days ago [-]

She will be 91 the next time she is up for re-election. Hopefully she'll finally retire instead. I've been hoping for that for over a decade.

Feinstein could not have been re-elected if it weren't for overwhelmingly strong support for her in Silicon Valley.

If she is up for re-election again, please vote for the other candidate (who will probably also be a Democrat!). Also, tell your friends.

She wants to ban encryption. Her voting record is further to the right than many republicans. She votes according to Trump's wishes more than any other Democrat in the senate.

shpongled(10000) 4 days ago [-]

Why would you be surprised? California has a long history of stepping on people's freedoms and rights

president(3560) 4 days ago [-]

Has anyone considered that maybe there are actually people in this world that want to be able to catch criminals and terrorists? Even if the bill does result in adding backdoors (which is all based on speculation anyway), you don't think people and technology would recalibrate itself to overcome issues caused by backdoors? I find it insane that people think technology is so much more important than the potential for saving human lives.

pphysch(10000) 4 days ago [-]

I'm all for _benevolent_ authoritarian governance.

Problems start to arise when the government pretends they _are_ the former and pretends they _are_not_ the latter.

antepodius(10000) 4 days ago [-]

It's Liberty vs. Security.

Shared404(4419) 4 days ago [-]

Alternately, we could recalibrate the methods used to catch criminals and terrorists so that we did _not_ have to yield up all of our information to the gov, as well as every script kiddie with an internet connection.

danboarder(894) 4 days ago [-]

Where are my elected representatives that represent my privacy rights on bills like this? Maybe I need to run for office since I care about end2end encryption, free software, and the rights of free people and information to freely travel around the world...

yummypaint(4226) 4 days ago [-]

You should call their offices. Give your qualifications and make their staff listen to you explain all the reasons why its a terrible idea. It can move the needle if enough people do it. If you have the energy to run for office then you should start by lobbying for these issues, it's good practice

AcerbicZero(10000) 4 days ago [-]

Awkwardly enough the only Senator I have even the smallest amount of respect for these days did add a bunch of amendments to try and fix parts of it....but I think its a shit bill from top to bottom and I expect him to vote against it anyway.

Perhaps instead of supporting politicians who 'say' they're going to do X, Y, and Z we should support politicians who have a track record of doing exactly X, Y and Z regardless of how politically convenient it is to do something else.

More laws will not fix the problem of having too many bad laws, in my opinion.

YetAnotherMatt(10000) 4 days ago [-]

In Europe quite a few 'pirate parties' popped up about a decade ago. None of them were too successful as far as I know, at least in The Netherlands they didn't mention to gain a single seat (out of 150).

mikestew(4078) 4 days ago [-]

You can run, and with that platform your opponent will conveniently turn it around into 'my opponent supports child molestors and terrorists'. Because if you think you'll simply walk up to the podium talking about something most folks could not give a working definition for, boy, are you going to be in for a shock when your opponent 'dumbs it down' for the audience.

Which is how we end up with stuff like the EARN IT Act that one would reasonably think would be shot down immediately, but isn't.

hristov(4236) 4 days ago [-]

It is interesting how this dovetails with the trade war with China. When the media got all hot and heavy about evil Huawei and how the would use their communication equipment to spy on America, my response was simple. They can't spy on us if we encrypt everything. If they want to give us cheap subsidized telecom equipment, let them, we will just use encryption for everything.

Then the government decided to ban encryption and then things started to get clearer.

dragonwriter(4447) 4 days ago [-]

> They can't spy on us if we encrypt everything.

Well, they can if it gets onto their hardware before being encrypted or after being decrypted, such as if it is encrypted or decrypted on their hardware, and since Huawei makes a lot of consumer gear...

justin66(2306) 4 days ago [-]

In one sense, the anti-crypto forces are not yet being as crazy as they have been at certain times in the past. People often forget that the federal government wanted to put Phil Zimmerman in prison over PGP in the nineties. [1]

In another sense, this is much crazier. There wasn't much at stake commercially in the nineties, but today, legislatively screwing up crypto could compromise trillions of dollars worth of commercial transactions if something goes wrong.

https://philzimmermann.com/EN/news/PRZ_case_dropped.html

upofadown(4402) 4 days ago [-]

It's more or less the same thing. PGP was used as an example of why the Clipper chip was pointless and futile. PGP is an example of why the Earn IT act is pointless and futile.

The Earn IT act is just another instance of the sort of incomplete thinking that produced the Clipper chip.

pochamago(10000) 4 days ago [-]

I don't really understand what about the bill is distinctly bad for encryption. Reading the Library of Congress summary, it's mostly just creating a body to draft guidelines. It seems like it's mostly just Congress punting the issue over to the executive branch, so there aren't any actual guidelines to oppose yet.

jagged-chisel(10000) 4 days ago [-]

The problem in this case is allowing the executive to draft "guidelines" with the force of law. If the whims of the executive decide what laws we abide by, it'll be an ever-changing landscape used to harass political opponents, various racial groups, and whomever are the current administration's less-desirables.

surround(3776) 4 days ago [-]

"A Parable" by Perry E. Metzger (1993)

> There was once a far away land called Ruritania, and in Ruritania there was a strange phenonmenon -- all the trees that grew in Ruritainia were transparent. Now, in the days when people had lived in mud huts, this had not been a problem, but now high-tech wood technology had been developed, and in the new age of wood, everyone in Ruritania found that their homes were all 100% see through...

> One day, a smart man invented paint -- and if you painted your house, suddenly the police couldn't watch all your actions at will...

> Indignant, the state decided to try to require that all homes have video cameras installed in every nook and cranny. 'After all', they said, 'with this new development crime could run rampant. Installing video cameras doesn't mean that the police get any new capability -- they are just keeping the old one.' [...]

https://cypherpunks.venona.com/date/1993/04/msg00559.html

(I recommend reading the full parable.)

—————

clock.org homepage (c. 1998)

> Some agencies of the United States Government (notably the Federal Bureau of Investigation (FBI)), want to prevent the deployment of encryption technology. They want either encryption so weak that just about anyone can break it, or they want a copy of every key used with strong encryption.

https://web.archive.org/web/19980113124617/http://www.clock....

Sound familiar?

—————

Mujahedeen Secrets (first release: 2007), Al-Queda's own encrypted messaging software. Those developers aren't going to respond to a US court order.

https://en.m.wikipedia.org/wiki/Mujahedeen_Secrets

RcouF1uZ4gsC(4241) 4 days ago [-]

> Mujahedeen Secrets (first release: 2007), Al-Queda's own encrypted messaging software. Those developers aren't going to respond to a US court order.

Even with that, you are now forcing the terrorists to roll their own encryption. The likelihood of finding an exploit in Mujahadeen Secrets is likely much higher than finding an exploit in Apple Messenger or WhatsApp or something by djb or Moxie, etc. In addition, Mujahadeen is likely having to be side loaded - you won't find it in the App Store. This gives a chance to try to hack a single storage location to have a trojan horse version spread through their network. Finally, it affects recruiting. Basically only someone already comitted will go to the trouble of having Mujahadeen Secrets on their device, whereas a lot more people could be expected to have a common end-to-end messenger on their device. So overall, thinking from a purely law enforcement, anti-terrorist perspective, it would still be better for them to not have e2e in widely used apps, even if the terrorists tried to develop their own app.

jjoonathan(10000) 4 days ago [-]

The people who want a skeleton key to open all electronic doors are the same people who compiled a blackmail database on their own high-clearance employees and failed to keep it safe (the OPM data breach).

chillacy(10000) 4 days ago [-]

> This would be equivalent to a town with no locks on the front doors, or where the sheriff has a copy of every door key (just in case he has to search the house).

To be accurate though that's not exactly true. The sheriff doesn't even need a key to enter your house, he can get a warrant and bust your door down.

This is a town where your house is indestructible and un-enterable without a key.

granitDev(10000) 4 days ago [-]

There's another good reason for term limits, how many of these VERY OLD people don't even understand what encryption is and couldn't possibly understand the concept that this is like passing a law demanding that you leave your house key at the local police station, unless you let them search your house every second Tuesday.

JoshTriplett(194) 4 days ago [-]

Analogies to physical security are not helpful, because they're easily countered with 'but you support someone getting into a house with a warrant, right?'. Encryption is nothing like physical security. There is no analogue to busting down a door. Correctly built encryption has no way to get in without the key; anything else has fundamental, unfixable security flaws.





Historical Discussions: India bans TikTok, WeChat, and dozens of other Chinese apps (June 29, 2020: 1482 points)
India bans TikTok, dozens of other Chinese apps (June 29, 2020: 55 points)
India Bans TikTok, Dozens of Other Chinese Apps (June 29, 2020: 10 points)

(1482) India bans TikTok, WeChat, and dozens of other Chinese apps

1482 points 6 days ago by samdung in 1543rd position

techcrunch.com | Estimated reading time – 5 minutes | comments | anchor

The Indian government on Monday evening said it was banning 59 apps developed by Chinese firms over concerns that these apps were engaging in activities that threatened "national security and defence of India, which ultimately impinges upon the sovereignty and integrity of India" in what is the latest standoff between the world's two most populated nations.

Among the apps that India's Ministry of Electronics and IT has ordered to ban include ByteDance's TikTok, which counts India as its biggest overseas market; Community and Video Call apps from Xiaomi, which is the top smartphone vendor in India; two of Alibaba Group's apps (UC Browser and UC News); Shareit; CM Browser, Club Factory, which claims to be India's third-largest e-commerce firm; and ES File Explorer.

This is the first time that India, the world's second largest internet market with nearly half of its 1.3 billion population online, has ordered to ban so many foreign apps. New Delhi said nation's Computer Emergency Response Team had received many "representations from citizens regarding security of data and breach of privacy impacting upon public order issues."

"The compilation of these data, its mining and profiling by elements hostile to national security and defence of India," it said.

The apps India is banning

Tarun Pathak, an analyst at research firm Counterpoint, said the order would impact roughly one in three smartphone users in India. TikTok, Club Factory and UC Browser and other apps put together had more than 500 million monthly active users in May, according to one of the top mobile insight firms.

And, 27 of these 59 apps were among the top 1,000 Android apps in India last month, according to the mobile insights firm — data of which an industry executive shared with TechCrunch.

It's unclear what exactly the "ban" means and how mobile operating system makers and internet service providers are expected to comply. At the time of writing, all of the aforementioned apps were available to download from Google Play Store and Apple's App Store in India.

Google said it had yet to receive the order from New Delhi. Apple said it was reviewing the order. The companies have traditionally complied with such app removal requests.

New Delhi said it had received "many complaints from various sources, including several reports about misuse of some mobile apps available on Android and iOS platforms for stealing and surreptitiously transmitting users' data in an unauthorized manner to servers which have locations outside India."

Monday evening's announcement is the latest standoff between the two neighboring nations following a deadly clash at the border earlier this month that stoked historical tensions. In recent weeks, custom officials at major Indian ports and airports have halted clearances of industrial consignments coming from China.

Jayanth Kolla, an analyst at research firm Convergence Catalyst, told TechCrunch the move was surprising and will have huge impact on Chinese firms, many of which count India as their biggest market. He said banning these apps would also hurt the livelihood of several Indians who directly or indirectly work for them.

Anti-China sentiment has been gaining mindshare in India in recent weeks, since more than 20 Indian soldiers were killed in a military clash in the Himalayas earlier this month. "Boycott China" — and variations of it — has been trending on Twitter in India ever since as a growing number of people posted videos demonstrating destruction of Chinese-made smartphones, TVs and other products.

Chinese smartphone makers command more than 80% of the smartphone market in India, which is the world's second largest. For SoftBank-backed TikTok, which has more than 200 million monthly active users in India, New Delhi's move is its latest headache. The Chinese firm has also faced scrutiny in Europe and the United States in recent quarters.

TikTok has been facing backlash in India since the second half of May after users unearthed and shared numerous recent TikTok videos on Twitter that appeared to promote domestic violence, animal cruelty, racism, child abuse and objectification of women. Many in India rushed to leave a poor rating of the TikTok app in the Google Play Store to express their disgust — and the Android-maker had to intervene and delete millions of comments.

Days later, an app called "Remove China Apps" gained popularity among some Indians. Google pulled the app later from the Play Store citing it violated its guidelines. A TikTok spokesperson did not immediately respond to a request for comment.

In April, India amended its foreign direct investment policy to require all neighboring nations, including China, with which it shares a boundary to seek approval from New Delhi for their future investments in India. The nation's Department of Promotion of Industry and Internal Trade said it was taking this measure to "curb the opportunistic takeover" of Indian firms that are grappling with challenges due to the coronavirus crises.

When TikTok app was blocked in India for a week last year, ByteDance had said in a court filing that it was losing more than $500,000 a day in the nation. In a statement on Tuesday (local time), TikTok said that it was working to comply with New Delhi's order.




All Comments: [-] | anchor

kalesh(10000) 6 days ago [-]

I have a conspiracy theory here. Alibaba & AliExpress are missing in that banned list. This seems more like a direct hit on ByteDance rather than China.

Facebook recently invested in Reliance Jio. There have been other investments from Silver Lake, Vista Equity Partners etc.

TikTok is a big threat to FB, Instagram & YouTube. ByteDance apps TikTok, Vigo Video, Helo have around 300 million users in India. Facebook user base is 280 million. ByteDance had more apps planned for this year.

Banning BD is a big win for Facebook & Reliance. Government also get's to score a few points in this coronavirus mess.

acd10j(10000) 6 days ago [-]

Alibaba and aliexpress are not popular in india, hardly anyone uses them in India. May be that's why they were spared banning.

mav3rick(4244) 6 days ago [-]

Its more to do with the border skirmish.

NicoJuicy(394) 6 days ago [-]

And it's just a conspiracy theory.

When military is getting active, they seriously don't care about a chat app.

Except if it's from the invader ofc.

vijaybritto(4361) 6 days ago [-]

China moved into Indian territory and has killed 20 Indian soldiers and this is how India is retaliating. The prime minister is not even mentioning China in his speeches or tweets. This has been a disaster for our country.

stevens32(10000) 6 days ago [-]

Don't worry, apparently India has 'snapped the necks of at least 18 PLA'

https://swarajyamag.com/news-brief/indian-soldiers-merciless...

sameerds(10000) 6 days ago [-]

What surprises me is that the discussion here is taking the original article at face value. Sure, the Indian govt is citing national security as the reason for the ban. But take a slightly cynical view, and focus your attention on this sentence:

'Jayanth Kolla, an analyst at research firm Convergence Catalyst, told TechCrunch the move was surprising and will have huge impact on Chinese firms, many of which count India as their biggest market.'

That's the real reason. This is just posturing between two neighbouring countries who are currently involved in a border dispute. There is already a lot of military posturing, a lot of diplomatic posturing and now a lot of market posturing, which includes the Internet.

rsanek(10000) 6 days ago [-]

It's probable that the border issue made them more willing to take actions that they might normally avoid in fears of unnecessarily angering China. I think of it less as pure posturing and more as, if we're already in a border dispute we might as well take the opportunity to do some things we planned to but put off.

factorialboy(2970) 6 days ago [-]

Am I the only one concerned about how TikTok is promoting child pornography by serving as a gateway for young girls and boys to prostitution apps such as OnlyFans?

I think this is a bigger concern than loss of privacy.

Those comparing loss of privacy for Indians on US apps / platforms compared to Chinese apps / platforms, China and India are openly hostile.

US and India are not. This is a big difference.

crispyporkbites(4391) 6 days ago [-]

I wouldn't compare onlyfans to prostitution, it's not the same as going out on the street. If anything it's empowering.

phoe-krk(1889) 6 days ago [-]

Clash of Kings? That online game? Why?

dmix(1519) 6 days ago [-]

Clash of Kings is owned by a Chinese company called Elex-Tech aka 'Beijing ELEX Technology Co'. https://en.wikipedia.org/wiki/Clash_of_Kings

Not to be confused with the more popular 'Clash of Clans' which is owned by a Finnish company https://en.wikipedia.org/wiki/Clash_of_Clans

tech_dreamer(10000) 6 days ago [-]

It has everything to do with current border skirmish. Also, 44% percent of tiktok users are from India.

qeternity(4397) 6 days ago [-]

Source on the demographics? That's really interesting.

op03(4147) 6 days ago [-]

The kids are going to be pissed with Mr. Modi :) I hope he has a plan of where this goes.

yyhhsj0521(10000) 6 days ago [-]

Kids can't vote.

tksb(10000) 6 days ago [-]

I've been passively thinking about this recently. Is there any pathway for individual users to (effectively) do the same for their devices or network, or both? Is IP blocking via hosts or otherwise still effective enough to be useful?

quantummkv(4392) 6 days ago [-]

For networks, a Raspberry pi with pihole would be great starting step that still works

mathnmusic(4308) 6 days ago [-]

I think this sort of response was inevitable and will be seen in more and more nations. China bans most of the popular websites and apps (including Wikipedia). The most recent trigger was the border standoff between India and China where Indian citizens could see and read both governments' responses, but Chinese citizens were only told Chinese govt's talking points.

This seems like yet another instance of the paradox of tolerance (reciprocity is a must have for a tolerant/liberal/globalized society): https://en.wikipedia.org/wiki/Paradox_of_tolerance

bad_user(3400) 6 days ago [-]

The paradox of tolerance is bullshit though, as it's not based on empirical evidence.

It's just armchair philosophy used as justification for intolerance by intellectuals, the mental gymnastic people need to get over their cognitive dissonance.

AFAIK Plato came up with it to justify autocracy. That says it all actually.

kanox(10000) 6 days ago [-]

How is 'tolerance' relevant to banning apps based on national origin?

This is nothing but good old fashion economic escalation.

xster(4445) 6 days ago [-]

Though if you apply your argument with rigor and reciprocity was in play, more countries should ban american social media which censors geopolitical rivals

https://thegrayzone.com/2020/01/12/us-pressure-social-media-...

cmonnow(10000) 6 days ago [-]

> paradox of tolerance

Unnecessary use of the 'paradox' label.

If a criminal shoots a cop, that is violence.

If a cop shoots the criminal back, is that the 'paradox of violence' ?

If a surgeon cuts open a patient with a knife, to treat a tumor, is that the 'paradox of violence' ?

Violence used to curb violence is peace. Peace used to ignore violence is violence.

There is nothing 'paradoxical' about not tolerating the intolerant. That is basic justice.

Aunche(10000) 6 days ago [-]

How does this have anything to do with tolerance? India has never been known to highly value tolerance, and this move is about national security rather than censorship on the Chinese apps.

apta(4077) 6 days ago [-]

> I think this sort of response was inevitable and will be seen in more and more nations. China bans most of the popular websites and apps (including Wikipedia).

Hopefully the rest of the world follows suit and bans Chinese apps and websites.

chrischen(2262) 6 days ago [-]

The China censors things so we should too stance just seems to strengthen their position of censorship...

rezeroed(10000) 6 days ago [-]

What about Chinese phones and their customised Android?

suyash(3843) 6 days ago [-]

Lot of countries are banning Chinese 5G hardware for the same reasons. Trump also issued ban on mobile device makes like ZTE etc for the same reason.

stevewodil(10000) 6 days ago [-]

How does this ban actually get enforced? How are the apps blocked on devices which already have them installed?

khuey(3936) 6 days ago [-]

Sounds like it'll be a combination of pulling them from the app stores in India and ISPs blocking connections to the relevant server IPs.

chvid(3748) 6 days ago [-]

Sad to see India degenerating into nationalism and populism.

pvelagal(10000) 6 days ago [-]

Why does China ban Google, Twitter, Facebook and Youtube ? Why ? What are they afraid of ? https://en.wikipedia.org/wiki/List_of_websites_blocked_in_ma...

stevens32(10000) 6 days ago [-]

I feel like this could be said for the rest of the world as well

zandroid(10000) 6 days ago [-]

InShot is missing.

eqtn(4267) 6 days ago [-]

Resso is missing

zachguo(2514) 6 days ago [-]

Surprisingly HNers are embracing the concept of cyber sovereignty that China and Russia are pushing.

sergiotapia(834) 6 days ago [-]

money makes it easy for them.

raghava(4164) 6 days ago [-]

This news must be seen along with what happened early this year.

https://www.medianama.com/2020/01/223-the-great-indian-firew...

A Great Indian Firewall is shaping up. Now that the sentiment is all 'anti-China!', people would celebrate instead of protesting the great firewall.

'Masterstroke!'* - Indian public.

*Basically, anything the current leader does, his PR team and his party spokespersons end up terming 'unprecedented' / 'masterstroke' / 'genius move'. Everything seems headed towards hardcore Soviet scheme of things.

thowme245(10000) 6 days ago [-]

> This news must be seen along with what happened early this year.

And why shouldn't we see it in the context of Chinese aggression and TikTok's reported spying?

pvelagal(10000) 6 days ago [-]

Everyone questioning India, should first Question China ! Why does this country ban these services ? https://en.wikipedia.org/wiki/List_of_websites_blocked_in_ma...

hnarn(10000) 6 days ago [-]

> Everyone questioning India, should first Question China

Or, you could question both. Whataboutism isn't a valid argument, so censorship of the Internet can still be wrong at the same time as China is a malevolent force both on the Internet in general terms and for specific pieces of software or services in more concrete terms.

SoulMan(3458) 6 days ago [-]

Interestingly even before this ban, there has been a movement (at least by armchair critics in Social Media) to uninstall Chinese apps. People started installing this unknown app 'Remove china apps' to get rid of the apps that Chinese phone makers like Xiomi does not let you remove.

People started removing Chinese apps not because of national security but because they thought it would stop boosting Chinese economy which in turn will come as defense budget

Bigger question is how do you stop the market penetration of mobiles like Oppo/Vivo/Xiomi etc and all the Startups like Ola/Flipkart which is backed by Chinese investment.

https://www.indiatvnews.com/technology/news-remove-china-app...

Avamander(10000) 6 days ago [-]

That 'Remove China' app was also removed from Google Play: https://gadgets.ndtv.com/apps/news/remove-china-apps-google-...

kumarvvr(10000) 6 days ago [-]

This is a right move, in my view.

Chinese infiltration takes a myriad of forms and data collection is one of the biggest.

And those apps form a basis for click-of-the-button hacking.

And ordinary users will find it very very difficult to determine if an app is Chinese made or not.

And the Chinese govt. will have it's fingers in everyone of them, one way or the other.

What I do worry now is that since China has been exposed, it will resort to even elaborate deceptive methods to hide itself and it's infiltration.

China is not to be trusted.

hatenberg(10000) 6 days ago [-]

You'll see the same attitude to the US from the rest of the world too

cmonnow(10000) 6 days ago [-]

How long will it take China to rebrand these apps under a different name ?

x87678r(10000) 6 days ago [-]

India smartphone marketshare:

Xiaomi 30%

Vivo 17%

Samsung 16%

Realme 14%

Oppo 12%

Others 11%

https://www.counterpointresearch.com/india-smartphone-share/

bg24(10000) 6 days ago [-]

Curious why can't India learn the lesson that China uses to change these statistics in 3 months? Of course, if it is related to long-term national security.

- Ban the companies for 3 months. Then allow selectively. - Reduce the tax from South Korea and Japan.

amrrs(2769) 6 days ago [-]

Covid cases are on rise in India. Government didn't handle lockdown well to accommodate guest (migrant) workers and they were stranded. On top of all these, lives of Indian soldiers died recently in a clash with Chinese soldiers and China is continuously testing limits at Galwan Valley. You need a disruptive breaking news to crush all these and prove Government is in control. This is simply that. This is nothing but fueling the anti-China sentiment growing in the country. This will temporarily relieve government from answering the other growing concerns.

It's also quite ironic that so many companies with strong Chinese funding are still operating. Chinese smartphone makers are also doing great in India despite economic slowdown. OnePlus recently did a flash sale and sold out.

It'll be interesting to see what's going to happen in Long term. But this one, it's just a spicy headlines.

sbmthakur(3761) 6 days ago [-]

Boycotting China is not a binary thing. Banning Chinese funded Indian startups and Chinese mobile companies which have big plants in India(and support thousands of jobs) is a complex thing.

On the other hand, banning spyware from the country you are currently in conflict is common sense and not that tricky. It's far more than spicy headlines.

https://news.ycombinator.com/item?id=23634138

https://news.ycombinator.com/item?id=23638129

dmix(1519) 6 days ago [-]

I'm always skeptical about these government does x to distract from y arguments.

I highly doubt the indian government woke up one day and said 'how can we distract the public from COVID today?' and someone replied 'lets ban a bunch of highly popular mobile apps and games our citizens use to entertain themselves on the internet'.

That seems like a great way to make them less distracted, not more.

TriNetra(10000) 6 days ago [-]

no. there's more to it. THis stand-off, as India realizes, isn't going to end soon so country is preparing for that [0]. China is attempting to threaten/bully India for many reasons one of which is stopping India from creating near-border infrastructure. Prior governments ignored border infra on the Chinese side to appease China while China continued to develop infra on its side of the LAC. Modi has changed this policy [1].

0: https://economictimes.indiatimes.com/news/defence/ladakh-sta...

1: https://www.geopoliticalmonitor.com/the-end-of-indias-panipa...

kumarvvr(10000) 6 days ago [-]

As i said earlier in the thread.

India must shift its trade integration into democracies and wean away from China.

Everything must have a beginning and this is the right time for India to wake up and stop its indirect economic support toan authoritarian bully.

saltedonion(10000) 6 days ago [-]

Might be a form of shifting the narrative and scapegoating for sure. But you have to admit that recent events have outlined a significant shift in Indian-Sino foreign policy. I don't see this being reversed any time soon.

mratsim(4293) 6 days ago [-]

Seems like they learned from Trump and the change in visa/immigration to distract the public.

saltedonion(10000) 6 days ago [-]

India is absolutely right in banning TikTok as it is a significant national securities threat.

As long as the platforms recommendation and ranking algos are a black box, there is no guarantee that China isn't conducting misinformation campaigns via the platform.

At the very least, the government should audit the algos and make sure China can't arbitrarily alter ranking results.

ex3ndr(4226) 6 days ago [-]

I bet this is just a corrupted ban for promoting local app of friends of friends of some minister.

moscovium(10000) 6 days ago [-]

China bad

zacharytelschow(10000) 6 days ago [-]

> As long as the platforms recommendation and ranking algos are a black box, there is no guarantee that [insert platform here] isn't conducting misinformation campaigns via the platform.

Razengan(4170) 6 days ago [-]

> India is absolutely right in banning TikTok as it is a significant national securities threat.

As opposed to Skype, Facebook, WhatsApp, Snapchat and all the others?

mrlala(10000) 6 days ago [-]

I was skeptical about this, but I think they are surely are.

I installed tiktok just maybe 2 or 3 weeks ago.. it was kind of fun at first and was amused by some stuff on there. What it was showing me I thought was very relevant to prior stuff I had seen in some regard (just the type of humor) so I was kind of impressed.

Then all the sudden it started showing me all this pro-Trump/pro-religion/antiBLM stuff. And look, a platform like this and the age demographic the amount of pro-trump & pro-religion stuff is TINY. The amount that was coming up was baffling. And it would just be inserted at the weirdest times, where it honestly felt like I was being targeted to see this stuff.

I'm not one to be a conspiracy nut at all, but there were some flags hitting in my head that something weird was going on.. so I promptly uninstalled.

Honestly it's too bad, I think tiktok is a very unique thing, and seemed to fill some type of space where people just want to create stupid little videos that either say a little message or are just funny. But there's something weird going on.

john_ackerman(10000) 6 days ago [-]

I'm chinese and I'm glad the indian government is taking action before the US/Europe, which still haven't rstricted the spread of Chinese spyware.

1. Tiktok and wechat are bigger threats than huawei as they taken user generated data and biometric data to build a large surveillance network of not only chinese, but also folks living outside of the great fire wall. 2. The chinese government essentially bans the top 20 websites outside of china. Not seeking reciprocal treatment is the extension of clinton era illusion that somehow china will open up as they get richer.

I unfortunately can't disclose details on zoom's infringement but hope yall can figure out what going on behind the scene

justicezyx(10000) 6 days ago [-]

Chinese immigrant in US here:

I don't think it's a stretch to state the risk of tiktok being mass propaganda machine, from India's perspective.

Additionally, I don't take this as a particular politically charged statement against China, as quite a few replies stated. The reason is that China and India are on a very delicate geopolitical environment. The history is long and ambiguous. The current rivalry is subtle and dangerous. You just cannot give any chance in this situation. After all, China do not have any foreign social network services anyway. There is no reason to gift the opponent an potential upper hand.

The Pandora's box was formally opened in the Arab spring already. It was a well intended start, followed with an ugly development and messy prospect left for generations to suffer and struggle.

Now the whole idea of social networking services as an actual helper of connecting people with different cultural background roughly reduces to nil. That really was a buffer.

Lastly, I don't think it makes sense for any sovereign government to force their country's corporation to serve them directly.

That would immediately destroy any chance of those organizations to expand beyond their home country. Someone might argue Chinese firms are OK to that because they had a big market already. That's a totally unreasonable imagination on Chinese business men's brain structure. I never encountered any such Chinese business man who believe loyalty to CCP is higher than their profit. Other argument is that Chinese law can coerce, but all the laws are saying the company ought to corporate when necessary for the security of the country. I cannot imagine any sane political personnel can convince anyone else that offensive propaganda in peace time is necessary for national security. At least I did not see any such behavior or even minor behavior with hint of such reasoning from past history.

zkid18(4433) 6 days ago [-]

All RS algorithms follows non-trivial logic based on the user's feedback. They constantly adapt and upgraded. You can add some fuzzy-rules based on your moderation principles — remove porn, videos about flat earth, abusive content, and etc. But the core algorithm hard for interpreting.

Letting government step into this space creates an alternative form of censorship. I would rather ask all content platforms for transparent open-sourced moderation rules.

At the meantime, TikTok has a remarkable traction that put in a row of the greatest product of the decade. Previously ByteDance tried to move HQ away from China, but I now I guess they might even consider selling their product.

I guess it is more political decision rather than Indian government really cares about data security. Seems that US and specifically Facebook become the absolute winner in India.

jkingsbery(4413) 5 days ago [-]

My reading of the article is that it has a lot more to do with the border clash with China that killed 20 Indian soldiers with more believed to be captured (https://www.nytimes.com/2020/06/16/world/asia/indian-china-b...) and a subsequent desire to distance themselves from Chinese goods and services than it has to do with algorithmic transparency.

coopsmgoops(10000) 6 days ago [-]

The ranking algorithm is the least of anyone's worries in my opinion.

bigpumpkin(3605) 6 days ago [-]

You are going to use the government to audit black box algos?

ekianjo(331) 6 days ago [-]

Wasn't TikTok also caught intercepting the clipboard contents of the phones on which it was installed? Not sure about malware, but it could certainly be qualified as spyware if it actually does that.

kanox(10000) 6 days ago [-]

> India is absolutely right in banning TikTok as it is a significant national securities threat.

This is just a lame attempt at economic warfare and 'misinformation campaigns' is a very weak excuse.

What concerns me the most is that in 2020 people are overwhelmingly in favor of governments and corporations controlling what their people watch and read and think.

numpad0(4432) 6 days ago [-]

Since this is HN, what's the easiest way to roll your own brainwashing machine like Facebook or TikTok? I mean isn't it going to be fun to plant your own ideas on people...

mytailorisrich(10000) 6 days ago [-]

Covid-19 is biting extremely hard in India. Handling has been very poor so far.

Stroking nationalism and pointing at foreign enemies in hard times is a tried and tested tactic.

kalesh(10000) 6 days ago [-]

There is no such thing as ABSOLUTELY right. We don't even know what's actually going on at higher levels in the government. Why isn't Alibaba & AliExpress banned?

mcs_(4449) 6 days ago [-]

But, if a nation can be threatened by an application, a chat, or a social network, can the ban solve the problem?

joshribakoff(10000) 6 days ago [-]

There is no guarantee any app is not doing this, should we ban all apps?

edoo(10000) 6 days ago [-]

I also bet the Chinese version of TikTok, if it is even allowed in their country, has a completely different content moderation system. They are more than likely promoting degeneracy in our country while promoting their own ideals at home.

ccktlmazeltov(10000) 6 days ago [-]

> it is a significant national securities threat

come on...

29athrowaway(10000) 6 days ago [-]

A reddit user reverse engineered Tiktok and listed some of the information they collect and the extent to which they go to obfuscate their data exfiltration.

https://www.reddit.com/r/videos/comments/fxgi06/not_new_news...

See the top comment as the video has now been removed from YouTube.

tjpnz(10000) 6 days ago [-]

What I fear is whether this data could be incorporated into the state facial recognition systems already well deployed throughout China. You're potentially helping the CCP in training a tool to aid in the violent suppression of ethnic minorities.

KKKKkkkk1(971) 6 days ago [-]

Honestly, I don't see the difference between TikTok and other social media apps and the old media. Why not ban Hollywood, the Beatles, and McDonald's then? Aren't they a source of pernicious American influence on this generation's youth?

akshaybhalotia(4322) 6 days ago [-]

Very interesting. I would like to know more about this thought.

Is the problem, let's say, with:

- the country of origin, so if this was the exact same app from US it would have been less harmful, or,

- the amount of time people spend on it, so if people spent time on something like facebook it would have been less harmful?

est(3160) 6 days ago [-]

Well technically China did it first. China banned tons of foreign apps or websites because 'significant national securities threat'

suyash(3843) 6 days ago [-]

US should follow suit.

happppy(4309) 5 days ago [-]

like fb, youtube, twitter, linkedIn algos are audited. Just because tiktok is Chinese app, audit it?? what kind of hypocrisy is this.

Nginx487(10000) 5 days ago [-]

I hope other countries follow this absolutely wise decision. China will not hesitate to use any measures for weakening national security of countries it believes to be its enemies

jlmorton(10000) 6 days ago [-]

> there is no guarantee that China isn't conducting misinformation campaigns via the platform.

And who cares if they are? We live in an open society, and in the US, freedom of speech is a foundational principle. This is among the trade-offs you face when prioritizing absolute freedom of speech.

esafwan(10000) 6 days ago [-]

This looks mostly like gimmick PR stunt to cover the recent setback India had with China in the border. For example see this news from a channel that supports govt. shorturl.at/acjoM This is not a comedy show, its a real prime time new show.

If India was serious about taking action it would have done something about the Chinese branded phone and Chinese infra project. All the largest phone brands in India are chinese. Same is true for Tv's. All unicorn startups in the country has considerable chinese interest. This ban carefully stays away from all apps, businesses and things that could really invite a response from China.

And it worked. As in the link above, Modi's followed have taken the bait. They are happy and all over social media celebrating this as a fitting reply for the soldiers life lost in the border.

gtm1260(4448) 6 days ago [-]

I kind of agree, but 99% of the videos I see on FYP are just absolutely silly, hilarious content.

coliveira(3392) 6 days ago [-]

The same can be said of companies like FB and GOOG. They operate black boxes with little scrutiny from society.

NicoJuicy(394) 6 days ago [-]

Yeah, when I saw my neece on TikTok I was feeling really bad.

Most people don't see it, but I hate it to realize that she's being slowly influenced by hidden propaganda :(

ascotan(10000) 6 days ago [-]

I doubt they can audit close source software.

gazelleeatslion(10000) 6 days ago [-]

Is this regulated for US companies?

learnstats2(10000) 6 days ago [-]

>As long as the platforms recommendation and ranking algos are a black box

All platforms recommendation and ranking algos are ultimately a black box. Why is TikTok special here?

mlang23(3651) 6 days ago [-]

Well, the same applies to FaceBook, just to name one prominent example. Or Google, for that matter.

slim(4224) 6 days ago [-]

we, non americans, think facebook is doing everything you describe and even worse.

sniperjzp(10000) 6 days ago [-]

This is funny, can you elaborate how come a ranking algorithm is a national security threat? It reminds me of the time when Google/FB/Instagram/Twitter still worked in China, Chinese govt used the same excuse to ban big tech social platforms.

Reason077(10000) 6 days ago [-]

> 'India is absolutely right in banning TikTok as it is a significant national securities threat.'

I'm surprised, actually, that Trump hasn't expressed a desire ban TikTok. Especially after they trolled him so badly at that Tulsa rally!

He's managed to ban Huawei over (unproven and strenuously denied) security issues. Yet TikTok has direct influence over something much more important than 5G radio equipment: the hearts and minds of a generation of American children!

StreamBright(2858) 6 days ago [-]

Same reason the EU should ban Facebook and Google right?

gbraad(4320) 6 days ago [-]

> At the very least, the government should audit the algos and make sure China can't arbitrarily alter ranking results.

Can we do the same for YouTube ?!

billfruit(4176) 6 days ago [-]

But I doubt any due process was done before the ban, if any and concrete proof of wrongdoing is established before doing such things. Most of the bigger app developers will surely goto court.

If any this sounds like a propaganda move aimed at giving the people a feeling that the government has done something about it.

Some of these may relaunched as a web service, some may relaunch under a different branding, there are any number of things the app makers can legitimately do to skirt around this.

Using summary executive powers to do such is a rather undemocratic move, and which time and again this Indian Government has indicated it has no qualms of using.

This is a politically motivated move, where as it should not be. If there was established wrongdoing then ban the apps, not because it is politically convenient to do it.

ianleeclark(4439) 6 days ago [-]

Are YouTube algorithms open? I don't really see a difference here. Nation state vs. private corporation is different on paper, but I don't see why. They're both going to react to material stimuli to increase their standings.

gdubs(1471) 6 days ago [-]

Just from a more farcical standpoint, the idea of one country attacking another via dance trends feels like a page out of Zoolander.

OkGoDoIt(4374) 6 days ago [-]

So if WeChat is basically the only messaging app allowed in China, and WeChat is not allowed to be used in India, is there any widely available platform that Indian citizens can use to chat with Chinese citizens? Making it impossible or at least very difficult to communicate with people in another country seems like it's only going to make any divide even worse in the long term. Think of the fallout of Facebook filter bubbles but to an extreme, by making it very difficult to even communicate with someone in a different culture with different viewpoints.

Andrew_nenakhov(10000) 6 days ago [-]

Email should work. Or XMPP.

tanilama(10000) 6 days ago [-]

Telegram....

president(3560) 6 days ago [-]

Have you ever thought that maybe China shouldn't banned all foreign social media and messaging services in the first place? They literally divided the world by putting up a wall between themselves and the rest of the globe, the Great Firewall. Besides, the divide between China and the rest of the world has already happened on the ideological level.

sa1(10000) 6 days ago [-]

Signal works inside China, as long as you installed it outside.

kortilla(10000) 6 days ago [-]

The language barrier already guarantees that. Only 12.6 percent of Indians speak English and only 6.4 percent of Chinese speak English. The overlap is worse with any other common language.

ppeetteerr(10000) 6 days ago [-]

You can send a text message...

ketzu(4259) 6 days ago [-]

> So if WeChat is basically the only messaging app allowed in China

How did you get that impression? There are a ton of chinese messanging apps, e.g., QQ, which is also on the list of banned apps.

jshah111(10000) 6 days ago [-]

WhatsApp is incredibly popular in India.

jumelles(2726) 6 days ago [-]

Would iMessage work between India and China?

Darmody(4350) 6 days ago [-]

I'm absolutely against banning things but TikTok should be banned until we figure out how to protect ourselves from bad actors like the CCP.

unishark(10000) 6 days ago [-]

I'm not sure that 'absolutely' is the right word for you here.

riverlong(4333) 6 days ago [-]

I'm seeing lots of comments about national security-style concerns, and framing this in terms of the recent India-China skirmishes. That all makes sense.

Where I think a lot of folks are missing the point is that this is also a tremendous boost to local Indian entrepreneurship. One of the really clever aspects of China's Great Firewall is that it keeps out international competition, which would crush local startups. By banning more advanced, foreign competitors, India gives its local entrepreneurs a chance to grow hugely successful domestic apps, which can then compete internationally.

systemvoltage(10000) 6 days ago [-]

While I agree about symmetric response to China, here is the problem:

> By banning more advanced, foreign competitors, India gives its local entrepreneurs a chance to grow hugely successful domestic apps, which can then compete internationally.

If India bans international apps from competing within India, wouldn't Indian apps from these new found enterpreneurs expect the same response from other countries? There is a paradox here.

Why should any country allow Indian apps if they cannot compete in India? You realize this is the exact same situation as what India is doing with China. Now replace China with India.

What hypocrisy!

jimbob45(10000) 6 days ago [-]

https://en.m.wikipedia.org/wiki/List_of_countries_by_GDP_(no...

India's GDP is much much smaller than most think: its smaller than California's GDP. That doesn't seem so bad since they're so close to other European countries but remember that India's population is also considerably larger.

lioeters(3719) 6 days ago [-]

This is a good point, for me the most intriguing aspect of this ban.

Not only is it a good idea from security perspective - as well as privacy as a selling point - by forbidding specific, popular foreign services, it opens up the market for domestic players to grow. This is what China has done, and now India is doing the same to them.

Overall, this seems like a win for the people in India.

arjun27(4063) 6 days ago [-]

- Banning TikTok would move traffic to Instagram, not a "local Indian entrepreneur"

- the Indian startup ecosystem is dependent on external capital, a lot of which comes from China. This protectionist attitude will lead to a overall negative sentiment about investing in Indian tech.

moreorless(10000) 6 days ago [-]

> Where I think a lot of folks are missing the point is that this is also a tremendous boost to local Indian entrepreneurship.

Last I looked, a good portion of the funding for Indian startups is from China.

krn(1811) 6 days ago [-]

A reddit user, who claims to have reverse-engineered the TikTok app[1], concluded:

> TikTok is a data collection service that is thinly-veiled as a social network. If there is an API to get information on you, your contacts, or your device... well, they're using it.

> For what it's worth I've reversed the Instagram, Facebook, Reddit, and Twitter apps. They don't collect anywhere near the same amount of data that TikTok does, and they sure as hell aren't outright trying to hide exactly whats being sent like TikTok is.

It doesn't seem surprising now, given that Zoom, which is also being developed in China, acts like a malware application, too[2].

I'm glad, that India is more aware of the possible consequences of using any software made in China than, for instance, the government of the UK is[3].

[1] https://old.reddit.com/r/videos/comments/fxgi06/not_new_news...

[2] https://news.ycombinator.com/item?id=22748204

[3] https://www.bbc.com/news/technology-52126534

LockAndLol(3531) 6 days ago [-]

> TikTok is a data collection service that is thinly-veiled as a social network. If there is an API to get information on you, your contacts, or your device... well, they're using it.

They might as well be describing Facebook or Google. They are data harvesting services - first and foremost. The actual applications are only the bait yet since they're owned by the country that makes those great movies and TV series, somehow, they aren't as bad.

sethammons(4262) 6 days ago [-]

I didn't realize Zoom was mostly developed in China! That seems bad.

> "Our product development team is largely based in China, where personnel costs are less expensive than in many other jurisdictions," Zoom wrote in a regulatory filing.

https://www.cnbc.com/2019/03/26/zoom-key-profit-driver-ahead...

kanox(10000) 6 days ago [-]

> > If there is an API to get information on you, your contacts, or your device... well, they're using it. For what it's worth I've reversed the Instagram, Facebook, Reddit, and Twitter apps. They don't collect anywhere near the same amount of data that TikTok does, and they sure as hell aren't outright trying to hide exactly whats being sent like TikTok is.

I find this unconvincing and reddit comments are not trustworthy at all.

Wouldn't data collection be limited by the mobile OS anyway? I actually have TikTok on my phone and it requested no special permissions, compared to most other apps which don't even let you view content without validating a phone number.

stevens32(10000) 6 days ago [-]

> The scariest part of all of this is that much of the logging they're doing is remotely configurable

How is this scary at all - much less the 'scariest part'? The vast majority of the bullet points also seem standard and not worth paying attention to. I also read the Penetrum paper he linked which was similarly unconvincing.

ccktlmazeltov(10000) 6 days ago [-]

the MOST SCARY stuff for me is apps that ask for your photos album permissions (so they can save to it, or upload a picture ). This permission basically gives them access to ALL your photos, including your dick pics, even when the app is in the background.

captain_price7(10000) 6 days ago [-]

About that reddit comment, when pressed for evidence, turns out that guy's own research is conveniently 'lost' because of a motherboard failure. And someone in the comment pointed out that most of the things he found suspicious about tiktok is actually regularly employed by apps like fb, twitter etc.

I'm not taking sides and I don't have the technical expertise to judge everything said there. It's just that I'd be much more comfortable if all of those 'evidences' came from a more trusted source & not a reddit comment from god-knows-who.

Honestly, as someone who doesn't belong to China/USA/India and genuinely curious about this, I'm tired of seeing all this 'but but communist evil' and not much in the way of actual evidence.

reactchain(10000) 6 days ago [-]

The 'openness' of the west is being blatantly exploited, and yet for some reason people are still hesitant to call it out for what it is. Almost every significant US internet company is totally blocked in China. Until the US completely blocks WeChat, TikTok, Zoom and so on, China will continue to have a major geopolitical advantage.

cmonnow(10000) 6 days ago [-]

How long will it take China to rebrand these apps under a different name ?

echelon(4134) 6 days ago [-]

> There's also a few snippets of code on the Android version that allows for the downloading of a remote zip file, unzipping it, and executing said binary.

If these claims are true, a remote state actor can now take over 40% of young American's phones.

Imagine if they decided to shut off everyone's ability to communicate. That would be an incredible capability to possess in the event that they wanted to launch an attack or distract us. (I'm not saying that they would, but that we should be wary of the possibility.)

This is incredibly dangerous.

Furthermore, this does not seem like an accident in TikTok's design. This app is very well put together. Given the expertise involved, I can't see this as an 'oops, we didn't know' oversight with respect to either alternative design choices or platform rules. It feels very deliberate.

Google should ban this app immediately for breaking the terms, and US legislators should make a law prohibiting it outright.

We have to do some more due diligence to make sure these claims are valid, but if they do turn out to be true, then we have some very serious issues to consider.

This is one of the few instances where I'll admit that I wish Facebook or Twitter had an answer for this.

sanmon3186(4441) 6 days ago [-]

This ban is primarily to satiate anti-China sentiment.

The global economy is too interwined with China. No country can afford to put a blanket ban on things that matter, without a viable plan-B.

cm2187(3592) 6 days ago [-]

Overnight, I agree. But I think the decoupling will happen. The outsourcing to China took about 10 years (mid 90s to mid 2000s). It will likely not take a lot longer to decouple. 10y is very rapid.

sdinsn(10000) 6 days ago [-]

And yet, China bans foreign apps.

God forbid other countries give China a taste of their own medicine.

sremani(3747) 6 days ago [-]

If we walk into the past a little in 2000 world is not intertwined with China, and very likely 2040 world is not intertwined with China.

If only people read about PBOC, CNY/CNH and how currency is a political tool within China, you realize they have successfully exploited the 'free market' impulse to effectively use state capitalism and dumping to create the world we are in Today.

Just because the supply chains got complex and intertwined with China does not mean it will that way forever -- it is going to be a painful 5 - 10 years, just like the ramp up, the tear down will take time.

kumarvvr(10000) 6 days ago [-]

De coupling with Chinese economy is good for India in the long run.

This is the first step.

India should engage more with democracies.

yumraj(3689) 6 days ago [-]

You have to start somewhere, this is a starting point.

It's better than just raising the hands and saying that it cannot be done.

TLightful(10000) 6 days ago [-]

Everyone is kissing China's netherlands for their hardware, and suddenly becoming precious over their software.

Hippopotamuses!

stevens32(10000) 6 days ago [-]

Hardware is also getting a fair share of international bans

kumarski(913) 6 days ago [-]

Disclaimer: This is not investment advice, this is lunatic gambling advice from a systems engineer with a background in polymers who is an American born Indian.

Tibet quenches the thirst of 3Bn people through ~10 rivers in SE Asia.

Kashmir and Tibet are red hot conflict zones and this is through and through a contest of freshwater.

This is a powder keg unlike any other and this scenario is collateral shrapnel.

China has 5% arable land and its water becomes more putrid by the day. 20% of industrial waste water pollution is from textile dye that's dumped into water, and China has cornered the textile manufacturing market.

China uses water for dams. They built something like 20k+ dams in 70 years.

China uses water for agriculture, extremely inefficiently.

China weaponized water data on the Brahmaputra river and it caused downstream deaths in India.

I'm aggressively invested in $DFEN and $BA.

Boeing has 400 vendors in Inda. This will heat up.

MOD and DRDO ain't no joke.

If you were a murderous dictator like Xi Jinping, it would be prudent to kill the muslim minority to ensure a long and stable CCP indoctrinated control of Tibet and all the water brouhaha.

pm90(4178) 6 days ago [-]

Yes, obviously, long term intense military conflict between nuclear powers leads in incredible security.

While both countries engage in Sabre rattling, its inconceivable that there would be an arms race of the kind that would cause a significant change in the stock price of defense contractors.

A more likely scenario is the upcoming Biden administration steps in and bullies China and/or India into backing down. The hollowing out of the US State department has probably been one of the most underappreciated casualties of the Trump administration.

neximo64(10000) 6 days ago [-]

Are any of you afraid it'll happen to your app? today its China tomorrow it could be another country

quonn(4360) 6 days ago [-]

But it won't.

zigh(10000) 6 days ago [-]

Any of you afraid that the once unconceivable WWIII is gradually become possible because of the irresponsible politician trying to mitigate their own misconduct/misjudgement by instigating hate and xenophobia.

baybal2(1750) 6 days ago [-]

Almost all (except 1) are Chinese made.

I'm sure it has absolutely nothing to do with this https://www.ndtv.com/india-news/china-has-intruded-423-metre...

chrisco255(4389) 6 days ago [-]

Yeah: https://uk.reuters.com/article/uk-india-china/indians-hold-f...

China is basically using these apps as spyware.

cbhl(3595) 6 days ago [-]

I wonder whether that's intentional?

China uses a different coordinate system inside the country (GCJ-02 vs WGS-84) and it results in discontinuities whenever you try to display a map of a border region (for example, the Hong Kong–Zhuhai–Macau Bridge).

kerkeslager(4429) 6 days ago [-]

The Sino-Indian border dispute is one of the most frightening things going on right now. Two nuclear superpowers are fighting with 'rods and swords'[1], restrained only by, well, restraint. Covid and police violence are bigger issues right now in my opinion, but a nuclear war could get worse than either one quite quickly.

That said, I'm glad to see this not getting much press coverage in the US. The political discourse here has shown a willingness to throw gasoline on any fire, and the last thing we need is our fearless leader weighing in on an already tense and dangerous situation.

[1] https://defencenewsofindia.com/ghatak-and-16-bihar-troops-to...

TulliusCicero(10000) 6 days ago [-]

I don't see a problem with targeting Chinese-made apps, given how much China blocks from other countries. Free trade has to work both ways.

duxup(4127) 6 days ago [-]

That incident ... I doubt it is that.

Larger tensions and security concerns, yeah I'm willing to bet it is a larger issue.

tomohawk(798) 6 days ago [-]

The Chinese government moved in martial artists and elite alpine troops to the area. Shortly there after, a number of unarmed Indian soldiers end up dead in the same area.

https://www.straitstimes.com/asia/east-asia/china-sent-marti...

This is small potatoes compared to the concentration camps and genocide and ethic cleansing being forced on Tibet and East Turkistan (what CCP calls Xinjiang). Or to the forced harvesting of organs from living political dissidents.

https://chinatribunal.com/

What will it take?

stevens32(10000) 6 days ago [-]

I read through the tribunal report expecting some sort of evidence, instead it uses guestimates to arrive at numbers for transplant volume then says there's no possible way the organs could've come from volunteers due to that volume. China has 4 times the population of the US, and right about 4 times the amount of transplants claimed in the report.

ryanmarsh(3999) 6 days ago [-]

The public health toll of social media will not be fully understood for another 30 years it seems.

People with adolescents and teen daughters please chime in here. The rates of suicide, cutting, and psychological problems are climbing. My wife and I find a charity that offers counseling and therapy to young girls. What is common place now was nearly unheard of in my generation (X).

yellow_postit(4186) 6 days ago [-]

Just because it was "unheard" of didn't mean it wasn't happening in GenX. I don't know if there's a way to ever fully compare them but there are panics that accompany every new medium of communication.

koopuluri(4186) 6 days ago [-]

I'm curious how this will affect the local startup scene. There's definitely a great demand for these products that now needs to be fulfilled by local / non-Chinese international players.

If this ban lasts:

My concern is that Facebook will introduce India-specific products to fill this void, while my hope is that local players significantly up their product quality and reach to build a strong domestic tech scene - the way China did a decade and a half ago w/ Tencent, Baidu, ByteDance, etc. This domestic talent expansion will help build a stronger domestic tech ecosystem.

Why the concern w/ Facebook? They already have a deathly grip on Indian consumers w/ WhatsApp, Facebook, Instagram. Tik Tok was the only major social platform used by Indians that wasn't owned by Facebook, and now they have a chance to potentially grab that as well. I'm not comfortable with one company owning that much attention, I don't trust their privacy policies with their data privacy track record and they don't iterate quickly enough on localized product-features meaning Indian users lag months / years behind western regions (e.g. Instagram in-app shopping experience).

searchableguy(3431) 6 days ago [-]

What do you think the local market needs right now?

Can you give some example?

didip(4239) 6 days ago [-]

Well.. besides the blatant privacy concerns, it seems fair to ban Chinese apps since they themselves also ban foreign apps.

scarlac(4371) 6 days ago [-]

I strongly disagree. We should prefer to not divide the world if we can. The sentiment 'If they can be bad, so can I' is not the recipe for a world I want to build. I believe the world would regress if we kept acting like that.

sloshnmosh(10000) 6 days ago [-]

To be clear, India has banned most of the apps listed for the members of their army years ago.

Also, many of the apps listed have been removed from the Google Play store several times for violating Googles developer policies and user privacy.

Anyone familiar with Android knows to avoid apps made by DU and CM.

Another little known factoid is that the "Beauty camera" line (Mitu?) has an admistrative level person that used to work at Cambridge Analytica.

balola(4340) 6 days ago [-]

Surprised a free tool app would want that guy.

sadfev(10000) 6 days ago [-]

It was brewing for a while. I am surprised that so many apps got b&. I was sure weChat and TikTok were gonna get axed but not others

geekrax(3347) 6 days ago [-]

> 'b&'

wow!

president(3560) 6 days ago [-]

Glad to see India understanding the threat to national security posed by these apps. I would like to see the US do the same.

KingOfCoders(4235) 6 days ago [-]

Yes and Europe doing the same to Chinese and US apps.

rllearneratwork(10000) 6 days ago [-]

while I absolutely won't allow any of those apps on my school-aged children phones and would never install one myself, it would be absolutely wrong for US government to dictate what software we can and can not use. Same goes for books and movies.

markovbot(3727) 6 days ago [-]

I would find it very problematic if the US government attempted to interfere with it's citizens downloading whatever software they want. I absolutely do not trust the US government to be the arbitrator of what app's are 'safe'.

pphysch(10000) 6 days ago [-]

CCP, NSA, tomato tomato.

Barrin92(10000) 6 days ago [-]

banning the usage of WeChat for public officials or national defense companies and so on would make some sense, I fail to see however how TikTok is a security threat. It's just a response to the clash over the border and nothing else, and not really a particularly effective one at that.

pvelagal(10000) 6 days ago [-]

US Senators already introduced a bill ! https://www.hawley.senate.gov/senators-hawley-scott-introduc...

connectsnk(4437) 6 days ago [-]

Sooner or later it will happen





Historical Discussions: Where Am I? NYTimes or Google? (July 04, 2020: 1082 points)

(1107) Where Am I? NYTimes or Google?

1107 points 2 days ago by rwoll in 4428th position

theinternetbytes.com | Estimated reading time – 3 minutes | comments | anchor

Take a look at the below screenshot from Safari for iOS. What website am I on?

Based on the contents of the page, I'm clearly on a NYTimes property, but based on the address bar I'm clearly on google.com. If I click in the address bar I see https://www.google.com/amp/s/www.nytimes.com/2020/05/22/technology/google-antitrust.amp.html. Confused, I consult Google's Safebrowsing FAQ:

How can I tell if a page is a fake?

The best thing to do is to check the page's URL to make sure it's actually controlled by the party it appears to be controlled by. The crucial part of the URL is the part between the http:// and the next slash ('/'). (If there's no slash, start at the end of the URL.) This is the part of the URL that determines site ownership. Some popular domains, for instance, are amazon, google, and ebay:

http://www.amazon.com http://www.google.com http://www.ebay.com

In some cases, URLs will be a bit more complex; be sure to check the name listed immediately to the left of the top level domain (.com, .net, .co.uk, etc.). For instance, http://www.google.com, http://news.google.com and http://www.google.com/firefox/ are all part of the same site. However, google.com.fraudulentdomain.com/login.html is NOT! Neither is www.g00gle.com (note that in this URL, the letter o is replaced by the number 0).

and determine I'm on a https://www.google.com—but the confusion remains since this is genuinely NYTimes content and branding.

This is a really dangerous pattern: Google serves NYTimes' controlled content on a Google domain. It confuses the user whether to trust the address in URL bar or the content of the page. This confusion is precisely why phishing attempts work so well. Humans trust visual indicators a lot. Google, with the AMP Cache Project, is confusing humans more and training them to trust visual content of the page over the URL in the address bar—despite telling them to do otherwise on different sites. This surprises me since Google spends a lot of time researching visual indicators of security in the address bar (like the padlock icon). In work security trainings and guides on the Internet, we are trained to look at the URL bar to help make a decision on whether to trust a site, but the Google AMP Cache requires contradictory assumptions.

Comments on the post can be viewed here: https://news.ycombinator.com/item?id=23729160.




All Comments: [-] | anchor

metalliqaz(4416) 1 day ago [-]

I don't like AMP and I wish we just fixed the problems it is designed to handle at the root cause.

jacquesm(45) 1 day ago [-]

Get rid of Google?

gorgoiler(4352) 1 day ago [-]

With the utmost respect to you and the other commenters here, when I see positivity about the abstract, hypothetical technical merits of something with a long history of, in practice, being part of an extremely controversial power play it reminds me a lot of the comments I see promoting a widely installed piece of process management software — one which a lot of people don't really want, whose subtle changes to layers of abstraction introduce new and unexpected bugs that can only be fixed by further coupling, and which can also be reasonably described as a single entity politically maneuvering itself to bring order to the chaos at the expense of living in, for want of a better term, a dictatorship.

Well at least under Google AMP, the pages loaded on time.

dang(192) 1 day ago [-]

We detached this subthread from https://news.ycombinator.com/item?id=23729479.

Clausinho(10000) 1 day ago [-]

I'm out of the loop, care to elaborate which PM software you are referring to?

0898(4235) 1 day ago [-]

Is this some kind of human buffer overflow?

gsich(10000) 1 day ago [-]

>Well at least under Google AMP, the pages loaded on time.

Yeah, because Google is cheating.

ElliotH(4162) 1 day ago [-]

I struggle to see how comparing people who like particular technical products to supporters of dictatorships shows any of the 'utmost respect' to any commenters that you claim at the start of your comment.

jevgeni(4410) 1 day ago [-]

That first paragraph is a single sentence... Tried understanding what you are saying a couple of times and I just can't.

catalogia(4437) 1 day ago [-]

I enjoyed reading this comment. The way you write is fun.

cosmodisk(10000) 1 day ago [-]

You might want to work on reducing the length of your sentences- it's very hard to get the meaning behind them.

simias(4386) 1 day ago [-]

I don't like systemd and I think it's an overengineered mess of dubious value but comparing an open source linux init system to the long game Google seems to be playing to act as a proxy for the web is absurd and doesn't make any sense at any level.

Your comment is pure flamebait without any insight.

reaperducer(4213) 1 day ago [-]

People have been railing against Google's Amp on HN for years, and I think I finally figured out what it's for.

It's Google way of combatting phone apps.

If all of the world's information — especially current news and similar information — moves from the open web into apps, then Google can no longer crawl, index, or scrape that information for its own use. The rise of the mobile phone app is a threat to Google on so many levels from ad revenue to data for training its AIs.

So Google comes up with Amp to convince publishers to keep their content on the open web, where it can be collated, indexed, and otherwise used by Google for Google's services like search and those search result cards that keep people from visiting the content creators.

Google's explicit carrot in all this is the user benefit of page loading speed. Google's implicit carrot in all of this is page rank. But Google's real motivation is to have all of that information available to itself.

Can you imagine what would happen if content from even one of the big providers was no longer visible to Google? New York Times, WaPo, or even Medium? It would create a huge hole in a number of Google products and services, make its search results look even weaker than they already are, and cause people to look for search alternatives.

That's my theory, anyway.

mclightning(4315) 1 day ago [-]

Who is really using the dedicated apps for each news site? Web is just way more practical; for translation, for copy-paste, for sharing.

Besides,you dont need the app on your mobile.

sillysaurusx(3293) 1 day ago [-]

Interesting theory. One hole is that companies want to be on Google's results. It hurts WaPo not to be in the top N results, so they have an incentive to make it at least possible.

satyrnein(4443) 1 day ago [-]

That's long been Google's stated reason for Chrome and much else, that pushing the web forward as a platform aligns with their interests as well.

summerlight(10000) 1 day ago [-]

Apple and Facebook really doesn't care if the web dies as long as their platform take the lion's share. But for Google, search as a product can exist only if the web itself remains relevant and this is why it's trying to keep display ads alive even though it doesn't really give them much money compared to search ads but all the privacy complication coming from third party tracker.

hortense(10000) 1 day ago [-]

Amp was a reaction to Apple News and Facebook News: using those applications to read the news was a much better experience than using the web. Why? Mainly for two reasons:

1/ Apple and Facebook were hosting all the content.

2/ The content did not come with megabytes of JS and other unnecessary crap.

Amp is an attempt at saving the web, and Google is interested in that for the reason that you gave: they make their money from the web.

ffritz(10000) 1 day ago [-]

Interesting, though the barrier for users to install a new app seems to be very high these days. Most people only install a few necessary apps and thats it. In addition, we are talking about publishers here. There are thousands of news sites, no user has more than a couple of news apps. That's why they have to keep up their website anyway, with or without amp.

Kevin605(10000) 1 day ago [-]

This has already happened in China, where Baidu (The Chinese equivalent of Google) can't crawl any articles from WeChat (The Chinese equivalent of Medium), as a result, the usefulness of its search result has deteriorated significantly. Recently, Baidu has been trying to start its own publishing platform with little success.

IfOnlyYouKnew(3568) about 14 hours ago [-]

That's Google's motivation for almost anything. Especially Chrome.

remus(10000) 1 day ago [-]

I think this is a fairly cynical take, as having news on the web is also pretty great for users.

Imagine if instead of having all news stories a quick search away you instead had to install apps from X different news sources (and inevitably grant them permission to access your location, contacts list, name of first born child etc.). It'd create lots of little silos of news with very little ability to go outside those silos.

Put another way, the web is a great platform for news. It does benefit Google, but it also benefits the billions of people who can freely access a huge range of sources.

w-ll(4199) 2 days ago [-]

The shenanigans Google been doing to the url bar is super hostile.

Trying to copy the domain of a url without the protocol just infuriates me.

marvindanig(2310) 2 days ago [-]

Hm. There's room for a new good browser to pick up the beans and run with it now...

trishmapow2(10000) 2 days ago [-]

Not defending that change, but when do you ever need to copy just the domain instead of the full URL?

sodascripts(10000) 2 days ago [-]

Disable this setting in chrome by going to chrome://flags and switching #omnibox-context-menu-show-full-urls to enabled. Then right click the URL bar and select 'alawys show full URLS'

jacquesm(45) 1 day ago [-]

Try sending someone a link of a PDF you found using google.

ehsankia(10000) 1 day ago [-]

Isn't it just an extra click? Click one, it highlights the whole thing, click a second time and you see the full URL with the protocol.

quadrifoliate(10000) 2 days ago [-]

IMO the core point of the article is false.

> To be blunt, this is a really dangerous pattern: Google serves NYTimes' controlled content on a Google domain.

No, 'Google serves NYTimes' controlled content' is an oxymoron. Google controls the content that is served, and that's all your browser is verifying. Google could very well make the NYTimes content on there display something else and your browser wouldn't show a warning. NYTimes could do nothing about that.

I disagree that this pattern is dangerous. While Google taking over serving the world's content is hardly a thing to celebrate, at least we're seeing that it's doing so here.

gsnedders(3797) 2 days ago [-]

With Signed HTTP Exchanges, for Google to modify the content that is served, Google would need access to a private key for a certificate for nytimes.com, no? Either nytimes.com has handed over that key or Google would have to create a key/certificate for nytimes.com. Believing Google would maliciously issue certificates seems a stretch to me.

I don't like AMP nor much of how Google has behaved with it (http://exple.tive.org/blarg/2020/05/05/the-shape-of-the-mach... largely matches my thoughts), but let's stick to what's actually happening with SXG.

rtsil(10000) 2 days ago [-]

The pattern is dangerous because it trains the user to dissociate URL and legitimate content, and the best tool at our disposal against phishing is still the ability to use the URL to ascertain the legitimacy of a content.

jacob019(10000) 2 days ago [-]

New York Times and all the other publishers don't have to participate in this crap. It's shameful that they cede authority over their content so easily in exchange for a vuage promise of more visibility. There are so many better ways.

untog(2555) 2 days ago [-]

It's not a vague promise, it's an extremely explicit one. Search results for news contain a "top carousel", a horizontally scrolling box that shows cards for different articles. On most phones it takes up most of the screen. If you want to be in the carousel (i.e. if you want your site to be visible near the top of search results) you must use AMP. No ifs and buts about it.

If NYTimes and every other news organisation refused to participate then yes, Google would be in trouble. But they can rely on good old divide and conquer: these news organisations all compete with each other. All it would take is for one to starting producing AMP content again and they'd vacuum up all the search traffic, and all the other sites would follow them immediately.

lazyjones(4368) 2 days ago [-]

> don't have to participate in this crap

It's a tempting Ponzi scheme.

dewey(1292) 1 day ago [-]

In an ideal world where they would not rely on ad revenue and page views but are supported by the readers that assumption would be correct.

But right now we are not living in that ideal world and because all other publications are doing that they have to follow if they don't want to risk losing visibility against the competition.

So of course they don't 'have to' but they also kinda do.

aronpye(4436) 1 day ago [-]

AMP is the main reason I switched to DuckDuckGo from Google. Webpage rendering often used to break on iOS, in particular scrolling where the page would just go blank.

collinmanderson(1574) 1 day ago [-]

Yeah AMP is one of many reasons I switched to DuckDuckGo. "Why am I giving Google this much power? Why am I contributing to them being a monopoly?" were the general reasons.

Hearing people mention low quality search results was what kept me off, but I've actually only needed to do a google search about once a week, far less than I was expecting.

nwsm(4150) 2 days ago [-]

This has been all over HN since amp was released, and this is a two paragraph article with no new info or opinion.

https://hn.algolia.com/?q=google+amp

mindfulhack(4329) 2 days ago [-]

I suppose 328 votes so far show the usefulness of repeat discussions. This article is a catalyst to keep this one going. The votes prove that it's an important enough issue to continue talking about.

IgorPartola(1605) 2 days ago [-]

I truly hope whoever keeps pushing AMP such as it is has to birth a hedgehog backwards. I mean, at least add some way to opt out of this shit. This is why I don't use Google as my mobile search engine. I honestly still would if my browser allowed me to disable or bypass it, but nope. No extensions for me because Apple won't allow for anything like that.

/rant

bitsoda(10000) 2 days ago [-]

For real. For the most part I love all things Google, but AMP makes my blood boil. How does it save me any time when I have to tap that little 'i' to get to the original website through all that janky scrolling? Ugh, infuriating. I feel like a walrus on a tar floor maneuvering through AMP pages. What an awful piece of technology.

/rant

fizwhiz(3678) 2 days ago [-]

> has to birth a hedgehog backwards

You have every right to be indignant, but I could have really done without that imagery.

plplok(10000) 2 days ago [-]

I'm wondering about this as well? Who was it?

I'm pretty sure there were many internal discussions as to why/how to push this change. I'm surprised that those haven't been leaked.

toastal(10000) 2 days ago [-]

The beta of Firefox on Android still hasn't allowed any add-ons to auto-redirect away from AMP-enabled sites. I was excited to see Privacy Badger enabled 2 releases ago, but even its tracking-token-stripping capabilities seems missing on mobile.

anonu(2353) 2 days ago [-]

Remember when Google's mantra was 'Don't be evil' ???

nullc(2193) 2 days ago [-]

> Remember when Google's mantra was 'Don't be evil' ???

Meh, they preserved 2/3rds of it.

lalos(10000) 2 days ago [-]

'Don't be evil', using primary colors in the logo to bring that kindergarten familiarity, fun doodles, a dumb funny movie and quirky April Fools projects were a great marketing strategy to distract your average person in lowering their guard and feel safe to give all the data to an advertisement company. I wonder if they currently teach this case in marketing/PR classes.

twhitmore(10000) 2 days ago [-]

The whole AMP thing seems anti-competitive and hostile to the open web.

It's a really bad look on Google's part to be pushing this.

lern_too_spel(4447) 2 days ago [-]

In what way is it anti-competitive? Google's competitors also consume AMP pages and prerender them using AMP caches. Anti-competitive would be requiring the publishers to integrate directly with Google like Apple News, not asking the publishers to publish pages that all link aggregators can consume.

earthboundkid(4222) 2 days ago [-]

There has been no regulatory action since Microsoft (which happened as Google was being born), so the tech giants have forgotten fear and no longer self-regulate out of simple self-interest.

raverbashing(3952) 2 days ago [-]

I am conflicted

Yes, AMP is an anti-competitive move by Google

At the same time AMP is 'faster' because it gets rid of all the nagware and JS crap that the original page has.

So yeah, I don't like what Google is doing but I don't like what NYT is doing neither

Abishek_Muthian(4130) 2 days ago [-]

I think the main issue is limited AMPCache providers and inability for the publisher to choose their own AMPCache providers. Which is being exploited the two search engines.

AMP project by itself is open-source and it explicitly states 'Other companies may build their own AMP cache as well'.[1] There are only 2 AMP Cache providers - Google, Bing. Further, 'As a publisher, you don't choose an AMP Cache, it's actually the platform that links to your content that chooses the AMP Cache (if any) to use.'[2]

Say, if Cloudflare provides a AMPCache and if the site publisher can choose their own Cache provider this can be resolved effectively as AMP by design itself is easy for a laymen to create high performance websites; of course there is no excuse for hiding URLs.

[1]https://amp.dev/support/faq/overview/

[2]https://amp.dev/documentation/guides-and-tutorials/learn/amp...

snowwrestler(4329) 2 days ago [-]

Can we please stop trying to pretend AMP is some sort of community-driven open source project? AMP was created by Google, for the benefit of Google. We are not obligated to play along every time a company says "open source."

lern_too_spel(4447) 2 days ago [-]

The link aggregator, not the publisher, must control the AMP Cache in order to prerender pages from it safely.

SquareWheel(3887) 2 days ago [-]

Did Cloudflare end their Amp Cache? They hosted one previously.

Kiro(4022) 1 day ago [-]

AMP is disliked by privileged people who have never experienced how truly awful browsing the web with a bad internet connection can be.

saagarjha(10000) 1 day ago [-]

Perhaps they instead see the "solution" to be problematic.

mikro2nd(2411) 1 day ago [-]

Can you elucidate what you'd consider a 'bad' internet connection?

I live Out In The Styx, in a Shithole country, at the end of an allegedly 2MB/s piece of wet string masquerading as an internet connection that seldom lives up to its adverted performance. AMP has never once made any significant difference to my web experience.

ridiculous_fish(4448) 2 days ago [-]

It's wrong to trust the URL bar. For example, this search [1] has as top link an ad that boasts 'google.com', and it really is! And if you click on it, you'll end up on a google.com site, which nominally helps with printers, but in reality it's a tech support scam.

So much of the distrust here is that google wants to be everything: to host their content and publisher content and user content; to broker ads and recommend links; to run their software on your computer and phone, to store your data on their servers. They serve too many masters.

1: https://i.imgur.com/HalErpIr.png

kevingadd(4013) 1 day ago [-]

'It's wrong to trust the URL bar' is true but only because the companies operating services like... Google... don't bother trying to protect their URLs. It's not hard to have a separate 'user content' domain for your user content, we've done it at places I used to work for. But for some reason people think it's enough to use a subdomain or get cute and use the same domain with a different TLD (looking at you, github.io)

So it is kind of frustrating to see someone offering to fix a problem they helped create in the first place through neglect or carelessness.

icebraining(3873) 1 day ago [-]

Agreed, that's problematic. But Google didn't even have to not host content, they would just have to use a different domain. They have such weird blind spots.

jacquesm(45) 1 day ago [-]

All this does is rapidly devaluate the google.com domain. Not a bad thing per-se.

tommek4077(10000) 2 days ago [-]

As an advertiser, you can write whatever you want into the url displayed there. This does not need to match the real target.

nokya(10000) 1 day ago [-]

I have my own proxy filtering all my desktop and mobile traffic, anything 'AMP' is filtered spot on. Sometimes nothing shows up, sometimes the original server responds after a few seconds. I'd rather not see the page at all than play this game.

hjek(3239) 1 day ago [-]

Try the Redirect AMP to HTML[0] browser add-on to get out of the game without getting blank pages. There's also Privacy Redirect[1] for getting out of the Youtube / Twitter / Instagram / Google Maps game.

[0]: https://addons.mozilla.org/en-GB/firefox/addon/amp2html

[1]: https://addons.mozilla.org/en-GB/firefox/addon/privacy-redir...

bobbydroptables(4375) 2 days ago [-]

AMP seems like a solution in search of a problem. Are people really having trouble with loading speed in 2020? I travel to remote areas in third world countries regularly for work and still don't really have problems loading pages with mobile data.

Even if it didn't have all of the problems associated with it I just don't get the point. I don't need Google to repackage a website with less useability. It's frequently not even faster.

ocdtrekkie(2962) 2 days ago [-]

AMP solves a problem, it just doesn't solve a problem for users. It's an anticompetitive play and it's helping exactly who it is supposed to help.

crazygringo(3985) 1 day ago [-]

> Are people really having trouble with loading speed in 2020?

Huh? Yes. Hugely. I'm on my fast home internet using a new iPhone I bought two months ago, and loading a NYTimes article just took 8 seconds. God only knows if it's bounded by network or CPU or both, if the problem is frameworks or ads or what. And it isn't even 'stuck' on anything -- I watch the blue loading bar in Safari move pretty smoothly across the top.

I did a search for a NYT article on Google, clicked it, and it appeared instantaneously.

That's an insane difference. I know everyone hates AMP here, but when I've got my user hat on rather than my developer hat... it's unbelievably more performant.

a254613e(10000) 1 day ago [-]

I do, and I don't even live in 3rd world country - I live in Germany in one of the largest cities in the country.

But even if I can load both pages at roughly the same time AMP experience is just so much better, they always load at the very least at the same speed as the original website, there's no weird scrolling implemented, there's no annoying popups, etc.

I always choose AMP pages when possible, compared to the 'native' ones - because I know for a fact that I'll get fast loading, and other stuff mentioned above.

smabie(4386) 2 days ago [-]

I lived in Africa and the only internet I had was cellular and by the gb. Amp is a massive improvement over the extremely large web pages we now have to endure.

It's also much faster to render, which makes a huge difference on the crappy Android phones that are everywhere. Hell, I'm using a $200 Android phone right now because my iPhone broke and browsing the web is painful on it. And with the terrible hauwei $40 phones that have taken over Africa, most of the web is unusable.

I don't like Google's control of Amp, but it exists because of the original sin of html and js. Everything about html is terrible: bloated, pointlessly verbose, etc.

I have a dream that we all just start using Gopher and dump the www, but it's never going to happen. Maybe even browser vendors could get to together and design a super light weight markup based on S-exps or something, but that's probably not going to happen either. Amp is the best we got and it solves a real problem. And it solves the problem well.

superasn(1660) 2 days ago [-]

Yes this has been a big issue for a very long time now. Google wants to push a release where it will display the hostname of the amp site even if the content is being served from google.com[1].

Mozilla (and Apple) are strictly against it and thank god for Mozilla. If Google had a bigger market share this would already be something we would have been living with. I'm sure there are better sources for this, but here is the first result:

https://9to5google.com/2019/04/18/apple-mozilla-google-amp-s...

nokya(10000) 1 day ago [-]

Just having this idea already tells how important it is to actively resist Google.

One should never forget that at a certain point, Google will likely invoke the looser's argument ('protect you from terrorists and pedophiles') to require proof of identity prior to granting access to any resource or service it controls.

Anything that helps them advance in that direction must be fought fiercely.

esafwan(10000) 2 days ago [-]

Have a look at this: https://blog.cloudflare.com/announcing-amp-real-url/

Cloudflare allow using of same domain to use AMP. In this case, content is served from Cloudflare CDN.

priyaranjan(10000) 1 day ago [-]

This has been discussed over & over again and there is no representation from amp team to make it any better. I was surprised to realise how much my life changed when I started using firefox + duckduckgo. Full time, at work & home, on macOS & android.

rpastuszak(4331) 1 day ago [-]

Remember when Google was telling us that third-party cookies are there to protect us, and Safari/Firefox/Edge are just reckless and pose a risk to users by blocking them?

Spivak(10000) 2 days ago [-]

I don't really think Google's plan is that weird. And it would be amazing for decentralized networks, archiving, and offline web apps. Google can't just serve nyt.com — they can serve a specific bundle of resources published and signed by nyt.com verified by your browser to be authentic and unmodified.

realusername(3897) 2 days ago [-]

Being completely against AMP for obvious reasons, I'm personally not against signed exchanges itself, this feature could spawn a whole new class of decentralised and harder to censor web hosting, that sounds like a great addition.

xg15(2635) 1 day ago [-]

Aren't we essentially reinventing http proxies with this?

smpetrey(3547) about 24 hours ago [-]

Holy mackerel

agumonkey(863) 1 day ago [-]

has the mainstream web jumped the shark ?

jeffbee(4357) 2 days ago [-]

Well, it's the going opinion of HN for years that the main problem with AMP is it shows the actual origin instead of the proxied origin. Lying about the URL is something hundreds of HN comments have angrily demanded.

m-p-3(10000) 2 days ago [-]

I'm still waiting for general support of addons for the next version of Firefox on mobile just so that I can have the Redirect AMP to HTML[1] addon.

[1]: https://addons.mozilla.org/firefox/addon/amp2html/

dqpb(4272) 2 days ago [-]

Isn't this basically like a CDN or a PoP cache?





Historical Discussions: Forget Google, time to end the Visa-MasterCard duopoly (June 29, 2020: 1097 points)

(1099) Forget Google, time to end the Visa-MasterCard duopoly

1099 points 6 days ago by CM30 in 2264th position

medium.com | Estimated reading time – 5 minutes | comments | anchor

Forget Google, Time to End the Payment Processing Duopoly

In the last few years, you've likely seen a fair few comments about monopolies in the tech industry, and suggestions for breaking them up. These comments usually revolve around the big FAANG companies (or at least, Google, Amazon and Facebook) and suggest the power they hold is unhealthy for society.

And they're right. It is unhealthy, and definitely acts as a good reason to break up said companies.

But they're not the only ones that need it, nor the most important cases. No, there are two others that are far more dangerous to the internet, businesses and society right now.

Namely, the payment processing companies, like Visa and MasterCard. These companies handle billions of credit card transactions a day, and have a level of power the likes of Google can only dream of.

As Reuters points out, their marketshare for credit cards is about what Google and Apple's is for mobile OSes

Like almost complete control over what can be sold online. Ever wonder why certain things are difficult to buy via PayPal or other similar systems? Why Patreon ban certain types of creators really easily? Why adult sites find it so tough to process card payments?

Because these guys set the rules, and those rules ban a lot of content that's either offensive or brings high risk of fraud. It's why adult sites usually keep switching payment processors, or running ads from less than savoury networks.

So you've got a huge part of the monetary system controlled by parties with zero interest in neutrality, and power beyond government control.

What's more, abuse of this power has already happened. Back in 2010, Visa cut off access to WikiLeaks, rendering the site unable to receive donations as a result. This was fortunately then overturned (by a Dutch court order no less), but it's a scary precedent regardless, and shows just how easily 'undesirable' media outlets and creators can be deprived of their income by companies like this one.

And that's not the only issue they bring either. No, their control over the system means outages are very serious indeed.

Just think back to the UK in June 2018, when Visa went down across the country. That caused a huge incident that took thousands of businesses and millions of credit cards offline.

Which in turn likely caused billions of dollars in economic losses, all due to a simple network accident. One small issue, whole economies on the verge of collapse.

And it could potentially have been even worse. Had their few competitors also gone down, literally all electronic payments would have broken at once. Had it been a worldwide issue, and basically the entire world economy would be unusable for hours or days.

This in turn also means it's a perfect target for malicious enemy forces. Theoretically like Iran, North Korea, etc. All of whom have a vested interest in their rivals losing some of their technological advantages over them.

So there's a potential cybersecurity disaster waiting to happen there.

Though it's not as if our governments likely aren't using these companies and systems for questionable purposes as is. They're a perfect way to target and cripple regimes we don't like at this point in time, and would almost certainly be cut off in the case of a war or major conflict.

Hence neither group really has reason to trust them. They're dangerous for our own societies, dangerous for our opponents and a huge security risk regardless of the situation, plus a force for censoring people who've done nothing other than annoyed the 'wrong' people overall.

But surprisingly, they're not worth breaking up.

And I know what you'll say. That I made a perfectly good argument for doing just that. That with all those downsides, breaking up Visa and MasterCard (and perhaps some competitors) is the best way forward.

However, here's the thing. It's not true.

Since at the end of the day, the problem isn't just that these companies are too big, it's that their industry has been left entirely privatised at all.

Cause payment processing for individuals and businesses isn't a competitive market. It's not some 'optional' product the world can exist without.

It's a crucial part of a large chunk of the world's businesses. It's crucial for the internet to function, and its absence makes it incredibly difficult to live in a modern society.

Hence the solution isn't just more competition. It's to regulate the whole industry as utilities, or even turn it into neutral, global infrastructure.

This would mean that in the same way everyone has access to water, electricity, or public transport, everyone has access to payment processing, regardless of their nationality, gender, ethnicity, sexual orientation, political views, wealth or anything else. Reactions to people asking to cut people off from receiving money should be treated like an attempt to say, remove a police station's access to water or electricity; called absurd and ignored.

It's time we called it a day for the large payment processing companies, and regulated their entire market as utilities like people are expecting us to.

Otherwise, individuals, organisations and businesses alike will suffer their ridiculous rules and limitations.

Thanks for reading!




All Comments: [-] | anchor

taurath(3949) 6 days ago [-]

Having worked in fintech, they're essentially the gatekeepers of any new technology. Have a great idea for a value add in the payment processing gateway? They will not let you on any platform for 4 years while they launch a (poor) competitor at scale.

rbrtl(10000) 6 days ago [-]

Sometimes the turn around is quicker, because rather than building their poor competitor they just buy one.

moralsupply(10000) 6 days ago [-]

> time to end the Visa-MasterCard duopoly

Why? Because monopolies are bad, and we don't like them, no matter if they provide value to people, if they are not built around government regulations that stop competition from happening.

notpushkin(10000) 6 days ago [-]

They would have to provide more value for less money, were there some competition. I don't see a way in which one can compete with those two (without resorting to aggression) on a global level though.

except(10000) 6 days ago [-]

So, how does one build a card network?

User23(3437) 6 days ago [-]

It's easy, you just convince a plurality of banks to honor your IOUs.

jb775(4429) 6 days ago [-]

Could Stripe swoop in and make a play to become a third major CC option? I feel like they have the core infrastructure in place and are still nimble enough to make it happen if they wanted.

francislavoie(4447) 6 days ago [-]

I feel like if they tried anything, MC/Visa would try to squeeze them until it becomes unviable. They would still need to support MC/Visa until they get enough market share to not totally hinder their existing business.

GordonS(475) 6 days ago [-]

Visa own a stake in Stripe, and are unlikely to be supportive of such a play.

julienb_sea(10000) 6 days ago [-]

There is literally direct competition with Amex. Sure not everywhere takes amex but the majority do, and you can speak with your dollars (and still hold Visa / MC in case amex fails you). Consumers derive huge benefit from the safety, consistency and reliability of the networks Visa and MasterCard have built out, and yes the consolidation is a certain price to pay for that. These companies invest billions into anti-fraud technology.

But they are at their core public companies that want to maintain their image. They are reactive to public pressure and political headwinds, and will cover their bases by running away from thorny messy situations. This crosses the aisle in every imaginable way - whether its stifling whistleblowers or stifling supposed hate speech. The reaction to public pressure is not going to be avoidable with any conceivable system, privately owned or otherwise.

The best defense is competition, and despite this post's assertions, there is competition in this space.

pfundstein(10000) 6 days ago [-]

> Sure not everywhere takes amex but the majority do

Perhaps in the US, but elsewhere around the world Amex is barely ever accepted.

That said this duopoly only really applies to internationally accepted credit cards, lots of countries have alternative payment methods, but these generally don't reach beyond the borders.

wolco(3904) 6 days ago [-]

American Express is not the answer. Let's not replace bad for worse.

dannyw(4147) 6 days ago [-]

I would rather see Visa, MasterCard, and other big banks regulated and require them to accept all legal businesses. Fraud can still be blocked but on an individualised, reasoned level.

No more denying legal pornography businesses, no more denying legal fireworks stores, etc.

basicplus2(2714) 6 days ago [-]

One option would be for people to start paying each other in gold.

Ratiofarmings(10000) 6 days ago [-]

So instead of sometimes one of the two not being accepted, you want me to carry 50 different ones and have 2/3 rejected?

No, thank you.

apexalpha(10000) 6 days ago [-]

The only reason some cards are not accepted somewhere is because Visa-Mastercard are so powerful they force 3%-5% margins on transactions for merchants.

In the EU the just use bank cards. And since there's regulations the fees are lows and everyone can use it everywhere.

rbrtl(10000) 6 days ago [-]

No... You have the option of any of 50, and they can all be accepted everywhere; rather than the option of only two if you want a chance of being accepted. I believe this is already law in the EU anyway, but there are few competitors to the big two.

ur-whale(3297) 6 days ago [-]

There is something that I've never managed to put my finger on: most companies as huge as Google / Apple / FB etc... or even more traditional ones (banks, oil, etc ...) are sort of 'well known' in the sense that they do PR, they have well known figureheads, etc ...

VISA has always struck me as a very nebulous entity, whose structure, governance, is not very well know by the general public.

I wonder how they managed to grow so large while managing to keep such a conspicuously low profile.

edit: and to answer my own question, the wikipedia page is quite informative : https://en.wikipedia.org/wiki/Visa_Inc.

thoraway1010(10000) 6 days ago [-]

If you make $10B/year on revenue of $20B/year (great margin at that scale) you keep quiet I think for good reason :)

redis_mlc(10000) 6 days ago [-]

If you don't work in the payments space, then there's a lot of entities that you're not aware of since their members/associates front them.

One fintech startup recently sold for $5 billion to VISA. They issused a PR statement that was completely false, but if you didn't work in payments you'd never know they didn't have 11,000+ clients (hint: that's the number of institutions in NACHA.)

https://en.wikipedia.org/wiki/ACH_Network

https://en.wikipedia.org/wiki/NACHA

xfour(4108) 6 days ago [-]

I think they were an 'association' until the spun out into a company in their own right recently.

https://money.cnn.com/2008/03/21/news/companies/visabanks/in...

ryanwatkins(3787) 6 days ago [-]

Their leaders may not be well known, but they spend an absolutely massive amount of money promoting their brands. They are extremely recognizable.

Mastercard so much so that the recently removed their name from their logo as its nolonger needed.

cactus2093(10000) 5 days ago [-]

You can't talk about credit cards in isolation without considering the rest of the consumer finance sector in the US.

A big part of the reasons credit cards rose to such prominence is that the banking and payment rails are so incredibly stupidly designed. The check system and ACH which is the highest volume way that money is sent between people with different banks requires you to give out the secret key every time you make a payment. Plus it takes at least 1-2 days to clear which makes fraud more difficult to deal with.

Interestingly the credit card system also suffers from the first part, your number is printed right there on your card. But they've managed to find and/or strong arm ways to reduce fraud on their network, and as a consumer I'm not liable for paying it which is a huge benefit.

The author really lost me at the end though, I don't see how regulation magically solves everything and they didn't explain it at all. There is already a ton of regulation in the payment industry. And this comically flimsy underlying system could easily be improved by existing regulators, how about starting with small steps and seeing how that goes? Instead of advocating for throwing out the entire payment system all at once under the very flimsy assumption that the government will do it better.

puszczyk(4424) 5 days ago [-]

In Poland there's really nice pay-by-link (you're being redirected to your bank website, log-in, they display the merchants data and amount and you confirm it) and basically all the merchants support it. There's also BLIK, which is a payment app started by a few banks which is gaining traction. This all is super convenient and easy to use.

However, I still prefer to use a credit card, because of the chargeback — the legal protections with credit cards are still higher.

—-

Edit: about "you're paying the fees" — maybe, but most of the merchants don't show different price or don't charge extra for credit card payment. So why not use the IMHO better option if it costs me the same.

vinniejames(4238) 5 days ago [-]

'not liable for paying it,' but you're still paying it at the end of the day via fees and increased merchant prices

m12k(10000) 6 days ago [-]

The experience of receiving credit card payments for a SaaS subscription sucks badly, even if you disregard the inflated payment processing fee. Banks can allow the charge to happen the first five months, then on the sixth charge, randomly turn on a penny and reject the charge with a completely opaque do_not_honor code, which probably translates to 'our fraud detection algorithm felt extra paranoid today'. There's a whole industry of companies who help other companies 'poke' their customers when their payments randomly fail, to prevent churn. This seems ripe for disruption. My dream tool for receiving payments:

- Isn't tied to a piece of plastic that expires

- Front loads the fraud prevention by requiring 2FA to make sure up front that the buyer actually wants to pay for this.

- Opt-in, per-transaction chargeback/escrow service (intended for shipment of physical goods) that you don't pay for if you don't activate. Transactions are clearly marked whether they include this during checkout. Otherwise transactions are final

- For subscriptions, allows you to specify the conditions under which future charges should also be accepted, and when they should be held and you get contacted to approve them instead.

- Basically anywhere the current system can fail/reverse a charge, the improved system would verify this up ahead, so the payment processor can be sure the buyer really wants to pay, and the merchant can be sure charges don't fail unless the buyer runs out of money in their account or actively cancels the subscription.

- Charge a much more reasonable processing fee, due to not needing to wade into disputes and fraud recovery all the time.

avianlyric(4272) 6 days ago [-]

Interesting this systems already exists. For Mastercard it's called MDES for Merchants, or to give it its full name Mastercard Digital Enablement Service for Merchants.

It's an extension of the technology that powers Apple Pay and Google Pay, and basically allows merchants to get a virtual card issued to them.

In the EU that would require 2FA to happen, along with describing to the customer what the billing schedule is (required for SCA but being implemented slowly).

The end result is the merchant gets a non-expiring virtual card to bill the customer, and the customer get the ability to disable certain merchants by asking their bank to destroy the card linked to a specific merchant.

Unfortunately all of this stuff is very new, and there are a bunch of issues that will prevent merchants from using the tech. But it is happening slowly.

Polylactic_acid(10000) 6 days ago [-]

The whole system seems to be designed in such a bad way and relies on a bunch of heuristics and insurance to make up for easy fraud.

Why is it that to pay for something you give the seller the keys to your account and they go in and withdraw the money they want instead of the seller sending you a request and you accepting to send the requested money..

Another plus side of your suggestion is it now becomes trivial to cancel any subscriptions since you can simply stop allowing the payment to go through rather than getting the seller to stop taking your money.

shuringai(10000) 6 days ago [-]

the ideal system you described is called bitcoin

ww520(3474) 6 days ago [-]

Amex/Discover, plus PayPal and a host of lessor money transfer apps/companies, like Venmo and Zelle.

adjkant(4428) 6 days ago [-]

Venmo is owned by Paypal. Zelle is owned by all of the big banks collectively (Bank of America, Capital One, Wells Fargo et al.). The closest thing to a 'small' player is the Cash app, owned by Square.

No commentary on the larger picture here or Amex/Discover, just pointing out your other examples don't exactly fit your point fully.

julius_set(4439) 6 days ago [-]

Just curious why PayPal or Venmo?

x87678r(10000) 6 days ago [-]

Amex/Discover cards exist too, you should use them!

jedberg(1967) 6 days ago [-]

I used to carry an Amex, but I also had to carry a Visa for all the times the merchant wouldn't take my Amex. Life got a lot easier when I got a Visa with all the same rewards as my Amex.

erostrate(4421) 6 days ago [-]

How come we haven't had big privacy scandals from Visa/ Mastercard?

Big companies, a rent-like business, no pressure to do things well, own lots of very sensitive personal data, clients giving the data away without realizing it, little regulatory oversight on the data front, etc.

I would expect them to sell or leak poorly anonymised personal data, leading to huge privacy issues.

Why hasn't this happened?

osamagirl69(10000) 6 days ago [-]

Because they know they have a golden goose and are not willing to risk it all pushing boundaries like the FANG companies are. They also have literally 50 years of experience with their data mining and the only think more powerful that incompetence is 50 years of bureaucracy....

partiallypro(4394) 6 days ago [-]

I feel like card competition is actually pretty steep. MasterCard and Visa may have the majority of the market, but there are still Discover and AmEx. I have been using things like NerdWallet to figure out what is the best card to go with, it is usually a healthy balance. I think the main thing is that most Debit cards are Visa or MasterCard...and you are locked in by your bank. Where as credit & charge cards are a free for all.

When it comes to advertising online though, you have only 2 choices...Google or Facebook. As a business I can't not go with those two or I'll lose immense business. I can go with a Visa, Mastercard, Discover, Amex...and it really has no effect on me outside of the benefits the card gives me. The fees charged to the banks, etc are essentially the same. So it's really no comparison as to which is worse for the market itself.

teej(2406) 6 days ago [-]

You're talking about different things. When you shop credit card offers, you are comparing issuers (eg Chase vs Capital One). That's not the same as the payment processor aka card network (Visa vs MasterCard).

Of course it's endlessly more complicated then that but it's important to make the distinction.

numpad0(4432) 6 days ago [-]

When you open a web shop, you pay to get SDK from payment processor to integrate them with the site.

When customer clicks "Checkout", they are sent to payment processor website to enter card number like 1234 ...5678... and whichever card it is, it goes through, and shop owner receive the payment later.

Which means payment processor has contracts and connects with every card networks, absorb API differences, and on top of that, obeys and agrees to everything CC network thinks or says, to be able to handle any cards customers may have with them.

So if a Visa or MasterCard exec thinks maybe he don't like Cheetos and make a call to processor CEOs how they think about it, no later than by Friday no one will be able to order a single bag of Cheetos, especially online, using any credit card because payment processors will have explicitly communicated that Cheetos had never been tolerated from the beginning and any store who let that happen will have accounts frozen.

It happens somewhat softer than that but kind of happening once couple years these days.

ajb(4345) 6 days ago [-]

Coincidentally I just read a post from a business that's finding ordinary bank transfers getting more traction from customers than they expected:

https://www.revk.uk/2020/06/beyond-credit-cards-is-this-way-...

peteretep(1809) 6 days ago [-]

This is very common in Thailand, in person and online. Shops often have QR codes than encode their bank details. Also, cash on delivery, and also you can finish some online purchases by taking money into 7-Eleven.

jpkoning(4078) 6 days ago [-]

This is good news on the competition front.

The traditional problem with bank-to-bank transfers in a retail, or point-of-sale, setting has been speed. Cheque was about the fastest that could be mustered.

But with developments like UK Faster Payments (which is mentioned in your link) bank-to-bank transfers are getting faster, in some cases instant. And so usability at the point-of-sale is now on par with cards.

In Holland, iDEAL is used a lot for retail purchases. It's an instant bank-to-bank payment option that competes with the card networks.

The equivalent in Sweden is Swish. It too is moving into point-of-sale payments.

As for the US, I suspect that at some point Zelle will pivot into retail point-of-sale payments. At which point the card networks will have a big competitor.

All of this is good news if you are worried about the card oligopolies!

exprZ(10000) 6 days ago [-]

Giving everyone access to payment processing means giving platform to unworthy people like misogynists, white supremacists and conservatives as well.

Maybe we are better off with VISA.

voldacar(4442) 6 days ago [-]

Can't tell if satire or not

seibelj(2424) 6 days ago [-]

Unpopular opinion here (from my experience), but crypto / blockchain is ever so slowly and deliberately destroying the moat around payments and the artificial barriers constructed. Probably 5 more years for some very serious mainstream business movements into it, and 10 more for non-technical consumers, but I believe with the rise of stablecoins (USDC) and Ethereum scaling via proof of stake (ETH2) their days are numbered.

erostrate(4421) 6 days ago [-]

And one year after that we will get self driving cars. And the next one will definitely be the year of Linux on the desktop :)

Sure, it could happen somewhere down the line but saying that blockchain is 'destroying the moat' today is a big stretch. Blockchain barely just started to realize that the moat is actually much larger than previously thought. And that parts of the moat exist for a reason. It's barely starting to understand the moat, still far from attacking it, let alone destroying it.

Hopefully it will but I'm not holding my breath nor my bitcoins.

throwawaysea(4145) 6 days ago [-]

This has been a problem for a long time and unfortunately the outrage around the duopoly died out the last time we confronted it, which was in 2010 when Visa/Mastercard blocked Wikileaks (https://www.forbes.com/sites/andygreenberg/2010/12/07/visa-m...).

It is still a problem in 2020. Gab was recently impacted by Visa blacklisting them (https://news.gab.com/2020/06/19/gab-blacklisted-by-visa/) and also drew parallels to the realities of the 'social credit score' system used in China (https://news.gab.com/2020/06/26/social-credit-score-is-in-am...). The vagueness of Visa's allegations stood out to me as problematic, given the lack of viable alternatives.

Ultimately, the lack of a provider who acts neutrally is a threat to freedom of speech and expression for all practical intents and purposes.

chejazi(2436) 6 days ago [-]

It's trickled into other mainstream/downstream services as well like Patreon. Patreon bans certain creators because they have a 'policy' on acceptable content that is backstopped by visa/mc policies.

Animats(2227) 6 days ago [-]

Visa is now a publicly traded company, but it used to be a chaord, owned by the banks that used it. It's a data network and a standards organization. It doesn't issue cards or handle the money, it just passes transactions from one bank to another. The banks settle up separately.

harry8(4418) 6 days ago [-]

https://www.forbes.com/sites/andygreenberg/2010/12/07/visa-m...

Note that this was a policy decision based on whim and a politician lobbying. No hearing, no evidence, no recourse, no rights and no meaningful alternative. Also a decision not taken by the banks nor, I believe, were they consulted.

The upopular person's (Wikileaks here) rights are your rights and my rights. If you think the've done the wrong thing, you're entitled to that opion and establishing that is literally what courts are for. Much the same way we might want law and courts involved for suspension of a driving license and not simply because a politician doesn't like you and lobbies a bit. 'Nobody who drives for UPS can drive on the roads because they love Putin' --not as ridiculous a fabricated politician's quote as it should be.

But going back to it being a data network and standards organisation that describes the classic long-run decreasing average total cost curve of the natural monopoly which makes privatising it for profit a pure 'rent-seeking' play.

So we can see how egregious the monopoly is two ways there. Practically with an example and according to classic and relatively uncontroversial micoreconomic theory.

So what about the classic monopolists' defence, which will come up again here. Define the market to be bigger and claim it's a small fraction of that bigger market. Can we just dismiss that as total B.S.? ie 'You can also use cash to buy things.' Try running a business or calculate the additional cost of buying the things you need without using the visa/mastercard network.

Unless you have multiple networks with very low switching costs it's a disaster. Disclosure: Ajit Pai disagrees with all that totally.

random3(10000) 6 days ago [-]

Cards still account for less than 18% of US payments (source https://go.plaid.com/rs/495-WRE-561/images/Plaid-Modern-guid...).

ur-whale(3297) 6 days ago [-]

18% is freakishly huge, we're talking about the whole of the US here.

What's worse, if you look at the details, there is a swath of products that are darn near impossible to buy without a credit card.

Laremere(4430) 6 days ago [-]

From your source 'More than 82 percent of the value of all U.S. payments goes through ACH'

Note that it's value, and not number of transactions.

Personally, I have only 4 things pay out of my checking account. Mortgage, HOA dues, and power are all bills that require checking account transactions for auto-payment to avoid credit card fees. These are very large transactions which will greatly skew any measurement by value.

The final thing paid out of my checking account is interesting: the credit card payment. Since any money I spend with my card necessarily is repaid from my checking account, that also greatly skews measurement by value.

If I spent 50% on housing, and spent all of the rest of my money on things with my credit card, then my personal ACH value percentage would be 66%. (1 unit house payment, 1 unit credit card transactions, 1 unit paying credit card bill)

This isn't even starting with business to business transactions. I'm unsure if the source is counting it in that metric, but it would further skew any value measurement. No factory is going to use credit card when buying $100,000 worth of parts from a supplier.

All those things considered, 82% seems about right, even if you assume something like 90% of consumer transactions use credit card.

kyrra(4047) 6 days ago [-]

(Googler, opinions are my own)

It really depends on the types of products and what the consumer experience is like. Google is an interesting beast in that we provide services across a wide variety of billing models.

You have immediate product purchases, where knowing that the transactions is complete immediately can be important (ex: Play store games, or movies). Credit cards are great for that instant guarantee.

For delayed billing or threshold billing (Ads), slower payment methods can work great (eg: ACH, wires, vouchers). Some of these also allow a standing instruction that a company that just keep paying money against to top-up their account (and Google will see it as a bank statement push payment).

So yes, Credit Cards are a small volume of US payments, they enable specific products that don't work great with ACH or wires. If the US ever gets 100% ubiquitous instant bank push payments, maybe that'll change, but our banking system is too disjoint to move at any kind of speed to do this.

DennisP(3711) 6 days ago [-]

Just to clarify, that says 18% of the value, not 18% of the number of transactions. From googling a few months ago, credit cards appeared to have the most transactions.

sm4rk0(4181) 6 days ago [-]

However small you think that %-age is, the absolute numbers are still huge (src: Wikipedia):

'100 billion transactions during 2014 with a total volume of US$6.8 trillion'

grawprog(3767) 6 days ago [-]

Bet those stats go way up in the next few months as more and more businesses get on the refusing cash bandwagon. Apparently it's for 'safety' but I live in a country where my money's made out of plastic or metal, that shit can be sprayed or dunked in iso. The excuse seems pretty flimsy too when again, places don't seem to bother cleaning off their pinpads before the cashiers or other customers touch them anyway.

ggm(4206) 6 days ago [-]

It always fascinated me that the process went to a semi regulated user pays cost recovery competition model, not to a regulated utility model.

At this point, the innovation stream has just about dried up. We're left with a need for micropayments that don't cost more to process than the value of the transaction, and almost all innovation has taken place to one side of card services.

If we fixed international funds transfer we'd probably get some incremental benefit. KYC is only part of the problem here, I recently did some IBAN transactions to the UK government from Australia and the expectations of fixed-field width (send this 15+ char reference string, but the input side has 12 chars for the reference field) were bizarre.

Huge amount of excess profit in TT.

Cheques? dead except for the USA.

Coins are dying of covid.

Remittence processes and the Islamic banking tradition is waiting to be somewhat unlocked. (trust is not transitive, unless you are a migrant worker from S.E.Asia and you have to get money back to mom and dad efficiently, without having any formal ID in the host country because your boss took your passport)

arcticbull(3938) 6 days ago [-]

> We're left with a need for micropayments that don't cost more to process than the value of the transaction.

I disagree completely.

There's no technical or business reason why we couldn't have micropayments tomorrow. It's been tried many times already. The thing about micropayments is nobody actually wants them. Each time you make a payment of any magnitude, your brain has to process a 'purchase' which carries a large mental burden, and eventually you get decision fatigue.

Micropayments are one of those ideas that people think we want, but in practice, nobody does.

As a thought exercise, why do you pay Netflix $13/month and deal with sporadic content disappearances, when you could pay Apple $1.99 for a perpetual license to whatever piece of content you could ever want? Nobody wants to make that purchasing decision each and every time they want something. They'd rather pay for an all you can eat buffet even if it's objectively worse and more expensive over time.

> If we fixed international funds transfer we'd probably get some incremental benefit.

Check out TransferWise! They've done a ton to solve this problem. They even have a currency agnostic bank account with local banking details in 6+ regions and supports 50+ currencies. [1] IMO they've largely solved remittences for the average joe.

If you've got a ton of money to move you can use InteractiveBrokers to exchange currencies at market rates for $20 per million (!!) in commission.

[1] https://transferwise.com/us/borderless/

IvanK_net(4069) 6 days ago [-]

The worst part is, that the fee for a card payment is shared between Visa/MasterCard and your bank.

Bank transfers within a country are often free of charge, but are quite uncomfortable to make (typing numbers) and usually are not sent immediately (so that you can not pay this way in a grocery store).

Banks have no interest in making bank transfers easier or faster, or opening up to cheaper competitors of Visa/MasterCard. Meanwhile, people happily keep their money in a bank, seeing that all bank services are free and that it 'can not get any better'.

I think, if anyone is to replace Visa/MasterCard, they should also make their own bank (a place where people keep money in a long-term).

spockz(4373) 6 days ago [-]

We have instant transfers between banks here in the Netherlands. It is coming for the whole EER.

Some payment terminals now can also generate a QR code that you can use to pay, in addition to the normal debit/credit card payments.

tappio(10000) 6 days ago [-]

This is a very us centric view. Most of the europe is so much ahead of US in payments that it is funny. There are dozens of alternative payment methods in the Europe. Especially in the Nordics card payments are on a big decline. There are basically two 'tracks' to move money - card networks and bank to bank. Most new payment methods use bank transfer as the method of moving money. In the Nordics, every country has a mobile payments system where your bank account is attached to your phone number, any anyone can issue a payment to your phone number and you receive it to your bank account. You can use this also in brick and mortar stores etc. Not to talk about all the bill-payment based companies like Klarna... And how about China? They don't use cards either, just look at Alipay.

mrweasel(4337) 6 days ago [-]

Just to clarify, the Danish mobile payment system (cleverly named: MobilePay) is based on debit cards, not bank to bank transfer.

The system that was designed to do bank to bank, without the card systems being involved failed horribly. It was late to market and the launch has horrible mismanaged and covered in unnecessary secrecy. MobilePay had already launched and crabbed a large share of the market, the secrecy was completely pointless and I believe it was partly to blame for the massive failure of the solution.

tappio(10000) 6 days ago [-]

There is also a huge difference in b2b payments. Cards account a very small percentage of B2B payments in Europe. I don't know how it is in the US, but I've understood that a much higher share of B2B payments go the card-track?

eloisant(10000) 6 days ago [-]

In some European countries maybe, but in France it's all Visa and Mastercard.

Even Amex has an even lower market share than in the rest of the world.

Ninn(10000) 6 days ago [-]

While it is true that we have more options, I do not agree that non credit card mobile payment options are a primary payment. These still lack a lot of usability improvement in comparison to the now widespread mobile credit cards supported by almost all banks too. So I still recognise visa and their support for Apple Pay helps maintain this power.

Heres to hoping that Apple Pay (and Android) will open up directly to banks or solutions such as Mobile Pay, which bridges to direct bank transfers for member banks and then utilise credit cards as a backup for unsupported banks.

BiteCode_dev(4319) 6 days ago [-]

Yes but concentrating that many things on your phone is not the solution. It causes different problems.

xfour(4108) 6 days ago [-]

While I agree with the sentiment, things have certainly opened up lately quite a bit with the mainstream-ization of the crypto currencies. One could conceivably avoid Visa / MC in a way that just wouldn't have been possible before.

Visa / MC remind me a bit of Ticketmaster in that they've got parties on both sides defending them because of kickbacks. You charge the merchant the 'interchange fee' plus some amount and that fee goes back to the 'card issuing' bank, so they like the system.

The merchant passes on the cost (generally) to the consumer, so they don't really notice, and the ease of moving the money in 99% of cases means everyone is happy.

sharemywin(2244) 6 days ago [-]

Pretty much the digital economy. As long as the consumer only get screwed a little per transaction, and consumers don't see the charge directly.

ryanwatkins(3787) 6 days ago [-]

Merchants certainly 'notice' and take any opportunity to use an alternative to the card networks. They only accept it because they must to avoid losing a purchase. They would rather customers pay in almost any other form due to interchange costs.

When talking to big merchants about any new payment product, the first question you will hear is often 'so, how does this lower my interchange cost?'

PeterisP(10000) 6 days ago [-]

I'm not seeing a 'mainstream-ization of the crypto currencies'. A few years ago, quite a few local businesses were experimenting with accepting cryptocurrency payments, and I could buy all kinds of stuff and services using bitcoin. I could order a pizza with bitcoin, I could buy electronics at a major retailer, I could buy plane tickets, I could pay for lunch in a local cafe.

That's not the case any more, by now all these local companies have stopped accepting bitcoin, because after the first hype, the volume simply was not there to make it worth their while. Some people (often the same people!) bought some stuff initially to try it out, but that was it, there was no sustainable mainstream business. It's still usable for some online services targeting the tech crowd, especially where anonymity might be a feature, but for everyday use of paying for physical goods and in-person services there has been the opposite of 'mainstream-ization' in my experience; by now the mainstream businesses have tried crypto and found it not useful. There's enough well developed infrastructure and service providers so that mainstream businesses could easily accept cryptocurrencies if they wanted, but they don't, because there's no significant customer demand outside specific niche markets.

lima(4283) 6 days ago [-]

> mainstream-ization of the crypto currencies

And modern Tendermint[1]/Cosmos SDK-based chains or Solana are basically distributed databases, using consensus models similar to Raft, except with byzantine fault tolerance and Proof of Stake for leader selection.

No mining, no forks, finality within seconds, large throughput. Pretty boring, actually, with little hype surrounding it - it just works, like a regular database, except there's no single entity controlling it.

One of the top five payment gateways in Korea - CHAI - uses the decentralized Terra[2] blockchain as their backend.

[1]: https://tendermint.com/docs/tendermint.pdf

[2]: https://terra.money

sm4rk0(4181) 6 days ago [-]

Let's end the duopoly, but please don't forget Google (or Facebook, Apple, Amazon,...)

seph-reed(4432) 6 days ago [-]

'Both' is word that will not make the Newspeak cut.

rvz(3476) 6 days ago [-]

Well one of the reasons why they (MasterCard and Visa) backed Libra is because they were after the potential of cryptocurrencies, but in a controlled fashion, which Libra was perfect for them, Unlike other alternatives until they themselves left Libra.

Anything this duopoly can't control is a big no-no to them which is why they detest Bitcoin and the other alternative cryptocurrencies. Some online services are beginning to accept cryptocurrencies, which is a start. Cryptocurrency ATMs are a thing to cash out money, thus one could say that you might have bypassed them.

A side note, for those offended by master/slave terminology perhaps now you can ask Mastercard to change their name. Since, its pretty much has somehow offended somebody out there. /s

sonicggg(10000) 6 days ago [-]

The terminology issue just shows us that people have got way too much spare time during the pandemic.

tiffanyh(3928) 6 days ago [-]

I am reading so much incorrect information in this thread it's maddening.

The card network (Visa, Mastercard, etc) do NOT make money on Interchange.

It's hard to have a healthy dialogue on this topic if folks don't understand the basics of how the card networks generate revenue.

Below is a decent primer.

https://www.investopedia.com/articles/markets/032615/how-mas...

dathinab(10000) 6 days ago [-]

True, also in many EU countries credit cards do matter much less. For example in Germany card payment is normally done with EC cards (girocard,vpay,etc), including NFC based payment. For long term recurring payments SEPA is common, for one time payments simple bank transactions over online banking. This includes online payment for services like Amazon. Oh an not to large local payments (e.g. restaurant) are also very often done in cash. Credit cards are only needed for a German person in two cases: Travel and non EU online shops (but which often have pay pal through which you can use your EC card!!).

EDIT: Warning in some German cities (e.g. Berlin) you will find a lot of cash only restaurants, mostly due to high costs of payment terminals not being worth it due to most people paying with cash anyway. Like the a local restaurant from where I live they bought a payment terminal it broke in some stupid accident no insurance want's to cover so now it's back to cash only.

vmception(4408) 6 days ago [-]

> It's hard to have a healthy dialogue on this topic if folks don't understand the basics of how the card networks generate revenue.

same for so many finance and legal topics

you should see the stuff people say in cryptocurrency land, you can even agree with their technology but still be surrounded by the weakest arguments

random3(10000) 6 days ago [-]

Well they 'do not' in the same sense inflation is 'not' a tax on anyone holding cash, right?

One way or another every fee must be absorbed by someone that pays.

morninglight(10000) 6 days ago [-]

( MASTER Card Ha! They are both making a SLAVE out of you.

jedberg(1967) 6 days ago [-]

I mean, technically no they don't. But that's just being pedantic. They charge the bank based on gross dollar volume, and the bank gets the money to pay for that from... interchange fees. The more volume the bank does, the more they pay and the more they get from interchange fees.

Interchange fee pricing is set based on how much mastercard charges the banks, so while technically true, the interchange fees are basically set by mastercard's pricing.

jariel(4435) 6 days ago [-]

The article does not indicate that VISA/MC makes their money directly from the 'interchange' but it's perfectly valid to indicate that VISA/MS are in fact the lynch-pin of the credit card 'system oligarchy'.

In particular, it is they that are able to set rates for transactions, and do anti-competitive things like ban e-retailers from offering relative discounts like 'save 2.5% if you use cash' etc..

It's absolutely an ancient cabal banking network, that would be disrupted in any normal, competitive system.

2.5% of a transaction considerably too much, were there efficiency, it would be less than 0.5%.

Edit: monopoly->oligarchy

craftydevil(10000) 6 days ago [-]

Already India start doing it, its UPI system enable us to avoid cards for all type of payments with processing fee. Also Rupay backed by GOI enable us to use those cards in abroad using vast connection.

rishav_sharan(10000) 6 days ago [-]

UPI is godsent. Probably the only good thing to come out of demonetization. I use it everyday, from buying groceries at roadside stalls to paying for my life insurance.

I have pretty much stopped using cash altogether and use cards mainly in those outlets which don't have upi.

crazygringo(3985) 6 days ago [-]

I don't get it -- Amex and Discover are still providing healthy competition, so the 'duopoly' the author complains about seems to be largely irrelevant. Competition between cards is thriving and consumers benefit -- witness the miniscule transaction fee the credit card companies keep after paying your rewards back of 2% to 5%, when you pay your bill on time.

The article's main point is that such large companies are a cybersecurity risk, and that the government should regulate/nationalize/globalize payment infrastructure.

Unfortunately, experience tends to show that governments would be far worse at providing secure, low-cost payment services. Also, credit cards already are highly regulated when it comes to consumer protections.

So, this article is just not making a lot of sense to me.

dpacmittal(10000) 6 days ago [-]

Amex and discover maybe popular in the US, but rest of the world still has a duopoly. I'd say 90%+ cards in India are visa/mastercard.

briffle(10000) 6 days ago [-]

Imagine if companies charged 2-5% less because they didn't have to fund CC company marketing tools like cashbacks.

matthewdgreen(10000) 6 days ago [-]

Every time I pay for something in cash at a store, I'm paying for those 2-5% rewards, even though I won't partake in them. This happens because the credit card companies, by dint of their crushing market power over small retailers, can force stores to charge identical cash and credit prices. I suppose a well-functioning competitive market would fix this problem, but weirdly enough Amex and Discover don't seem interested on competing to offer a better deal to retailers.

Credit card fees and rebates are such an amazing example of how systems can fool human beings. The entire system clearly could not exist in a frictionless, well-functioning marketplace —- but people seem hard pressed to actually figure out how the whole thing works. It's kind of brilliant.

biaachmonkie(10000) 6 days ago [-]

But since the retailers can't offer discounts for cash transactions, people who pay with cash are being over charged and subsidizing those fees.

ummonk(4395) 6 days ago [-]

The competition between credit card issuers is healthy. It's the payment processor side that isn't (though micropayments seem poised to disrupt the space).

argonaut(4047) 6 days ago [-]

I don't get it -- bringing up Amex and Discover is like bringing up Bing and DuckDuckGo when people talk about the Google monopoly.

giancarlostoro(3104) 6 days ago [-]

Sure Amex and Discover provide alternatives but if you want to start a credit card company you dont get to tag team with those two at all as far as I can tell. Which I think is a bigger problem. If you as a bank want to start your own credit card you have to go for the most supported provider to even attract potential credit card holders.

dayaz36(4279) 6 days ago [-]

Did you even read the article? Visa and Master Card own 90% of the market. Discover has ~1%. I wouldn't call that competition let alone THRIVING competition...

ravar(4361) 6 days ago [-]

As someone who has never used visa or mastercard and only discover for credit cards (I do have a visa debit card) I second this sentiment. There is no duopoly from my vantage point.

analyte123(10000) 6 days ago [-]

The article (particularly the linked tweet) somewhat addresses your point: Amex and Discover may offer a superficial degree of competition for consumers, but banks have to obey Mastercard and Visa's rules even when it comes to offering bank accounts to merchants, otherwise they (and therefore all their other account holders) can't participate in the 80% of card transactions that are on the Mastercard and Visa networks.

If you're a bank who wants their business customers to receive payment via Mastercard, you are not allowed to give a bank account to or otherwise handle payments for anyone on the MATCH list, otherwise Mastercard will shut off your bank's access to their network completely, probably putting you out of business. There is little transparency about how entities end up on this MATCH list. The article does not propose an exact regulation, but the end goal would be that a single company can't arbitrarily shut people out of most of the financial system.

adultSwim(10000) 6 days ago [-]

No one uses or takes those.

pradn(4441) 6 days ago [-]

CC rewards seem like a indirect regressive tax. Wealthier consumers who can afford to pay on time and have better credit ratings are subsidized by poor consumers who actually use the 'credit' part of 'credit card'.

badrabbit(3595) 6 days ago [-]

Would be happy to buy a usable cryptocoin card. I was thinking about interest-free crypto-loans where the debtor would speculate on the value of the currency at the time it will paid off (like options trading). If you loan 100btc you would loan with the speculation that for example it would be 10% more valuable in a year. If in a year your speculation is correct, you break even, if the price is 10% lower, the debtee still has to pay using your speculated valuation, if the price is 10% higher you lost potential money you coulf have gained had you hold onto it. Either way, you have insurance against loss the debtee takes on risk but they don't get endlessly canibalized by interest payment. Escrow or credit rating is something I have not figured out.

Regardless, an acual card and a payment processing network would cost billions to deploy.

bluesign(10000) 6 days ago [-]

You have 1 btc lets say 10k usd.

So in one year i have to pay you 11k.

It is basically 10% interest.

milkytron(4277) 6 days ago [-]

That seems extremely risky.

logicalmonster(10000) 6 days ago [-]

I'd love to see some action here but I'm skeptical. It seems like attacking payment processors is the main avenue that the thought-crime-police use to attack alt-tech sites that don't play ball with censoring. If there was any mass protest movement to push change here, the media would absolutely flood the public with scare stories about (insert evil monsters here) using payment networks to finance (evil activity here) and undermine the protest movement. It doesn't help that so many politicians are old lawyers and activists who generally don't have the technical knowledge to wade through the BS.

derefr(3913) 6 days ago [-]

I feel like it'd be fine if every single payment processor were individually part of the surveillance-industrial complex, as long as there were enough players in the space that one could get running a service by hopping around between them. For grey-market businesses to succeed, they don't really need air-tight legal protection; they just need their actions to be illegible.

This is basically the situation VPN service providers are in. Yes, in theory, any given provider could be beholden to the state. However, when a new VPN provider can spring up so easily (spin up some DigitalOcean instances, stand up a WordPress/Shopify e-commerce frontend), it's unlikely that any given new player in the space has been gotten to yet by the state (unless, of course, it's a new marque of an existing company, set up specifically to serve as a honeypot for switchers.)





Historical Discussions: Why we won't be supporting Sign in with Apple (June 29, 2020: 1068 points)

(1069) Why we won't be supporting Sign in with Apple

1069 points 6 days ago by dirtae in 2322nd position

blog.anylist.com | Estimated reading time – 10 minutes | comments | anchor

Starting June 30th, Apple will be enforcing a new rule in the App Store requiring many apps to support Sign in with Apple. AnyList is one of the apps affected by this new rule, which means that we must either implement Sign in with Apple or make other changes to our app. After considering the merits of Sign in with Apple, we have decided not to support it. We understand that this may surprise some of our customers, so we'd like to explain in detail why we made this decision.

Third-party login systems like Sign in with Apple cause many user experience and customer support headaches. People don't remember which login system they used to create their account. ("Hmm, I created this account a couple of years ago. Did I use my email address? Facebook account? Sign in with Apple?") Simple questions like, "How do I reset my password?" no longer have simple answers and depend on which system you used to create your account, if you can remember. And if you get locked out of your account and used a third-party login system, we may not be able to help you ourselves and will instead have to direct you to another company, with all of the hassles that entails.

In addition to these customer experience problems that are common to all third-party login systems, Sign in with Apple introduces several more that are unique to it.

One problem is that most Apple IDs are tied to an iCloud email address. So most accounts created via Sign in with Apple will use an iCloud email address. But many of those iCloud email addresses are unused and unchecked, because a customer's "real" email account is their Gmail, Yahoo, or Hotmail account. If we try to contact a customer using their iCloud email address, they may never see our message. We used to run into this problem constantly with customer support, back when AnyList used the built-in iOS email compose interface for sending support requests. This interface often defaults to using an iCloud email account. So people would ask for help, we'd reply, and they'd contact us again later, angry that we never replied. Our reply was going to their iCloud email account, but they didn't see it because they only ever looked at their Gmail account, in the Gmail app.

Another issue is Sign in with Apple's "Hide My Email" feature. With this feature, if you create an account with us, Apple will generate a special email address just for that account. So rather than your email address being [email protected], we will see your email address as something like [email protected]. While this is an intriguing idea that provides a measure of privacy, in practice it creates numerous support and user experience headaches. Here are a few:

  • If a customer contacts us asking for support, and we need to look up something in their account, typically we can just ask them for the email address on their account. But with "Hide My Email" that wouldn't be easily possible, because the customer would have to figure out the privaterelay.appleid.com email address used for their account.

  • Furthermore, if there are platforms where AnyList doesn't support Sign in with Apple, like Android, and someone wants to log into their account, they'd have to know their privaterelay.appleid.com email address. (And that certainly won't be easy to find if you no longer have an iOS device.) And then they'd have to create a password with us, since they wouldn't be able to sign in using Sign in with Apple.

  • Finally, for a service like AnyList, which is heavily focused on sharing lists with other people, the "Hide My Email" option greatly complicates collaboration. Typically, customers share a list by typing in the email address of the person they want to share with. If that person already has an account, the list is instantly shared. But with the "Hide My Email" option, your spouse or friends obviously won't know your privaterelay.appleid.com email address, so when they enter your email address, our systems will believe that you don't have an account. At that point, you'll get an email from us asking you to create an account. If you accidentally create a new account, it won't include the work you've done in your existing account created via Sign in with Apple. And if you manage not to make that mistake, then there would be a link between your email address and the account you created with Sign in with Apple, negating the value of hiding your email address.

We agree with Apple that privacy is a fundamental human right, and understand that the "Hide My Email" option in Sign in with Apple is well-intentioned, but it feels like Apple didn't really think through all of the implications for basic user experience, customer support, and collaboration. At AnyList, we respect your privacy. We're a small company that makes money when people like our app and pay for it. We do not make money with creepy tracking or by selling your information. When you provide us with your email address, it is never sold, shared, or used to invade your privacy.

Beyond customer experience, there are also many problems that Sign in with Apple creates for us as developers, which has knock-on effects for our customers.

First, implementing support for third-party login systems like Sign in with Apple significantly increases the complexity of handling user accounts in our systems. Instead of having one system for common operations like creating an account and signing in, supporting third-party login systems can quickly turn account management code into a rat's nest, with special logic necessary to handle each different login system. This is especially true when supporting multiple third-party login systems (e.g., Facebook Login, Google Sign-In, Sign in with Twitter, etc.). This makes maintenance more difficult and error-prone, and if there's any place where you cannot afford to have any errors, it's in security-critical code like account management.

It's also time consuming to implement support for a new third-party login system, particularly for a small company like ours that supports our app on multiple platforms. It's not enough for us to add Sign in with Apple to our iOS app. We also have to add it to our web app, Mac app, and Android app. (Creating even more complexity, Apple does not provide any real solution for supporting Sign in with Apple on Android, see below.) So if we choose to support Sign in with Apple, that means that we have to spend a significant amount of time to get it working everywhere, rather than spending that time improving the core list, recipe, and meal planning functionality of our app.

As developers, we always have to do our due-diligence in evaluating the security implications of our work. In the last month a massive security flaw was reported in Sign in with Apple, so serious that Apple paid $100,000 to the person who found it. If you read the linked report, you'll see that this serious flaw was also very simple, which doesn't provide a lot of confidence. For something as critical as a service for logging into accounts, it doesn't seem wise to use an immature service (less than a year old) that has recently been the subject of a serious security flaw.

Another sign of Sign in with Apple's immaturity is the sad state of the documentation for it. Good documentation is critical to facilitating developer adoption of any service. Since Apple is expecting developers to adopt this service by June 30th, it seems reasonable to expect decent documentation. Sadly, like most of Apple's recent developer documentation, it's sorely lacking. For example, Apple vaguely states that you can implement Sign in with Apple on Android, but there is no direct documentation on how to do it. We understand that Apple probably doesn't care much for Android, but if they are going to provide a login system, and are going to force developers of multi-platform apps to adopt it, then providing no real support for a major platform that these multi-platform apps run on is not acceptable.

Finally, from a policy perspective, Apple explicitly states in their usage guidelines, "Apple reserves the right to disable Sign in with Apple on a website or app for any reason at any time." If customers cannot log into their accounts, then they can't use our service. Giving a third-party such powerful control over a core part of our service when it's not absolutely required is unnecessarily risky, in our view.

What about Facebook Login?

At this juncture, you may be thinking:

  • "You have a point, but I use AnyList and it offers to let me sign in with Facebook. Doesn't it suffer from many similar problems?"
  • "Anyway, don't the new App Store Review Guidelines require you to support Sign in with Apple, since you offer Facebook as a sign in option?

These are both excellent points, and it's absolutely true that some of the arguments above apply to creating an account via Facebook. That's why we're also announcing that we'll be removing Facebook Login from AnyList. Our support for signing up using Facebook was begrudgingly added years ago as an experiment in offering another signup option, but we were never enthusiastic about it. That's become even more true as time goes on, since Facebook constantly seems to be upping the ante with creepy privacy practices. We use the Facebook SDK to provide login functionality, and every new release of the SDK seems to add new tracking options that are turned on by default, which we have to take action to disable. Furthermore, the Facebook SDK has quality problems, and recently caused a huge number of iOS apps to crash due to a misconfigured server. You can expect to see an AnyList app update soon that removes Facebook Login.

We hope this post has helped to explain why we won't be supporting Sign in with Apple. We'd love to hear your feedback on this post. If you have any comments or questions, please contact us and we'll get back to you.


Want to be informed when a new post is available? Sign up to be notified via email. Infrequent updates, no spam:




All Comments: [-] | anchor

persona(4403) 6 days ago [-]

This seems to be a common problem, made more visible when using third-party authentication, that your application has taken the concepts of 'Account' and 'Authentication Method' as if they were the same thing.

It appears that the 'account ID', 'preferred contact method+address' and 'authentication ID' are all the same here - which then creates the 'account management code into a rat's nest' scenario they describe in the post.

If an Account is, by design, it's own entity - you should be able to have 100 different authentication methods linked to that same account without impacting any other flow or part of the application.

Turn on and off authentication methods would also allow for seamless transition for users, without worrying about when one method is about to be killed.

dukoid(4289) 6 days ago [-]

+1 Any authentication method that I have enabled for my email should work -- there shouldn't be a need to remember which one is associated with a particular service...

nikitaga(10000) 6 days ago [-]

With rare specialized exceptions, pretty much nobody needs all this complexity. Certainly not a simple app like AnyList. All this flexibility is not free, it comes at the cost of obviousness, as they described. It's not worth it much of the time.

The real problem is Apple shoving their proprietary, poorly designed services down everyone's throats.

No, I don't want to use icloud email, I already have an email address. No, I don't want to provide a 'real' email address after I provided an obfuscated one. No it's not my fault that messages sent to the obfuscated one will go to some icloud inbox that I didn't create and I don't read. No, it's also not my fault that when I contact support I do it from my normal email address and not from the obfuscated one (how would I even do that). It's not the support's fault that they can't connect the two.

It's not the user's fault, and it's not the developer's fault. Apple is the sole designer of this mess. There is no excuse.

dahfizz(4429) 6 days ago [-]

I feel like they address this in the article. What you say makes sense in a technical UML diagram point of view, but that's not how people work.

The examples they give are getting support and sharing things with another person with an account. As a user, both of those things are easier for me if there is an email associated with the account.

Said another way, the Account needs some human friendly global identifier. The email you use to log in is an obvious choice, and anything else would require extra work from the user to set up. You could have usernames, for example, but that complicates the signup process and still makes sharing things hard. I know my friends emails already, but I don't know what username they ended up with on this site.

gitgud(2152) 6 days ago [-]

I think the problem is that many apps, use the same form for login/signup. So a user thinks they're logging in, but they're actually creating a new account with a new authentication method...

mr_toad(4225) 6 days ago [-]

I don't see any technical reason this couldn't be done, but it would be more work for both the app developers and the user.

lowmemcpu(3709) 6 days ago [-]

> One problem is that most Apple IDs are tied to an iCloud email address. So most accounts created via Sign in with Apple will use an iCloud email address. But many of those iCloud email addresses are unused and unchecked, because a customer's "real" email account is their Gmail, Yahoo, or Hotmail account.

Wow, this is a really good point. I just checked and yup -- my AppleID is directly linked to my icloud email, and I've never once checked my icloud email account. I wonder what's in there. Meh, too lazy to go check it

pwinnski(10000) 6 days ago [-]

My Apple ID was initially tied to my gmail address, and then at some point Apple forced me to change it, so I use my yahoo address. I never check that address, since I only use it for my Apple ID.

That has never been an issue.

stephenr(4254) 6 days ago [-]

... I'm not sure I understand this scenario?

So you have an AppleID, which is a full iCloud account (i.e. not just an AppleID using a Gmail address.. So you login to iCloud on some device, and then specifically go untick the 'Mail' option in iCloud preferences? Really?

mns(10000) 6 days ago [-]

This seems strange to me. My iCloud account is my Gmail address (even though I also have an iCloud one on the account) and when I use Sign in with Apple I get an option to redirect all emails to Gmail.

But the system is indeed weird, I signed up for an account in an app with bike routes and wanted later to check it in the browser and had no idea what or how to find out what my account is or how should I sign in (could be also the app/website didn't implement this properly).

WorldMaker(10000) 6 days ago [-]

> Furthermore, if there are platforms where AnyList doesn't support Sign in with Apple, like Android, and someone wants to log into their account, they'd have to know their privaterelay.appleid.com email address. (And that certainly won't be easy to find if you no longer have an iOS device.) And then they'd have to create a password with us, since they wouldn't be able to sign in using Sign in with Apple.

The easy answer is: they should just support 'Sign in with Apple' on every platform. (That absolutely works. Sign in with Apple is a [mostly] standard OpenID Connect provider and has a web frontend that should work on every non-Apple platform just fine, just like FB/Google/etc.)

You wouldn't think to only support 'Sign in with Google' only on Android devices? Maybe 'Sign in with Facebook' should only apply to web browsers?

It's an interesting misconception or miscommunication that so many developers think 'Sign in with Apple' should only show up on Apple devices.

dahfizz(4429) 6 days ago [-]

They directly address this point. In the article they say they considered adding sign in with Apple everywhere. Besides the fact that it's more code to write and test, they say the documentation is very poor for other platforms, as it's not even great for iOS.

l3s2d(10000) 6 days ago [-]

I agree in principal, but in practice this isn't as easy as one would hope. Each IdP has slightly different requirements and parameters for connecting clients. There may be significant code non-overlap across providers, not to mention across platforms.

Facebook, for instance, doesn't actually implement OpenID Connect, but has a custom layer on top of OAuth. Their recommended method of connecting is a client SDK for each platform.

tech-historian(2563) 6 days ago [-]

Sure, except there is no documentation for it. From the article:

'For example, Apple vaguely states that you can implement Sign in with Apple on Android, but there is no direct documentation on how to do it. We understand that Apple probably doesn't care much for Android, but if they are going to provide a login system, and are going to force developers of multi-platform apps to adopt it, then providing no real support for a major platform that these multi-platform apps run on is not acceptable.'

Karupan(4373) 6 days ago [-]

That's what I was thinking as well. Why not just treat it like any other OpenID provider and show it for all platforms?

stevepike(4329) 6 days ago [-]

Heck, I've been happily using 'Sign in with Apple' on linux anywhere I can.

scarface74(4013) 6 days ago [-]

Your iCloud account does not have to be paired with an iCloud email address. Mine is paired with my regular email address.

snazz(3516) 6 days ago [-]

Yes, but many are. This app in particular needs an email address that is actually checked.

forrestthewoods(3554) 6 days ago [-]

> Furthermore, if there are platforms where AnyList doesn't support Sign in with Apple, like Android, and someone wants to log into their account, they'd have to know their privaterelay.appleid.com email address. (And that certainly won't be easy to find if you no longer have an iOS device.) And then they'd have to create a password with us, since they wouldn't be able to sign in using Sign in with Apple.

I'm an avid iOS user with a Windows desktop. I will never use "Sign-In with Apple" for this reason. It's not useable unless you exclusively use Apple devices. Which I don't.

WorldMaker(10000) 6 days ago [-]

Sign-In with Apple is a (mostly) standard OpenID Connect provider that works everywhere (and works just like sign in with FB, Google, et al from a technical implementation), including the Web and Android. It's an interesting miscommunication or misunderstanding (and I was not surprised to see it in this article) that applications and developers think the 'Sign in with Apple' button should only show up on Apple devices. It should show up on the web and in Android apps, Apple only requires it on Apple devices because that's the only devices that they control.

I wish Apple communicated that better and/or developers better understood that Sign in with Apple really is a FB/Google/social login button like all of the others and should be supported everywhere, not just Apple devices.

(I'm in the iOS/Windows dual mode user team myself these days and find that I trust Sign-In with Apple, but I've definitely had to already email developers to request that they add the Sign-In with Apple button to websites and explain why they would/should.)

oramit(10000) 5 days ago [-]

After reading the article and comments i'm honestly a bit baffled.

Why is email address obfuscation an important component of online privacy? There are so many other more invasive and pernicious privacy concerns to worry about. It seems like we're spending an enormous amount of time to build far more complex authentication systems that are brittle and confusing just to avoid sharing an email address. Why?

Email addresses are supposed to be semi-public. If I share it with you I want you to contact me. People do abuse this, of course, but the open nature of it is exactly its best quality. I can sign up for new services easily, they can contact me, and if they bother me I block them.

I've had the same email address for almost 20 years now and have never had issues managing it. I cannot say the same for Facebook connect and Google Auth. I actively avoid signing up for services if I have to use a 3rd party auth service.

ric2b(10000) 5 days ago [-]

An e-mail address is as close to a unique id of a user as you can get online.

It makes cross-site/service tracking very easy.

thealistra(10000) 6 days ago [-]

For me it sounds like they didn't like the additional work Apple made them do, that would actually benefit the end user - I love sign in with apple, the best part is the unified workflow without typing on a phone.

I dream of a world when I won't have to type passwords on a phone anymore.

tech-historian(2563) 6 days ago [-]

Their blog post is well written and explains point by point why implementing it would be problematic for them AND for end users. Strange that you arrived at the conclusion 'they didn't like the additional work Apple made them do'

Angostura(3881) 6 days ago [-]

> One problem is that most Apple IDs are tied to an iCloud email address.

Is that actually true? None in my household are.

withinboredom(4404) 6 days ago [-]

It depends mostly on the demographic in my experience. For example, my parents have icloud email addresses that they occasionally email me from and never reply to for the reasons mentioned in this post.

intellirogue(10000) 6 days ago [-]

Worth noting that AnyList automatically subscribed me to a marketing list without double opt-in or any kind of consent, which is exactly the kind of behaviour that makes me not want apps to have my real email address.

nashashmi(4263) 6 days ago [-]

This.

The blog post was long and winded. And it brought up some very desperate arguments, like the bug bounty offered to hackers when they report security vulnerabilities.

They want people's email addresses, period.

They say that no email breaks the sharing feature. True. But that's something that can be offered later when someone actually does share something with you. They say that the emails will go to the a seldom checked account. True. But users can change email addresses. They say it breaks support service for looking up accounts without email addresses. Again true. But what's another way of looking up accounts? Username. What is another? Apple ID.

They are email network harvesters. Plain and simple. And this is their business model.

Smoosh(10000) 6 days ago [-]

The article says 'When you provide us with your email address, it is never sold, shared, or used to invade your privacy.' So, one of you is lying. I don't have the means to determine who, but I don't see what your motivation for lying would be, but can see what theirs may be.

And to extend that, if they are a spammy company, that would be exactly why they would be complaining about SIWAI.

mafriese(10000) 6 days ago [-]

That's exactly what I'm thinking! Further I can't find the EULA of the app on the internet - maybe I'll find it later. But I could bet that they sell the data for marketing purposes. And I could also bet that they removed Facebook not because they don't want to use it but because they had to implement Sign in with Apple which would result in the Data being not so valuable because the standard option is to obfuscate the mail address.

mxcrossr(10000) 6 days ago [-]

They mention customer support so many times in that article, but it's a grocery list app! When is the last time I asked for support for the sticky note attached to my refrigerator? I don't doubt that there are indeed customers who need support from time to time, but surely it's a small minority.

These seem to just be contrived arguments to protect their customer data selling bottom line.

ativzzz(4449) 6 days ago [-]

Having any kind of subscription payment massively increases complexity and support requirements in your app.

nuker(10000) 6 days ago [-]

> automatically subscribed me to a marketing list

Side note, disabling 'load remote content' in email client stops all spam in a while, they think emails are not read.

kergonath(4306) 6 days ago [-]

And their whining about not getting email addresses is exactly why I would not want to give mine.

danpalmer(3943) 6 days ago [-]

Another issue with Sign in with Apple is the fact that their private relay has a pre-set allow-list per app for sending email to relay addresses.

This means that you must either prove ownership of domains, or pre-add email addresses to Apple's systems. I understand why they have done this, it will reduce spam considerably, but the private relay system is already designed to empower users to do this and this extra step may be impossible for some developers.

Take for example a retailer – they need to dispatch goods and use different carriers in different countries. When the user buys something they very likely want email notifications about delivery, a feature that most carriers provide. For the carriers to send those notification emails you'll need to pre-add them all to Apple's systems. You can't prove domain ownership because fedex.com isn't your domain, but where are those emails going to come from? Better hope your carrier doesn't change sending address at some point or the email goes into a black hole.

Apple also limits the number of domains and addresses you can send from. In the original documentation it was '10 domains and addresses' (not sure if 10 of each, or 10 total). This was raised to 100 I believe, but that's still probably an issue for larger multi-national companies, or those who necessarily have to integrate with many external services.

The really hard-line privacy stance is that the retailer shouldn't share the emails and should do the notifications themselves, but for many this is prohibitively difficult to do, or at least detracts from places where the retailer can actually add value. The benefits are also very small, as the contracts with carriers typically protect user data, require deletion quickly after delivery, and retain most privacy benefits while allowing for a good UX.

rossjudson(4361) 6 days ago [-]

That's a good point, but I'd be surprised if Apple doesn't have (or is building) a mechanism to allow certain well-known domains to be trusted sender, in the circumstances you note. Like, have 'enter your custom domain', but also 'checkboxes for fedex.com, etc'.

surround(3776) 6 days ago [-]

What about vendor lock-in? Apple forced many apps on the App Store to use Sign In with Apple for increased "convenience" but it also makes it very inconvenient for people to switch to Android.

dangoor(4031) 6 days ago [-]

Sign In with Apple works a lot like Sign In with Google or Facebook. There's no reason it can't be used on Android also.

https://github.com/willowtreeapps/sign-in-with-apple-button-...

scarlac(4371) 6 days ago [-]

Yes, that's mentioned in the article:

> if there are platforms where AnyList doesn't support Sign in with Apple, like Android, and someone wants to log into their account, they'd have to know their privaterelay.appleid.com email address.

djrogers(1938) 6 days ago [-]

This makes perfect sense from their standpoint - especially since they've had similar problems to what they outline with Facebook sign-in and are now dropping that as well. This is also a win for Apple & end-user privacy, as there's one less app using FB's login feature now.

I think Sign in with Apple is a great step forward even if all it does is eliminate apps that require Facebook and/or Google accounts to log in. I hate that - I actually ran into a feature on my mesh router system that required a FB/G login, which made it a useless feature for me. Fortunately I didn't need it..

muro(10000) 6 days ago [-]

I've never seen an app that required a FB or Google login. It was always possible to use email+password.

alerighi(10000) 6 days ago [-]

I mean it can be better for privacy if you think about Google/Facebook loging. But it will prevent adding all third party login services, potentially even ones that are more privacy respecting than Apple.

Also there are cases where a 'sign in with <particular provider>' is the only option that makes sense because you really want to integrate with the API of this provider. Take for example a 'sign in with GitHub'. Or in case of services correlated, take for example Instagram where you obviously can sign up with a Facebook account.

I'm more for letting the developer choose what it prefers for authenticating the user and not having a authentication system that gets imposed by Apple.

stickfigure(3349) 6 days ago [-]

For my last two companies (both B2B), I implemented login via Google accounts only. Google login has a number of advantages:

1) Identity is an email address. If I wanted to rip out Google, or Google kicked me off the platform, all I need to do is add passwords and put a 'forgot my password' link and my customers continue business as usual.

2) It's not a google-specific email address. You can create Google accounts for any email address.

3) Google login effectively lets other businesses federate their auth system with ours. When they terminate their ex-employee's @example.com account, the employee loses access to their resources at my company.

I don't think you could get away with this for a consumer company; too many people have strong feelings about FB/G/Apple/whatever. But it's fantastic for B2B.

czzr(10000) 6 days ago [-]

Buries the lede. They've chosen to drop support for Facebook login rather than also support Apple login. So working as intended!

mikeryan(4420) 6 days ago [-]

I'm going to be fascinated to see what this does for conversions. My company built the Neil Young Archives, when doing so we initially launched with Social log ins and at one point Neil decided Facebook and Google were evil and wanted to remove the access. According to our logs a full 2/3s of all users were registering with a social account and we were having great success getting folks to log into a free service (We had 250k sign ups over the first weekend, we thought Auth0 might turn us off as we started on a tier capped around 40k)

We assumed this success would quickly taper off if it wasn't 'one click' to sign up with your Google/Faceboook account an talked Neil off the ledge.

philistine(10000) 6 days ago [-]

Bingo ! This is Apple's really hard bargain here. Implement us or get rid of all the other garbage sign-in.

marmada(10000) 6 days ago [-]

It is very unclear to me why removing user options is a good thing. I use sign in with Google/Facebook.

guessmyname(707) 6 days ago [-]

Either the changes are only for the mobile app or they forgot to remove Facebook Sign In in the web [1][2].

[1] https://www.anylist.com/auth/sign-in

[2] https://i.imgur.com/3uC0E9G.png

buboard(3142) 6 days ago [-]

aren't they going to be asking for the real email though?

email remains the best and free-est login identifier. Most people who are not complete internet plebs have a second email for things they dont trust.

gramakri(4427) 6 days ago [-]

Not an Apple use but I had a question. If one signs up with Signin with Apple, how can that user move over his stuff to say an Android app? Or even Desktop app? (Or does the Apple ID keep track of all this anonymous id to app mapping?)

faet(10000) 6 days ago [-]

You still have your apple account on your PC/Android phone/etc. You just login on apple's website with your apple account and it passes the token/id to whatever app needs it.

For example you can goto dropbox on your pc/whatever and click signin with apple to see how it works.

AshamedCaptain(10000) 6 days ago [-]

I found it jarring that Apple would present themselves as the vanguard defenders of privacy by announcing what is basically an email relay service. Most privacy-conscious people wouldn't exactly think of their emails going through Apple as any particular win in privacy.

And even for the 'general user' I find the argument very weak, since it doesn't look as being any easier than using any other email relay, and there is a huge obvious conflict of interest for Apple here (they get data they may not have had otherwise PLUS have yet another tool to bind you to their services).

It reminds me of the days where everyone in the www was making OpenID providers but no one was actually willing to do an actual OpenID _consumer_. So that I could actually use _my_ identity provider on a server of _my_ choice instead of going through the hoops of yet another large company for no reason.

WiseWeasel(4442) 6 days ago [-]

How is it anything but a win to have an additional easy option to use a more trusted relay rather than not? Many people have never heard of a relay, and wouldn't understand its benefit if the option wasn't presented to them like this.

blakesterz(4416) 6 days ago [-]

That was an interesting read. Also, they close with...

'These are both excellent points, and it's absolutely true that some of the arguments above apply to creating an account via Facebook. That's why we're also announcing that we'll be removing the Facebook Login from AnyList.'

dilap(3476) 6 days ago [-]

Well, that, and also if they don't remove FB login, then the would be required to also implement Sign in with Apple.

They didn't want to implement Sign in with Apple, so they had to remove FB login.

qwerty456127(4201) 6 days ago [-]

There are 3 major problems with this kind of sign-in from the user perspective they apparently omit: whenever you sign-up for a service B with an account at A (usually Google, probably applies to Apple, Facebook and the rest as well) 1. A will block your account at B at any time as soon as its (A's) algorithms realize they don't like you for some stupid reason they won't even tell you (which is icky but understandable given how many users they serve) 2. A tracks your usage of B (obviously). 3. The most overlooked - A discloses many additional details about you (like your contacts, your location, your birth date, your real name etc.) to B. Sign up to some shitty website once and they immediately have enough data on you to apply a wide range of social engineering / identity theft attacks with ease.

I actually can consciously accept the 2 in many specific cases but 1 and 3, each alone, are enough for me to avoid using this kind of sign-in.

lrem(10000) 6 days ago [-]

Why is 2 so obvious? I imagine many implementations would do that, but basically once you exchange your SSO token for the site-specific token, you could stop making any requests to the SSO provider. The only part of your SSO provider's offering that inherently needs to track you is running detection if you're a bot.

stickfigure(3349) 6 days ago [-]

If you implement Sign In With Apple, you don't have a relationship with your customers anymore. They're Apple's customers, and Apple can take them away at any time.

djrogers(1938) 6 days ago [-]

How is that different from FB/Google's login services? The only apps that are required to support Sign in with Apple are those that also support FB/G/etc sign in, so those companies have already chosen a path...

danudey(4435) 6 days ago [-]

It's good that you read Hey's blog post and are repeating it here, but it's not very accurate in this case.

Specifically, if you implement Sign In with Apple, then they are still your customers as much as ever, they just might choose to hide their information from you because they don't trust you, which means that the power in the relationship is transferred to the user instead of the app developer.

scarface74(4013) 6 days ago [-]

I don't want to have a relationship with every app developer. If you want a relationship with your customers that you control, you are free not to implement any third party sign on and implement your own.

elicash(4418) 6 days ago [-]

Not every app maker WANTS to store this user data.

Some makers just want things to work and to keep the process as simple as possible for the user.

Avamander(10000) 6 days ago [-]

They will get removed. They don't have the power to fight Apple.

dilap(3476) 6 days ago [-]

No, they're fine -- if your only login is first-party, you don't need to support Sign in with Apple. (They're taking out FB login to comply.)

fulldecent2(4410) 6 days ago [-]

I usually uninstall apps with a 1-star review if they provide zero functionality until you provide your email address.

Apps and devices serve me, not the other way around.

t0astbread(10000) 6 days ago [-]

So you don't use social media on your phone?

mssio(10000) 6 days ago [-]

Developer should just ask for user's email after new sign up using third party provider. Even Facebook does not require email for user to sign up. They should separate the email used for the account & the email used for third party sign in. Because 1 user can have multiple third party sign in, all with different email.

I think they did not need to care about customer who did not check the reply email from support, because customer can also have multiple email and did not use their primary email to sign up to your service.

esperent(10000) 6 days ago [-]

What's to stop someone signing up with a fake email and sharing it on a forum so that many people can sign in with it?

Yes, I know that there are ways to reduce this by scanning IPs and so on, but by using third party auth you offload that onto the auth providers.

tboyd47(3670) 6 days ago [-]

There's a subtle sense of exuberance shining through in this article that makes it a gratifying read, even if you've never heard of the company before. Kudos to them for their decision not to bow to Apple's demands. Please tell us how it goes!

dave5104(10000) 6 days ago [-]

Well, they're also getting rid of the 'Log in with Facebook' option too, which is arguably giving into the demands of Apple.

ogre_codes(10000) 6 days ago [-]

> bow to Apple's demands.

Apple 'demanded' companies either add their privacy friendly sign in option, or give up on the data-slurping Facebook google sign-ups. This company gave up FB sign in, which they acknowledge is pretty gross and bloated.

scarface74(4013) 6 days ago [-]

That works as intended - they took out FB integration.

no_wizard(4393) 6 days ago [-]

A lot of the points they make here are real points, and I think AnyList has validity in their actions.

I also think it's not as unmanageable as it seems.

Let's analyze this quote, from the article, as it highlights what I imagine are a big crux of this issue:

> with the "Hide My Email" option, your spouse or friends obviously won't know your privaterelay.appleid.com email address, so when they enter your email address, our systems will believe that you don't have an account

Since you know this to be the case, why not have an onboard if flow they Sign In with Apple where you have them A) choose a visibility email used for sharing/communication etc. and B) allow for this email to be their backup email? So if they forget their login or whatever you could just transfer the account to this email instead? Of course this should be opt-in but you can always Under good faith explain benefits there in.

It's more work, but I don't believe that it's going to run issue with Apple and provides end users with flexibility.

Of course this may not be worth it, at all. This is just a consideration worth thinking about as an app developer

edit: Of course another alternative here is they just make users aware of what their sharing email is and allow users to optionally change that, if they want to. This most definitely wouldn't run counter to this I'd think

duxup(4127) 6 days ago [-]

I wonder if Apple would be ok with asking them to give up their email?

Apple clearly likes the idea of the hide option... personally I would expect a less than positive reception from Apple.

I get where both AnyList (if they asked) and Apple (if they didn't like it) would be coming from here.

It does seem to be a shortcoming here where outside of a user one time sign up situation... you don't want to have to burden the user with coming up with silly names and codes to use social like features that require someone else knowing an identifier for you that isn't email.

I don't want to go back to a time where we have to remember / pass along everyone's ICQ number. ...

danudey(4435) 6 days ago [-]

> Since you know this to be the case, why not have an onboard if flow they Sign In with Apple where you have them A) choose a visibility email used for sharing/communication etc. and B) allow for this email to be their backup email? So if they forget their login or whatever you could just transfer the account to this email instead?

I'm fairly certain that detecting someone hiding their e-mail from you and then making them pick a different e-mail goes against the spirit, if not the rules, of Sign In with Apple.

That said, it would be extremely beneficial to pop up a screen saying 'Hey, is this the e-mail you want to use for communications?' and let the user decide.

That said, removing third-party sign-in is also a fine solution, almost definitely a better one, and simplifies things immensely for everyone involved (assuming their sign-in form in the app supports saving passwords to the keychain).

m_eiman(3735) 6 days ago [-]

> with the "Hide My Email" option, your spouse or friends obviously won't know your privaterelay.appleid.com email address, so when they enter your email address, our systems will believe that you don't have an account

Just create an invitation or 'share list' link and let the user send it in any way they prefer, be it AirDrop, email or SMS.

The recipient clicks the link, and the service can connect the two accounts as needed (allowing the potentially new user to create an account as needed).

bberenberg(4446) 6 days ago [-]

Do you know if this is allowed within the scope of Sign In with Apple policies? A company I work with is implementing Sign In with Apple and said they can't do this. Not sure if they're right or if this is their weird interpretation.

envolt(10000) 6 days ago [-]

Having a user name solve this. Github collaboration works the same way.

elondaits(10000) 6 days ago [-]

In a perfect world people would share things with you without entering your private email address in other people's systems. I don't want to be in the database of whatever app or system my friends decided to join, nor I want to receive spam from these companies.

Many of the objections come from wanting to do things the old way, without privacy and responsible handling of private data.

crazygringo(3985) 6 days ago [-]

As they point out at the very bottom, all their arguments apply to all third-party sign-ons, so they're removing Facebook as well.

So there's nothing specifically against Apple, despite the title seeming to imply it -- just that they're taking the move right now because of Apple's new policy coming into effect.

I've got to say, I really wish there were a way to know whether I already used Facebook, Google, or Apple to log into a site or app before. My password manager is usually pretty good at letting me know if I've got a 'normal' account with user/password, but it doesn't do anything to remind me if I ought to log in with one of the other services.

Every time I'm occasionally asked to sign into Spotify, Pinterest, Medium, Quora, etc. -- it's like, I'm pretty sure I've signed up with something before, but who even knows which one, or multiple?

If password managers could start saving that you've got accounts associated with Apple/Facebook/Google and highlight the relevant button on sign-in, it would be a big feature improvement.

jborichevskiy(3751) 6 days ago [-]

Yes! Especially once you start juggling multiple accounts for different companies and projects. Becomes a guessing game, and each wrong guess creates another account magically. Infuriating

mdavidn(10000) 6 days ago [-]

I cope with this confusion by avoiding third-party login whenever possible. Why volunteer additional information about myself to Google or Facebook?

ourcat(4443) 6 days ago [-]

> 'I really wish there were a way to know whether I already used Facebook, Google, or Apple to log into a site or app before'

You can. On each of the places you mention (Google and Facebook, certainly), somewhere in a settings page/window, you'll find your list of 'authorised apps'.

These will be a list the login systems to the third-party sites you've used to log in with.

You should then see a way to 'revoke' their access to your data.

Thorrez(10000) 6 days ago [-]

> all their arguments apply to all third-party sign-ons

What about the argument that users check their gmail addresses regularly but rarely check their icloud email addresses?

joshspankit(4444) 6 days ago [-]

Having had this same problem multiple times, I actually made it a point to save a "login" for those sites that when I autofill it reminds me which auth service I've used.

Username: Log in with FB

Password: <blank>

envolt(10000) 6 days ago [-]

Not all.

1. Apple obfuscate email - this complicates the support system, and as per them Apple hadn't thought about it thoroughly. Collaboration is obstructed. Password recovery is not an easy process. 2. Cross Platform - The post states that Apple vaguely says that sign in on Android is possible, but doesn't state how it is to be done.

ryanianian(10000) 6 days ago [-]

I create dummy logins in my password manager for sites that use external auth. Username of just "LOGIN WITH GOOGLE". Hacky but it makes me smile every time it gets filled in.

dustinmoris(722) 5 days ago [-]

TBF if single sign on is implemented correctly and you use the same email address across your accounts then it shouldn't matter which SSO you're using.

When you log in for the first time it should request permission to 'see your email address'. Then you authenticate with your provider and get redirected back at which point the website should create an account for you on behalf of that email address. If next time you log in again via a complete different provider which has the same email address then it should just work. I mean that is the whole point of this...

thaumasiotes(3886) 6 days ago [-]

> As they point out at the very bottom, all their arguments apply to all third-party sign-ons, so they're removing Facebook as well.

That section of the post was surprising. If they're not supporting Sign in with Apple, then obviously they're going to remove support for all other third-party sign-ons, because those third-party sign-ons are what trigger the obligation to support Sign in with Apple.

Ending their post about 'why we won't be supporting Sign in with Apple' with a note that they're also ending Sign in with Facebook on the merits of third-party sign-in is quite disingenuous. It doesn't matter at all what they think about the merits of Sign in with Facebook; those thoughts are completely irrelevant to their decision.

neya(775) 6 days ago [-]

Did you actually bother to read the post? The post specifically explains the headaches associated with Apple sign in. In particular -

'Another issue is Sign in with Apple's "Hide My Email" feature. With this feature, if you create an account with us, Apple will generate a special email address just for that account. So rather than your email address being [email protected], we will see your email address as something like [email protected] While this is an intriguing idea that provides a measure of privacy, in practice it creates numerous support and user experience headaches...'

meerita(3723) 6 days ago [-]

I don't support social login anymore. It's better for the customer.

rahimnathwani(2522) 6 days ago [-]

'all their arguments apply to all third-party sign-ons'

No they don't. Other sign-on options don't obfuscate the email address.

They are likely removing FB login as otherwise their next app update will be rejected by Apple for supporting third party login but not Apple login.

kripy(4249) 6 days ago [-]

Perform an audit on yourself. Both Facebook [1] and Google [2] have pages where you can check third-party apps that you have connected with. You might be surprised what you find.

[1] https://www.facebook.com/settings?tab=applications&ref=setti...

[2] https://myaccount.google.com/security

jasonlingx(4008) 6 days ago [-]

> As they point out at the very bottom, all their arguments apply to all third-party sign-ons, so they're removing Facebook as well.

Nope, not all their arguments. Only some.

cbhl(3595) 6 days ago [-]

> I've got to say, I really wish there were a way to know whether I already used Facebook, Google, or Apple to log into a site or app before.

Bookwalker (from Japan) draws a big red box around the login you used last on a given device. Presumably they store a cookie/sharedpreferences with it. It doesn't look pretty, but it helps.

grawprog(3767) 6 days ago [-]

>My password manager is usually pretty good at letting me know if I've got a 'normal' account with user/password, but it doesn't do anything to remind me if I ought to log in with one of the other services.

Doesn't it somewhat defeat the purpose of using a password manager if you use one account to sign into multiple sites?

Sign on services from main accounts seem like security flaws. If you use one main account resonsible for all your 'main things' to sign in to all the 'other things' that gives one vector of attack to enter or compromise 'all the things'.

Password managers exist to make the management of many things as easy as one thing, not to adapt to using one thing for everything, that's pretty much the opposite of what a password manager does.

Sign on services don't exist for convenience, despite being marketed that way, they exist to increase data collection abilities. Password managers exist to make using multiple accounts as easy as using a sign on service, that's the point. They should be separate from existing providers. They are an alternative to them.

cj(2897) 6 days ago [-]

> So there's nothing specifically against Apple

The one thing that is specifically against Apple is the new App Store policy that if an app uses google/Facebook sign in, the app _must_ also use Apple Sign In.

habosa(3243) 6 days ago [-]

I worked on a product that can do that (Google Smartlock for Passwords) but these "identity provider" hints were extremely confusing to both users and developers. The UX definitely could have been better but overall I just don't think it works.

dalore(10000) 6 days ago [-]

Paperspace has this with their Google integration. And I've implemented this pattern before in web dev.

You return to the site and if you have logged in with social media site before, and it detected you are still logged in, it will auto login for you.

spullara(1384) 6 days ago [-]

This is as close as we got trying to do it from the app itself: https://www.lukew.com/ff/entry.asp?1906

I absolutely agree that password managers could remember this stuff. Single-signon is pretty easy to identify and you could setup the relationship.

mcintyre1994(4438) 6 days ago [-]

> As they point out at the very bottom, all their arguments apply to all third-party sign-ons, so they're removing Facebook as well.

Nope, some of them also apply to Facebook, and Facebook has the additional destruction of privacy concern. They have to remove Facebook or support Apple too because of the policy and have chose neither instead of both.

Some of their concerns specifically don't apply to Facebook/Google/anything directly tied to your real email that you'd otherwise choose to sign up with. You add a bit of complexity to your database to record different login types, but you can easily reconcile them to an existing user if the emails match, and provide the features they want like searching for a user by email.

imron(4436) 6 days ago [-]

> So there's nothing specifically against Apple, despite the title seeming to imply it

From the article...

> In addition to these customer experience problems that are common to all third-party login systems, Sign in with Apple introduces several more that are unique to it.

TheSpiceIsLife(4288) 6 days ago [-]

This happened to me recently where, in a hurry and distracted, I logged in with one or another third party auth service then realised that wasn't the correct login as it displayed a new account.

alister(3310) 6 days ago [-]

> Apple reserves the right to disable Sign in with Apple on a website or app for any reason at any time.

Holy cow, how is this acceptable to any app developer or software company? This is reason enough for me to never use Apple/Facebook/Google sign-on as a developer -- huh, or even as a user. Apple/Facebook/Google could lock out all your users and literally destroy your business in a split second for an arbitrary policy violation, without explaining why, with no way to contact a human being. Haven't we seen enough HN headlines where an independent developer or a small software company is begging for help because <LargeCorporation> canceled their account or locked them out of something with no recourse?

EDIT: I know that AnyList is dependent on Apple's app store. This is still no reason to give Apple (or Google or Facebook) even more power over you.

nexuist(4138) 6 days ago [-]

You know the reason. It's on the front page of HN today even. Apple will support Sign in with Apple until it is co-opted by 'white nationalists' or other scary characters, at which point they will disassociate with that website or app and prevent their services from working with them.

Which is their right of course, but at the end of the day it means we get changes like these in the fine print. Gatekeepers like Apple and Google gain more control over what is allowed on their platforms, and subsequently what is allowed for the majority of the population to see.

TazeTSchnitzel(2344) 6 days ago [-]

It's not like other popular login systems can't also arbitrarily terminate your account, but the really problematic thing here is "Sign in with Apple"'s email hiding, which removes the lifeline of emailing your customers when you lose your sign-in provider.

ramraj07(4086) 6 days ago [-]

This seems like the only bummer rule of all. I really would like to use this service on my next project but this dictator rule cannot be tolerated.

rorykoehler(3793) 6 days ago [-]

I've got a new sign in flow I'll be using for all my indie apps. It solves 2 problems I have with Apple.

1) no PWA support for notifications

2) forcing stuff like this on everyone

I use a telegram chat bot. After signing up via the bot that sends you a link to set your password, you then also request a short expiry sign in link everytime you wish to sign in. The chatbot doubles as a notifications channel. I'm thinking of enhancing notifications do you can interact with them directly from the chatbot interface too.

The signin flow is great as it has 2fa built in by default.

kergonath(4306) 6 days ago [-]

I am pretty sure I would never create an account on a website that wants me to chat with a bot on Telegram. This sounds particularly user-hostile.

sharp11(3676) 6 days ago [-]

Could you elaborate? Doesn't this require users to have telegram installed?

TheArcane(4374) 6 days ago [-]

> Another issue is Sign in with Apple's "Hide My Email" feature. With this feature, if you create an account with us, Apple will generate a special email address just for that account. So rather than your email address being [email protected], we will see your email address as something like [email protected]

Ironically, this is also why I use Sign Up with Apple at every opportunity I can

Smoosh(10000) 6 days ago [-]

The article also implies that if anyone can guess your email address, they can send you/share with you a list. I wonder what anti-spam measures AnyList implements?

renewiltord(10000) 6 days ago [-]

How is this ironic? It is by design and obviously they know why people do it because the very next sentence says that. Why on Earth would you'd want to use a list-sharing app that uses email as the addressing system and then not share your email.

jackson1442(10000) 6 days ago [-]

If they really needed a user ID, just have account holders create a username after the Apple sign-in flow. Most people have a go-to username, and those are easy enough to remember and give to a support associate.





Historical Discussions: DuckDuckGo browser seemingly sends domains a user visits to DDG servers (July 01, 2020: 820 points)

(820) DuckDuckGo browser seemingly sends domains a user visits to DDG servers

820 points 4 days ago by commotionfever in 3652nd position

github.com | Estimated reading time – 6 minutes | comments | anchor

First of all - I am a big fan of DDG

  • I promote it whenever I can
  • I use it anywhere possible
  • This comment is meant to help the DDG team to understand this PR issue. Even tough - we (OS_Community) are very concerned that this explanation is necessary in the first place! We normally do not rise up from our cellars - because we code. But now where the realms of management and C0D3 collide again, and the management does not show any signs of understanding - we have to take action.

This is why I will now explain, (again) what the problem with your answer is and what it indicates.

Hi all, CTO of DuckDuckGo here. I thought it might be helpful if I briefly shared some of our internal thinking around this issue so folks can see how we got here and how we plan to move forward.

Your 'internal thinking' is what has raised this issue in public! It is basically the bare reason for our concerns! Because it shows, that you do not even understand our privacy argument! It is not about the bug, it is your action and the, like @calimeroteknik puts it, 'curios wording', that makes it appear, like you have not understood our critic at all.

Time of Events

  1. @Tritonio found a curios line of code that sends unnecessary requests to one of your backends
  2. @tagawa said: 'The purpose of the request you observed is to retrieve a website's favicon so that it can be displayed in certain places within the app or on the results page' https://github.com/duckduckgo/Android/pull/878/files#diff-63ac5c0d645555fe179e72977d9c1728
  3. Then @stefan01 and later we said, please stop this! There is no reason to send data in the first place! A big part of privacy focused software is, to avoid to creating unnecessary data in the first place. It is not necessary! Please do not send our browser history or any unnecessary request, to any server! No matter how 'secure' this server is. Please load the favicon locally, we love you and we do not like heartbreaks! In the worst case - security is only a illusion - only the hacker with root access knows, what is really going on and maybe like @Tritonio indicates Microsoft.
  4. DGG show no real understanding and argues in a 'curios way' - and now you @nilnilnil continue

Bugs can happen! Bad decisions can happen! But arguing with 'trust us, it anonymous', it 'saves user badwidth', and its 'the only visible attribute', on top of that 'we already have a anonymous [...] service'! It is pure poison for trust, because it indicates, you entirely missed the point! Either on purpose or not. Both would be alarming!

The logic behind how we've been displaying favicons in our apps is a function of how we operate our private search engine. Since we already have an anonymous favicon service through our search engine, using it has a number of benefits: it avoids more requests to known non-anonymous websites that are visited, it's way faster since it runs server side, saves user bandwidth, and the only externally visible attribute is that the app is connecting to DuckDuckGo.com (as the favicon location is actually encrypted in the path in transit). To our team, utilizing this anonymous service we had made for the search engine seemed like an optimal principled choice across a set of criteria.

Its not that we do not understand your reasoning or the reasoning of @tagawa. The problem is your reasoning, seems to entirely miss our point!

Now you continue - as CTO! This indicates no real insight at all!

We want to be clear that at no point was the actual visited domain otherwise exposed. This favicon service is fully anonymous on our end, and URL parameters (like the favicon domain) are encrypted in transit, just like the search engine (with search queries). This is also why when this issue was raised in the past, we decided to keep the solution as is. At no point was this ignored.

We said, please do not mix the search engine code and your browser code! They should not relate on each other if not necessary (for example if the user actually wants to do a search on DDG). Arguing 'we already have a anonymous [...] service' is very hard to understand from a technical point of view. It might make sense for a CFO, but not for nerds or hackers!

However, we understand that there is an alternative method of getting the favicons locally that a lot of folks prefer while still maintaining our privacy standards. We also believe that method is in line with our product vision of 'Privacy, simplified.', considering its a somewhat simpler method than the one we had been using.

Like @calimeroteknik already said: Luckily higher security standards meet your standard! -_-

So, we went ahead today and implemented the change for both Android (#878) and iOS (duckduckgo/iOS#667) that will move this logic onto the client, and we will no longer be using the favicon service in our apps. These changes are currently in the release phase and are rolling out live now.

You only took this action after enormous public pressure, there was no sign of insight or internal intention to resolve this! This is why you have been compared with BigCorp, because they act like this. (Or like pointed out by some people, are even doing a better job than you, handling these kinds of issues)

[LARP MODE ON] [Loading Game Data...] You currently have (3) quests! [(1) Closed Quest] Even under enormous public pressure, do not remove the criticized code, before at least one year has past! [(1) Open Quest] Understand the whole point, and communicate it in a acceptable way. [(1) Bonus Quest] Build trust back by transparent actions in the future, which reflect that you understood what we said [LARP MODE OFF]

We really appreciate the feedback and exchange of ideas on this topic, and on ways to further improve the privacy of our products in general.

You are welcome! I hope this helps!




All Comments: [-] | anchor

eightlimbed(10000) 4 days ago [-]

Can someone please explain like I'm five how this line of code sends the domain a user visited to DDG's servers?

oplav(4439) 3 days ago [-]

Every time you visit a website on the Android version of the browser, instead of requesting the favicon from https://example.com/favicon.ico, the app is calling out to https://icons.duckduckgo.com/ip3/example.com/favicon.ico.

Since DDG owns the icons.duckduckgo.com service and the domain you were interested in is in the request to icons.duckduckgo.com, you've sent the domain to DDG's servers.

bangonkeyboard(1005) 4 days ago [-]

EDIT: I didn't notice that this topic was about the DDG browser (which I didn't know existed) and responded assuming this was about the site/extension. For a browser, yes, a client-side solution is possible and probably preferable. Please check and upvote other comment trees.

This makes sense to me and is not alarming. Getting favicons actually is difficult to do robustly; many applications and websites use Google's service to do so, which then leaks the request to Google: https://www.google.com/s2/favicons?domain=ycombinator.com

Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?

DuckDuckGo hosting this functionality themselves is also a positive. They have previously been burned when the Web of Trust service they were originally using was found to be farming data, and turned it off immediately once discovered. Processing, hosting, and serving the icon themselves prevents that from happening again.

This is not to say that DDG is perfect: links you click do seem to be redirected through a /l/ page on their domain, which can cause problems: https://lapcatsoftware.com/articles/duckduckgo.html

detaro(2148) 4 days ago [-]

Which other browser doesn't fetch favicons locally?

And I have no clue how search results come into this? Of course the browser isn't fetching favicons on search result pages, that's the servers job, but that's not what this is about.

It's really odd that this is the top comment right now, given that even the headline makes it clear its about the browser, not the search results.

mdoms(4413) 4 days ago [-]

> Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?

I don't see why not. Browsers are constantly dealing with these very issues. It's one of their core competencies.

harry8(4418) 4 days ago [-]

> Getting favicons actually is difficult to do robustly

So if you're a privacy browser. Don't. Favicons are not essential. Or, at the very least make them 'opt in' and explain it.

stefan_(4448) 4 days ago [-]

I don't need the favicon for a domain I never visited just because it adds some visual cool to your UI. That is not a thing. Cache it whenever I end up visiting the page and be done with it.

fiddlerwoaroof(10000) 4 days ago [-]

Don't they already have access to all the search results anyways? This doesn't seem to be any privacy loss relative to that.

richdougherty(4442) 4 days ago [-]

> Putting this logic in the client is not feasible.

> You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?

Looking at DuckDuckGo search results and visiting a page that you navigate to are two different things.

1. DuckDuckGo search results:

DDG already returns the search results so there's no privacy violation to return the favicon or the URL for the favicon in the list.

2. Any page that isn't a DDG search results page:

Use client side logic to locate the favicon. This means worse performance but better privacy - which aligns with DDG's goals.

If you want to optimise this then DDG could send the client some precalculated Bloom filters with info about known sites. The client could use these to try certain methods of favicon retrieval first.

tananaev(4332) 4 days ago [-]

For the search results DDG can do whatever they way. As far as I understand from the title, this is for the pages that you actually visit. For those pages it totally make sense for the browser to figure out favicon locally on its own.

As for complexity, I'm sure users of privacy oriented software would prefer to not have favicon in small percentage of corner cases rather than leaking browsing history.

pwdisswordfish2(10000) 4 days ago [-]

'This is not to say that DDG is perfect: links you click do seem to be redirected through a /l/ page on their domain ...'

I am surprised the user is not complaining about this instead of the favicons. Their privacy policy goes on about the privacy implications of Referer headers and instead of calling out browsers for sending Referer by default, they instead give themselves power to record all the user's clicked results themselves. The Referer problem is something that can be solved by the user at the browser level through, e.g., using a client that does not send Referer, browser extensions/plug-ins that can control headers sent, or perhaps with a local proxy to remove the Referer header.

Unless DDG has changed, these prefixed result URLs are the default. It is possible to get unprefixed result URLs using the 'lite' version of DDG however that is not the default. 'Privacy-focused' search engine chooses less private default. News at 11.

I recently noticed that DDG has started redirecting queries submitted via POST to /lite/. The redirect is to the same domain. No explanation. I have a custom client that does not follow redirects and I now have to submit two sets of HTTP headers instead of one.

These guys are trying to make money from advertising just like everyone else. They have to be very particular in the methods they use to do it -- check the exceptions in their privacy policy -- but it is the same game. Ads and affiliate links. That sort of business and privacy are always going to be at odds with each other.

dmitrygr(1693) 4 days ago [-]

> Putting this logic in the client is not feasible

Wait, what? Why?

niftylettuce(3333) 4 days ago [-]

Formerly worked with DuckDuckGo

My advice:

Install ungoogled-chromium: https://github.com/Eloston/ungoogled-chromium

Install these extensions: https://github.com/gorhill/uBlock https://github.com/ilGur1132/Smart-HTTPS

There is also a Chromium extension that lets you install from Chrome Web Store: https://github.com/NeverDecaf/chromium-web-store

Set duckduckgo.com as your default search engine with a blank home page. But you could also use @pkrumins home pages of https://techurls.com or https://finurls.com as nice home pages.

Use Mullvad VPN: https://mullvad.net/ (They are EVEN available on F-Droid now, which is AMAZING)

Security harden your Android device: https://niftylettuce.com/posts/google-free-android-setup/

Security harden your Mac: https://gist.github.com/niftylettuce/39597a7b3bc0660ffe1e09d...

P.S. If you need email forwarding for your domain name, you can use something I made. https://forwardemail.net - it is 100% open source.

Follow me @niftylettuce on GitHub and Twitter for more

RandomBacon(4310) 4 days ago [-]

Why ungoogled-chromium over Firefox?

morpheuskafka(4402) 4 days ago [-]

ForwardEmail is a great service, thanks for making it!

adjagu(10000) 4 days ago [-]

Could you please elaborate as to why you suggest ilGur1132's Smart-HTTPS instead of EFF's HTTPS-Everywhere?

the_duke(3481) 4 days ago [-]

Seems a bit off-topic for the concrete issue.

Advertising your Twitter for the advice of 'switch to somewhat well-known browser X, install these very common extensions and use a VPN' is also a bit ... odd.

bluehatbrit(4211) 4 days ago [-]

Your 'Security harden your mac' guide has a bunch of amazon affiliate links, it would be good to call that out when piggybacking on a post about privacy at least.

livre(10000) 4 days ago [-]

> Security harden your Android device: https://niftylettuce.com/posts/google-free-android-setup/

I haven't checked all links but some things need to be updated, for example Skimmer Scanner is gone from the Play Store and Yalp Store is abandoned and doesn't work anymore, you should be using Aurora Store. I'd also recommend Aegis or FreeOTP+ over FreeOTP for 2FA. NewPipe is better installed from this[1] repository until this[2] issue is solved.

[1] https://archive.newpipe.net/fdroid/repo/

[2] https://github.com/TeamNewPipe/NewPipe/issues/1981

awinter-py(1221) 4 days ago [-]

seems like the ticket author found this by reading code (presumably was grepping for duckduckgo.com URLs)

this would never happen with a consumer-facing product from apple or google; someone would have to MITM their whole OS to discover phone-home

LeoNatan25(3577) 4 days ago [-]

It actually happens very often. People monitor their networks often, and pay close attention to such problems. Just search for "Apple phone home" and you'll find many cases where people complain about Apple's various services making worrying requests.

tomtomtom777(4422) 4 days ago [-]

I have a hard time understanding the problem.

The favicon is acquired from DDG servers for the result you've just retrieved from DDG servers.

How is this leaking anything? What additional privacy would you gain from getting the favicons from the domains directly of search results delivered by DDG?

marcinzm(10000) 4 days ago [-]

This is the DDG mobile browser app, NOT the DDG search engine.

yegg(3842) 4 days ago [-]

Hi all, Founder and CEO of DuckDuckGo here. I'm literally just waking up and reading the comments here.

I'm new to this issue and happy to commit us to move to doing this locally in the browser and will have us move on that ASAP.

That said, I want to be clear that we did not and have not collected any personal information here. As other staff have referenced, our services are encrypted and throw away PII like IP addresses by design. However, I take the point that it is nevertheless safer to do it locally and so we will do that.

fierarul(4367) 4 days ago [-]

> That said, I want to be clear that we did not and have not collected any personal information here.

So, you do collect information just not the kind you would classify as 'personal information'. I wonder if my personal domain with my full name qualifies?

There is no way this feature would be created in a company built on privacy considerations.

Like, wow!

bluesign(10000) 4 days ago [-]

I think it would be more clear if you come up with a statement like:

'we never used this data, other than showing favicons'

mv4(4447) 4 days ago [-]

Good. Thank you.

Terretta(1602) 4 days ago [-]

To be more clear, your staff, and you, have said PII 'like IP addresses', and have said 'thrown away' some places and 'not collected' others.

Contrary to this framing, it's not possible to not incidentally become aware of every single browser users' usage timing and user IP addresses if the browsers are phoning home this way — a colloquial understanding of 'collect', not the James Clapper NSA dodge definition of 'collect'. Most normals think of collect as become known not as permanently store. You knowing it means others can know it if you break trust or are required to comply with authorities.

And regardless of end-to-end encryption, that this user is phoning home to your fave icon endpoint, when, and from what IP, is revealed to every ISP in the chain. You're leaking browser usage telemetry to every single party to that traffic — the source IP address PII you mention is in unencrypted metadata.

The fact this browser connects to that endpoint reveals demographics (choice of privacy browser) and behaviors (when and how much web surfing) to e.g. ISP or nation state firewall operators who are certainly not bound by your 'just trust us' privacy policy.

Privacy policies are a patch for insufficient privacy engineering.

To be a strong privacy browser you could consider what it would take to be "NSL proof" such that if handed a national security letter with gag order, you cannot comply. That is not the case with this faveicon telemetry endpoint.

jsnell(148) 4 days ago [-]

Thanks. What kind of process changes are you putting in place to prevent things like this from happening again?

worble(4363) 4 days ago [-]

It's a little confusing to see 2 accounts, with very different usernames (epi0Bauqu and yegg), that appear to be posting as Gabriel Weinberg . Are these both legitimate? And if they are, what's the reason for two of them?

ddevault(327) 3 days ago [-]

Thank you for re-opening and prioritizing this.

However, this problem demonstrates gross incompotence for a browser team supposedly concerned with privacy. Will you please do a post-mortem on how this code made it through your code review process in the first place, as well as how it managed to stay in place for a full year after it was pointed out that it represented a privacy problem?

'Sends every URL you visit to the vendor's servers' is the single worst thing DuckDuckGo could have done for privacy in this web browser, and that needs to be accounted for. There was a major failure in the code review process, ticket review process, and in how you treat your community. A standard marketroid 'by design' response with washy promises that 'we'll take very good care of this highly sensitive personal data, just trust us' is not something I want to see in the future from this team.

[reposted from GitHub]

ryuukk_(10000) 4 days ago [-]

liar

pelliphant(10000) 4 days ago [-]

Thank you, this is the response people here want to hear.

And if this gets fixed in a reasonable timeframe, this is just one of those 'everyone makes a mistake one in a while'-things, no big deal.

jacquesm(45) 4 days ago [-]

Thank you Gabriel. This was what I expected.

MfromB123456(10000) 4 days ago [-]

Why do you develop features without considering privacy in the first place? Every developer on earth could have told you, that your favicon service is potentially violating users privacy. One could guess you don't know much about how to keep users privacy. Other guess would be, you implemented this service that way deliberately.

chrispauley(10000) 4 days ago [-]

This response validates my trust in DDG as has happened so many times before. Seriously cool company you've built here.

_hl_(10000) 4 days ago [-]

I've already posted this somewhere else, but I'll copy it here again as well:

It's not immediately obvious whether it is more privacy preserving if the client automatically makes a request to each site in the search results while scrolling through the results, especially since you're already trusting DDG when performing the search.

Maybe this should be an opt-in rather than an opt-out feature?

All in all its really not as big of an issue as people here make it out to be.

m463(10000) 4 days ago [-]

Thank you, keep up the good work!

(I always knew there was a business model for privacy and I'm glad someone is working to figure it out)

bad_user(3400) 4 days ago [-]

Speaking of leaks, I never understood why people use DDG's bangs.

By using bangs you're sending your search history to DDG even when using search engines that aren't DDG.

Nextgrid(4086) 4 days ago [-]

Because it's the easiest (and sometimes only) way to get this functionality on major browsers and browser developers have no plans to implement this functionality natively.

pelliphant(10000) 4 days ago [-]

but.. that is obvious?!?

If I don't want ddg to know what I google, ofc I won't do it by typing in !g in ddg...

I actually use bang functions because I want to help ddg out by informing them when I'm unhappy with the results I got from them.

tome(4213) 4 days ago [-]

> you're sending your search history to DDG

I was astonished by this at first, but I think you must mean 'you're sending all searches performed with bangs to DDG'. I worried that you meant somehow the browser search history was being sent to DDG, but that seems impossible.

mkl(4339) 4 days ago [-]

I only use bangs when regular DDG search has failed me, and I'm quite happy to tell DDG of that failure.

tagawa(1919) 4 days ago [-]

DuckDuckGo staff here. As mentioned in the linked page, the purpose of the request is to retrieve a website's favicon so that it can be displayed in certain places within the app or on the results page. We use an internal favicon service because it can be complicated to locate a favicon for a website. They can be stored in a variety of locations and in a variety of formats. The service understands these edge cases and simplifies retrieval within our apps and our search engine.

Like our search results, the favicon service adheres to our strict privacy policy[1] in that the requests are anonymous and we do not collect or share any personal information.

[1] https://duckduckgo.com/privacy

leowoo91(4407) 4 days ago [-]

favicon has fairly enough known meta tags in case favicon.ico url is lacking

bluetwo(4371) 4 days ago [-]

Can you tell, for instance, how many of your users visited site A?

Can you tell how many visited site A and also site B?

bluesign(10000) 4 days ago [-]

Sorry but this is not enough reason. There is a simple question you should ask to yourself.

- Would you be ok to use a third party for this with same privacy policy?

ViViDboarder(10000) 4 days ago [-]

Complicated code can run just fine on device.

I've been an avid DDG user for years and it worries me that DDG staff don't see why this is an issue. We shouldn't have to trust your privacy policy if you minimize exposure.

sm2i(10000) 4 days ago [-]

so, essentially you blew privacy because of favicons? favicons??

beshrkayali(1970) 4 days ago [-]

I'm surprised at how you're handling this. DDG is supposed to be friendly to privacy-aware users. You're dismissing people's valid points and asking them to trust you, just like any other privacy-non-friendly service would do.

Edit: I'm speculating here. But specifically because of the way you've replied here and on Github, my actual level of trust in DDG team went down.

Mandatum(4190) 4 days ago [-]

What a bizarre potential privacy flaw to introduce for a tiny little icon nobody cares about. I understand it, usability and UX is important - but you guys are DDG! Come on! Your customers all have tinfoil hats!

byteshock(10000) 4 days ago [-]

I'm very disappointed in how you guys responded to this. As a privacy focused company I would not have expected an answer that sounds like it came from a data collecting company like Google.

I just switched to DDG browser a week ago and will now be looking for a new browser now. I hope you know this is not an appropriate response to the situation. Especially because all you guys do is preach about how much you protect your users' privacy. Now you're here asking us to trust you not to abuse our data and just linking us your privacy policy. I'm sad to say that my faith in the DuckDuckGo company and team is now lost.

MrGilbert(10000) 4 days ago [-]

Maybe I'm too old for this, but wasn't a favicon supposed to be located at 'fancy.url/favicon.ico', or alternatively as a '<link rel='shortcut icon' \>'?

Curious to know why this is an issue.

dna_polymerase(3641) 4 days ago [-]

> We use an internal favicon service because it can be complicated to locate a favicon for a website

That must be the worst justification for this possible. Favicons. Complicated to locate? Who are you trying to fool, 5 year olds?

iandanforth(4234) 4 days ago [-]

This is obviously an insufficient answer. Why risk your one selling point on such a trivial bit of code?

bn7t(4352) 4 days ago [-]

Please choose another hill to die on. This is just not worth it. Clearly it's possible to do this on device like mozilla did it.

NotSammyHagar(10000) 4 days ago [-]

I appreciate you answering, probably knowing you'd face some negative feedback. Saying 'we should trust you, it's for a good reason' is what google and everyone else says. You'll be better off if you just end this. The loss of the fav icon is less important than keeping your credibility.

rezonant(10000) 4 days ago [-]

Doubling down is fairly ridiculous when one has to imagine the original reason for doing this was to save on time and engineering for the app by leveraging what DDG had already built, but a mindful response would be that you are aware of the downsides of this approach and you'll be working to change it.

Besides, how do you handle Intranet, VPN sites, and auth-only sites where DDG's god-tier favicon parser in the cloud couldn't fetch the URL anyway?

gap(10000) 4 days ago [-]

It's amazing how tone deaf technologists can be when it comes to privacy, even when they have nothing to gain by exploiting the user's data. DDG's response reminds me of Mark Shuttleworth's argument that they "have root", so we can trust them with our life.

Dear DDG, you are getting complaints on GitHub and Hacker News. This is not the general public, it's people who understand the issue. You should definitely reconsider whether you're doing something wrong.

st3fan(2492) 4 days ago [-]

Take the code from Firefox iOS or Android-components. We spent a lot of time on these and it is all on device.

https://github.com/mozilla-mobile/android-components

https://github.com/mozilla-mobile/Firefox-iOS

ComodoHacker(3773) 4 days ago [-]

Technical aspects aside, don't you agree it's a legitimate privacy concern from the user's point of view?

jacquesm(45) 4 days ago [-]

Does Gabriel know about this? If not could you please clue him in and get some guidance because you are absolutely getting roasted here and are wrecking DDG's carefully built up reputation. I can easily see how this might seem to be a good idea to you and other DDG engineers but it goes 180 degrees against DDG's stated mission. In other words: you may be well outside your paygrade on this.

fiddlerwoaroof(10000) 4 days ago [-]

This doesn't make sense for a browser: just embed the service's logic in the browser, the browser has all the same information the service could get.

mysterydip(4449) 4 days ago [-]

Are all these edge cases part of the html/w3c/whichever standard? If not, let the edge cases fail. I'm not going to lose sleep over an icon not showing for a site I'm visiting once.

ximeng(2534) 4 days ago [-]

You're repeating what's on that page, which is exactly what everyone is worried about in the first place.

Igelau(10000) 4 days ago [-]

At this point we're all well aware why the app phones home. Continuing to spout that like it's some ward against the fact that this is a very real vulnerability is an insult. Trust me, your target audience doesn't give a crap if favicons work; they care that DDG acknowledges the risk of a glaringly obvious vulnerability. Who do you even think you're arguing with on HN and GitHub? My children can't multiply yet but they'd be able to understand why this is bad practice.

The repeated handwaving that no one in your company is ever going to do something bad or stupid when the browser phones home for what amounts to a cute sticker is extremely suspicious.

ajnin(4296) 4 days ago [-]

You really can't use 'we promise we won't misuse the information' as an argument, that's what everyone says whether it's true or not, and the whole point of using a privacy-centric browser is that as a user you can't trust those kinds of promises.

trustmeimdrunk(10000) 4 days ago [-]

What a wussy excuse. This should have been a no-brainer decision: do we quietly compromise privacy so that our users can have little icons on their browser tabs? How absurd.

Duckduckgo chose compliance with the inconsequential minutae of bigtech over its primary pain point. This is indicative of misalignment between stated values and the values demonstrated through actions. If you guys made this call, sacrificing privacy for something so banal, it doesn't bode well for what's going on in the rest of your operations. That means the problem isnt technical, its cultural, which are unfixable, therefore it's pretty much over. Good luck to regain trust once you've been outed as a put on.

sujesh(4379) 4 days ago [-]

come on dude. I thought you got a better answer.

SimeVidas(4074) 4 days ago [-]

How can users turn this off?

alistproducer2(3510) 4 days ago [-]

The reviews are in for this response and they are bad. It's concerning that given the react it got, there's no edit addressing the concerns. The HN audience has to be the power user, bread and butter of a product like this and when you see a company ignore the concerns of a key constituency like this, their future almost never looks bright.

cheez(10000) 4 days ago [-]

I wonder how tone deaf you have to be to leave this response and think you're doing the right thing.

jtdev(4381) 4 days ago [-]

I don't care if I get the favicon or not... can I just opt out of this functionality?

izietto(3673) 4 days ago [-]

You are just repeating what's already written in the link, it isn't very useful. Try to address users' concerns instead.

ddevault(327) 4 days ago [-]

I'll state this in no uncertain terms: this is not acceptable, and you need to stop doing it. It makes sense on your search engine, but adding it to your web browser is very much over the line.

I have read your explanations in good faith and they don't cut it. This behavior cannot continue. Good privacy promises are not based on trust - they're based on not ever handling private data in the first place. If you don't quickly admit your mistake and roll this back, it will jepoardize your entire brand - and rightfully so. If you believe this behavior is okay, then it demonstrates incompetence; if you don't believe this behavior is okay but do it anyway, it demonstrates malice.

This is the one thing you Should Not Have Done.

dvdkhlng(4432) 4 days ago [-]

In germany we have the words 'Datensparsamkeit' (data parsimony) and 'Datenvermeidung' (data prevention) [1]. Which wikipedia merely translates as 'Privacy by design' [2].

DDG is unneccessaryly producing (aggregating), transmitting (and collecting?) very sensitive user data here, which is just the opposite of data protection. I can't even understand why they try to justify their actions. It's like omitting the seat-belt in a car, then telling customers that this was required to make the in-car entertainment system more usable.

[1] https://de.wikipedia.org/wiki/Datenvermeidung_und_Datenspars...

[2] https://en.wikipedia.org/wiki/Privacy_by_design

aronpye(4436) 4 days ago [-]

The road to hell is paved with good intentions. At the end of the day, your privacy policy is just a bunch of words with nothing to actually prevent you from abusing the data collected. Instead of us relying on DuckDuckGo to act ethically, just don't collect the data in the first place.

scoutt(10000) 4 days ago [-]

This answer and other replies from yourself in this page seem copypasted from here: https://help.duckduckgo.com/duckduckgo-help-pages/privacy/fa...

why?

zzo38computer(10000) 4 days ago [-]

Does it have an option to disable the internal favicon service and/or an option to disable favicons entirely? (I disable the favicons on my own computer, since I don't use them.)

Sir_Substance(10000) 4 days ago [-]

I don't know how you can misunderstand your core demographic this badly mate.

If you think the next time I hit the shitter I'm not going to be looking for a new browser, you're dead wrong.

Just do the basic checks and then fall back to a DDG logo, no one cares that much about the favicon.

speedgoose(4449) 4 days ago [-]

This is a bit concerning that for a company with a marketing so focus on privacy, you guys thought it was a good idea to have such a service.

Nobody in the company at any point thought that it could be a problem?

Your strict privacy policy mean nothing by the way, because you are a USA company and you must respect your local laws such as the patriot act, which are not very privacy friendly.

bambax(3768) 4 days ago [-]

> the purpose of the request...

... is completely irrelevant. Even if they were trying to save babies from a fire (which they really aren't) it wouldn't excuse the fact that they're doing something orthogonal to their stated policy and sole reason for existing.

Everyone makes mistakes, that's not the point. The point is to correct them when they're found, instead of digging one's heels in the ground and pretending it's nothing.

zeckalpha(3319) 4 days ago [-]

Seems a bit much, but k-anonymity could work here. Hash the domain, take the prefix, get a batch of favicons back. They won't know which you visited, but still get the benefits of consistent favicon support.

bad_user(3400) 4 days ago [-]

And how would they get the 'batch of favicons' in the first place?

smolder(10000) 4 days ago [-]

That's an elegant fix, but for a problem that wouldn't exist if the feature was properly designed. (IOW, designed as a local function.)

tananaev(4332) 4 days ago [-]

Very weak argument for why they do it. Using a service to retrieve a favicon? Surely there's a way to implement the same logic locally.

heavyset_go(4427) 4 days ago [-]

BitWarden does this, too.

tagawa(1919) 4 days ago [-]

We had already had created this anonymous favicon service for our private search engine. In addition, doing it this way avoids another request (and potentially multiple) to the end site.

The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.

ghewgill(3851) 4 days ago [-]

So if favicons were gathered locally, then you would prefer that your own browser would reach out and make multiple requests to every (potentially dodgy) site listed on the search result page?

Note also that these are favicons for results that DDG has already given you. This isn't tracking your clicks. The list of sites that appear on the search result page is not new information to the search engine that just gave them to you.

EDIT: Like other commenters, I was not previously aware that DDG had a browser, and my comments were about this behaviour for the search engine results page.

renewiltord(10000) 4 days ago [-]

Haha, amazing to witness. This is the problem with catering to this crowd: your audience is suddenly full of people who just want to see you fail. Good luck, DDG.

mhaberl(10000) 4 days ago [-]

I don't agree.

I really like DDG - it works good, it is fast and it does not use my personal search for giving me 'better ads' or 'better search results' that put me in a filter bubble.

But there is a different issue here at play; because of errors like this the whole DDG brand gets a bad rap - and thats not only bad because of the risk of people losing a google alternative but because it is real easy to exploit situations like this for google-like companies to give an impression that 'all this privacy thing is bs, all companies work in a same way'. There are a lot of people that are not really sure is this 'privacy thing' is worth the inconvinience of swiching to some other search engine/browser/app and situations like this one are not helpful in that regard.

Lot of folks are aware of this and are displeased for risking brand confidence of such a visible privacy-concerned company for miniscule gains like performance gains for fetching a favicon for the first time - just fetch the favicon after you display the rest of the page and cache it, maybe dont even try to fech it if the connection is poor - who cares really

throwaway_pdp09(10000) 4 days ago [-]

There's an interesting disease showing up here in the responses.

I accept DDG's statement that this is about a favicon and that they 'do not collect or share any personal information', and despite that, I also agree with others that DDG should be on the safe side and just stop doing this small thing. It's just the safer and more moral thing to do (So DDG, as many are suggesting, plz stop doing it. Today is good).

But... the reaction here is 'they made a mistake, let's pile on like kids in a playground' ignoring the genuinely huger issue of the amount of info and mining that google et al. do. There's no measure of proportion in the responses, someone is making a mistake then there's a wolfish, pack-like desire to get stuck in and hurt someone.

Which is why politicians rarely admit mistakes, because it's taken as a sign of weakness, not strength, to admit you were wrong. DDG isn't the big evil on the web but from reading some of these you'd think it was the 2nd google.

This isn't about DDG, just the proportionality of responses in public errors and what society you'd like to have.

(no affiliation to DDG)

classified(3877) 4 days ago [-]

After we have been stung for the umpteenth time patience starts to wear thin.

the_cramer(10000) 4 days ago [-]

Many argue that 'other things are worse'. This argument is invalid in my opinion. We're talking about this special case and if people feel like they need to openly show their opinion about it this is okay. Even if the response is big in numbers.

That just shows how much also the small things matter.

If you only care about 'the biggest' or 'the worst' you'll never get anywhere...

neya(775) 3 days ago [-]

If you claim to fight for privacy and rise to popularity by shaming your competitors' evil anti-privacy (or shady) practices and do something exactly what the rest do, you deserve every bit of criticism. Why should you receive a free pass and the competitors shouldn't? Simply because of your marketing stance??

And for the record, collecting your browser history just to display a stupid favicon is the most ridiculous excuse I've heard in a long while. And I am not going to blindly believe them because they said that's what they use it for.

slightwinder(10000) 4 days ago [-]

> But... the reaction here is 'they made a mistake, let's pile on like kids in a playground'

It's not one mistake, but several. Other then the initial mistake there is also the sloopy reaction and the fact they just closed the issue without bothering to fix it. And this was 1 year(!) ago. Nothing changed in the meanwhile. Now someone pushed it to public and after just some hours they reopen the issue and promise to fix it.

This is the reason why people react loud, because it works. And often it's even the only way that works.

nsomaru(3333) 4 days ago [-]

Google don't sell privacy.

Anyways, what's this got to do with Google? "Privacy browser violates basic privacy to do something useless" is actually ridiculous. And the response is even more ridiculous! How does literally every single other browser do it?

I'm disappointed because I put my reputation on the line to recommend DDG to users based on...privacy. But here we see they actually do not hold true to their stated values. And they don't even seem to care.

dwild(10000) 3 days ago [-]

> But... the reaction here is 'they made a mistake, let's pile on like kids in a playground' ignoring the genuinely huger issue of the amount of info and mining that google et al. do.

What about genocide too? Please people, stop with these 'but the other bigger unrelated issue should get more visibility'.

> Which is why politicians rarely admit mistakes, because it's taken as a sign of weakness, not strength, to admit you were wrong.

Can you point me where DDG admitted they were wrong doing this? They didn't... they just explained why they did it but completely ignore the greater issue because they consider themselves 'good'. Just like that politician you may talk about, or Google, or whatever.

This is about DDG.

dwild(10000) 3 days ago [-]

> But... the reaction here is 'they made a mistake, let's pile on like kids in a playground' ignoring the genuinely huger issue of the amount of info and mining that google et al. do.

What about genocide too? Please people, stop with these 'but the other bigger unrelated issue should get more visibility'.

> Which is why politicians rarely admit mistakes, because it's taken as a sign of weakness, not strength, to admit you were wrong.

Can you point me where DDG admitted they were wrong doing this in their first response? They didn't... they just explained why they did it but completely ignore the greater issue because they consider themselves 'good'. Just like that politician you may talk about, or Google, or whatever. They are part of that bigger issue you mentions.

This is about DDG.

Luckily that pile of kids in a playground made them realize that mistake, they would have ignored otherwise (like they did on their first respond).

fierarul(4367) 4 days ago [-]

You're right mr throwaway, let's be adults here and stop using DDG.

pelliphant(10000) 4 days ago [-]

I disagree, or maybe we read different responses, but the ones I read where more critical of how (a) ddg (employee) handled this.

Didn't see anyone claim that this was on a google-level of bad, more like pointing out that google started out as a small company wanting to 'do no evil', but slowly turned into what it is today.

Is it really that weird that people are worried that this might be the first of many small steps down the slippery slope?

christophilus(3986) 4 days ago [-]

> There's no measure of proportion in the responses

Man. You just described 2020.

Anyway, I'm now using the DDG browser, which until today, I didn't know existed. I think DDG will do the right thing, ultimately.

bartread(4429) 4 days ago [-]

> But... the reaction here is 'they made a mistake, let's pile on like kids in a playground' ignoring the genuinely huger issue of the amount of info and mining that google et al. do. There's no measure of proportion in the responses, someone is making a mistake then there's a wolfish, pack-like desire to get stuck in and hurt someone.

Sadly people simply derive satisfaction from piling on like this. The pop psychology explanation is that it's because people are dissatisfied with their own lives and are lashing out at anything that allows them to vent that underlying frustration. It sounds plausible, but it also sounds like it might be an over-generalisation.

I think it's certainly fair to say - as depressing as it is - that it may be somewhat in our nature to behave in this way, that most or all of us may possess this characteristic to a greater or lesser extent, and that the current political, cultural, economic, and media climate is only serving to amplify that tendency.

warpspin(10000) 4 days ago [-]

I think what angered people was actually that a company saying to hold privacy high was simply refusing to change something after a mistake was pointed out and instead kept on defending it with a technical argument, which makes no sense at all.

The reaction would have been actually a lot different if someone from the company admitted the mistake and promised it will be changed.

Update: Gabriel Weinberg has promised to change it, linking it here so it does not get buried in the pile of comments: https://news.ycombinator.com/item?id=23711597

jsnell(148) 4 days ago [-]

Oh, please. There's a million threads on HN complaining about Google and Facebook. Doesn't even matter if they did something, any post will still have those comments. You can't consider the proportionality of the response by just one thread.

It's really quite amazing that when a company that's hitched it's brand entirely to privacy first commits a big privacy faux pas, hides it for a year, and then doubles down on it not being a problem, you have somehow managed to turn the top voted thread to a discussion on the failings of other companies instead. Bravo.

Frost1x(10000) 3 days ago [-]

My guess is that DDG is being held to a higher standard because it's the liferaft competitor in a world of Google data harvesting, targeted manipulation and selling customer data.

Most complaining or being heavily critical about DDG are probably already upset to the point of abandonment with other services and they don't want the same trend to happen to this competitor (DDG). This sort of reaction is, IMHO, due to poor diversity of viable competitors.

In our societal structure, competitive options are the only things that keep power in check. I'm personally not entirely convinced you can have a reasonable amount of diverse competition in our economic system and there are some inherent equilibria that we tend to converge on over and over again in a market space (without corresponding massive social equalibria shifts).

If you do have any sort of faith left in our economic system, then you certainly want competitors like DDG to be different and be successful. Even if you don't have much faith, outside of say stringent regulation, supporting these sort of competitors is really the only practical option we have in the current state of affairs.

hanspeter(10000) 4 days ago [-]

> ignoring the genuinely huger issue of the amount of info and mining that google

By using DuckDuckGo you're doing the opposite of ignoring privacy issues in Google products.

And hence the reaction. Why use DDG at all, if they're not safely protecting your private data 100%?

marcinzm(10000) 4 days ago [-]

This is concerning because it indicates a lack of care in terms of privacy and understanding that the best privacy is achieved by knowing the least. Does this approach permeate their backend as well?

Stevvo(4366) 4 days ago [-]

A privacy focused browser shouldn't even have a backend.

markholmes(10000) 4 days ago [-]

Does this only occur in DuckDuckGo's Android browser?

coronadisaster(3694) 4 days ago [-]

I don't really trust DuckDuckGo, but I use their search service because I trust Google less... I still trust Firefox more for a browser although it won't take much at this point to make me switch.

stan_rogers(10000) 4 days ago [-]

There's always Startpage. You use Startpage; Startpage uses Google for you, with their own user agent - no history, no tracking.

projomni(4424) 4 days ago [-]

a paranoid android-:)

bad_user(3400) 4 days ago [-]

Google is a big target for law enforcement though. The EU for example can't wait to slap them with another huge fine.

Go here and turn your 'web and app activity', your 'location history' and 'ad personalisation' off, if they aren't already:

https://myaccount.google.com/data-and-personalization

If you do that, there's no evidence that Google is more privacy invading than DuckDuckGo, since you're left with taking their word for it. And frankly I trust a big, bureaucratic company more than I trust a startup. The ideal would be to trust technology (e.g. end to end encryption, open source) but that's not the case here.

Note that I also use DuckDuckGo in my Private Mode, as Firefox allows setting different search engine for Private Mode. I do that because it's better to compartmentalize your online personas, plus I keep my 'web and app activity' in Google on, with deletion after 3 months.

kovac(10000) 4 days ago [-]

There's Qwant, a search engine. I'm using that. It's not as fast as Bing, but it does the job and seem to be trustworthy.

TabbyCatKirk(10000) 4 days ago [-]

Everyone is missing the point here. Let me break this down as simple as I can:

1. End user does a DDG search for 'food' 2. The 'food' query returns a list of search results, these results have each have a link, DDG wants to display the favicon for each link. 3. To be clear, DDG does not store or log the IP address of the user doing the query. They do, however, know what was queried, so they know 'somebody' somewhere searched for 'food'. They have to know this, they are a search engine after all. 4. Since DDG wants to show the favicon 'privately', and they dont want to put that logic/work on the client side (which could leak your IP), so instead DDG finds the favicon internally. 5. A DDG server, completely separate from anything search-related is then tasked with finding the favicon for your 'food' query results, lets say the #1 result is www.allrecipes.com, so a DDG server goes to www.allrecipes.com and finds the exact favicon location. 6. The 'found' favicons are then stored in a cache, and displayed from the cache like this: https://external-content.duckduckgo.com/ip3/www.allrecipes.c... (and if no favicon is found in the local cache, you get a grey arrow by default) 7. I'd like to note, even with all this action, DDG doesn't know if you actually 'visited' www.allrecipes.com, they simply know that some anonymous user did a search for 'food', www.allrecipes.com was a search result, and a favicon was displayed. They dont know who searched for it because the users IP is not stored anywhere, they dont know if you visited www.allrecipes.com, they prevented you from leaking your IP to allrecipes.com since they didn't force the end user to load the favicon.

So whats the issue? What am I missing here?

PS: You know this works because after doing all these searches for food and seeing allrecipes.com (and even clicking allrecipes.com result in the DDG Mobile App or browser extension), guess what? allrecipes.com doesn't follow you around with re-targeting ads! Why? Because DDG prevented that from happening!

tonyaiken(10000) 4 days ago [-]

The issue is with browser, not the search service.

ddevault(327) 4 days ago [-]

This is not duckduckgo.com. This is the DuckDuckGo-branded end-user web browser.

lopmotr(10000) 4 days ago [-]

Nevermind privacy. How are favicons so complicated that they need a special service that understands edge cases. Just do it one standard way and if a minority of websites don't work, then exclude them. We've been through this mess before with all kinds of web standards devolving into mess.

aronpye(4436) 4 days ago [-]

Agreed, creating work arounds for non-standards compliant websites just eliminates the motivation and incentives to be standards compliant in the first place.

fiddlerwoaroof(10000) 4 days ago [-]

The standard way involves a meta tag, right?

CivBase(10000) 4 days ago [-]

This is a bad look for a company that is trying to build its brand on privacy and trust. Even though I don't use the DDG browser I hope they own up to this, rectify it quickly, and learn from it.

dabbernaught420(10000) 4 days ago [-]

In my view, anyone who trusts ddg is a bit silly - founder has a bad track record on user privacy. Founded Names Database[1], a social media website designed to collect user information as aggressively as possible, before selling all the information to classmates.com.

[1]https://en.wikipedia.org/wiki/Names_Database

projomni(4424) 4 days ago [-]

well... i'd much rather fix their deep search so it's as good as google's and not worry about this like this:)

davidhyde(10000) 4 days ago [-]

Ubiquity did the same thing with their routers. They couldn't understand why users had such a problem with their phone home feature that was on by default when the purpose of it was to ultimately "improve" the user experience. I didn't buy their router as a result. I also removed kaspersky from my computer because I didn't like their phone home feature. Turns out they were selling my data despite holding my trust as a security company. DDG, don't turn this into a PR nightmare. We don't trust anyone anymore. Privacy policies are worthless. Nobody cares about favicons anyway.

Source: https://www.theregister.com/2019/11/07/ubiquiti_networks_pho... https://palant.info/2019/08/19/kaspersky-in-the-middle-what-...

pelliphant(10000) 4 days ago [-]

wow, didn't know that, thnx for the heads up.

Time to check what my ubiquity router is up to once I get home from work.

akent(3843) 4 days ago [-]

Do they release the source of the webservice? Seemingly not. This is extremely shady.

Carpetsmoker(4380) 4 days ago [-]

Would it matter? I can release all the source I want, but what guarantees this is also what's running on the actual endpoint?

donedealomg(10000) 4 days ago [-]

the owner of duckduckgo is extremely shady.

he made big money in the past selling user data to the highest bidder. you can google all about it. I never understood why people trusted duckduckgo

Geee(2815) 4 days ago [-]

I guess they collect statistics of the sites that people visit. This is anonymous but valuable information.

jacquesm(45) 4 days ago [-]

Your guesses are both wrong (see upthread) and irrelevant.





Historical Discussions: Reddit shuts down subreddits including r/The_Donald and r/ChapoTrapHouse (June 29, 2020: 794 points)

(794) Reddit shuts down subreddits including r/The_Donald and r/ChapoTrapHouse

794 points 6 days ago by catacombs in 1485th position

www.washingtonpost.com | | comments | anchor

(Clockwise from top left) Stevan Molinar, left, with two squad leaders in 2016; Frank Scafidi prepares for a mission in Vietnam; Lt. Col. Harry Lesher pins a First Lieutenant bar on Tom Knutsen in 1971; Victoria Chamberlin at the NCO academy in South Korea in 2014. (Courtesy of Stevan Molinar, Frank Scafidi, Tom Knutsen, Victoria Chamberlin)

Many said they were moved to tears reading about the experiences of 1st Platoon. Others questioned how the soldiers could blindly follow orders to shoot at civilians.




All Comments: [-] | anchor

blisseyGo(4442) 6 days ago [-]

Spez is lying because The_Donald had been inactive for 4 months when the Reddit admins banned their top mods and they all left and built their own website thedonald.win

So Spez claiming TD has been continuing to post rule breaking content is literally a lie.

asdf21(10000) 6 days ago [-]

If you follow spez at all that shouldn't surprise you.

Hell he directly edited the comment database on the_donald one time

jacobwilliamroy(4356) 6 days ago [-]

This may be the first ban of a sitting U.S. president. Reddit is making history over here and I get to see it. Cool.

c3534l(4345) 6 days ago [-]

They didn't ban Trump, nor did they ban all subreddits about Trump. They banned a single subreddit which repeatedly violates reddit rules and refuses to moderate according the rules.

samfriedman(3895) 6 days ago [-]

The title should be updated, as T_D was just one of a large number of subs banned today; even then, it has been largely irrelevant since the admins' earlier decision to quarantine it and impose other restrictions.

Other larger banned subs include /r/GenderCritical (anti-Trans) and /r/ConsumeProduct which was ostensibly for criticism of consumerism and product promotion, but hid a large strain of antisemitism below the surface in a similar though less-direct fashion to previously banned /r/clownworld (barely-veiled antisemitic and racist cartoons/commentary). Also banned is /r/chapotraphouse, notably the biggest (only?) left-leaning sub on the list. Its users were known to be relatively cantankerous and tended to kick off a lot of brigading on Twitter and such; though some will say its banning was a 'both sides' maneuver from the management.

In my own opinion this was a long time coming, and Reddit has long since shown that the original hands-off model is woefully inadequate in the face of communities that are willing to expend the effort to argue continuously in bad faith, organize to influence and control opinion in other communities, and attack the platform itself in their campaigns for hateful speech. Just ask /r/BlackLadies if you think these users 'stay in their containment areas'. Hopefully Reddit is turning the page to better empower its communities to protect their users and keep hate off the platform.

oska(446) 6 days ago [-]

> /r/GenderCritical (anti-Trans)

This is not a fair portrayal of that sub.

Reddit can do what it likes about banning political subs but the banning of r/gendercritical is quite concerning, in my opinion.

dang(192) 6 days ago [-]

OK, I've updated the title to highlight the both-sides aspect above.

wnevets(10000) 6 days ago [-]

a number of subreddits like r/ConsumeProduct were taken over by the hate mongers from the_donald. You would have post brigaded to the front page that had absolutely nothing to do with consumerism or product promotion

throwaway0xb(10000) 6 days ago [-]

/r/cumtown was Chapo adjacent and also got the axe. Part of the dirtbag left or whatever they go by now.

wyoh(4394) 6 days ago [-]

I liked /r/ConsumeProduct, it was funny to see them critics Reddit blattant consumerism and pornography.

rjbwork(10000) 6 days ago [-]

>In my own opinion this was a long time coming, and Reddit has long since shown that the original hands-off model is woefully inadequate in the face of communities that are willing to expend the effort to argue continuously in bad faith, organize to influence and control opinion in other communities, and attack the platform itself in their campaigns for hateful speech. Just ask /r/BlackLadies if you think these users 'stay in their containment areas'. Hopefully Reddit is turning the page to better empower its communities to protect their users and keep hate off the platform.

All this will do is push these folks out to different platforms where they will organize brigading and bad faith participation in the dark. There are gobs of matrix/discord/irc rooms where people organize the manipulation of social media, and this move just removed visibility of it from reddit.

seibelj(2424) 6 days ago [-]

My perusal of Chapo Trap House was a lot of people un-ironically wishing that gulags came back and certain people were thrown into them. Just because communists did them, doesn't mean you aren't advocating for concentration camps, left-wing or not.

jbob2000(10000) 6 days ago [-]

/r/ConsumeProduct was antisemtic? I never saw that in the posts of theirs that made the front page. It was always memes about over-consumption, they might make fun of disposable make-up wipes for example ('And when you're done, just throw it away!' cue gallery yelling in a robotic voice: CONSUME PRODUCT).

Genuinely curious what the link to antisemitism is. Is the thought that consumerism = big business, and big business = 'the jews'?

Edit: I made this comment early in the thread, before others started to weigh in on why this subreddit was banned. I'm willing to engage in a discussion if you reply instead of downvote.

gtk40(10000) 6 days ago [-]

/r/gendercritical was also banned, an active and well-moderated radical feminist sub.

I am not a radical feminist, nor was I particularly welcomed there as a man, but I found the sub interesting to expand my perspective.

eclat(10000) 2 days ago [-]

Yeah this is odd. I didn't spend much time there as it's not a topic I particularly care about, but from what I saw it was mainly discussion around transgender and women's rights from a perspective critical of TRAs, which is the standard view in the west from my admittedly limited understanding.

asdf21(10000) 6 days ago [-]

You literally can't debate any controversial issue / cultural topic without it being called "hate" now...

abellerose(4437) 6 days ago [-]

What did that subreddit do besides hating on trans people? I never saw anything but countless posts targeting hate against trans people.

softwaredoug(1997) 6 days ago [-]

Both Reddit's and /r/The_Donald speech protected from the outcome. From the article is that that the moderators/participants already have a new place to discuss whatever they want on a reddit clone. Reddit as a platform/company doesn't have to choose to be associated with this hate speech.

Both sides have free speech rights and both have outcomes that support that speech...

arcticbull(3938) 6 days ago [-]

> Both sides have free speech rights and both have outcomes that support that speech...

'Free speech rights' don't apply between two private entities, only between the government and a private entity.

kylebenzle(4200) 6 days ago [-]

Is there a 'real' definition of hate speech? It seems like a hard to define rule. I also hate Trump but am allowed to say any horrible thing I want to about him, but if I were to same the same things about Hilary (I also hated her as a politician) it would be hate speech.

nordsieck(10000) 6 days ago [-]

> Both sides have free speech rights and both have outcomes that support that speech...

Free speech has 2 distinct meanings:

1. The legal right to be free from government interference to most speech.

2. The cultural norm to tolerate speech you disagree with in order to foster political inclusion.

You're correct under meaning 1; you're not correct under meaning 2.

koolba(609) 6 days ago [-]

This final banning of /r/The_Donald is part of a concerted effort to eliminate any sense of normalcy for being pro-Trump. Even with no content for the past few months (after the mod removal), it stood out as a historical reference point to show thousands of people that supported Trump's 2016 campaign and presumably would continue to support him in 2020.

They had a choice to leave it up and locked, but instead chose to erase it from the Internet and that's no accident.

nullc(2193) 6 days ago [-]

Weird. If anything I'd be more prone to believe that The_Donald was some kind of false-flag operation used to make trump supporters look bad. I'm really surprised to see someone say that it made trump support look normal.

dangus(10000) 6 days ago [-]

As a counterpoint to that, there is really nothing "normal" about supporting Trump. He really is just as bad as his critics say.

This is the guy that retweeted a video of someone yelling white power in support. This is the guy who called COVID-19 the "Kung flu." This is the guy that said there were "fine people on both sides" where one side was neo-Nazis. This is the guy that encouraged both police and rally-goers to get violent with the "wrong" type of people.

Trump is deeply, unacceptably racist, and pro-violence, full stop. Trump is on the side of Christian white nationalism and there are plenty of people who voted for him who don't even recognize it.

The fact that your typical upper middle class white suburban family with generally moderate views might have voted for this guy isn't normal. They just can't hear the dog whistles and have been, amazingly, convinced that that the alternative is worse.

andrewla(3242) 6 days ago [-]

Why is the article so cagey about mentioning what the successor is, while mentioning two other sites, Gab and Parler, that are equally controversial? It seems like a basic failure of factual journalism to say '[r/The_Donald's] volunteer moderators and much of the community had left in recent months, moving to a website that mimics Reddit's system of conversation and user voting on content but appears to have fewer rules.' If they're trying to preemptively de-platform thedonald.win, then why mention Gab and Parler?

floatingatoll(3888) 6 days ago [-]

They take perverse glee in being named, because it attracts more bad actors to their cause and strengthens their ability to harm others. By naming their site openly here, you too are now helping them market to HN readers. Why would you do so voluntarily here?

jl2718(10000) 6 days ago [-]

There's no winning this game. Soon they will be considered liable for anything they didn't censor, especially from the extreme left, as they are making the appearance of support. I hope they're paying Michael a massive amount of money to associate himself with this dumpster fire.

ggggtez(10000) 6 days ago [-]

Bizarrely, the reason that there is even a discussion of making them liabile for user speech to begin with is because of prostitution, pedophiles, and terrorism.

I would hardly call those issues 'extreme left'. This change is actually from the conservative view point a lot of the time, especially as it comes to speech about sex. See cases against Backpage or Craigslist and prostitution.

Once a company is criminally liable for user speech, you're going to get this situation where it's easier to just ban speech that even approach the line. But I don't think anyone is going to say that a company should be willfully blind to pedophiles on their platform. So this is the world we get.

chippy(645) 6 days ago [-]

a strange game - the only winning move is not to play.

basch(4449) 6 days ago [-]

Section 230 protects you from liability when you choose to censor.

mFixman(10000) 6 days ago [-]

That's the same slippery-slope argument people user to complain about Reddit banning pedophile subs in 2012.

Extremist subs and their brigading were the biggest treat to free speech and creativity in the site.

fzeroracer(4322) 6 days ago [-]

Trump literally signed an executive order which was threatening social media companies to take on liability from anything their users posted. When that happens said companies will simply outright ban the subs that are causing problems, which are usually the far-right subs because they insist on making threats of violence and racism.

The 'extreme left' has little to no political power compared to the president telling companies to either obey him or risk having their site go down in flames.

catsarebetter(10000) 6 days ago [-]

Never checked it out. Don't regret it

doopy-loopy2(10000) 6 days ago [-]

lol, the left in a nutshell.

MINIMAN10000(10000) 6 days ago [-]

I did and I was left satisfied.

I wanted to see their viewpoint on at the time 'the hot topic' and confirm my suspicions that I truly don't consider their viewpoint rational or valid. It's like they start from a conclusion and work backwards to excuse it.

josefresco(4198) 6 days ago [-]

I visited, shared my viewpoint and was promptly banned. Good for you to stay away (for your mental health) but seeing first-hand the sort of hate and bigotry posted daily was both shocking and informative.

TrackerFF(10000) 6 days ago [-]

The problem with T_D was that it mainly functioned as an echo chamber. And I don't mean that as a liberal person just disagreeing with their politics - they'd literally ban and delete _anything_ even slightly critical of Trump.

When you cross that point, it just becomes another channel for propaganda. No different than what you'd see in actual dictatorships.

But that kind of culture / behavior is not unique to T_D, every time someone posted anything remotely critical of Bernie on /politics, they'd get downvoted at the speed of light. Same with pro-Biden threads before Sanders threw in the towel.

But T_D is probably the worst offender. It just functions as pro-Trump worship, and 'owning the libs' memes. Everything else will get you banned.

tzs(3100) 6 days ago [-]

Checked it out, and promptly got banned. There was some discussion going on about how the liberal press criticized Trump for wanting to criminalize flag burning but didn't criticize Hillary Clinton for the same thing even though she had actually cosponsored legislation to do that when she was in Congress.

I had never actually heard of that, so looked up the bill to see what it proposed. I posted the summary from congress.gov in case anyone else was curious too:

> Flag Protection Act of 2005 - Amends the federal criminal code to revise provisions regarding desecration of the flag to prohibit: (1) destroying or damaging a U.S. flag with the primary purpose and intent to incite or produce imminent violence or a breach of the peace; (2) intentionally threatening or intimidating any person, or group of persons, by burning a U.S. flag; or (3) stealing or knowingly converting the use of a U.S. flag belonging to the United States, or belonging to another person on U.S. lands, and intentionally destroying or damaging that flag

Banned for posting anti-Trump material.

zozbot234(10000) 6 days ago [-]

Press F to Pay Respects.

BTW, the unofficial successor (as listed on the subreddit itself before the ban) is alive and well at thedonald.win

toomuchtodo(1035) 6 days ago [-]

Appears to be fronted by Cloudflare (as of 1593450845). Curious to see how long that lasts in the current tumultuous environment.

augustt(4420) 6 days ago [-]

Let's see how they're doing... highest post, stickied:

> title: you are far more likely to die from shootings in nyc than the 'rona

> headline 1: The New Normal in NYC? 11 People Shot In Under 12 Hours.

> headline 2: New York Reports Just 8 Deaths As Cuomo ...

Create however many platforms you want, can't fix weapons-grade stupidity.

danso(4) 6 days ago [-]

So Voat never took off?

extr(10000) 6 days ago [-]

RIP /r/cumtown

sketchyj(10000) 6 days ago [-]

o7

ve55(3115) 6 days ago [-]

Worth taking note that in today's climate, you really cannot win when you are in a position to moderate important things that a lot of people use.

There is constantly tremendous pressure on you to perform opposing actions, and even making no decision at all will cause you significant mental stress and harassment, regardles of what the issue is.

You have to pick who you want to cave to, and to what extent, and no matter how good of a job you try to do, a lot of people will really really hate you.

This is even more apparent when you see that Reddit has been taking action against a lot more subreddits recently, some of which are listed in the article, and many of which clearly have little to do with the president. The attitude of some of these communities may be abhorrent, but they are still communities, and people do not react well to their communities being deleted, whether a company had the legal authority to, or was justified in doing so, or not.

It's very tough and I wish that we didn't have to go through these things to begin with, and could have more federated and decentralized platforms, or at least more client-side filtering inside of centralization curation. I can always dream.

dustingetz(2752) 6 days ago [-]

the only reason they are centralized at all is to make a profit, so it's the profit motive which is causing them to hoover up and own all this content and thereby being vulnerable to the risk

amadeuspagel(4387) 6 days ago [-]

The fundamental problem here is that people discover things that make them angry. Platforms need to figure out how to prevent that, how to let open-minded people discover others who think differently, while keeping crybullies in a comfortable bubble.

For reddit, part of the problem is that everyone who isn't logged in sees the same frontpage, which means they see posts from these controversial subreddits and get angry. They should make it possible to subscribe to or block subreddits without making an account.

wefarrell(10000) 6 days ago [-]

Social media companies should not be in this position where they are the arbiter of what is offensive/explicit/hate speech vs acceptable free speech. Despite what they state, they will ALWAYS make the decision that generates more profits or power.

There really needs to be effective legislation given the importance of free speech for democracy. Unfortunately government agencies are so politically charged that I trust them with enforcement even less than social media companies.

duxup(4127) 6 days ago [-]

Would something decentralized help?

I don't think people want that. The_Donald involved a lot of spreading their message / fake news sites / spamming across other subs and etc.

Do people want to be on a platform where other folks on the same platform are targeting their community with dishonest and often bigoted content?

I'm also not sure how much The_Donald was as a community. The sheer volume of users at its height seemed to involve a huge amount of brand new accounts you never saw again / never posted again (except when they show up as a group again), and etc. Or those without brand new accounts show up and post strange dishonest lead in type posts that sort of try to lead folks down a bigoted path ... that you can blatantly see in their post history.

So you'd be in your other community and one day they all show up and down votes and the vitriol begins... it was no mystery that this happened, it was spoken of openly in The_Donald for a time.

Who wants to deal with that?

CM30(2264) 6 days ago [-]

The best solution is not to cave to anyone. It's to come out, say what you think is best, then tell anyone who disagrees to either take it or leave the platform.

If enough companies, groups and individuals do that, this sort of pressure will stop, since people will learn it doesn't work. Most people won't stop using a platform over this stuff anyway.

tbabb(3915) 6 days ago [-]

> You have to pick who you want to cave to

Why does the narrative have to be about 'caving' to someone instead of weighing principles against each other and taking the action which protects what's most important?

untangle(3864) 6 days ago [-]

> '...I wish that we...could have more federated and decentralized platforms'

We did. They were called newsgroups and forums. Reddit is just 1000+ forums stuck together to build audience and make money.

matchagaucho(10000) 6 days ago [-]

> a lot of people will really really hate you.

That's an anticipated cost of removing hate.

dman(3781) 6 days ago [-]

I think a lack of geographical diversity is coming back to bite tech companies. For better or for worse conservatives believe they are not fairly treated by tech companies. The fact that most US tech companies are on the coasts which happen to be fairly liberal further feeds into this us-vs-them tribal arguments. Curbing divisiveness and helping bring people together has never been more important, I hope we find a way out of the current quagmire collectively as a society.

jakobdabo(1358) 6 days ago [-]

So, why moderate a public forum?

Why should platform owners decide what's right and what's wrong? If something is legal then why should we censor it just because it may seem hateful to someone, or somebody feels offended? So what if a post offends somebody? They can deal with it like an adult. Ignore the trolls, and discuss if there is a discussion, offending or not.

If people want a safe private place with no intruders, then they can create a closed/invite-only group and moderate it themselves.

raxxorrax(10000) 6 days ago [-]

If they had not caved to calls for banning, they would have less problems. The ambitions to ban more subs won't subside anytime soon.

Of course this hands off approach would get you hate. That is price you have to pay. Worth it in my opinion.

nullc(2193) 6 days ago [-]

Not just hate you, but stalk, threaten and harass you.

There are some people where even just saying-- as a moderator-- 'Hey, can you chill out and at lease pretend to treat other posters with some respect' will trigger a full on war against you.

There aren't many people like this, but it only takes a few because even a single obsessed person can spew a lot of hate.

enraged_camel(1455) 6 days ago [-]

>> I wish that we didn't have to go through these things to begin with, and could have more federated and decentralized platforms

Decentralization simply leads to the creation of echo-chambers, except with no one to keep the toxicity in check.

lenkite(10000) 4 days ago [-]

The_Donald was censored and taken over by reddit moderators for 'threats made against police'. It was laughable considering the extraordinarily explicit explicit threats made against the police in other more popular reddit forums. Equal rules were not applied.

If reddit was consistent in their application of rules, it would be nice. But we all know that is not the case. Some are more equal than others.

nordsieck(10000) 6 days ago [-]

> Worth taking note that in today's climate, you really cannot win when you are in a position to moderate important things that a lot of people use.

While this is probably true in most cases, given the Reddit CEO's history with the_donald[1], this doesn't seem like it applies in this particular case.

___

1. https://techcrunch.com/2016/11/23/reddit-huffman-trump/

kylebenzle(4200) 6 days ago [-]

Users won't allow for (use) decentralized platforms until there is a good way to censor them. Once that happens, what would be the point of switching.

Facebook and Reddit are successful BECAUSE they are censorship machines. Facebook spends far more man hours on 'curating' content compared to engineering.

Same with Reddit. Add up all the 'moderation' time spent removing negative or controversial posts and I bet much more time is spent on content compared to programming the site.

You are not wrong, but even if a perfect decentralized clone existed tomorrow you'd only get young men and programmers to use it, the average TicTok users wants to see beautiful people doing stupid things, they don't care how it happens.

barnesto(10000) 6 days ago [-]

You can win if you apply rules equally, but you have to be willing to stand up for your platform and not cave to a small group of people.

r/The_Donald - they shut down r/Fuckthealtright - open for business

And by open for business, I mean openly calling for killing conservatives.

rvz(3476) 6 days ago [-]

The whole cancelling of the opposition is the reason why those who do it, become their own echo-chamber and will continue to find every. single. thought-criminal. and suffocate them in livelihood, privacy and financial healthiness.

Twitter has been the ground digital war of the liberal-left and (some of the far-left) over the conservative right and (some of the actual far-right) and the right-wingers retreated to their own chambers with the former have Twitter under its control. As soon as 'the right' we're told to 'create their own platform' which they have done many times, cancellation was always the goal from those from the left.

Having echo-chambers creates one-sided discussions but unfortunately they are left with no choice given that cancellation is inevitable where-ever they go. If both sides are yelling, screeching and howling at each other they're both not listening to one another which is nothing like a reasonable debate.

dgzl(4407) 6 days ago [-]

Almost like being a police officer?

tanilama(10000) 6 days ago [-]

> could have more federated and decentralized platforms

I would absolutely not going to a platform with no moderation. The quality of the content will be abysmal.

Honestly, r/D or something like that should just go to host its own website/forum, it shouldn't be Reddit's problem to begin with. Their presence brings toxic attention/traffic that is hard to monetize anyway

hhhhhhhjjjjjjju(10000) 6 days ago [-]

I run a video game clan with a fairly active discord channel -- the stance I've taken is: there are no rules beyond:

1. nothing illegal (includes cheating at games) 2. no racism / homophobia / transphobia 3. no harassing other members 4. everyone has different beliefs and come from different parts of the world -- if you are offended, ask the offender to stop, otherwise see a mod

so far there are around 50 people in the chat and almost no issues even though we have members who are rep/dem/other, religions ranging from Christian to Satanist, and we regularly post political and religious memes and such.

I don't know how but thus far we've had no real issues.

SI_Rob(10000) 6 days ago [-]

> we [..] could have more federated and decentralized platforms, or at least more client-side filtering inside of centralization curation

We should start a center to discuss and curate these ideas, a place where we could get together and define what it means to achieve decentralization.

We could call it something snappy and semantic like, say, the Center for Decentralization.

Svperstar(10000) 6 days ago [-]

As the head mod of a subreddit this really resonates with me.

SmokeyHamster(10000) 6 days ago [-]

>Worth taking note that in today's climate, you really cannot win when you are in a position to moderate important things that a lot of people use.

You kind of can, though. Sure, group X hates group Y and wants the admins to ban them. Group Y hates group X and wants the admins to ban them. The smart play would have been to be neutral and set out an explicit set of rules everyone has to follow.

Reddit instead did it the worst possible way imaginable. They laid out no clear rules, banned thousands of subs for vague 'hate' reasons that you can't define, much less verify, and drastically reduced the scale and appeal of their site.

If you think group X will now be happy and stop complaining now that group Y has been banned, I think that's naive. The admins will still be pressured, but the goalposts will just be moved to ever more extremes until Reddit's such a niche echo-chamber that it appeals to too few people to remain financially viable.

seph-reed(4432) 6 days ago [-]

There is a right way to do this which minimizes the issues, but it's counter-intuitive: reflect the negative aspects of The_Donald.

It's called 'closed loop hypocrisy' and the theory is that all negativity or opposition is inherently hypocritical (given we are all very similar in the end). But being hypocritical does not mean opposition is bad, if anything the issue is long loop hypocrisy where someone pretends to be righteous when really it's just hypocrisy with extra steps.

Most people solve problems with long loop hypocrisy and those who support them can easily choose to overlook the hypocrisy by focusing on the convoluted steps and rhetoric in between. Meanwhile, it's obvious to those being attacked just how hypocritical they are.

In contrast, closed loop hypocrisy is about applying the form of negativity to itself. It seems kind of dumb, but that's the point: to show how dumb X form of negativity is directly to X form of negativity.

It's a form of inception and it's hard to pull off, but there is a better way.

ipython(10000) 6 days ago [-]

It's funny, what I see is not politics but simple market effects. The truth is that most people value some level of moderation. Surprise! A cesspool of trolls does not provide for an inviting community.

I find it hilarious that we have congresspeople whining about being "censored" on the very platform they claim is censoring them. I would think the first step of a competent censor would be to censor complaints of censorship itself.

smolder(10000) 6 days ago [-]

This resonates with my experience moderating a local facebook group.

I get attacked and threatened by racist instigators and BLM supporters alike because people feel that any moderation against them is solidarity with the other side. I point to the group rules violated and they still assume bias on my part. I AM biased (in favor of BLM) but still get treated like crap when I'm doing my best to moderate fairly, for free, in my spare time, for the benefit of the people treating me like crap.

Joeri(4389) 6 days ago [-]

If you make enemies regardless of what position you take, you are free to take your own position without guilt or additional consequence. Moderators should take positions based on what they personally believe moderation should be (or those deciding the rules for moderators should do so). Just because their position may align with the interests of a particular group does not mean they are caving to that group.

champagneben(10000) 6 days ago [-]

If black people (or perhaps more specifically, subreddits focused on issues pertinent to black communities) were banned on reddit, but they had a much smaller website where they could discuss issues, would it be an outcome that supported their free speech?

cultus(10000) 6 days ago [-]

You are drawing an equivalence of discrimination against those with dark skin to discrimination against the profoundly immoral and hateful. You are a racist, and are not engaging in good faith arguments.

Jweb_Guru(10000) 6 days ago [-]

Trump is the president of the United States, Republicans have both the Senate and the Supreme Court, and I can barely go two seconds without hearing the perspective of people who voted for Trump. There is an entire news network that acts in an almost official capacity as the Republican party press arm. What on earth makes you think Reddit is the only major place that people could discuss 'issues'?

throwaway010120(10000) 6 days ago [-]

Good job galvanizing Trump supporters. I'm sure silencing them online will silence them at the voting booth.

CyberDildonics(10000) 6 days ago [-]

bots can't vote

Press2forEN(10000) 6 days ago [-]

Since the 2016 election the left has been consolidating their vast cultural power by exiling opposing views that don't pass their ever-shifting purity tests.

It remains to be seen whether or not this will bury conservative thought or cause an underground resurgence.

ipnon(4405) 6 days ago [-]

Social media companies and 'the left' are not the same. Although they surely have some overlap (just as social media companies and 'the right' surely have some overlap), conflating them entirely with each other is simply false. We cannot assign political conspiracies to entire industries.

Miner49er(4448) 6 days ago [-]

The largest lefist subreddit was banned today. The T_D ban was purely symbolic, it was already dead.

This isn't 'the left' this is liberalism.

newguy1234(10000) 6 days ago [-]

I don't know how many will see the comment but I was a member of the_donald sub and subsequently migrated to the new thedonald.win

There is a significant shift going on that people aren't realizing. The days of the 'platform' being a monopoly over users is coming to an end. The cost of running a website, even a bandwidth heavy website is going lower. As major social networks and platforms crack down on controversial content, these communities don't vanish. They simply move to some other place. In the conservative community this idea of 'censorship' is increasing to the point where it is becoming a serious movement. Many alternatives to mainstream media are actually gaining a decent userbase. Here is what is going on that I can see:

Twitter is replaced by parler

Youtube is replaced by bitchute

Reddit subs are replaced by thedonald.win (you can get your sub hosted by them now)

Twitch is replaced by dlive

Just to show you the growth of some of these platforms. Just look at the growth of thedonald.win. Apparently it is ranked at something like 1,250 for the USA by Alexa. That's staggering considering this website is only a few years old.

If you are a web entrepreneur, take note because there is a growing market for 'conservative' or 'censorship-free' social media now.

joshschreuder(2644) 6 days ago [-]

It's ranked 8,648 from what I can tell

https://www.alexa.com/siteinfo/thedonald.win

I have never heard of the other sites you mention, so I can't speak to their popularity. But as far as I can see this all seems very fringe for lack of a better word. There doesn't really seem to be a groundswell of movement towards alternate platforms because of censorship.

parliament32(4442) 6 days ago [-]

Whether you agree with the move or not, it's funny how Reddit's stance has changed over time. Under the old ownership, 2012:

'At reddit we care deeply about not imposing ours or anyone elses' opinions on how people use the reddit platform. We are adamant about not limiting the ability to use the reddit platform even when we do not ourselves agree with or condone a specific use. We have very few rules here on reddit; no spamming, no cheating, no personal info, nothing illegal, and no interfering the site's functions.'

BoorishBears(4424) 6 days ago [-]

The Donald would have been deleted under the old leadership too...

They got in hot water not for supporting Trump (there are plenty of pro-Trump subreddits) but actively breaking the most basic rules of Reddit, maybe you should actually look at their history.

Spamming the same images to the subreddit, interfering with other subreddits, trying to out whistleblowers in federal cases, gaming the content discovery system with sticky posts.

I'm tired of this narrative that The Donald was banned for censorship reasons, they were banned for being a virus that was infecting Reddit.

They could have been supporting the Dalai Lama and their behavior would have been against the basic expectations of Reddit, and that's exactly why they were quarantined/banned

-

https://en.wikipedia.org/wiki/R/The_Donald#History

In true fashion of people trying to push a made up point, every reply is nitpicking over which one of their many incredible misbehaviors was the deal breaker, from doxxing people to brigading other subreddits, instead of actually providing a rebuttal

It doesn't matter which one broke the camels back, anyone who can read their history of behavior and pretend Reddit didn't have a reason to take action, even by those 2012 rules, is in denial.

foogazi(10000) 6 days ago [-]

> and no interfering the site's functions

Spamming the r/all homepage due to eliminating downvotes certainly interferes with the sites functions

pjc50(1356) 6 days ago [-]

That was effectively a challenge: how bad can you be without explicitly violating the rules?

(I call this 'I'm not touching you Fascism', after the popular children's pastime. The goal is to enrage people without ever crossing anything that might be considered a clear line. Most systems evolve a meta-rule somewhere that 'pushing the boundaries of the rules is itself against the rules'. At the very least 'ban evasion' is itself bannable.)

theLastVoice(10000) 6 days ago [-]

Thoughtcrime and crimethink is 'illegal' now.

/s -> or is it...

slantaclaus(10000) 5 days ago [-]

Reddit was a smaller community back then, and just like the article implies, smaller communities are easier to manage and of generally higher quality overall

NelsonMinar(1198) 6 days ago [-]

Makes sense to me; the Reddit folks have learned from their experience. Maximalist free speech policies have failed all over the Internet, the only places that adhere to them are Nazi cesspools like Gab.

ironman1478(10000) 6 days ago [-]

I don't really expect people's opinions/beliefs to stay constant when presented with new facts. People are horrible on the internet and there needs to be moderation. It's evidenced time and time again. The consequences of not doing this weren't obvious then (maybe, I wasn't really on reddit then), but they are obvious now.

chaostheory(256) 6 days ago [-]

Living under this rule would enable reddit to survive and adapt far longer in a democratic republic. This philosophy and the sub-reddits were reddit's strength. If anyone didn't like what reddit currently had, anyone could form their own sub-reddit. That's over. I can't put all the blame on reddit leadership when SESTA and FOSTA exist as laws.

The bad content isn't going to go away. It's just going somewhere else. imo this is the start of a diaspora The only question is where will everyone go?

fourstar(3894) 6 days ago [-]

AKA: Please come over to our platform and contribute content so we can profit. Once we have profited, we will make sure to virtue signal on the highest level.

I created /r/lifeprotips and got shadowbanned for promoting a website I built around it after fostering the community for YEARS/spending my own money to give 'gold' to reward community members. reddit is a scam.

minimaxir(74) 6 days ago [-]

The internet (and Reddit itself) was rather different in 2012.

kmfrk(442) 6 days ago [-]

Perhaps they are consistent in only acting based on convenience, and they might finally have reached the inflection point where doing all this was the easiest, laziest thing they could do.

komali2(10000) 6 days ago [-]

And then the site exploded, and it had to learn how to handle things like actual nazis calling for the deaths of jewish people.

I'm not sure I see where the joke is - this was inevitable.

dkdk8283(10000) 6 days ago [-]

That's the Reddit and Youtube we need again.

blisseyGo(4442) 6 days ago [-]

From Reddit:

https://www.reddithelp.com/en/categories/rules-reporting/acc...

> Marginalized or vulnerable groups include, but are not limited to, groups based on their actual and perceived race, color, religion, national origin, ethnicity, immigration status, gender, gender identity, sexual orientation, pregnancy, or disability. These include victims of a major violent event and their families.

> While the rule on hate protects such groups, it does not protect all groups or all forms of identity. For example, the rule does not protect groups of people who are in the majority or who promote such attacks of hate.

The majority based on what? An individual state? The US? The west? The world? Men are the minority in many countries but the majority world wide. White people are the majority in the west but a minority world wide.

Does that mean people can attack white people with impunity, even though they're a global minority? Can I crap on women to my hearts content because they are a majority in the UK? Can people in California shit all over Hispanics because they're the majority in that State?

Will they assess a users state/country/continent of origin before deciding whether or not they're being hateful towards a specific group?

We are watching Reddit die. Aaron Swartz, the cofounder of Reddit warned about this a decade ago.

SmokeyHamster(10000) 6 days ago [-]

>the rule does not protect groups of people who are in the majority

Wow. Like, holy shit wow. So racism and hate is ok with Reddit but only as long as it's directed at 'the majority'?

What the hell is the 'majority'? I'm white. Yet I live in a city where blacks are the majority. Am I still the majority, meaning it's ok to harass me and send me hateful racist insults on Reddit, or would Reddit grant me honorary minority status? This is legit sickening.

mitchdoogle(10000) 4 days ago [-]

Is there another link? I just checked that page and did not see anything such as the line about 'the rule does not protect groups of people who are in the majority'

I supposed they might have changed it, which would be good to hear, since that wording is obviously misguided

ppf(10000) 6 days ago [-]

Reddit just boarded the one-way express to crazy town. How can that policy possibly be enforced? Even if they do decide on some workable definition of what 'majority groups' are, and how to prove which group you are in, I can't figure out what 'actual and perceived race' means. So if someone makes an offensive comment to me because they think I am of a certain race, but it turns out I am not, it's still racism?

haunter(2273) 6 days ago [-]

It means /r/sino won't be banned which is a CPC propaganda sub

tibiapejagala(10000) 6 days ago [-]

Well, the world should just accept morals of the tech workers in California.

thinkingemote(3591) 6 days ago [-]

This really is the key point. Posts which on Twitter or Facebook would be deemed as hate are allowed now on Reddit. They were not before.

Indeed in most EU countries, Reddit steps over the line into hate speech by allowing it against certain groups. There is an important caveat as it still would not allow violence, harassment or bullying.

drocer88(2386) 6 days ago [-]

20th Century Dystopian novels are now operations manuals.

ipnon(4405) 6 days ago [-]

The difference between 1984 and 2020 is censorship is committed by corporations instead of governments. Does this difference matter? All Americans have to participate in government, whether directly by voting and holding office or indirectly by simply living in America's society of laws and its enforcement. Using Reddit directly is done voluntarily.

daveevad(10000) 6 days ago [-]

> as Internet pioneer John Gilmore puts it, 'The Net interprets censorship as damage and routes around it.'

snowwrestler(4329) 6 days ago [-]

I don't see it. Usually when the dystopia is overthrown, the survivors find themselves with the freedom to build what they want. Well, that's already the situation today. No one needs Reddit's permission to build what they want.

skrowl(4293) 6 days ago [-]

It's very sad how many leftist are cheering on the censorship.

Forcible silencing of opposing views is a central tenant of actual fascism (not the kind of 'fascism' that kids on social media accuse people of just for disagreeing with them).

History repeats itself, it would seem. This is a modern day book burning.

madars(2812) 6 days ago [-]

Full list of subreddits banned today: https://www.redditstatic.com/banned-subreddits-june-2020.txt (with 'blurring' after top 10)

donatj(3683) 6 days ago [-]

Why the blurring?

blisseyGo(4442) 6 days ago [-]

3 of some of the biggest right wing YouTube channels, Trump's Twitch account, and 2000 other subreddits including The_Donald all being banned in the span of less than an hour is just a coincidence. More of a criminal conspiracy.

blisseyGo(4442) 6 days ago [-]

Twitter also banned Sidney Powell, the lawyer of 3 star general Michael Flynn.

hartator(3848) 6 days ago [-]

To be fair, The_Donald was already shutdown for a while. Between quarantine that makes it not show up in feeds and the moderation team being replaced by Reddit's, it wasn't up and running anymore.

minimaxir(74) 6 days ago [-]

A de facto shutdown is not the same as an actual shutdown.

shiado(10000) 6 days ago [-]

Reddit really had a good run as an interesting site, but it's really angling for a FB style consumer ad experience now. I think the final blow will be when they go after porn like tumblr did, then it will die forever. Too bad the alternatives are pretty extremist. I think Dread is really the best one because people who want to buy drugs on the internet aren't there completely for the politics.

WorkingDead(10000) 6 days ago [-]

Hobby specific communities were a big attraction point to reddit, but even those have devolved into heavy censorship. There isnt really a good draw to the site anymore.

fortran77(342) 6 days ago [-]

I tried some of the alternatives. I figured if the people I don't like stay in their subgroups, I'm fine. But they don't. They invade every group. I don't want to see extremely racist posts in my 'anti-bicyclist' group (for example). They'll take every post and make it about race.

nullc(2193) 6 days ago [-]

I'm confused by the article. My understanding was that reddit replaced the moderators of /r/The_Donald months ago after initially making in quarantined.

The Wikipedia article seems to agree with my understanding:

> In February 26, 2020, Reddit administrators removed a number of r/The_Donald moderators 'that were approving, stickying, and generally supporting content in this subreddit that breaks [Reddit's] content policy' and called the remaining moderators to choose new ones from a list of Reddit-approved individuals.[77] About the same time, Reddit placed r/The_Donald in 'Restricted mode', removing the ability to create new posts from most of its users. Since then, the subreddit's community has moved to thedonald.win, an independently hosted site based on Reddit's old user interface.

Which all sounds fine to me, but the article's quote 'We're not the ones who shut down the community. The moderators are the ones who shut down that community.' seems disingenuous to me.

What's wrong with just saying that it's their platform and they have the right to set the rules?

john-shaffer(10000) 6 days ago [-]

> What's wrong with just saying that it's their platform and they have the right to set the rules?

1) They don't want to alienate people who care about free speech. 2) A lot of people liked reddit because it was user-driven, and don't like it when the content they see is selected by admins and mods. They want to keep the illusion of being user-driven as long as they can.

intopieces(4445) 6 days ago [-]

> What's wrong with just saying that it's their platform and they have the right to set the rules?

That would be a major shift in strategy for the site. It has always been billed a freewheeling place that belonged to the users, with the company that owns it being something of a benevolent overlord that keeps the lights on. That is what attracts users and content. But of course that version of the web is dead. Any website that relies on advertisers for revenue will necessarily be limited in its tolerance for controversy.

In the end, though, I doubt it matters too much for the site's revenue. Reddit has been dying for years according to many of the posters and yet it sees growth YoY.

belltaco(1479) 6 days ago [-]

>About the same time, Reddit placed r/The_Donald in 'Restricted mode', removing the ability to create new posts from most of its users.

I think this is wrong, IIRC it was the mods who chose to do that, not Reddit admins. The mods wouldn't choose other mods even though the rules made sense, like having a certain karma threshold in t_d etc.

SmokeyHamster(10000) 6 days ago [-]

>What's wrong with just saying that it's their platform and they have the right to set the rules?

Because the admins still need people to believe they're the 'good guys'. And we all know the good guys don't censor people. That's what evil fascist governments do.

I would routinely visit the_donald. It was definitely a ruckus pro-Trump circle jerk, but it was hardly racist or abusive. It was mostly just silly pro-Trump memes. There's far more black Trump supporters than most people realize. The media does everything they can to hide them, so a lot of people would love to post photos of them with funny captions like 'Oh look, another white supremacist Trump supporter!'

The admins couldn't have that, but they couldn't outright ban them without appearing draconian. So they instead tried just banning most of the mods, hoping the remaining mod would agree to the admin's terms of appointing only admin-approved puppet mods. Of course, the mod balked at that and just locked the sub and forwarded everyone to thedonald.win, the backup site.

So the admins ended up with the worst of both worlds. They still looked like the bad guys who shut down a sub on a whim, but now there's still a huge trove of funny non-racist pro-Trump memes sitting on their site for any Google searcher to stumble upon, which will only reinforce just how draconian the admins have become. By banning the sub outright, all that content gets deleted and hidden from search engines, so now when they vaguely label it 'hate', you can't refute it.

Orwell nailed it. We've always been at war with ~~Eastasia~~ Eurasia.

Miner49er(4448) 6 days ago [-]

Yeah, the ban of T_D is only symbolic; it has been dead for months. I think the real news here is the ban of /r/ChapoTrapHouse, (arguably) the most active leftist subreddit there was.

Rebles(10000) 6 days ago [-]

Since the donald moderators removed the new moderators that Reddit added (that was selected from and voting on in the donald subreddit), it was clear the moderators weren't interested in rescuing their subreddit to Reddit's standards.

I think Reddit is trying to avoid appearing capricious when it banned subreddits. Reddit users do want a rythme or reason to Reddit admins actions, otherwise, there'd be further user discontent.

bmarquez(10000) 6 days ago [-]

You're right. The_Donald has been dead for months due to the above-mentioned restrictions on submissions, moderator removal, and moving to a different site.

If they're getting banned now it's for something that was done months ago.

teej(2406) 6 days ago [-]

Here is what Reddit said directly, vs through a media outlet - https://www.reddit.com/r/announcements/comments/hi3oht/updat...

el_cujo(10000) 6 days ago [-]

The_Donald was neutered a long time ago. The real story is the 200 other subs getting shut down and Reddit essentially saying the wild west days on their site are actually over now. Several publications have focused on The_Donald at least in the headline because I assume it makes for a sexier, more political story than about a website cracking down on moderation.

anonu(2353) 6 days ago [-]

The internet remains a bastion of free speech. Reddit, Facebook, Twitter, remain privately managed companies with a duty to police their platforms. If /r/The_Donald contributors are upset - they can go almost anywhere else. Vitriol, hate, disinformation, etc... just doesn't need to be on public display.

TheGrim-888(10000) 6 days ago [-]

The problem is that what hate is, is in the eye of the beholder. People can say Trump wanting to enforce rule of law is hate, I don't think so. People can say Trump wanting to enforce our borders is hate, and I don't think so. You have politically motivated people who think that anything that disagrees with their political viewpoints is hate, and justification to be memoryholed from the internet. I don't agree.

txcwpalpha(4437) 6 days ago [-]

About time. However, this move was mostly symbolic. r/the_donald has been inactive for months without a single new post on it. AFAIK the community had already moved on to another site off reddit.

Much more significant IMO is that the ban also includes r/ChapoTrapHouse, which is a 'left'-leaning subreddit accused of much of the same stuff r/the_donald was.

GoodJokes(10000) 6 days ago [-]

what was chapo accused of? The original quarantine was due to the sub posting about John Brown (killer of slave holders). Reddit didn't like that. Slave holders are people too.

pozdnyshev(10000) 6 days ago [-]

r/Chapotraphouse was literally the largest online aggregate and forum for leftists, nothing else came close.

robotron(10000) 6 days ago [-]

I haven't seen this mentioned in any stories but r/antifa is also banned. It could have come about as something else but I took a peak in it Thursday and it was there.

phjesusthatguy3(10000) 6 days ago [-]

Whatever else you want to want to say about our current president, 'The People' did not vote him in.

The fact that he's President really is going back to the founding fathers' intention, and why we have an Electoral College in the first place:

So that underrepresented voters would have a say in who the President was. And they did, so he is. And now he's pretending that he's the voice of our country, when he so clearly is not. The People spoke, and they requested someone else.

A reasonable person would look at his election results and create an agenda from there, but not Trump. I am hopeful that is his downfall. I'm not holding my breath about it, though.

afiori(3538) 6 days ago [-]

But there is not 'one true way' to coordinate a democratic election. Especially when considering the differences between presidential republics and parliamentary republics.

Or even concepts like 'one person one vote' force systems to converge towards a few stable establishment parties.

It would still be democratic if voter could grade candidates and the winners were chosen by some appropriate statistical measure, or if voters could select arbitrary subsets of okeyish candidates and the winners were those with most votes, or any combination of many other systems.

The real objective of most democratic electoral systems is governability and meaningful representation. Pretending that popular vote is the only meaningful way to elect representatives is naive.

jeremiahhs(10000) 6 days ago [-]

The article is incorrect.

'racism, misogyny, anti-Semitism, glorification of violence and conspiracy theories that flourished there.'

That can be seen in all political subreddits - even the favored /r/politics.

Those forms of hate were not condoned in T_D.

But if washpo wants to rewrite history, they can try. T_D was banned because reddit hates the policies Trump was elected to enact.

psychometry(10000) 6 days ago [-]

There's a difference between hate speech occasionally being found in a subreddit and a subreddit existing principally as a platform for hate speech. the_donald was unambiguously the latter.

chris_wot(1512) 6 days ago [-]

Reddit shut down the subreddit because of repeated violations of their usage policy.

It is interested to me that violations of their reasonable policy correlate with the direct actions of Donald Trump himself.

runawaybottle(4409) 6 days ago [-]

Without getting into the weeds, I just don't see why we accepted that the internet should be one giant message board. We seem to do this over and over with stuff like Reddit and Facebook, but once upon a time communities would be their own site with their own message board.

buboard(3142) 6 days ago [-]

Because it is and always has been , from the start. Without this liberation of having a public , anonymous board, we 'd be so much poorer culturally. Reddit banning some stuff is a good thing - new venues will arise because people arent just going to stop talking.

annexrichmond(4444) 6 days ago [-]

r/RightWingLGBT was also banned because it supposedly promotes 'hate'

What is hate, anyways?

cjones26(10000) 6 days ago [-]

When someone disagrees with you

mFixman(10000) 6 days ago [-]

A big chunk of that sub was transphobic posts.

abnry(10000) 6 days ago [-]

If the rationale for banning is policy violations (doxxing, racism, vulgarity, calls for violence, etc), as long as the standards are consistently enforced I'm okay with it. Although my caveat is that racism is defined differently for different people. I E., racism has to be directed towards a marginalized group vs derogatory statements based on racial characteristics.

This is my main problem with Twitter. The standards aren't consistently enforced. I've seen so many leftist blue checkmarks justifying violence that get a pass.

And on the racism front, Sarah Jeong comes to mind.

SV_BubbleTime(10000) 6 days ago [-]

> racism has to be directed towards a marginalized group vs derogatory statements based on racial characteristics.

Here I am old school just thinking the color of a mans skin should have no more significance than the color of his eyes...

I don't believe in the oppression olympics and that hate is ok because group A has it worse than B.

Edit: HN 2020, ideals of Haile Selassie are rejected in the name of "tolerance" I guess? What a wild ride!

ipnon(4405) 6 days ago [-]

The issue is more pronounced when applied to online communities. We have to assume there are individual members of any online community both breaking and following the rules. What is the threshold of rule breaking that triggers a blanket deactivation of an entire group?

The rights of social media companies are so vaguely defined now. Can they escape this crisis without anti-trust litigation or a Supreme Court ruling on the fundamental rights of companies when hosting content online?

duxup(4127) 6 days ago [-]

It was largely empty after the quarantine.

I always wondered what the actual population of that sub was, I felt like I saw some strange patterns where whole hordes of users would appear, disappear, mob another sub and so on. It felt like a very non organic community in many ways.

The issues surroundings that sub weren't just some folks who had a sub to talk to each other, they were very busy mobbing / taking over other subs and etc.

Like most of those subs that leaned right it seemed like a front for sort of a rabbit hole of fear, hate, open bigotry, and blatent calls for violence / division and etc.

zionic(10000) 6 days ago [-]

Well they broke off and made their own website after the quarantine and it's on it's way to the top-1000 US sites (alexa rank).

SketchySeaBeast(10000) 6 days ago [-]

> I felt like I saw some strange patterns where whole hordes of users would appear, disappear, mob another sub and so on. It felt like a very non organic community in many ways.

Wouldn't that match the claims that it was largely just full of bots?

teej(2406) 6 days ago [-]

Reddit Admins posted this[0] list which had r/td at less than 8k daily active users before being shut down. I imagine that's an order of magnitude below its peak.

[0]: https://www.redditstatic.com/banned-subreddits-june-2020.txt

blisseyGo(4442) 6 days ago [-]

I used to spend a lot of money on Reddit gold until few years ago when Reddit blocked any discussion about the gay club shooting at Pulse nightclub. The_Donald was the only sub which allowed discussion of it and seemed to be the only ones who could sympathize with the gay community on Reddit at that time.

Before that, I used to believe right wingers were a bunch of homophobes and what not but if that was the case, why would would they be the only one discussing and sympathizing with the families of the victims?

Why was the supposed homophobic and violent sub The_Donald the only ones which were allowing stickies for blood donations towards the shooting victims while the /r/news sub deleting any comment which talked about blood donations to the victims? PEOPLE LITERALLY DIED and for all the virtue signaling Reddit and the front page subs did, they didn't care about the victims?

https://web.archive.org/web/20160612232812/http://i.imgur.co...

https://web.archive.org/web/20160612215205/https://www.reddi...

https://web.archive.org/web/20160612215019/http://reddit.com...

https://web.archive.org/web/20160613172842/http://reddit.com...

Here's people literally pleading mods of /r/news to not delete blood donation comments:

https://web.archive.org/web/20160612212229/http://media.brei...

When people called out the mods of /r/news to stop censoring, the mod told them to 'kill yourself':

https://web.archive.org/web/20160616024508/http://imgur.com/...

Here's mods of /r/news deleting a comment about a missing friend:

https://web.archive.org/web/20160920213229/https://i.imgur.c...

This made me re-think about whether my side was the bad guys? Since I am a brown immigrant myself, I was always told right wingers hate me and other minorities. I had never looked deep enough. The Orlando Pulse nightclub shooting which killed 50 gays made me re-think. Why is the side which supposedly hates minorities the only one talking about blood donations to the victims?

This also made me question why was '/r/rightwingLGBT' banned? Are gays not allowed to be conservative?

Since then, I started browsing The_Donald more often and despite whatever media says, there can sometimes be tongue in cheek memes and comments on there but they are not evil people as media and Reddit makes them out to be. There's a huge community there with gays, hispanics, blacks and every minority group which media claims right wingers hate. When hurricane Maria hit Puerto Rico 3 years ago, The_Donald was the only sub where people were sharing how a lot of the FEMA supplies were going missing. Everyone else was calling this a conspiracy theory but there were people literally sharing videos of garbage trucks and warehouses filled with perfectly well and brand new supplies. It was quite sinister. Then the truth finally came out earlier this year that they found 8 warehouses full of FEMA supplies which were rotting away and never given to the Puerto Rico people who were literally starving and dying. These are the types of incidents which have changed my mind since then that there's evil in media and many people who are actively willing to let people suffer and die if it makes right wingers look bad.

I might get downvoted for this comment but I hope few people pay attention to what's happening and dig a bit deeper than what media and big tech are telling them. I used to be a liberal - I am still a liberal on most things but I can no longer identify with the current left. It breaks my heart on what people have become.

emerged(10000) 6 days ago [-]

This is my experience as well. I was fully liberal until election night. The veil was lifted watching CNN anchors react to their preferred candidate losing. I started digging and wound up at T_D. My experience there has been polar opposite of the perspective you hear claimed by many.

ThA0x2(10000) 6 days ago [-]

They literally greenlit racism against Whites:

'Marginalized or vulnerable groups include, but are not limited to, groups based on their actual and perceived race, color, religion, national origin, ethnicity, immigration status, gender, gender identity, sexual orientation, pregnancy, or disability. These include victims of a major violent event and their families.

While the rule on hate protects such groups, it does not protect all groups or all forms of identity. For example, the rule does not protect groups of people who are in the majority or who promote such attacks of hate.'

mitchdoogle(10000) 4 days ago [-]

Can't find any reference to the line about majority on their website

frankzen(10000) 6 days ago [-]

It's about time now to declare the 'social media' experiment dead. We're basically learning that there is value in excluding. The all inclusive buffet only brings you down to a neutered lowest common denominator. You're not really going to get any real thought or discussion when you're conversation is monitored by outside moderators. It was doomed to fail anyway.

floatingatoll(3888) 6 days ago [-]

Do you consider this failure to include this social media site, "Hacker News"?

superkuh(4196) 6 days ago [-]

I have no love for the political ideology or behavior of this group but they weren't breaking any laws or even any more rules than other groups on reddit that still operate. They were just politically unviable.

Centralized corporate means to communicate (like, say, this one we're using) always go bad eventually. It's the natural lifecycle of online forums. Once money involved it's only a matter of time before the profit and drama-avoidance incentives of the corporation win over the wishes of the users. Reddit hasn't been usable since 2013.

root_axis(10000) 6 days ago [-]

> They were just politically unviable.

We all know reddit leans a bit left, but the ban also included /r/chapotraphouse which is a leftist subreddit that was also notorious for rule breaking. Frankly, it seems reddit went out of its way to accommodate the_donald for a while now.

Pfhreak(10000) 6 days ago [-]

> they weren't breaking ... any more rules than other groups on reddit

Do you have data that backs this assertion? Because it seems to run counter to the stated reasoning for the removal of the subreddit.

Consultant32452(10000) 6 days ago [-]

I generally agree with everything you said. The phrase 'politically unviable' has taken on a totally different meaning to me this year. I don't watch his show, but Tucker Carlson has the most watched cable news show at the moment, and yet somehow it's becoming politically unviable to advertise on his show. Who would have ever thought that ideas with a plurality of support/viewership would be politically unviable. We are living in clown world.

13415(10000) 6 days ago [-]

I've written about it elsewhere, so just a quick comment: The best thing is to get rid of all political zealots on a forum. The majority of users are not interested in the political opinions of these people (who usually defend fringe views), and political agitation in online forums is not needed or any way beneficial for democracy. It just annoys the vast majority of normal users.

haunter(2273) 6 days ago [-]

I wonder how they will fight against the other kind of systematic propaganda now but guess nothing will happen. Subs like /r/bestof became absolute shit

See the current top post there which is absolutely crazy but hey it fits the agenda.

tenpies(4431) 6 days ago [-]

Reddit is basically a Left-wing propaganda outlet at this point. The amount of clearly targeted sub-Reddits with the aim of radicalising their target group is staggering and would make even the most nefarious of insurgents blush at how open and obvious it is.

secondcoming(10000) 6 days ago [-]

Great, I hope r/politics is next. And then they can start censoring anti-semitic far-leftists.

MrZongle2(4274) 6 days ago [-]

Don't hold your breath, especially regarding any action taken against /r/politics or an effort to make it actually politically neutral.

zimac(10000) 6 days ago [-]

New rule change is essentially no hate subs except if it's hate against a majority aka white people. As a minority I'm tired of defending white people from other white people and being shouted at by white people for doing so. I've tried for 6 years to push for equality for all and had the most insane rebuttals against that.

SkyBelow(10000) 6 days ago [-]

Oddest thing is that many groups deemed a majority are not if one looks at a global scale. It really seems the terms majority/minority are selectively defined as needed to target certain groups.

maceurt(4403) 6 days ago [-]

Yeah I feel you. The problem is that the majority of white people agree with you, and do not believe its okay to use hate against them. However, there is a minority of white people who think they can speak for their whole race and will attack anybody who doesn't fall in line.

Maybe its because I don't live in a city like Portland or LA, but I have never talked to a white person in real life one on one who agreed with the typical narrative on places like reddit of, 'you can't be racist against white people'. The more time I spend on reddit, the more I realize is that the people who comment are getting less and less human. Back 5 years ago, reddit actually felt human. Now it feels like it is just filled with either bots or seriously mentally ill people. I think its about time for me to stop going on that site entirely.

praveen9920(4433) 6 days ago [-]

I'm sorry but I feel that the posts with paywalls should be separated out from top page.

floatingatoll(3888) 6 days ago [-]

You'll need to email the site moderators using the footer Contact link if you wish to have any influence on their views in this matter. Posting about it here will have no effect at all.

electrotype(4430) 6 days ago [-]

I remember how was Reddit 10 years ago... It was a very open and permissive place. Only very bad things were prohibited at that time (pedophilia, rape, murder, etc.).

It is now ran by people/corporations with a political agenda.

sschueller(2641) 6 days ago [-]

Yep, r/videos is so locked down you don't get anything relevant other than some old cat video repost.

reaperducer(4213) 6 days ago [-]

I was mostly OK with the previous restrictions on Mr. Trump, because they were directed at a person acting irresponsibily.

This, however, may cross a line for me. I'm not a Reddit user, but it's my understanding that this is a discussion group about a person. I'm less OK with restricting the discussions of topics.

It reminds me of something I heard in a forensics class years ago: Attack the problem, not the person.

I need to think about this one for a bit.

dunnevens(10000) 6 days ago [-]

There are pro-Trump, pro-Conservative subreddits which haven't broken the rules and haven't been particularly hateful. Those subs are still active.

Giorgi(4062) 6 days ago [-]

Day before someone reported that on Monday Reddit was planning wave of censorship.

Follow reddit self-destruction here: https://www.reddit.com/r/WatchRedditDie/

chippy(645) 6 days ago [-]

could you link the report?

maceurt(4403) 6 days ago [-]

And the news traveled pretty quickly to so many different subreddits before the ban wave. All the subs that had a chance of getting banned were making posts warning about an upcoming ban, and I even saw a post in r/againsthatesubreddits like 6 hours before the banwave that was talking about the post on watchredditdie in a way that sounded like they knew it was true.

I highly doubt that all these mods just saw one post from a medium sized subreddit and started preparing for a ban. Seems to me like all the mods within certain subreddits knew and it got leaked to other mods.

raldi(660) 6 days ago [-]

> a community for 4 years

admin1010(10000) 6 days ago [-]

Do you all think Steve is ok? Like on a personal health level? The constant onslaught of 'you're a nazi and racist' and being overwhelmingly hated by your user base must take a toll. How do CEOs in positions like this not burn out or avoid major stress-related health problems?

asdf21(10000) 6 days ago [-]

I met him years ago, he was never ok

flyingfences(10000) 6 days ago [-]

> How do CEOs in positions like this not burn out or avoid major stress-related health problems?

Money.

lumberingjack(10000) 6 days ago [-]

I was still using the site but now my front page is empty I tell you empty not one sub I had is LEFT

duxup(4127) 6 days ago [-]

That's a pretty weird thing to have happen. How many subs could you possibly been following?

Not that many subs get banned...

bradlys(10000) 6 days ago [-]

Chapo was just banned too. No warning AFAIK.

peruvian(3972) 6 days ago [-]

I think after getting quarantined we all knew it was coming, and this ban was known ahead of time. We just didn't know for sure.

Miner49er(4448) 6 days ago [-]

Really this is what it's all about. T_D has been dead for months and its ban is mostly symbolic. Chapo has still been active.

TheGrim-888(10000) 6 days ago [-]

There's already multiple clones of that subreddit, and there will be many more created, and they'll all be allowed to thrive.

otterley(3585) 6 days ago [-]

'...and nothing of value was lost.'

bastardoperator(10000) 6 days ago [-]

trash in / trash out

submeta(2618) 6 days ago [-]

I wonder how someone like Hitler would have been treated. I guess free speech advocates would have helped him to spread his sick ideas and rise to power? Free speech needs to have limits somewhere. If others are insulted, smeared, de-humanized, if hate is spreaded, if under the protection of free speech people rise to power who poison the political climate, then the whole system is in danger of falling apart, with all the good elements, even the right for free speech.

NE2z2T9qi(10000) 6 days ago [-]

So you think people who are fierce advocates of everyone's fundamental rights and liberties would--on net--enable the Nazis? The idea that we still have rabid censorship advocates in 2020 is dumbfounding. How many 'heretics' have to burn before people realize that policing ideas is deeply illiberal?

50ckpuppet(10000) 6 days ago [-]

The protection of the whole system is a sound education that teaches people how to think not what to think. you should try it sometime.

ummonk(4395) 6 days ago [-]

The Weimar Republic had hate speech laws, and prosecuted Nazis for violating them. This simply resulted in more polarization and Nazis becoming more committed to their cause.

anm89(3612) 6 days ago [-]

Throwing Hitler out as a boogie man does not nullify the entire legal system of our country or the intellectual tradition behind it. Yes Hitler would have had free speech too although most of what is viewed as problematic by Hitler would have been considered violent hate speech in our current legal system and would still not be allowed.

As a jew, I support free speech for hypothetical Hitler. Stop using these lazy arguments.





Historical Discussions: Abstract Wikipedia (July 02, 2020: 793 points)

(793) Abstract Wikipedia

793 points 3 days ago by infodocket in 312th position

meta.wikimedia.org | Estimated reading time – 3 minutes | comments | anchor

Announcing a new wiki project! Welcome, Abstract Wikipedia[edit]

Hi all,

It is my honor to introduce Abstract Wikipedia, a new project that has been unanimously approved by the Wikimedia Foundation Board of Trustees. Abstract Wikipedia proposes a new way to generate baseline encyclopedic content in a multilingual fashion, allowing more contributors and more readers to share more knowledge in more languages. It is an approach that aims to make cross-lingual cooperation easier on our projects, increase the sustainability of our movement through expanding access to participation, improve the user experience for readers of all languages, and innovate in free knowledge by connecting some of the strengths of our movement to create something new.

This is our first new project in over seven years. Abstract Wikipedia was submitted as a project proposal by Denny Vrandečić in May of 2020 [1] after years of preparation and research, leading to a detailed plan and lively discussions in the Wikimedia communities. We know that the energy and the creativity of the community often runs up against language barriers, and information that is available in one language may not make it to other language Wikipedias. Abstract Wikipedia intends to look and feel like a Wikipedia, but build on the powerful, language-independent conceptual models of Wikidata, with the goal of letting volunteers create and maintain Wikipedia articles across our polyglot Wikimedia world.

The project will allow volunteers to assemble the fundamentals of an article using words and entities from Wikidata. Because Wikidata uses conceptual models that are meant to be universal across languages, it should be possible to use and extend these building blocks of knowledge to create models for articles that also have universal value. Using code, volunteers will be able to translate these abstract "articles" into their own languages. If successful, this could eventually allow everyone to read about any topic in Wikidata in their own language.

As you can imagine, this work will require a lot of software development, and a lot of cooperation among Wikimedians. In order to make this effort possible, Denny will join the Foundation as a staff member in July and lead this initiative. You may know Denny as the creator of Wikidata, a long-time community member, a former staff member at Wikimedia Deutschland, and a former Trustee at the Wikimedia Foundation [2]. We are very excited that Denny will bring his skills and expertise to work on this project alongside the Foundation's product, technology, and community liaison teams.

It is important to acknowledge that this is an experimental project, and that every Wikipedia community has different needs. This project may offer some communities great advantages. Other communities may engage less. Every language Wikipedia community will be free to choose and moderate whether or how they would use content from this project.

We are excited that this new wiki-project has the possibility to advance knowledge equity through increased access to knowledge. It also invites us to consider and engage with critical questions about how and by whom knowledge is constructed. We look forward to working in cooperation with the communities to think through these important questions.

There is much to do as we begin designing a plan for Abstract Wikipedia in close collaboration with our communities. I encourage you to get involved by going to the project page and joining the new mailing list [3]. We recognize that Abstract Wikipedia is ambitious, but we also recognize its potential. We invite you all to join us on a new, unexplored path.

Yours,

Katherine Maher (Executive Director, Wikimedia Foundation)




All Comments: [-] | anchor

crazygringo(3985) 3 days ago [-]

I think a consistent multilingual Wikipedia is a fantastic goal.

But I'm not sure this is the right way to do it.

Given that most of the information on Wikipedia is 'narrative', and doesn't consist of facts contained in Wikidata (e.g. a history article recounting a battle, or a movie article explaining the plot), this scope for this will be extremely limited. The creators are attempting to address this by actually containing every single aspect of a movie's plot as a fact, and that sentences are functions that express those facts... but this seems entirely unwieldy and just too much work.

What I've wished for instead, for years, is actually an underlying 'metalanguage' that expresses the vocabulary and grammatical concepts in all languages. Very loosely, think of an 'intermediate' linguistic representation layer in Google Translate.

Obviously nobody can write in that directly in a user-friendly way. But what you could do is take English (or any language) text, do an automated translation into that intermediate representation, then ask the author or volunteers to identify all ambiguous language cases' -- e.g. it would ask if 'he signed' means made his signature, or communicated in sign language. It would also ask for things that would need clarification perhaps not in your own language but in other languages -- e.g. what noun does 'it' refer to, so another language will know to use the masculine or feminine version. All of this can be done within your own language to produce an accurate language-agnostic 'text'.

Then, out of this intermediate canonical interpretation, every article on Wikipedia would be generated back out of it, in all languages, and perfectly accurately, because the output program isn't even ML, it's just a straight-up rule engine.

Interestingly, an English-language original might be output just a little bit different but in ways that don't change the meaning. Almost like a language 'linter'.

Anyways -- I think it would actually be doable. The key part is a 'Google Translate'-type tool that does 99% of the work. It would need manual curation of the intermediate layer with a professional linguist from each language, as well as manually curated output rules (although those could be generated by ML as a first pass).

But something like that could fundamentally change communication. Imagine if any article you wanted to make available perfectly translated to anyone, you could do, just with the extra work of resolving all the ambiguities a translating program finds.

miracle2k(4447) 3 days ago [-]

Is the Grammatical Framework something like the thing you are imagining?

SkyBelow(10000) 3 days ago [-]

>The creators are attempting to address this by actually containing every single aspect of a movie's plot as a fact, and that sentences are functions that express those facts... but this seems entirely unwieldy and just too much work.

Doesn't this also get into issues where facts aren't clearly defined. I can think of a lot of interpretation of meaning from my literature classes, but there are also questions such as ownership of land at contested borders, if something was a legal acquisition or theft, or even coming up with a factual distinction between when something is grave robbery vs archaeology. A personal favorite would be mental illness, especially with some of the DSM V changes that have largely been rejected (or outright ignored) by society. And there are all sorts of political disagreements.

And as this applies to different languages, and different languages are likely aimed at different cultures and different nations, this gets messy. I could see some differences in an article written in Hindi vs Chinese when concerning issues involving both China and India. Creating a common language will force a unification of these differences that currently might exist with a sort of stalemate with each linguistic side by maintained by the dominant country for that language.

amirouche(3083) 3 days ago [-]

> But what you could do is take English...

Then it is not english but a subset of english. They are translation systems that work the way you describe already using restricted natural language grammars.

noema(10000) 3 days ago [-]

>What I've wished for instead, for years, is actually an underlying 'metalanguage' that expresses the vocabulary and grammatical concepts in all languages.

We've been down this road before... https://en.wikipedia.org/wiki/Characteristica_universalis

sitkack(4376) 3 days ago [-]

Even if you wrote the articles in a subset of English like the https://simple.wikipedia.org/wiki/Main_Page and then used ML to translate into other languages and then formed a feedback loop with the translation so that the original author could have some assurance that the translated texts were valid, this would be huge.

ComodoHacker(3773) 3 days ago [-]

I don't quite get what problem it's trying to solve. Save labor? Improve factual consistency across languages?

20after4(4289) 3 days ago [-]

'Knowledge Equity.'

To increase the availability of knowledge for speakers of less popular languages. Once encoded in Abstract form, it can be made available in every human language.

That is an improvement over the current situation where knowledge is concentrated in just a few of the most popular languages.

random04243(10000) 3 days ago [-]

This is of course an interesting idea, but it has a number of huge technical hurdles to overcome. Here is the biggest:

Right now, if you want to become an editor of Wikipedia, you simply need to have a passing familiarity with wikitext, and how the syntax of wikitext translates into the final presentation of the article.

However, if you want to become an editor of Abstract Wikipedia, you'd need to have an in-depth knowledge of lambda calculus, and possibly a Ph.D. in linguistics. Without a quantum leap in editing technology and accessibility for beginners, there's little hope for this to gain any traction.

memexy(1727) 3 days ago [-]

Why do you need a PhD in linguistics to write code?

abbe98(2639) 3 days ago [-]

> Right now, if you want to become an editor of Wikipedia, you simply need to have a passing familiarity with wikitext

Wikipedia have had an WYSIWYG editor for years.

> if you want to become an editor of Abstract Wikipedia, you'd need to have an in-depth knowledge of lambda calculus, and possibly a Ph.D. in linguistics

No this is not how it's intended. First the data itself is supposed to be from Wikidata which is super simple to edit. Secondly surly they can come up with an UI for the other parts.

memexy(1727) 3 days ago [-]

> The project will allow volunteers to assemble the fundamentals of an article using words and entities from Wikidata. Because Wikidata uses conceptual models that are meant to be universal across languages, it should be possible to use and extend these building blocks of knowledge to create models for articles that also have universal value. Using code, volunteers will be able to translate these abstract "articles" into their own languages. If successful, this could eventually allow everyone to read about any topic in Wikidata in their own language.

This is a great idea. I bet the translations will be interesting as well. I was wondering about how the translation was going to work and it looks like they thought of that as well. They're going to use code to help with the translation.

> Wikilambda is a new Wikimedia project that allows to create and maintain code. This is useful in many different ways. It provides a catalog of all kind of functions that anyone can call, write, maintain, and use. It also provides code that translates the language-independent article from Abstract Wikipedia into the language of a Wikipedia. This allows everyone to read the article in their language. Wikilambda will use knowledge about words and entities from Wikidata.

zozbot234(10000) 3 days ago [-]

Pretty-printing the abstract content into an arbitrary target language (a better way of putting it than 'translation') would be quite the challenge, because 'conceptual models' do vary by language. One can attempt to come up with something that's 'as abstract/universal as possible' but it remains to be seen how practically useful that would be.

For that matter, making the source model 'logical' and 'compositional', as implied by the Wikilambda idea, only opens up further cans of worms. Linguists and cognitive scientists have explored the idea of a 'logical' semantics for natural language, even drawing on the λ-calculus itself (e.g. in Montague grammar and Montague semantics), but one can be sure that a lot of complexity will be involved in trying to express realistic notions by relying on anything like that.

Jasper_(3676) 3 days ago [-]

So do people find Wikidata that impressive? Here's what Wikidata says about Earth, an item that is number 2 in the ID list, and also on their front page as an example of incredible data.

https://www.wikidata.org/wiki/Q2

I struggle to find anything interesting on this page. It is apparently a 'topic of geography', whatever that means as a statement. It has a WordLift URL. It is an instance of an inner planet.

The first perhaps verifiable, solid fact, that Earth has a diameter of '12,742 kilometre', is immediately suspect. There is no clarifying remark, not even a note, that Earth is not any uniform shape and cannot have a single value as its diameter.

This is my problem with SPARQL, with 'data bases', in that sense. Data alone is useless without a context or a framework in which it can be truly understood. Facts like this can have multiple values depending on exactly what you're measuring, or what you're using the measurement for.

And this on the page for Earth, an example that is used on their front page, and has the ID of 2. It is the second item to ever be created in Wikidata, after Q1, 'Universe', and yet everything on it is useless.

bawolff(10000) 3 days ago [-]

I dont think its interesting in itself so much as in applications. I remember talking to someone once who was working on a project where you stick a probe in some soil, and then it uses wikidata to tell you the best type of plant to grow. I have no idea whatever happened to this project, if it worked or not - but it always struck me as a great example of the enabling value of wikidata - that you can use it to power ideas totally unrelated to the original purpose the data was collected for.

ricardo81(10000) 3 days ago [-]

I've worked with the Wikidata set a bit. On first glance the entries do seem to lack any useful information as it's all heavily abstracted into other items and properties - as well as containing a bunch of references and qualifiers to validate the facts.

Once you start connecting the items to other items and properties, you begin to see better information and context.

A lot of the 'snaks' of items are units of measurement, so no worries converting them into other languages. This project should help in generating articles in other languages based on these facts.

visarga(3945) 3 days ago [-]

I find it pretty well stuffed with appropriate information. You're looking at an ontology, not a wikipedia article, it's supposed to be dry (subject, relation, object). It's being used to disambiguate concepts, named entities and support machine learning models with general knowledge in a standard format. There are plenty of papers on the topic of link prediction, auto-completion and triplet mining.

Also, if you look:

> radius: 6,378.137±0.001 kilometre

> applies to part: equator

So it clearly states how the radius was measured.

blablabla123(10000) 3 days ago [-]

Q2 is just an id, probably one shouldn't interpret too much into it except that it defines an entity. Regarding the diameter, probably it depends how you define it. For instance according to Wikipedia one can generalize it as sup { d(x,y) }, seems legitimate to me although Wikidata's referenced diameter definition (P2386) isn't that general, probably it should be updated... But to be fair, Earth (Q2) has the shape (P1419) oblate spheroid (Q3241540) under sourcing circumstances (P1480) approximation (Q27058) :-)

To me Wikidata (and similar projects like OSM) shine because they tend to have so many details.

ImaCake(10000) 3 days ago [-]

People might be interested to know that semantic web ideas have been more successful in some niches than others. Computational biology for example makes extensive use of 'ontologies' which are domain specific DAGs that do exactly what Abstract Wikipedia is attempting. Much of the analysis of organism's genomes and related sequences relies on these ontologies to automatically annotate the results so that meaningfull relationships can be discovered.

There are of course HUGE issues with the ontologies. They are not sexy projects so they are often underfunded and under resourced - even though the entirety of bioinformatics uses them! The ontologies are incomplete and sometimes their information is years behind the current research.

For the curious, the Gene Ontology is the golden child of biology ontologies. See here: http://geneontology.org/

jchook(4358) 3 days ago [-]

Amazingly fascinating field. I have learned a (very) small amount about this from Dr. David Sinclair's book Lifespan

aaron695(1174) 3 days ago [-]

This has been tried and failed many times before.

Why is this different?

What is the fundamental structural difference that will allow this to work?

ricardo81(10000) 3 days ago [-]

There are many more English articles than any other language on Wikipedia, even though there's more non-English speakers in the world.

To me, it seems this project will allow for at least 'stub' articles in essentially every-other-language which at the very least provides some basic information about each entity to a reader in their preferred language.

young_unixer(10000) 3 days ago [-]

Might be a good idea, but the multilingual argument doesn't convince me one bit. If this project is any useful, it won't be because of its multilingual part.

Any person worth reading in STEM fields already knows English, and I don't know why anyone would want to read Wikipedia in any language other than English.

I'm Latin American. I used the Internet in Spanish in my early teens before learning English, and it's a joke compared the English Internet. I don't even like English from a grammatical and phonetical point of view, but trying to cater to the non-English-speaking public seems like a waste of time in 2020. Just learn English already if you don't know it, it will be a much better use of your time than reading subpar material in another language.

rtpg(2950) 3 days ago [-]

> Any person worth reading in STEM fields already knows English, and I don't know why anyone would want to read Wikipedia in any language other than English.

> Just learn English already if you don't know it

Some people are merely fine at English, or uncomfortable reading 'casually' in their second/third languages...

It's actually not unreasonable for someone to want learning content in their native language. And there are loads of opportunities to try out new content when people in different places are writing content in different languages, with new angles and takes

For example the best intro to LaTeX is a book originally written in French[0].

And sometimes content just makes better sense in other languages because primary materials will be in that language (if you had the choice, would you rather read about the great Tokyo Fire in English or in Japanese?)

Sure, having access to English content is really important! But trying to have multilingual content is normal.

[0]: https://www.latexpourlimpatient.fr/

082349872349872(4435) 3 days ago [-]

I often read wikipedia in several languages, because the differences between the articles sometimes offer almost as many bits of information as the commonalities.

What amazes me are the small-audience ones. For instance, who uses https://pdc.wikipedia.org/wiki/Haaptblatt given that most of the native speakers of that dialect adhere to a religion which mandates that cell phones belong, not on one's person, but in the barn, and furthermore be used strictly for business?

(I did first learn about the Mennonite origins of the https://pdc.wikipedia.org/wiki/New_Holland_Machine_Company from this wikipedia)

To the main point, I expect the lojban wikipedia to profit immensely from this project :-)

https://jbo.wikipedia.org/wiki/uikipedi%27as:ralju

yellowapple(4412) 3 days ago [-]

> I used the Internet in Spanish in my early teens before learning English, and it's a joke compared the English Internet.

This project seems like one decent way to try and fix that.

sajforbes(10000) 3 days ago [-]

> Any person worth reading in STEM fields already knows English, and I don't know why anyone would want to read Wikipedia in any language other than English

That's not the goal of Wikipedia. The goal is to make knowledge freely available to as many people as possible. That's fantastic that you were given the opportunity to learn English and have made the best of it. That is not everyone though. It's not reasonable to expect a farmer in Malaysia to know English or to learn it purely to take advantage of what's there.

numpad0(4432) 3 days ago [-]

Hello from Japanese internet.

nemoniac(3811) 3 days ago [-]

On the contrary, Wikipedia is a wonderful multilingual resource.

English is one of my mother tongues but I regularly read Wikipedia in half a dozen other languages to improve my knowledge of them. Being able to cross-reference what you're reading to the English version of the text, even though it's not a literal translation, gives valuable context as well as perspective on how a topic is viewed by different language groups.

It's also often more useful than a dictionary for finding the name of a flower or fish in another language. Or even some topics that you wouldn't find in a dictionary at all.

thom(3662) 3 days ago [-]

Anything that gives a boost to Wikidata is great. Being able to run queries over wiki remains one of the most magical things on the internet:

https://query.wikidata.org/

yorwba(3525) 3 days ago [-]

I recently learned that words and translations from Wiktionary are in Wikidata's graph as well, which enables e.g. this simple lemmatizer: https://tools.wmflabs.org/ordia/text-to-lexemes (The Wikidata query it uses is linked at the bottom.)

visarga(3945) 3 days ago [-]

It's magic but the SparQL language is very hard to learn.

simple_phrases(10000) 3 days ago [-]

Thanks for sharing this, didn't know it existed.

blahedo(2358) 3 days ago [-]

Anyone who has studied old-school AI will know that this is an incredibly ambitious project; it is essentially throwing itself at the problem of 'knowledge frames', i.e. how to encode information about the world in a way that an AI system can access it and, well, be intelligent about it. (Also at the problem of natural language generation, but as hard as that is, at the moment it seems like the easier of the two.)

But...

One of the biggest problems with a lot of the old 'Big AI' projects that were developing some sort of knowledge frames (and there were several, and some of them still exist and have public faces) was, who the hell is going to get all the info in there in a way that's complete enough to be useful? Now you have a learning problem on top of the knowledge representation problem. But throw the wikimedia community at it and crowdsource the information?

This actually starts to seem plausible.

dannyw(4147) 3 days ago [-]

Even if it's not successful, there's certainly enough interest in it to make it worth trying.

Maybe we only get 30% of the way. So what? That's 30% more than zero!

tasogare(10000) 3 days ago [-]

Yes, and the paper do only lists 2 relevant references regarding the decades of work in this domain.

heavenlyblue(10000) 3 days ago [-]

How is that different from Wolfram Alpha?

skybrian(1888) 3 days ago [-]

It seems more similar to an elaborate version of the internationalization and translation of messages done in any program that targets multiple languages? If you think of it as a principled template language for generating text from the results of canned database queries, it starts seeming a lot more feasible. The templates themselves do need to be translated into every language, much like the messages in internationalization.

Ideally this enables something like an improved version of the ICU library, with a lot more data available.

cochne(4448) 3 days ago [-]

If successful, this could open huge doors in machine translation and NLP. Very cool.

coding123(3808) 3 days ago [-]

It kinda would, basically a huge library of labelled NLP data may come available as the result of this.

ethhics(10000) 3 days ago [-]

This appears to be an attempt to make a Wikipedia using the semantic data from Wikidata. The semantic web ideas of Tim Berners-Lee may be catching on.

Schoolmeister(3920) 3 days ago [-]

That already happens, that's what Wikidata does. [0]

[0] https://www.wikidata.org/wiki/Wikidata:RDF

ppur(10000) 3 days ago [-]

It's also interesting to note that, already, bots on some Wikipedias are the largest contributors of articles in that language. The Swedish, Waray, and Cebuano Wikipedias already have an estimated 'between 80% and 99% of the total' all written by one bot, Lsjbot [1].

[1]. https://en.wikipedia.org/wiki/Lsjbot

renewiltord(10000) 3 days ago [-]

I wonder if lsjbot has increased single contribution users. Wikipedia (or EN Wikipedia anyway) gates article creation but not editing. If other Wikipedias do that as well, then single edit users won't be able to create an article and hence can't contribute. But if lsjbot has created the stub then people can contribute.

lihaciudaniel(4114) 3 days ago [-]

Reminder that wiki means quick, so when you read Wikipedia you only have surface knowledge,

shadowgovt(4438) 3 days ago [-]

As the set of information encoded into Wikipedia approaches the sum total of human knowledge, there's no particular reason that needs to remain true.

Ninjaneered(10000) 3 days ago [-]

For reference, this is from the same developer [1] that created Semantic MediaWiki [2] and lead the development of Wikidata [3]. Here's a link to the white paper [4] describing Abstract Wikipedia (and Wikilambda). Considering the success of Wikidata, I'm hopeful this effort succeeds, but it is pretty ambitious.

[1] https://meta.wikimedia.org/wiki/User:Denny

[2] https://en.wikipedia.org/wiki/Semantic_MediaWiki

[3] https://en.wikipedia.org/wiki/Wikidata

[4] https://arxiv.org/abs/2004.04733

9nGQluzmnq3M(1392) 3 days ago [-]

As a long-time Wikipedian, this track record is actually worrisome.

Semantic Mediawiki (which I attempted to use at one point) is difficult to work with and far too complicated and abstract for the average Wiki editor. (See also Tim Berners-Lee and the failure of Semantic Web.)

WikiData is a seemingly genius concept -- turn all those boxes of data into a queryable database! -- kneecapped by academic but impractical technology choices (RDF/SPARQL). If they had just dumped the data into a relational database queryable by SQL, it would be far more accessible to developers and data scientists.

jameshart(3008) 3 days ago [-]

There could be some very interesting meta analytics that could be done on knowledge structured in this way. For example, this research which identifies the structural differences in the fact graphs of conspiracy theories vs accurate accounts: https://phys.org/news/2020-06-conspiracy-theories-emergeand-...

xtacy(885) 3 days ago [-]

Interesting link, thanks for sharing. I wonder what this means precisely:

  If you take out one of the characters or story elements of a conspiracy theory, the connections between the other elements of the story fall apart.
I guess I have to read the paper, but what are these 'connections' and what does 'fall apart' actually mean?

EDIT: I just skimmed the paper https://journals.plos.org/plosone/article?id=10.1371/journal...

The connections capture context-specific relationships, such as co-occurrences. The 'fall apart' part comes from the fact that conspiracy theories rely on hidden, unsubstantiated, subjective interpretations of intent or actions whose validity can be questioned. If they are key pillars of the narrative, then their falsity can negate the truth of the narrative.

This reminds me of a philosophical discussion around what 'truth' means. Coherent theory of truth: truth is defined as a property that's coherent among a set of beliefs. It can also be used as an epistemic justification -- that is, any set of internally consistent beliefs can be taken as true. Of course, in practice, certain truth statements have to correspond to reality, which is where the correspondence theory of truth comes in.

samizdis(953) 3 days ago [-]

Great article. Thanks for the recommendation/link.

xvilka(1872) 3 days ago [-]

Semantic Web[1] reborn (after alleged[2] death)? Also I wonder how helpful Prolog infrastructure could be since they provided some useful frameworks [3][4] for that.

[1] https://www.w3.org/standards/semanticweb/

[2] https://twobithistory.org/2018/05/27/semantic-web.html

[3] https://www.swi-prolog.org/web/

[4] https://www.swi-prolog.org/pldoc/doc_for?object=section(%27p...

memexy(1727) 3 days ago [-]

Twobithistory article is pretty good. Outlines several things I wasn't aware of like DBPedia.

YeGoblynQueenne(292) 3 days ago [-]

>> Also I wonder how helpful Prolog infrastructure could be since they provided some useful frameworks [3][4] for that.

That's a good point, because looking at the working paper on the proposed architecture of the project [1], the example of a 'constructor' in Figure 1 is basically a set of frames and it has a straightforward translation in Prolog and the example of a 'renderer' in English is basically a pattern with holes that also has a very straightforward Prolog implementation via Definite Clause Grammars. In fact the whole architecture reminds me a lot of IBM's Watson - the good bits (i.e. the Prolog stuff they used to store the knowledgebase).

________

[1] https://arxiv.org/pdf/2004.04733.pdf

tgv(10000) 3 days ago [-]

That kind of AI has been tried, 30 years ago, and it doesn't go far enough. It's really difficult to get that out of the toy domains.

LukeEF(4161) 3 days ago [-]

The prolog renaissance is unstoppable!

JimmyRuska(4256) 3 days ago [-]

We actually looked into SWI prolog semantic web package for corporate work! We ended up finding RDFox ( https://www.oxfordsemantic.tech/ ) which is the bleeding edge in research on inference databases and linked data. Unfortunately COVID changed the plans but we were really really impressed with the capabilities.

Semantic web is used broadly; Google Structured data you see for reviews and infoboxes, wikidata. Data is broadly available, even if jobs in semantic technologies are not.

We're familiar with common databases like key value stores, OLAP, OLTP, but reasoning technology offers unique properties many people aren't aware of. For example you can have your business logic integrated with your database in a way that's much more flexible that stored procedures. You express your business rules as logic programs, the automatically run multi-core, they run as soon as data is inserted into the database and there is no function call; the data does not need to be aware of what logic is in the database, logical rules are applied incrementally so that adding new data or new rules does not trigger re-computation of all the data, business rules can use data produced by other business rules, and finally you use the explain command to get a mathematical proof of why an outcome happened.

Reasoning technology may be old but recently this idea of automatically stating things in a declarative form and having the application reconcile the differences has been the differentiating factor for the most popular software out there; kubernetes, teraform, ansible, react, graphql, flutter. Without the declarative reasoning capabilities, these tools may not be considered some of the best.

Think postgresql 12 generated columns except infinitely chainable, recursive and connectable to other tables. Think pre-computed materialized views, but automatically updated as new data is inserted (no refresh needed).

yewenjie(4422) 3 days ago [-]

Can really please ELI5 what the end product would look like? Couldn't understand anything concrete from the article.

SquishyPanda23(10000) 3 days ago [-]

This article from the SignPost is much more informative:

https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...

dankohn1(145) 3 days ago [-]

Sorry to be the typical pessimistic HN commenter (e.g., Dropbox is just ftp), but this seems ambitious enough to remind me of https://en.wikipedia.org/wiki/Cyc.

8bitsrule(3576) 2 days ago [-]

Agreed. '[since 1982,] by 2017 [Lenat] and his team had spent about 2,000 person-years building Cyc, approximately 24 million rules and assertions (not counting 'facts') and 2,000 person-years of effort.' https://en.wikipedia.org/wiki/Douglas_Lenat

memexy(1727) 3 days ago [-]

Why is thinking of an analogy pessimistic?

zozbot234(10000) 3 days ago [-]

Even Wikidata today is already a lot more usable and scalable than Cyc. The latter always seemed like a largely-pointless proof of concept; Wikidata by contrast is very clearly something that can contain real info, and be queried in useful ways. (Of course knowledge is not always consistently represented, but that issue is inherent to any general-purpose knowledge base - and Wikidata does at least try to address it, if only via leveraging the well-known principle 'many eyes make all bugs shallow'.)

carapace(3022) 3 days ago [-]

I don't think that's pessimistic, more like cautionary. For a project like this it behooves them (IMO) to do a 'related work' review, eh?

dredmorbius(140) 3 days ago [-]

I think Patick Cassidy (dict) and his MICRA project might be vaguely similar as well.

http://micra.com

csande17(3866) 3 days ago [-]

A Wikipedia Signpost article[1] gives a more detailed overview of the goals of the project, but it also made me think of an interesting failure case. From the article:

> Instead of saying 'in order to deny her the advantage of the incumbent, the board votes in January 2018 to replace her with Mark Farrell as interim mayor until the special elections', imagine we say something more abstract such as elect(elector: Board of Supervisors, electee: Mark Farrell, position: Mayor of San Francisco, reason: deny(advantage of incumbency, London Breed)) – and even more, all of these would be language-independent identifiers, so that thing would actually look more like Q40231(Q3658756, Q6767574, Q1343202(Q6015536, Q6669880)).

But Q1343202 doesn't mean 'denial' as in 'preventing someone else from getting something', it means 'denial' as in 'refusing to accept reality'. (See [2].) The two concepts are represented by the same word in English, but they might not be in other languages.

It seems like it'd be kind of tricky to create an interface that ensures other English-speaking editors indicate the right meaning of 'denial'.

[1] https://en.m.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost...

[2] https://m.wikidata.org/wiki/Q1343202

bawolff(10000) 3 days ago [-]

I think the answer is be as clear as possible in the interface, but also accept mistakes will be made. People make grammar mistakes in (normal) wikipedia all the time, then other people come along and fix them. I expect the same will occur here.

keane(4022) 3 days ago [-]

Example notation for the project, called AbstractText:

————

Input 1:

Subclassification(Wikipedia, Encyclopedia)

Result 1:

English: Wikipedias are encyclopedias.

German: Wikipedien sind Enzyklopädien.

————

Input 2:

  Article(
   content: [
     Instantiation(
       instance: San Francisco (Q62),
       class: Object_with_modifier_and_of(
         object: center,
         modifier: And_modifier(
           conjuncts: [cultural, commercial, financial]
         ),
         of: Northern California (Q1066807)
       )
     ),
     Ranking(
       subject: San Francisco (Q62),
       rank: 4,
       object: city (Q515),
       by: population (Q1613416),
       local_constraint: California (Q99),
       after: [Los Angeles (Q65), San Diego (Q16552), San Jose (Q16553)]
     )
   ]
 )
Result 2:

English: San Francisco is the cultural, commercial, and financial center of Northern California. It is the fourth-most populous city in California, after Los Angeles, San Diego and San Jose.

German: San Francisco ist das kulturelle, kommerzielle und finanzielle Zentrum Nordkaliforniens. Es ist, nach Los Angeles, San Diego und San Jose, die viertgrößte Stadt in Kalifornien.

————

I didn't understand quite what the proposal was until I saw these examples from https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Examples

IAmNotAFix(10000) 2 days ago [-]

How does it go beyond the headline and general info?

StavrosK(521) 3 days ago [-]

I wonder what happens in more literal languages, where 'center' doesn't mean 'main area'.

LudwigNagasena(10000) 3 days ago [-]

English and German have very similar vocabulary and syntactic structure. So this example is not very elucidating. Comparing it to Chinese, Turkish or Javanese would probably be better.

aaron695(1174) 3 days ago [-]

This example is quite child like.

It's what bright children in the 70's learning about computers thought.

50 years later they haven't solved it because it doesn't work that way.

Is there a real example not using proper nouns?

A city changes, a population changes depending on the country and language and time. Town X having a population Y might be considered a Village X and Population Z, because in some countries population includes the rural parts, the population of San Francisco might be different in another country.

The rabbit hole goes on forever, and more importantly it's been tried constantly for over 50 years.

Unlike Machine translation which is amazing compared to 50 years ago, and getting better, and you could see how you could integrate it better with Wikipedia (It's already used) yet it's tossed out in the white paper for no real good reason I can see. There's also lots of stuff like Duolingo style methods that you could look at.

knolax(4442) 1 day ago [-]

What a horribly myopic way to organize information. They seem to have unthinkingly copied from vernacular English various loosely defined concepts like 'city'. What do they mean by San Francisco? The City and County of San Francisco? What about Los Angeles? Is that the entire LA metro or just LA county? Is Santa Monica a part of Los Angeles or a seperate settlement? How is the concept of 'city', 'metro', and 'town' going to translate into '市', 'Burg', and 'Grad'?

gorgoiler(4352) 3 days ago [-]

The syntax looks well optimized for human editing.

The example seems like it would be machine generated though.

I hope the syntax learns from SQL, and allows for easy generation by either man or machine, preferably a little of both.

jbob2000(10000) 3 days ago [-]

This is getting very close to the Universal Language that Umberto Eco describes in his book The Search for the Perfect Language. I wonder what he would think about this if he were alive today...

blondin(4026) 3 days ago [-]

truly love the mobile design of wikipedia and find myself adding '.m' to every link that i visit on wikipedia. it has larger fonts, more readable copy (for me at least), and works great on mobile. surprisingly the trick worked with this one as well!

how come the mobile design is not the default?

Kuinox(10000) 3 days ago [-]

On mobile I struggle to remove the m. on every link. The mobile version does not work well with the research feature, and I never took time to know where do you switch to another language.

magnio(10000) 3 days ago [-]

The math template (the HTML one, not MathJax) and many other templates don't work well with the mobile version, and the talk page isn't there.

On desktop, I suggest installing Stylus and using a more readable and elegant Wikipedia themes. I like wikipedia.rehash by krasjet.

notatoad(10000) 3 days ago [-]

I'm not very involved in wikipedia politics so i might be wrong here, but my perception is that the desktop wikipedia has a lot of eyes on it and any change is received with a 'aahh, change is scary' response.

the people who react negatively to change are also the people don't like mobile versions of websites, so the mobile site is more free to experiment and evolve their design.

pvg(4446) 3 days ago [-]

Ironically, switching to different language versions of the same page is irritating on the mobile version.

forgotpwd16(10000) 3 days ago [-]

Because it is limited in design. It misses the sidebar which has useful links, the discussion page, the history page, doesn't have account links. Also the more compact text the desktop version is preferable to some.

You can automatically redirect to the mobile version if you want using a user script. I'm using a similar one for the reverse.

Edit: Here you're. https://gist.github.com/leakypixel/1b0a30fbdc815016c14264b82...

hannasanarion(10000) 3 days ago [-]

Try Wikiwand. It's a browser extension that gives all wikipedia pages a mobile-like design.

jholman(10000) 3 days ago [-]

Truly hate that I have to deal with deleting the .m off other people's links. It's so ugly, it has these disgustingly oversized fonts, and because all the sections are hidden by default, it works badly on mobile and even worse on a real computer.

How come I can't just stop seeing that loathsome mobile design, both on my desktop and on my phone?

And no, I'm not kidding.

bawolff(10000) 3 days ago [-]

Note the mobile skin is separate from the mobile site. If you set in your preferences the skin Minerva Neue, you will get it on desktop (just the skin, there are non skin differences with the mobile site you wont get).

As for why not default- i imagine partially because the mobile site tends to be unpopular with power users (of course opinions vary)

sitkack(4376) 3 days ago [-]

HolyShit!

> The goal of Abstract Wikipedia is to let more people share in more knowledge in more languages. Abstract Wikipedia is an extension of Wikidata. In Abstract Wikipedia, people can create and maintain Wikipedia articles in a language-independent way. A Wikipedia in a language can translate this language-independent article into its language. Code does the translation.

from https://meta.wikimedia.org/wiki/Abstract_Wikipedia

Will this mean that knowledge is encoded in machine readable format and that we can start to write programs over this knowledge graph? This is huge.

remolacha(4260) 3 days ago [-]

Very cool. I'm fascinated by the Wolfram Language paradigm of Knowledge Base+Programming language=Computable everything (demo: https://youtu.be/3yrVuM2SYZ8). But I could never get into the Wolfram ecosystem because it's totally proprietary. This makes me think, does Wikidata's model (ontologies?) provide a way to recreate the Wolfram computable everything concept as an open community project?

afandian(4069) 3 days ago [-]

The Abstract Wikipedia idea is a great advance.

> knowledge is encoded in machine readable format and that we can start to write programs over this knowledge graph

But surely that abilty has been the goal of wikidata from the start?

zelphirkalt(10000) 3 days ago [-]

The semantic structure is the structure of the language (or the independent thing between languages). This does not automatically facilitate machine Understandable knowledge. You would need to write the code to understand it first, which is probably almost as difficult as understanding English for example.

Only when relations are defined in a kind of Prolog style in those examples it will be usable as knowledge about things other than language.

A computer might spit out texts which it guessed are connected to some question you ask it though. Does not mean it understands the relations of things.

But when can we ever really say 'a machine understands' something? So perhaps there is not much of a difference?

yters(2130) 3 days ago [-]

Isn't this the same as expert systems that led to the first AI winter?

amrrs(2769) 3 days ago [-]

I'm sorry if I didn't understand. Wouldn't a json or xml type data structure (where some Wikipedia stuff is already stored) would support this?

msla(4394) 3 days ago [-]

It assumes articles will say the same thing in every language, which to me means that edit wars can now proceed on a more global basis. You're no longer fighting only the people who feel comfortable enough with your language to edit in it, you're fighting anyone who can edit the article at all around the world.

Do the Hebrew Wikipedia and the Arabic Wikipedia agree on the status of Israel?

miket(2393) 3 days ago [-]

Hi, founder of Diffbot here, we are an AI research company spinout from Stanford that generate the world's largest knowledge graph from crawling the whole web. I didn't want to comment, but I see a lot of misunderstandings here about knowledge graphs, abstract representations of language, and the extent as to which this project uses ML.

First of all, having a machine-readable database of knowledge(i.e. Wikidata) is no doubt a great thing. It's maintained by a large community of human curators and always growing. However, generating actually useful natural language that rivals the value you get from reading a Wikipedia page from an abstract representation is problematic.

If you look at the walkthrough for how this would work (https://github.com/google/abstracttext/blob/master/eneyj/doc...), this project does not use machine and uses CFG-like production rules to generate natural sentences. Works great for generating toy sentences like 'X is a Y'.

However, human languages are not programming languages. Many natural languages, like German and Finnish, are so syntactically and morphologically complex that there is no compact ruleset that can describe them. (those that have taken grammar class can relate to the number of exceptions to the ruleset)

Additionally, not every sentence in a typical Wikipedia article can be easily represented in a machine-readable factual format. Plenty of text is opinion, subjective, or describes notions that don't have an proper entity. Of course there are ways that engineer around this, however they will exponential grow the complexity of your ontology, number of properties, and make for a terrible user experience for the annotators.

A much better and direct approach to the stated intention of making the knowledge accessible to more readers is to advance the state of machine translation, which would capture nuance and non-facts present in the original article. Additionally, exploring ML-based ways of NL generation from the dataset this will produce will have academic impact.

Vinnl(381) 3 days ago [-]

> Many natural languages, like German and Finnish, are so syntactically and morphologically complex that there is no compact ruleset that can describe them. (...)

> Additionally, not every sentence in a typical Wikipedia article can be easily represented in a machine-readable factual format.

It doesn't seem like the goal of this project is to describe those languages, or to represent ever sentence in a typical Wikipedia article? The goal doesn't seem to be to have all Wikipedia articles generated from Wikidata, but rather to have a couple of templates to the order of 'if I have this data available about this type of Subject, generate this stub article about it'. That would allow the smaller Wikipedia language editions to automatically generate many baseline articles that they might not currently have.

For example, the Dutch Wikipedia is one of the largest editions mainly because a large percentage of its articles were created by bots [1] that created a lot of articles on small towns ('x is a town in the municipality of y, founded in z. It is nearby m, n and o.') and obscure species of plants. This just seems like a more structured plan to apply that approach to many of the smaller Wikipedia's that may be missing a lot of basic articles and are thus not exposing many basic facts.

[1] https://en.wikipedia.org/wiki/Dutch_Wikipedia#Internet_bots

riku_iki(10000) 3 days ago [-]

> Of course there are ways that engineer around this, however they will exponential grow the complexity of your ontology, number of properties, and make for a terrible user experience for the annotators.

So, the obvious solution is to create robo-annotators, and that's what your company is supposedly trying to do?

YeGoblynQueenne(292) 3 days ago [-]

>> Many natural languages, like German and Finnish, are so syntactically and morphologically complex that there is no compact ruleset that can describe them.

Is that realy true? If natural languages have rules, then there exists a ruleset that can describe any natural language- the set of all rules in that language. Of course, a 'rule' is a compact representation of a set of strings, so if natural languages don't have such rules it's difficult to see how any automated system can represent a natural language 'compactly'. A system without any kind of 'rules' would have to store every grammatical string in a language. That must be impossible in theory and in practice.

If I may offer a personal perspective, I think that the goal of the plan is to produce better automated translations than is currently possible with machine translation between language pairs for which there are very few parallel texts. My personal perspective is that I'm Greek and I am sad to report that basicaly translation from any language to Greek by e.g. Google Translate (which I use occasionally) is laughably, cringe-inducingly bad. From what I understand the reason for that is not only the morphology of the Greek language which is kind of a linguistic isolate (as opposed to, say, Romance languages), but also that, because there are not many parallel texts between most languages (on Google Translate) and Greek, the translation goes through English- which results in completely distorted syntax and meaning. Any project that can improve on this sorry state of affairs (and not just for Greek- there are languages with many fewer speakers and no paralle texts at all, not even with English) is worth every second of its time.

To put it plainly, if you don't have enough data to train a machine learning model, what, exactly, are your options? There is only one option: to do the work by hand. Wikipedia, with its army of volunteers, has a much better shot at getting results this way than any previous effort.

YeGoblynQueenne(292) 3 days ago [-]

This is addressed in the white paper describing the project's architecture:

10.2 Machine translation

Another widely used approach —mostly for readers, much less for contributors— is the use of automatic translation services like Google Translate. A reader finds an article they are interested in and then asks the service to translate itinto a language they understand. Google Translate currently supports about a hundred languages — about a third of thelanguages Wikipedia supports. Also the quality of these translations can vary widely — and almost never achieves thequality a reader expects from an encyclopedia [33, 86].*

Unfortunately, the quality of the translations often correlates with the availability of content in the given language [1],which leads to a Matthew effect: languages that already have larger amounts of content also feature better results intranslation. This is an inherent problem with the way Machine Translation is currently trained, using large corpora. Whereas further breakthroughs in Machine Translation are expected [43], these are hard to plan for.

In short, relying on Machine Translation may delay the achievement of the Wikipedia mission by a rather unpredictabletime frame.

One advantage Abstract Wikipedia would lead to is that Machine Translation system can use the natural language generation system available in Wikilambda to generate high-quality and high-fidelity parallel corpora for even morelanguages, which can be used to train Machine Translation systems which then can resolve the brittleness a symbolic system will undoubtedly encounter. So Abstract Wikipedia will increase the speed Machine Translation will become better and cover more languages in.

https://arxiv.org/abs/2004.04733

(Theres's more discussion of machine learning in the paper but I'm quoting the section on machine translation in particular).

zelly(10000) 3 days ago [-]

What's the point of this with the current high quality of state-of-the-art machine translation? Don't we expect machine translation to surpass humans in the near future?

People who are domain experts in various fields don't know how, don't care to, and shouldn't code. They should just edit the articles in natural language.

A lot of the content of Wikidata isn't numbers and is natural language also, so you'd still need to (machine?) translate it. But this time the machine translation algorithm would not have the benefit of the long-term context from the encompassing paragraph.

There are too many reasons why this is a bad idea. Almost makes me mad.

moriturius(10000) 3 days ago [-]

They are not solving your problem. They are solving theirs.

This is an experimental project so there is nothing to be angry about. Time will tell if this was a good idea or not.

_-___________-_(4229) 3 days ago [-]

Where is the high quality machine translation? I spend most of my time in countries where I don't speak the same language as the majority of people and text that I encounter, so I am using machine translation many times per day. My experience of the average quality of machine translation is extremely low. It garbles meaning a majority of the time, and in a significant minority of cases destroys meaning completely.

To me the idea that you could translate an encyclopedia, where accuracy of meaning is critical, using such technology in its current state is horrifying. By contrast the abstract/semantic approach seems to have some potential, although I can't imagine it working well for all articles.

yellowapple(4412) 3 days ago [-]

> People who are domain experts in various fields don't know how, don't care to, and shouldn't code.

My impression (correct me if I'm wrong, though) is that this is less 'domain experts writing code' and more 'editors can export a Wikidata 'subject' to a prebuilt translated page that domain experts can later expand'. The aim of Wikidata is to collect information in a language-agnostic way, whereas the aim of Abstract Wikipedia seems to be to take that information and turn it into autogenerated pages in whatever language (even if that page is more-or-less a stub).





Historical Discussions: Zettlr – FOSS markdown editor for personal knowledge management and publishing (July 03, 2020: 658 points)
Zettlr: Markdown editor for notes, Zettelkasten, and IDE (May 14, 2020: 4 points)

(661) Zettlr – FOSS markdown editor for personal knowledge management and publishing

661 points 2 days ago by DerWOK in 3796th position

www.zettlr.com | | comments | anchor

[[Connect]] Your Knowledge

In modern times, getting a hold of the flood of information is almost as hard as inserting a USB drive the right way on the first attempt. Zettlr allows you to connect pieces of information using state of the art Zettelkasten methodology. Links? Check. File IDs? Check. File tagging? Also check.

And the best is: Unlike many competitors, Zettlr never locks you in. Zettlr supports almost every conceivable way to create links and identify your files. In other words: No matter where you come from — all Zettelkästen are beautiful and supported by Zettlr. Out of the box.

See what's possible



All Comments: [-] | anchor

darkstarsys(10000) 2 days ago [-]

Not a popular opinion I'm sure, but I'm all-in on OneNote. Works everywhere (at least basically) and it is just so rich. Full pen support for drawing (vital for me), tables, equations (sort of), multiple text blocks on a page (also key!), internal & external links, fast search (as of last year). Search is good enough that I rarely use tags anymore.

Yes, it's totally vendor locked in and I do hate that. And no syntax highlighting for code is annoying. Lack of markdown is a pain. And it's bug-ridden and closed source.

But I've been using it for my work daily journal and knowledge capture for a few years now, and it's so fluid and easy to jot down or scribble a quick note and find it later that it's hard for me to imagine going back to a basic Markdown editor. It's the closest thing I've found to a searchable paper lab notebook.

And btw I'm a hardcore Emacs user for the last 40 years. Org mode is great, but for me, OneNote kills it in expressiveness and fluidity of idea capture and recall.

If someone makes a Markdown editor that supports pen/tablet ink drawing and multiple text blocks on a page, I'd be interested.

ralls_ebfe(10000) 2 days ago [-]

Regarding drawing: have you seen org-krita? https://github.com/lepisma/org-krita

smilekzs(4435) 2 days ago [-]

Long-time OneNote user chiming in. I've briefly switched to Evernote but due to their frequent screw-ups I switched back to OneNote. Stylus input is a big part for my dev logs, and one good thing about OneNote is that it accepts both handwriting and text without prejudice.

Also, pages in OneNote are allowed to have 2 levels of indent. More than that you can organize with traditional folder structures. The difference is that a less indented 'parent' page can still have content; a folder does not contain content on its own.

I would strongly recommend OneTastic as it adds quite a few missing features from stock OneNote (crop image, calendar view, etc.)

bluenose69(4449) 2 days ago [-]

zettlr does not seem to be able to import markdown files, which is a problem for people like me who have lots of such files. Maybe there is a way around that, but I am not motivated to spend much time seeking it, since I find vimwiki to be sufficient for navigating through links in my files. (Also, vimwiki makes it easy to create links, with the strike of a key.)

My scheme does not offer me a nice GUI, but I prefer to see simple text anyway, and I like how vimwiki lets me navigate through my cross-linked notes without my fingers leaving the home keys. Markdown permits images, etc., and if I want to see them I can just open a terminal to a subdirectory and use pandoc.

I don't see a way in vimwiki to get a 'what links to this page' item, which I imagine an application like zettlr would offer, but it be easy to write a python script to do that, and to add a a line to my crontab file to update things every so often.

The good thing about my setup is that the markdown format is not tied to any particular application. That's important, if you want your database of notes to last for a long time.

elric(4410) 1 day ago [-]

You can just dump the markdown files in whatever directory zettlr is using, and you're done. There's no magic required here, zettlr is just a markdown editor.

fastball(4302) 2 days ago [-]

Shameless plug: I've been building a similar tool for the past couple years, although it is not FOSS :(

The major differentiator is that content is based around notecards rather than documents/files, and there are multiple ways to structure these cards.

The most powerful way to organize things is with multi-parent nesting, where you put cards inside of other cards, and each card can have any number of parents. You can share these cards with others, and they can add their own parents to it that don't interfere with yours, allowing you to have shared cards that exist within entirely different hierarchies that are unique to each individual user.

That's a feature that I think is unique in the space, but we also have the links/backlinks and tags that you will see elsewhere (though tagging is similarly powerful in that shared cards can have tags that are public vs tags that are private to the user).

You can check it out here:

https://supernotes.app

crooked-v(4436) 2 days ago [-]

A couple of quick bits of feedback, from trying it out for the first time:

- The app locks itself to the browser's light/dark mode setting, with no way to change it. This is a huge pain for people like me who prefer a dark UI but light page content.

- 'where you put cards inside of other cards' - how to do this was totally unintuitive until random clicking around revealed that the edit button turned into a plus button, and under that was 'add parent' (but not 'add child'). The total lack of drag-and-drop functionality with the card focus seems really weird.

- The site advertises the ability to share cards, but the UI gives me zero idea how to do that, and the site seems to imply I can only share with logged-in users, which eliminates most of the reason I'd want to be easily share things in the first place.

emptysongglass(4329) 2 days ago [-]

I love the metaphor of a massively hyperlinked card. It reminds me a little of https://mochi.cards, which I've enjoyed spending time with.

But these things can only ever remain idle curiosities to me when they're closed source. We're talking about a knowledge graph that only really flexes itself with years of dedicated entry, roaming and combing through its interlinks for that electric flash of idea-sex.

Why should I commit to a company's knowledge box when I can't trust your team's bus factor? You're one human being, any number of calamities could befall you -- I have to assume Supernotes isn't wired to an open-source dead man's switch.

You're also an engine of capital beholden to its single truth: print more money at all costs. We've already seen apps like Roam jack up their price. Is there a ceiling you can guarantee me?

m-p-3(10000) 2 days ago [-]

Is there support for diagrams, like Mermaid? That's something I use with Joplin for my personal knowledgebase.

DerWOK(3796) 2 days ago [-]

Yes mermaid is built in. Just start a tripple apostrophe codeblock with code style mermaid and it will render the graphics when your cursor leaves the code block.

DerWOK(3796) 2 days ago [-]

Some admin removed the 'version 1.7' from the post headline? Why that? The news is that the version 1.7 - after 4 month of work - was just released a few hours ago... ️

dsissitka(10000) 2 days ago [-]

I imagine it's because the link points to Zettlr's home page and so the discussion ended up being about Zettlr in general rather than the 1.7 release.

steveklabnik(51) 2 days ago [-]

It's great to see such an explosion in tooling here. It also seems really hard to write comparisons since there's so much development going on. I haven't tried Zettlr yet, but it's on my list...

input_sh(3807) 2 days ago [-]

I've tried it. I do like the built-in features, but the design makes me despise it.

It doesn't follow any system preferences, even replacing the standard minimize/maximize/close buttons (seriously, why?). Some of the stuff should definitely be a plugin instead of a core feature. Don't need a pomodoro timer / readability feature? Tough luck, you can't even remove it from the interface, let alone remove it completely.

There's also no plugin support, so you can't extend the features in any way.

I'm using Obsidian[0] instead. A little rough around the edges, but it doesn't have these issues.

[0] https://obsidian.md/

bbx(1208) 2 days ago [-]

One feature I have yet to find in any Markdown editor is a simple 'block' mover, which I describe here: https://twitter.com/jgthms/status/1225513837379641350

Underneath, the data structure would remain straightforward Markdown. So the data wouldn't be stored as separate blocks; the moving would act similarly to Sublime Text's 'Swap Line Up/Down'.

cellularmitosis(4404) 2 days ago [-]

I added your tweet to Panicz's excellent twitter thread of structured editors, cheers! https://twitter.com/cellularmitosis/status/12791132604076851...

enobrev(3686) 2 days ago [-]

I wouldn't call notion.so a markdown editor, but it's very similar to one (understands the syntax) and the 'block mover' concept is inherent in their UI.

NateEag(10000) 1 day ago [-]

I kinda hate to be that guy, but one of the many reasons I love Vim keybindings is that they have support for working in blocks like built in.

'dap' deletes the entire paragraph my cursor is in, then I type '/' to search for where I want to move it to, and paste.

And this also works for sentences, and lines, and words. And more.

In every file type I edit.

(Emacs with Evil is a better vim than vim, in a lot of ways. )

codethief(10000) 2 days ago [-]

This (and showing/hiding blocks) is the one feature I miss from org-mode.

dr_zoidberg(10000) 2 days ago [-]

In VSCode anytime you have text selected you can alt+up/down to move the whole selection (or current line if no selection) up/down one line. I assume STs 'swap line up/down' works the same.

Is that different from your 'block mover' idea? I mean, if I select the paragrap I want to move and use it it sounds similar, thought it'd happen one line at a time, and I understand you'd mean to move it 'whole blocks' at a time.

But probably a plugin can be made to achieve that behavior.

koheripbal(10000) 2 days ago [-]

Maybe I'm missing something, but what's wrong with cutting and pasting blocks of markdown up/down?

black_puppydog(4314) 2 days ago [-]

I can't believe nobody bothered to link TiddlyWiki [1] here... Especially since this crowd here should be able to run it directly from npm, which makes it much easier (conceptually, for me) than 'a self-modifying html file'. :P

For Zettelkasten (and research more generally) Stroll [2] is a flavour of TiddlyWiki that has many of the features you'd like, including (most crucially for me) backlinks.

Edit: the reason I brought up TiddlyWiki here is because I tried Zettlr and while I see how some people would like it, I certainly didn't.

[1]: https://tiddlywiki.com/

[2]: https://giffmex.org/stroll/stroll.html

lstmemery(10000) 2 days ago [-]

I spent about an hour trying to set up a TiddlyWiki for a Zettelkasten, but couldn't get it working. I was overwhelmed by the number of editors available. I now use Zettlr.

Is there a tutorial you used to help your set up?

Erwin(4338) 2 days ago [-]

I switched to TW for journalling, but it's still not perfect. Before that I had just a text file I could press a button to insert a date and add some text in at the end.

I switched to monthly journals rather than daily as with daily there were just too many small tiddlers. I think you need a lot of discipline to categorize every thought you get in a nice hierarchy, and that takes time. I've started using the 'Excise' command for that in TW, that lets you pick some existing text and move it to another tiddler, maybe adding a link (here the backlinks would help, so I might upgrade to Stroll, if possible)

I_am_tiberius(10000) 2 days ago [-]

Joplin and Standard notes are great as well. I lost one week of data due to some synchronization issue with Joplin though - but the markdown editor is great (they even have a new WYSIWYG editor). Standard notes is good but it's missing good file/image support which is really annoying.

electricEmu(10000) 2 days ago [-]

What happened with your Joplin sync issue?

I am interested in using Joplin exclusively and use both apps currently. The Standard Notes development team is nice but refactors happen too frequently and estimates are rarely (never) produced, even though the product is paid Open Source.

lazyant(4428) 2 days ago [-]

I'm using Joplin but looking to change. Issues I have with Joplin:

  - when I type towards the end of a document, the page jumps out of focus, very annoying. The rest I can kind of live with.  
  - I'm not loving there's no G Drive sync  
  - the local disk sync are not original MD files plus a temp file but a bunch of files.  
  - I can't resize the mid partition in order to see more original md  
  - Can't type on right view panel  
  - panels not sync'd horizontally at the line we are at (this may not be possible)
codr7(4127) 2 days ago [-]

Two words, Org mode:

https://orgmode.org/

Add Magit and you're good to go:

https://github.com/magit/magit

Emacs is a quirky beast, but these two packages alone make it worth the effort to learn.

ralls_ebfe(10000) 2 days ago [-]

Switched to emacs(spacemacs) from vim last year, because I wanted to know what the craze with org-mode is all about. Am using mu4e, elfeed, rcirc, magit, org-mode and started using org-roam this week. I still suck at elisp, but emacs with org-mode really is a blast. I love my agenda: whenever I get confused what I was about to do, it is only some keystrokes away. I feel insanely organized while forgetting everything all the time.

alexktz(4429) 1 day ago [-]

Don't forget org-roam!!

I swear all these other folks trying tool x, y, z are only doing so because they haven't found org. It's a bit cryptic to get started but totally worth the effort.

lifeisstillgood(1633) 2 days ago [-]

I think this trend of better knowledge tools is missing two very important pieces of human nature

1. If we have time to enter something into a knowledge base of any kind - then we have time to just jot it on a piece of paper.

2. If we dont have time (or think it is important at that moment) then what solves the problem for us is not a knowledge base, but search.

You see the thing about Google and Facebook etc, is that if they were collecting all this information about me, and it was treated like medical information about me it would be far more useful to me (and far less useful to Advertisers).

I want a web browser that remembers every single page I have visited (#) and then lets me search them. Then someone could write a spaced reminder thingy for me - spent more than 5 minutes on a web page did he - he will want to refresh that page in 2 weeks and then 4 months.

Yes, knowledge bases are excellent for clearly defined study efforts - like y'know, university, but for the rest of life, explicit note taking is a cost that we need some activation energy barrier for.

Put it this way, once upon a time I had a study book for a new programming language, and i took notes of interesting examples on a ring binder. But the last time I learnt a new language I just relied on Google finding me the relevant StackOverflow pages - my cost/benefit line had changed.

(And notes just got dumped into a text file.)

(#) Ok maybe not those pages

thundergolfer(10000) 2 days ago [-]

I agree that search is a key direction for growth in this tooling ecosystem. I don't think a tool should index every visited webpage though. I think knowledge base construction should be intentional and methodology, as the process provides its own value.

I've begun building a knowledge base that is built around a Google-search interface but with closed-world assumptions, such that it can only answer questions and provide results that I should know myself.

In this sense it's another take on "second brain" tools, but to me it's potentially a novel angle, using search with closed-world restrictions (ie. much fewer documents, personally stored and indexed) and a question-answering focused interface.

Most other tools focus on the authoring side of things, but I want something that will superpower my memory.

It's all closed source until I get it going well, but I've set up http://rememex.org.

mftrhu(10000) 2 days ago [-]

> I want a web browser that remembers every single page I have visited (#) and then lets me search them.

Did you check out WorldBrain's Memex <http://worldbrain.io/> browser extension yet?

cborenstein(4418) 2 days ago [-]

> 1. If we have time to enter something into a knowledge base of any kind - then we have time to just jot it on a piece of paper.

Great point. I often see people sharing strategies for organizing notes but I rarely see people discussing how they get stuff down in the first place. And there's a lot of improvement we can make here.

I wanted to add more modularity to my jot-downs so that it would be easier to organize them later, one small step at a time. I found that I can get this modularity by writing my notes as if I'm texting myself.

(Working on a tool [1] that has this sms-style interface with keyboard shortcuts for purging my notes and organizing. would love your feedback).

[1] https://bytebase.io

bachmeier(3977) 2 days ago [-]

> what solves the problem for us is not a knowledge base, but search

There are problems with search: you have to know to search for something, and you have to know how to search for it. In some cases this is an issue, in some it isn't. I have had many times over the years where I reviewed my notes and reminded myself of things I had completely forgotten. Search is useless in that case.

But in any event, there's no conflict between a knowledge base and search. They are different things and you can search a knowledge base.

jeffbee(4357) 2 days ago [-]

I ran a private system for a while that proxied all my browser traffic and indexed it into Lucene/Solr. This was cute at the time but with the rise of javascript apps it became increasingly useless. There wasn't anything worth indexing in the responses. These days something like Zotero Connector is probably what you want.

Fiveplus(4358) 2 days ago [-]

Good points, especially point 1. About the second part here - I remember seeing a (rather creative) guy post a personal project based on that, on here, a few days ago. He wrote a comprehensive post about his addon which aims to remembers exactly in which context has a user encountered said article/author/keyword in the past. I'll see if I can link it.

---

quick update, found it: https://news.ycombinator.com/item?id=23668507

lukevp(10000) 2 days ago [-]

I agree, the most useful thing these tools can provide is fast data entry and robust search for most users. That's the focus of the tool I'm building, https://NoteBrook.com . Reach out if you want to be in on the beta, it has been delayed a bit as I've been refining the search and editor experience of the alpha version and building super fast apps on every platform, but it should be launching in the next week or two!

badassiel(10000) 2 days ago [-]

Indirectly related - you can try https://web.hypothes.is. You can annotate web pages (public, private, group) and have page notes.

techntoke(4445) 2 days ago [-]

If we write it down on a piece of paper and there is a fire or natural disaster it easily lost or destroyed. If you factor in writing it down on a piece of paper, scanning and making it searchable then it is easier just to input it on an electronic device.

marvinblum(4246) 2 days ago [-]

I find it hard to get people to write down anything. And if they do, they often times don't put enough effort into it, like writing down some bullet points without context or screenshots without adding text. This is a personal and cultural problem at companies and in teams. A lot of people don't know how to generate value from taking notes/documentation and aren't good writers in general. I'm still thinking about how you can teach people what's important when it comes to writing and when to take notes.

I couldn't agree more with your second point. Search is probably even more important than taking notes in the first place. I find myself searching through my notes multiple times a day, even if it's something you can find at Google at the same speed. But my notes are mine, meaning I know what I can't remember or in which form I need some piece of information. Sometimes, I write a long article about something and other times some bullet points are good enough. When I share my notes on a team/organization I try to make sure the context is clear. It's a bit like code: it's read by an order of magnitude more often than written/changed.

We built Emvi [1] to make note taking more collaborative and with focus on searchability and ease of use. I hope I can write more about this topic on our blog. In case you have ideas on how to solve these cultural problems, please let me know.

[1] https://emvi.com/

hrktb(4446) 2 days ago [-]

> 1. If we have time to enter something into a knowledge base of any kind - then we have time to just jot it on a piece of paper.

There is a kind of generation gap here I think.

I have 3 devices at arms reach while writing this, but it would take me 5 min to find a working pen and a blank piece of paper.

If I used it pen and paper more it would be closer to me now, but it would be the same wherever I go. Then I'd need to capture that paper or bring it with me until it has no value anymore, which is a PITA.

Our current society is tending to a paperless state, writing stuff on paper is just not fast, natural or reliable act anymore.

jeffbee(4357) 2 days ago [-]

Is it the generally-accepted practice to run an app like this in a network namespace or sandbox? How do I know it's not pilfering my research? I'd like to use a system like this for my work but I don't see how it can be trusted. I don't have the time to read the source. To be honest, I prefer a hosted solution because then I get a written contract about the privacy of my data.

cellularmitosis(4404) 2 days ago [-]

Not getting your data stolen vs. being able to sue after the fact are two different concerns which are somewhat in conflict.

If you want to be able to sue, then yes go with a hosted solution with a written contract.

But if the priority is to not have your data stolen, handing your data over to a third party is a non-starter.

estacado(10000) 2 days ago [-]

I've tried a lot of note-taking apps, I've settled on Simplenote. It's lightweight, syncs, and searchable. It's text-only. I find linking media and other stuff is just cumbersome, and takes a lot of effort to organize when all I want is a quick way to jot down notes. https://simplenote.com/

gregwebs(3983) 2 days ago [-]

I preferred InkDrop on my laptop, but I had to switch back to SimpleNote because SimpleNote does both sync and mobile extremely well. Doing mobile well just requires it to never take time to load. InkDrop would usually have to refresh its contents when switching to it.

douglaswlance(10000) 2 days ago [-]

There is nothing that compares to using your preferred text editor to write personal knowledge documents like this. I swapped over to [Foam](https://foambubble.github.io/foam/) in VSCode recently when it was released, and it's like a breath of fresh air. I can use my keyboard shortcuts, extensions, and snippets. Nothing else can compare.

arcticfox(4357) 2 days ago [-]

100% in the same boat. I have historically used notes products for hours and then quit; I am days into Foam and I love it more all the time.

deepersprout(3641) 2 days ago [-]

How do you organize your notes? Do you have one big file for everything, one file for one note, one file for argument, or something else? Do you keep a date on your notes?

Maybe you can share a bit about how you organize your written down knowledge?

stevesimmons(10000) 2 days ago [-]

In the last week, I've started using Foam too.

It hits the sweet spot for me:

- Plain Markdown files stored in git; exactly what I am used to for code.

- Editor is VisualStudioCode, which I always have open, and can configure however I like.

- No vendor lock-in.

rollinDyno(4176) 2 days ago [-]

I'm hoping this will include inline text transclusion soon. I'm sure there are other solutions for viewing images too.

unsungNovelty(10000) 2 days ago [-]

If anybody is sad about Zettlr not having the graph view of like the one in Obsidian or RoamResearch, please be patient. A PR is already open for it. -> https://github.com/Zettlr/Zettlr/pull/921

OpenSource powaaaaa! :)

grok22(10000) 2 days ago [-]

Does anybody find that graph view (like in Obsidian) useful? I play around with it for a while, but I don't really find it useful for any particular thing. Also overtime that graph grows too big to be visually easy to see patterns etc.

greenie_beans(4056) 2 days ago [-]

I read some blog post within the past year are so, written by some young engineer for shopify or spotify. They detailed the way they built their zettelkasten. It was well-written and more detailed than most blog posts in that genre. I can't find it on HN or Google. Anybody know what I'm talking about?

gdevenyi(10000) 2 days ago [-]

What's with all the weird communist references?

dxdm(10000) 2 days ago [-]

They're in the eye of the beholder.

I'm assuming you're talking about the name of the project? To me, Zettel is a German word, meaning a slip of paper useful for taking notes.

jtanderson(3960) 2 days ago [-]

I've been a heavy Standard Notes user for a couple years now, and the added function of Zettlr looks extremely appealing to me (images, linking, much nicer rendering, etc.). However, to really give this a test drive to see if it's a suitable replacement, I downloaded all my Standard Notes as plain text and tried importing them. This caused a ton of bugs/errors when trying to navigate and use the results. First, it complained about not being able to detect the file type -- again, these are all plain text that end in .txt... Second, it seems to have a ton of trouble with renaming folders: it works the first time, renames it on the filesystem, but then doesn't keep the change in the app? Then I try to reload and rename the folder back and now it throws null variable errors left and right? Then, I try to create sub-folders to start organizing the mess of notes I just imported and... big choke, can't create the folder, sometimes it gives an error and other times it just does nothing. The performance (speed opening tabs, scrolling notes) seems to degrade quite a bit with the number of notes I have.

So... this looks like something that could be really great! But there's a lot of friction still to having it get out of the way and let me be organized.

httpsterio(10000) 2 days ago [-]

Zettlr uses virtual folders basically so that your physical organization might not reflect how it's organized in Zettlr. Also, Zettlr is for markdown so you basically have to rename your files as plaintext files are not supported.

matthiasv(3893) 2 days ago [-]

I am also a long-time SN user but I've never really liked the editor. So, I used a bit of free time to turn a C GTK SN client PoC into a somewhat proper Rust app (https://github.com/matze/iridium) and hope to put in more effort the next few weeks. You might like it if Linux is your platform.

wooptoo(4325) 2 days ago [-]

I can recommend Zim wiki which is a GTK+ desktop app that works as a personal wiki / notebook. Has a WYSIWYG editor and can export to different formats, it can even render the wiki as html and serve it. Also supports plugins for extra things like tables and charts if you wish. You can have separate notebooks for each project like home/work/etc. You can commit to git from the UI, have git hooks set to automatically push to a remote on every commit. Not fancy but very functional and pleasant to use. https://zim-wiki.org/

space_ghost(10000) 2 days ago [-]

I've been using Zim for daily notes stashed to a Dropbox folder for many years.

hysan(3320) 2 days ago [-]

Really love Zim wiki and have been using it for over a decade. Sadly, the move to py3 and gtk3 broke it for non-linux platforms. macOS how has a huge wealth of [reported] bugs that break common keyboard shortcuts at the OS level and introduced a lot of performance problems. This isn't the fault of Zim as these are all bugs from gtk3 where it seems like the priority for non-linux bugs is 0. Meanwhile, Windows no longer has a prepackaged installer as the maintainer stopped doing that post transition.

I've tried hacking my way around this but to no avail. It's gotten to the point that I've finally started seriously looking and testing alternatives. Nothing has come close to replicating the feature set and UI workflow.

mkbkn(4021) about 9 hours ago [-]

I like Zim Wiki but haven't used much though. Can you suggest a way to embed images with sync? I mean if you want add images to the text notes, how does it work?

My use case - I want to add images to the text and then export it in some kind of everyday format like html and send it to my friends. I sent it to my friend one of such but he was not able to see any images.

fluder(4153) 2 days ago [-]

I recommend https://fsnot.es Native and blazing fast for iOS and macOS.

balladeer(4050) 1 day ago [-]

I have used it for a while and it's not very stable or rather say 'finished' yet (neither Mac nor iOS). But it's open source, so I'd favour it over, Bear, given a chance, which it resembles the most. So I just keep it installed and keep checking it once in a while.

Though I'd just be happy with something like nv (+ Simple Note + Dropbox) which is actively developed (Simple Note just for the iOS).

lastgeniusua(10000) 2 days ago [-]

Downloading page is down now, I guess?

renjith1(10000) 2 days ago [-]
garfieldnate(10000) 1 day ago [-]

What are the open source options for storing PDF annotations? These are just as important as free-form notes for me.

alltakendamned(3902) 1 day ago [-]

Have a look at Zotero ?

Keyframe(4025) 2 days ago [-]

So, is this like a Scrivener clone? Scrivener is awesome and one of the things I miss when I dumped MacOS altogether.

Wowfunhappy(3953) 2 days ago [-]

Scrivener is not Mac-only anymore! https://www.literatureandlatte.com/scrivener/download

supersrdjan(10000) 2 days ago [-]

What triggered the sudden surge of interest in these knowledge management platforms in your opinion?

rmujica(4025) 2 days ago [-]

HackerNews, obviously

slezyr(10000) 2 days ago [-]

This page loads one CPU to 100% in Firefox

captn3m0(1260) 2 days ago [-]

Its the constellations on the top.

bloopernova(4435) 2 days ago [-]

I'm really happy that the 'knowledge base management' type of tools are getting a lot more attention these days. In my opinion, the more brains that look at this area, the better the whole ecosystem will get.

I'll have to download this and give it a try, and compare it to my current workflow.

(I use org-roam on Emacs. I'm not sure if people are sick of org-mode and Emacs being mentioned on HN? I worry about becoming the stereotype of 'how do you tell if someone is a Vegan (or uses Emacs)?' 'Don't worry, they'll tell you'. I don't want to derail any discussion though!)

For those of you wondering about Zettelkasten and knowledge management, I suggest you start by reading 'How to Take Smart Notes' by Sönke Ahrens: https://takesmartnotes.com/ and https://www.goodreads.com/en/book/show/34507927-how-to-take-...

mxuribe(4253) 2 days ago [-]

> I'm really happy that the 'knowledge base management' type of tools are getting a lot more attention these days...

I share this happiness...It often feels like so many people assume this is a solved sort of area; But i believe it is not.

esperent(10000) 2 days ago [-]

I do think that the area of knowledge management is very interesting and worthy of discussion. When it comes to tooling though, I've tried out a couple and I don't find anything superior to a folder full of markdown files + your favorite text editor (I use VSCode, but I don't think that's especially important).

Systems like Zettelkasten are interesting to read about, but again, everyone's brain is different and for me a bunch of folder categories and one misc/daily folder is just as good.

On a purely personal level, any tooling or system more complex than what I already use is unnecessary.

komarov_om(10000) 2 days ago [-]

It would be awesome to see these kind of tools taken to the next level. So far we are 'just' doing Zettelkasten in computers.

What I mean by the next level is something like:

- when I pick up a new e-book, show me the chapters that have new ideas that I'm not familiar with yet, so I can skip the things I already know

- let me see how my ideas/opinions on a given topic were evolving through time, which encounters have influenced them

- let me publish a slice of personal knowledge base for others to explore, let me see what my friends published (better social media?)

koheripbal(10000) 2 days ago [-]

I've just been using Google Docs. It's simple, searchable, and I can read/edit my notes while sitting on the can.

The only markup I use are: links, lists, bold/underline/strikeout, embedded images?

What am I missing?

daffy(10000) 2 days ago [-]

Do tell if you find this has any advantage over Org mode.

hjek(3239) 2 days ago [-]

As a vegan Emacs + org-mode user, I found your comment hilariously spot-on!

mcdevilkiller(10000) 1 day ago [-]

Do you also use GNU/Hurd?

/s

scribu(960) 2 days ago [-]

The drawback to working directly on Markdown files is that it's hard to synchronize your notes from/to mobile.

If I didn't care about sync, I would use org-roam or some Vim plugin, personally.

DerWOK(3796) 2 days ago [-]

For syncing I'm using a private github repo, then syncing is easy and on iOS / Android I use GitJournal or WorkingCopy (iOS only IMHO).

regularfry(4153) 2 days ago [-]

Syncthing ought to be an answer to this, given the right editor.

ResidentSleeper(10000) 2 days ago [-]

I've recently started using GitJournal [1] to access my notes on Android, but I haven't used it enough to be able to say whether it's worthwhile. One annoying drawback is that it requires a monthly subscription to get access to all of the (convenience) features.

[1] https://play.google.com/store/apps/details?id=io.gitjournal....

harritaylor(10000) 2 days ago [-]

I use resilio sync for this. https://www.resilio.com/individuals/ The iOS (and I assume android) syncing is fine, and it hooks in with the files API so I use pretext/ whatever markdown editor to edit / look things up on the go. For some reason resilio isn't talked about highly on HN though, not entirely sure why.

sanchitnevgi(4357) 2 days ago [-]

I've been using Obsidian (https://obsidian.md/) for the past few weeks and it has been really great.

inakarmacoma(10000) 2 days ago [-]

By the creator of DynaList as well... Another worthy competitor that outshines workflowy in my opinion.

AtHeartEngineer(10000) about 13 hours ago [-]

I need something like this for Android, that's where I got down most of my ideas. For a while I was using Google keep and transferring my notes to scrivener occasionally but that is just time consuming.

DerWOK(3796) 2 days ago [-]

Yes, I also looked into Obsidian. And I really liked it. But their license scheme would have forced me to buy a subscription to also use it in work environment.

So I switched to Zettlr and never looked back.

Would they have allowed the personal one-time bought license to use in professional working, I would have sticked.

dmytton(2184) 2 days ago [-]

Note-taking seems to be a hot topic lately. I used Apple Notes for a long time because it was very lightweight and minimalist, but recent releases of macOS have been very buggy, so I decided to review all the options. I wrote this up at https://davidmytton.blog/the-best-note-taking-apps-for-mac-m... which has become one of the most trafficked post on my blog in the last few months!

The key for me is a) plain text files I can manage myself i.e. no database or mandatory custom sync; b) markdown.

Apps will come and go. You might decide to switch platforms and maybe a new editor will come along sometime. This means you want a format that can be opened by anything (plain text) but with lightweight markup that the editor can parse to make it look nice, but you can also parse with your eyes and get a reasonable sense of the document structure (markdown).

Then it's all about search. There's no point making notes if you can't find them. This is where something more than Markdown - that allows you to link notes - is handy. It's what is appearing more and more in the likes of Roam, Obsidian, etc.

I ultimately chose iA Writer on macOS because it is lightweight and really nicely designed, plus has good native support for Markdown. I sync using OneDrive but you can use anything because they're all individual files. iA Writer is also native, and I find most Electron apps to be slow and/or buggy. There are exceptions e.g. VS Code, but I prefer native where possible.

gaebn(10000) 2 days ago [-]

It would be great to have a tool to backup all the data from Apple Notes to a collection of markdown files or SQLite db.

fluffything(10000) 2 days ago [-]

I use typora to write markdown on macos and am very happy.

To sync, I use a git repository (you can set a watchpoint on macos and linux to any directory to invoke a command on changes, so I just invoke git and commit).

tweetle_beetle(10000) 2 days ago [-]

Seeing as your list is pretty comprehensive, it might be worth having a look at Writemonkey. It's been in development since 2006 - so long before the markdown editor trend in recent years, and probably even pre-dating the distraction-free trend about a decade ago.

The most recent incarnation (v3) fits your two criteria well and its plugin system is very versatile. Sadly cross platform supports only came with version 3 and that means saying goodbye to the small Windows native downloads, but it's not an Electron monstrosity.

Just a long time fan and always feel a bit disappointed that it never comes up in these kind of discussions :)

http://www.writemonkey.com/wm3/

(And in a pinch, away from my own machine, StackEdit is great.)

selykg(10000) 2 days ago [-]

I had a lot of weird issues with Notes. Notably folders wouldn't delete and would randomly come back after deletion.

Just strange stuff like that.

I ended up switching to Bear for notes.

I use iA Writer for writing longer form stuff like documentation and where I need to work with files directly. It's a great app.

tifadg1(10000) 2 days ago [-]

Could someone enlighten me what is the biggest advantage or this and similar tools over libreoffice writer? I've been using it extensively for years for technical documentation, have hundreds of bitrot-free documents, and am extensively using:

* tables (2 or 3 columns depending on type, often using sort by column 1 or column 1+2 to keep relevant information grouped);

* preset formatting for different styles (snippets, commands);

* navigation using ToC (on a sideway navigation pane which is always visible);

* auto-generating anki flashcards from the content with no modifications;

* inserting external media;

I've used different methods to keep a single synchronized copy depending on work tech restrictions, i.e. nfs over ssh, sshfs, vpn via vm. Nowadays working from home I just keep everything locally.

What are the selling points to drop all that and move to something else?

cellularmitosis(4404) 2 days ago [-]

I wasn't aware you could create links from one libreoffice writer document to another?

henrikeh(4439) 2 days ago [-]

How do you generate flashcards from LibreOffice and what do you mean with "no modification"?

wenc(4420) 2 days ago [-]

Folks who are familiar with Zettelkasten:

Would it be correct to say that most of these tools are identical to Wiki software with one exception: the ability to see 'what linked to this'?

scribu(960) 2 days ago [-]

It's not that. Wikipedia, for example, has a separate 'pages that link to this' view.

What makes Roam Research different is 'block addressing', i.e. you can get a reference to a single paragraph and use it in other docs (either referencing it or embedding it).

(and of course a nice UX for editing)

mikro2nd(2411) 2 days ago [-]

So Zim-Wiki with the Backlinks plugin, then. Notes (wikipages) are stored as ordinary text files in a directory structure that is whatever hierarchy you've used in the notes. (I tend not to use hierarchy much for my every-day notes.)

Zim doesn't use Markdown as it's native or input syntax, but can copy text as md if you so wish.

Honestly one of the most underrated, understated tools I've ever come across, and one I would hate to have to live without.

sradman(10000) 2 days ago [-]

> Would it be correct to say that most of these tools are identical to Wiki software...

It would be correct to say that a personal Wiki is a modern way to implement a Zettelkasten system [1]. Zettelkasten (Slip-Box is the literal translation from the German) is a paper-and-pen system where personal notes are written on index cards and stored in a box (like the library card catalogs of yesteryear). The trick to building the knowledge graph is giving each card a unique identifier and having a convention for referencing other cards.

Apple Notes and other similar products are simply missing the ability make links to other notes.

[1] https://en.wikipedia.org/wiki/Zettelkasten

marvinblum(4246) 2 days ago [-]

Hm yes and no. They are quite similar, but I would say a Zettelkasten is focused on connections rather than the 'articles' itself. Cards probably won't contain as much information as a fully fledged article. I wrote about it here: https://emvi.com/blog/luhmanns-zettelkasten-a-productivity-t...

maddyboo(4423) 2 days ago [-]

Neuron is a new Zettelkasten project that shows a lot of promise. The developer is very active and responsive. I like that it is an editor-independent cli tool, with plugins currently for Vim and Emacs.

https://neuron.zettel.page/

https://github.com/srid/neuron

tjstankus(10000) 1 day ago [-]

Thanks for sharing this, I've been thinking about building something like it, now I may not need to.

brunoqc(10000) 2 days ago [-]

Do you know if neuron will ever supports mobile?

app4soft(782) 2 days ago [-]

> Zettlr is powered by Electron,

Uh, I would prefer VNote editor + Viki instead.

[0] https://github.com/tamlok/vnote

[1] https://github.com/tamlok/viki

thomasfl(4093) 2 days ago [-]

«Developing Open Source Software is a Political Act.» damn sure is!

BeetleB(4268) 2 days ago [-]

Anything one does can be construed as a 'political act', making the whole concept pointless.





Historical Discussions: EFF and heavyweight legal team will defend Internet Archive against publishers (June 30, 2020: 599 points)

(599) EFF and heavyweight legal team will defend Internet Archive against publishers

599 points 5 days ago by toomuchtodo in 1035th position

torrentfreak.com | Estimated reading time – 5 minutes | comments | anchor

In March and faced with the chaos caused by the coronavirus pandemic, the Internet Archive (IA) launched its National Emergency Library (NEL)

Built on its existing Open Library, the NEL provided users with unlimited borrowing of more than a million books, something which the IA hoped would help "displaced learners" restricted by quarantine measures.

Publishers Sue Internet Archive

After making a lot of noise in opposition to both the Open and Emergency libraries, publishers Hachette, HarperCollins, John Wiley and Penguin Random House filed a massive copyright infringement lawsuit against the Internet Archive.

Declaring the libraries little more than 'pirate' services that have no right to scan books and lend them out, even in a controlled fashion, the publishers bemoaned the direct threat to their businesses and demanded millions of dollars in statutory damages.

Earlier this month the IA announced the early closure of the NEL, with IA founder Brewster Kahle calling for an end to litigation and the start of cooperation. There are no public signs of either. Indeed, the opposing sides are preparing for action.

EFF and Attorneys Team Up to Defend IA

Last evening the EFF announced that it is joining forces with California-based law firm Durie Tangri to defend the Internet Archive against a lawsuit which they say is a threat to IA's Controlled Digital Lending (CDL) program.

The CDL program allows people to check out scanned copies of books for which the IA and its partners can produce physically-owned copies. The publishers clearly have a major problem with the system but according to IA and EFF, the service is no different from that offered by other libraries.

"EFF is proud to stand with the Archive and protect this important public service," says EFF Legal Director Corynne McSherry.

"Controlled digital lending helps get books to teachers, children and the general public at a time when that is more needed and more difficult than ever. It is no threat to any publisher's bottom line."

Durie Tangri partner Joe Gratz agrees, noting that there is no issue with the Internet Archive lending books to one patron at a time.

"That's what libraries have done for centuries, and we're proud to represent Internet Archive in standing up for the rights of libraries in the digital age," he adds.

With Gratz on the team, the IA and EFF are clearly taking matters seriously. His profile states that he's as "comfortable on his feet in court as he is hashing over source code with a group of engineers", adding that he represented Google in the Google Book Search copyright cases.

Also on the team, according to the lawsuit docket, is Harvard Law School graduate Adi Kamdar, who was an affiliate with the Berkman Klein Center for Internet & Society. Before that, Kamdar was an EFF activist advocating on issues of privacy, speech, and intellectual property policy.

Publishers Have Brought in the Big Guns Too

The docket reveals some prominent veterans acting for the publishers too.

Matthew Jan Oppenheim, for example, served as lead counsel in the record-breaking $1 billion jury verdict against Cox Communications for the music industry, and the $34 million verdict against Book Dog Books for the publishing industry.

A former partner at the music industry law firm Jenner & Block, Oppenheim previously worked at the RIAA, handling landmark cases against Napster and Grokster.

Meredith Santana represented Miley Cyrus in the "We Can't Stop' copyright infringement lawsuit while Linda Steinman represents and counsels content providers on how to protect their work from "challenges ranging from aggregators to ad blockers."

War or Peace – Too Early To Say

It's still not too late for the parties to reach a negotiated settlement but given the legal forces now massing on both sides, that is becoming a more distant prospect.

The stakes are high for all parties and beyond, with either side coming out on top having the potential to affect how the public can consume scanned and borrowed content in the future.




All Comments: [-] | anchor

loughnane(10000) 5 days ago [-]

It's well established at this point that IA took a big misstep here and is too important and singular an entity to run afoul of current laws.

To those who think IA is wrong not just tactically but morally on the basis of 'violating the law', I'd like you to consider the following:

Unjust laws exist.

- Shall we be content to obey them?

- Shall we endeavor to amend them, and obey them until we have succeeded?

- Shall we transgress them at once?

aaron695(1174) 5 days ago [-]

You missed the step a lot of us believe this law is moral.

So this isn't about not following unmoral laws.

More so, on top of this, many people also think it's ok in some circumstance to break moral laws.

If someone steals food they need, then many people think that's ok. If someone breaks copyright when they wouldn't have normally payed for the product, then some people also think this is ok.

But I think this is none of the above.

This is a food bank openly stealing food from Walmart. Heck, if some anarchist group broke into Walmart and stole food, even that would have more support. But when they say what they are doing is ok and do it openly in front of police, that's when they lose me.

bzb3(10000) 5 days ago [-]

Before we can answer those questions we have to decide whether copyright is unjust or not.

timwaagh(10000) 5 days ago [-]

the reason they are doing this is the fact they are going to lose. libraries are a relic of the past too in my opinion.

aahhahahaaa(10000) 5 days ago [-]

well this is the bleakest thing I've read today

mellosouls(4234) 5 days ago [-]

For anybody interested in the IA's claimed ethical and legal justifications, they summarise and link to them at the following page:

http://blog.archive.org/2020/03/30/internet-archive-responds...

AmericanChopper(10000) 5 days ago [-]

I can't see a single legal justification put forward here or anywhere else. They talk extensively about CDL, but this lawsuit isn't about CDL, it's about un-CDL. Their moral justification is quite straightforward (though not without its own issues), but the legal system doesn't exist to enforce a system of morals, it exists to enforce a system of laws.

I can't imagine what their legal strategy could possibly be...

snapetom(10000) 5 days ago [-]

Wow, that still seems rather weak and reckless.

First they describe what Controlled Digital Lending is - one physical copy == one lend. They cite legal experts that agree. That's fine, no one's disagreeing there.

Then they go on with this:

'Additionally, fair use ultimately asks, 'whether the copyright law's goal of promoting the Progress of Science and useful Arts would be better served by allowing the use than by preventing it.' In this case we believe it would be. Controlled digital lending as we conceive it is...'

So basically they went off and created the National Emergency Library based on their own interpretation of Fair Use. You better be ready for a huge fight if you're going to that. I don't know how they couldn't have expected this.

ogennadi(4288) 5 days ago [-]

> Yes, we've had authors opt out. We anticipated that would happen as well; in fact, we launched with clear instructions on how to opt out because we understand that authors and creators have been impacted by the same global pandemic that has shuttered libraries and left students without access to print books. Our takedowns are completed quickly and the submitter is notified via email.

m0zg(4431) 5 days ago [-]

Y'all are focusing on the wrong thing. This is categorically not about 'library' or whatever they are being sued for right now. That's just a smokescreen.

It's a political takedown, plain and simple. It's much harder to gaslight the public when an archive of your taken down tweets, stealth edited or censored articles, etc is a single hyperlink away from being re-surfaced.

TouchyJoe(10000) 5 days ago [-]

I sympathize with the desire of average joes to take down footage of their smelling hair and rubbing people the wrong way. We need the internet archive.org. Some of the best website are only to be found there.

TheHeretic12(10000) 5 days ago [-]

I recognize some of the names here. This will be a case to watch, which ever way it goes the constructions allowed here will percolate through the law for years to come. This case hinges on the exact wording of the law that authorizes the National Emergency Library. IIRC, it basically gives copyright impunity during a declared national emergency. As far as I am aware, the coronavirus declared national emergency is still in effect. IA loses by attrition, is my guess. Many rounds of 'Preliminary' injunctions and orders will stop thier income streams, then they die.

tgsovlerkhgsel(10000) 5 days ago [-]

Isn't the income stream mostly donations, and is getting publicity from being targeted by unpopular entities a great way to get more donations?

WaltPurvis(4417) 5 days ago [-]

> This case hinges on the exact wording of the law that authorizes the National Emergency Library. IIRC, it basically gives copyright impunity during a declared national emergency.

There is no such law. What on earth are you talking about?

Vespasian(4037) 5 days ago [-]

What law does explicitly allow the NEL?

If I recall correctly, the Internet Archive decided unilaterally that they would do this due to many libraries being closed.

I (not in the US) do not remember any changes to the copyright act being passed to allow this.

Edit to add: They offered this globally and therefore disregarded any laws in countries which have already settled this issue. (Hint: to my knowledge not in a way that favors their interpretation)

jjcon(10000) 5 days ago [-]

EFF is on the wrong side of both the law and common sense here and they will lose no matter how much they pay their legal team. Internet archive didn't just steal from 'publishers' they are also stealing from authors big and small. You don't get to give out free digital copies of books to people without permission and expect to get away with it.

This exact digital/physical equivalence idea has already been through the courts (for movies/television) and it failed miserably:

https://en.m.wikipedia.org/wiki/VidAngel

> In June 2019, a jury ordered VidAngel to pay over $62 million in damages for copyright violations.

sp332(344) 5 days ago [-]

The lawsuit is only asking for damages for 127 books, for a theoretical maximum of $19 million according to https://www.vox.com/platform/amp/2020/6/23/21293875/internet... And with books from the last 5 years excluded, and the opt-out, and the fact they were partnering with multiple other libraries and could probably include their copies, the damages the authors could show will probably be relatively small.

gpm(10000) 5 days ago [-]

Even if you broke the law you don't just role over and give the other side whatever they want. They're inevitable going to ask for more compensation than they are entitled to (why wouldn't they), and you need to push back to make the penalty fair. That's just how our legal system is set up to work.

The EFF is on the right side of the law the same way a lawyer for a criminal is. The lawyer (eff) is fulfilling a necessary function of the judicial system even when the party they are representing is in the wrong.

The EFF has a keen interest in this case because the publishers will be doubt use this case to push for copyright maximalist interpretations of the law, and letting those interpretations become case law unchallenged is bad for society.

Edit: At a glance the vid angel lawsuit you quote is about circumventing DRM, something that is not relevant to this case since the physical books they scan have no DRM.

elliekelly(2133) 5 days ago [-]

It isn't always about winning. Most of the time it's more about not losing too much money. EFF's legal team will help IA mitigate the damage wherever possible so that IA will have a shot at continuing to exist.

kstenerud(4286) 5 days ago [-]

'EFF is on the wrong side of both the law and common sense here'

Are they? We can armchair critique all we want, but I find it doubtful that an organization with so much litigation and advocacy experience under their belt would naively throw away a ton of money on a losing battle.

AnthonyMouse(4340) 5 days ago [-]

> EFF is on the wrong side of both the law and common sense here and they will lose no matter how much they pay their legal team.

I don't understand how people have such certainty about this.

You have a situation where there are libraries who have bought millions of books from authors/publishers. The authors have been paid. Then, there is a pandemic and the libraries are closed, so the public has lost access to all the books they've paid for.

Giving them back access to the books they've paid for is obviously wrong? They're definitely going to lose?

It would be one thing if they had lent out more books than existed in all of the closed libraries, but is there any evidence that this is the case? That seems quite implausible, considering the massive number of books that have been paid for in all of those libraries.

agustif(10000) 5 days ago [-]

I won a civil law case against the MAFIAA being the owner of a site where links to download movies and tv shows as user-generated content.

I won, but I was in Spain, not the US.

HeadsUpHigh(10000) 5 days ago [-]

If by common sense you mean ethically, then a lot of people would disagree. This is a worthy battle and future case law.

ocdtrekkie(2962) 5 days ago [-]

Indeed. IA's defense is mostly about controlled digital lending, while the National Emergency Library was definitely uncontrolled digital lending, and really has no good precedent to base upon. I think it's unfortunate that this case is likely to blowback on CDL as a concept, even though the issue was uncontrolled lending.

maxlybbert(10000) 5 days ago [-]

I don't remember the outcome of the particular case mentioned in https://www.theregister.com/2005/12/06/eff_needs_to_die/ , but I know their track record isn't as good as this article's headline implies.

wolco(3904) 5 days ago [-]

Fair use is a term and redefining is a big hill to climb but a worthly one.

All copyrights expire and all works have fair use clauses. It's just a matter of when and what country is issuing the copyright.

You can give away free digital books. Project Gutenburg has been doing it for years. Even if the author's family doesn't like it. It's not stealing either. It's simply waiting for the copyright to expire.

In this case they are using borrowing like a libruary as a defensive with covid hardship thrown in to justify removing a limit on the amount of books they can lend.

gjsman-1000(4337) 5 days ago [-]

Agreed. However, I would expect the EFF to (possibly) be good negotiators. They might be able to make a settlement that keeps the IA's website open while satisfying the publishers - or at least making the lawsuit complicated enough that the publishers won't keep going.

Without this representation, the publishers could almost certainly bankrupt the IA, even if they are given the minimum infringement fine of $200/ea.

tzs(3100) 5 days ago [-]

I may be misunderstanding this case, but the impression I get is that they are not defending the unlimited lending that IA did as part of their 'National Emergency Library' (NEL). They are just defending the earlier 'Controlled Digital Lending' (CDL) program where the number of copies lent out at any one time was limited to the number of physical copies in the physical libraries [1].

The lawsuit is going after both NEL and CDL. The defense is trying to save CDL.

I don't think VidAngel is necessarily relevant here because VidAngel was making derivative works.

[1] If someone checked out a physical copy, did they reduce the number of digital copies that could be out until the physical copy was returned?

devit(10000) 5 days ago [-]

If the books were not being sold as digital copies, maybe they could invoke fair use due to the pandemic making other ways to access the books impossible.

That said, the best avenue would seem to be a PR campaign with a call for a boycott and public shaming of the publishers (Hachette, Penguin Random House, John Wiley & Sons, and HarperCollins) that are putting their profit ahead of public access to the books.

In fact, it seems pretty surprising that the publishers would risk such an hit to their reputation given that the Internet Archive probably doesn't have much money to give them as compensation anyway.

dehrmann(3388) 5 days ago [-]

Remind me not to donate to EFF. This money and time would be better spent on fighting EARN IT.

The Internet Archive's best bet is to settle and apologize very publicly.

mtgx(176) 5 days ago [-]

> You don't get to give out free digital copies of books to people without permission and expect to get away with it.

As hard as big content producers try to make copyright synonym to 'property', it's not and it never was. Why? Because ideas can't be copyrighted and all ideas are remixed. Nothing is 100% original. NOTHING.

Not to mention copyright used to apply for only 14 years. But corporations, which have bought just about everything else in the government, have also bought copyright laws that now last life + 70 years. And they keep trying to extend it all the time.

Relevant: https://www.everythingisaremix.info/watch-the-series

garmaine(4448) 5 days ago [-]

Well EFF won't be getting any more donations from me, just like I cut off IA when they started this.

Don't ask for money to fight a legal battle where you are so obviously in the (legal) wrong.

zucker42(4445) 5 days ago [-]

The EFF could be just trying to limit the damage. Indeed, the article mainly focuses on CDL and not the unlimited lending of the NEL. Even a partial victory could protect the right to digitize old books not out of copyright.

myself248(4425) 5 days ago [-]

How do you go about challenging a corrupt and unjust law, then?

hyko(10000) 5 days ago [-]

How do we back up the Internet Archive? If they lose this case then it will be like losing The Great Library of Alexandria.

This is our culture. This is our heritage!

Edited to add: when I say "back up", I mean preserve the data and the archival mission (minus the legal quagmires).

The technical challenges can be solved, and we should do this before it's too late. It seems there was a previous effort, but it has lost momentum: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

mantap(10000) 5 days ago [-]

It's possible that over the long term it could be backed up via torrents. However, in the short term the only way to save the archive is for IA to sell it. I hope they swallow their pride and do so.

_sbrk(3923) 5 days ago [-]

This is abuse of copyright.

IA did bad, and should not be rewarded for this behavior.

mellosouls(4234) 5 days ago [-]

There are many other digital archives, and that is how the 'single point of failure' problem you envisage is solved.

The Internet Archive is AFAIK the biggest, and the first - the important thing is for others to continue its good work (many have done this for years) - and not for us to rely too much on it or any other single archiving initiative.

JustSomeNobody(4205) 5 days ago [-]

Might find an answer in:

https://www.reddit.com/r/DataHoarder/

rakoo(10000) 5 days ago [-]

This is how you backup the Internet Archive: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

38e7eeeh(10000) 5 days ago [-]

Seriously, how will people carry on their lives without the IA? Unfathomable.

mindslight(10000) 5 days ago [-]

I'd love for the Internet Archive to prevail here, but honestly it was quite boneheaded to do this legal experiment under the same corporate umbrella as their archival work. At this point they should proceed with a damage mitigation strategy of selling off their servers and storage to a second entity at fair market value (maybe 'Archive Cloud'), and renting continued access. This way even if the IA organization is bankrupted, the archive itself will still remain intact - the archived data isn't under IA copyright and thus wouldn't be part of the bankruptcy estate.

Vespasian(4037) 5 days ago [-]

I still wonder what their plan was in anticipation of this confrontation.

Worst Case: Chirping (Not seeing this coming at all)

Naive Case: 'Surely they will come to recognize our 'superior' 'moral' arguments and cease their evil ways'

Historically ignorant case: 'The people will stand up and join us. We are invincible' (aka every rebel ever...most fail)

enriquto(10000) 5 days ago [-]

> it was quite boneheaded to do this legal experiment under the same corporate umbrella as their archival work.

I disagree. This is a wickedly good idea, and a very good hill to fight on. It is not a 'legal experiment' but a major battle against evil people. Everybody loves the Internet Archive, it is our sacred castle. If a decisive battle is to be won against the publishing parasites, it may be likely this one, and we have to be all on the same side! After Aaron Swartz, the book parasites have never been so potentially hated by everybody. We must go all-in.

The error is on the side of the book parasites for having decided to fight against an institution that is so loved by mankind. They cannot but lose the battle (socially), regardless of what the short-time legal outcome is.

yyyk(4447) 5 days ago [-]

I recall from the comments here and in Ars Technica that there was a significant debate whether IA could do that or whether this will be considered bankruptcy fraud or whether the publishers could still claw back the servers despite the separation.

There's another last ditch option though - IIRC, IA has copies in US, Canada, Netherlands and Egypt. Let's say that some government where an IA server is hosted is convinced to nationalize the server. They have legal authority - there's much legal latitude for acting in the name of national security, and there's an obvious case to be made here: if we consider disinformation to be a security threat than a server hosting reliable and trusted internal history is an obvious asset (e.g. [0]).

Such an act would easily override any legal attempts by the publishers, and can be pushed for even after a conviction and bankruptcy and without involving the IA foundation at all.

The danger is that post-nationalization the information would not be considered reliable - this can be ameliorated if the servers are immediately rerendered to a private party, but it's enough of a downside to make this a last option to push for. That said, I suggest keeping this option in mind.

[0] https://theconversation.com/dominic-cummings-how-the-interne...

Gaelan(4425) 5 days ago [-]

[I typed this up as a reply to a comment that has now been flagged. I think it's a conversation worth having, so I'm putting it at the top level instead.] I'd argue that 'intellectual property' is fundamentally different from physical property, and can't really be 'stolen' per se.

If I steal your car, I have a car and you don't have a car. If I 'steal' your book, we both have the book. I'm better off, and you're no worse off. In an 'ideal' society, the free sharing of knowledge would be not just allowed, but encouraged.

I imagine there's been measurable harm to our society's cultural and scientific advancement because copyright forces us to recreate others' work instead of building off of it.

But what does that world look like? Quite frankly, I'm not sure. Of course, we can't just abolish copyright with no replacement; it's certainly necessary to reward the creation of this work somehow.

One thing I've thought about is a sort of 'crowdfunded patronage' model, where the creator of a work declares upfront how much money they want to make off of it, and releases it into the public domain once they've received that much money.

Another possibility is drastically reducing the term of copyright to 20 years or so. That'd still give 20 years to exploit the work—and I imagine most copyrighted works make most of their money in the first 20 years—while allowing others to reuse the work much more quickly.

ksdale(10000) 5 days ago [-]

While I agree in principle that intellectual property is somehow fundamentally different from physical property, in the case of the 'stolen' book, the creator is not worse off only if the 'thief' wouldn't have purchased the book otherwise.

It's certainly true that the creator isn't out of pocket per person who reads the book, but that's not necessarily the same as 'no worse off.'

markvdb(4397) 5 days ago [-]

There's a real cost to government and society enforcing the copyright monopoly for the copyright owner. So why not have the copyright holder pay for upholding copyright?

Imagine differential pricing for enforcing copyright on a work depending on parameters:

- grow with every passing year - change with the level of protection required

  - zero for public domain and free (BSD/GPL/permissive CC style licensing)
  - symbolical amounts for CC no derivatives or non-commercial licensing
  - fixed amount plus percentage of revenue for other, more traditional licensing
Rebelgecko(4404) 5 days ago [-]

>If I 'steal' your book, we both have the book. I'm better off, and you're no worse off.

That's only true if 0% of copyright infringers would've paid for the book (either directly or via checking it out from a library that pays publishers per-borrow). That may be true for some individuals, but there's no way that it's true in the aggregate. There are some cases where copyright infringement might help with the bottom line (like if someone torrents a band's CD which inspires them to attend expensive concerts), but I for books I think that effect is drastically overpowered by the # of people that just don't want to pay.

Thorentis(4435) 5 days ago [-]

Most people support copyright because they think content creators should be rewarded for their work. Unfortunately, the copyright laws that we have do much more than that, and that's a problem.

I support minimalist copyright laws, where for a set period of time (shorter than current copyright) an author/creator is the only person who can profit from work. For instance, an author publishes a book, and for 10 years is the only person able to sell copies, sell the manuscript to a publisher, etc.

But after 10 years, that book is essentially public knowledge. It's been out there, maybe some people have memorised portions of it, it's been read in a library far more times than it's been purchased, etc. At that point, I see no societal benefit to maintaining copyright. My own moral framework does not require an author to be rewarded indefinitely for their creative work (imagine if a builder and their estate received royalties every time somebody entered a building they constructed for 70 years? Absurd.)

I also have a huge problem with creative 'rights'. Take Harry Potter as an example. Everybody knows about Harry Potter, anybody could tell you the story, some people could probably reproduce a very rough abridged version from memory. It is part of our culture, cannon, consciousness, whatever. It's popular.

Now, if I want to make a Harry Potter game, or write a play, or do a live dramatisation, etc, I owe JK royalties. That to me is not fair. She has been rewarded for her work many times over. The ideas and the characters are now in everybody's minds. We know them. If I want to adapt them or discuss them through a creative work, I should be free to do so.

I often use the example of Ancient Greece to explain this. Almost all the written stories and poems we have from that time are simply written versions of stories told aurally. The Illyiyd as we have it is just one version that somebody wrote down of possibly hundreds of spoken versions that existed at the time. The Greek plays were part of their culture and were freely shared. Anybody could build on them, change them, and referencing them in new works was an honor to the original creator (if they were even known). It is very sad that the West has now favored infinite profit (I.E. copyright that lasts a lifetime) over the spreading of ideas and the sharing of common cultural references freely.

ALittleLight(4365) 5 days ago [-]

I am made worse off if you steal a copy of my book though. Suppose I sell the book for X dollars and the probability that any person without my book will buy it is Y. People who don't have my book have a value to me, specifically, YX dollars.

When you steal my book, you directly cost me YX dollars, which is probably an infinitesimal amount, but if everyone was allowed to freely copy the book, then it would be approximately 100% of the money to be made by writing a book.

Of course there are complications, like, you might buy it after pirating it, or, you weren't going to buy it anyway, or maybe you'll recommend it to a friend, but I think those considerations should be evaluated by the people with skin in the game - i.e the people who will make or lose money if they get it wrong.

isodude(10000) 5 days ago [-]

I think the dilemma is more important than how we do the copyright laws right now. Sharing is always better as a model, it demands less restrictions for one, less oversight. I don't have a number how much the world population have grown since the laws were inacted, but it's at least in the 10 fold? What works on a 'small' number of people might not scale. We as programmers know that. So what gives if everyone wants to be a millionare, which in a sense promotes that ignoring values/others around is a good idea if you yourself get a good life. In an ironic twist the most shared idea is the one that anyone could be rich, and that it is a good thing to be much richer than those around you.

So, what is a good life? I don't see living in a mansion or castle as a good way of life if measured that way. I rather see it how many people I can help along during my stay here, that's my measurement. Of course I absolutely hate copyright and the notion of IP.

Maybe the laws and lengths are good for the life views that people have right now. But maybe there's other incentives than the Right to the work, maybe good work could be measured in an other sense and rewarded more directly in your community. Globalisation when done badly is so bad for us since the ones in that bubble optimize for their own profit instead of the greater good (which is the best scenario in long sight).

Maybe I'm weird, but I am soo tired of the idea that we should only focus on ourselves all the time. Get good ideas out that will solve big problem, reward those that produce those ideas and those that want to do the miles. Less of the, look I changed color of this thing and now I got a patent on it and I will sue you. Reward ideas that makes sharing easier. We as a planet just don't have the time if we want to make it as a whole.

swayvil(4324) 5 days ago [-]

The only relevant question here is, 'which benefits society more?'

The ip serves the author by paying his rent

The ip serves society by informing and empowering its members.

thrownaway954(4318) 5 days ago [-]

'If I steal your car, I have a car and you don't have a car. If I 'steal' your book, we both have the book. I'm better off, and you're no worse off. In an 'ideal' society, the free sharing of knowledge would be not just allowed, but encouraged.'

you are so wrong in this mentality. the bottom line is, that someone took the time and energy to write and publish that book to make money, so that person should be paid. with your mentality, it can be argued the same as downloading a movie or a song or an application. now... aren't all those things illegal and a form of stealing??? so why is downloading a book any different?

i really think that alot of people just don't think things though before they open their mouth and this is one time where you should have thought through your answer instead of jumping on the 'knowledge should be free' bandwagon.

izacus(10000) 5 days ago [-]

> But what does that world look like? Quite frankly, I'm not sure. Of course, we can't just abolish copyright with no replacement; it's certainly necessary to reward the creation of this work somehow.

Why can't we? Wouldn't even that state be better than the current one, where publishers outright steal (in true meaning of the world) revenue from YouTube content creators because there might be a sound that's similar to some song written 30+ years ago?

The current state is so one-sided and abusive that it's not obvious that no-copyright world is worse. Heck, down this same thread we have people calling people who put books on the internet for everyone to read 'criminals'. How messed up is that?

WalterBright(4355) 5 days ago [-]

> Of course, we can't just abolish copyright with no replacement; it's certainly necessary to reward the creation of this work somehow.

You say 'of course', but let's examine that. Consider the free software movement. I work full time on D, and anyone can use D for any purpose they want to and don't owe me anything. Just like all the other open source software out there.

And yet, OSS is thriving and is often best-of-breed.

jjcon(10000) 5 days ago [-]

> I'm better off, and you're no worse off. In an 'ideal' society, the free sharing of knowledge would be not just allowed, but encouraged.

I agree that theft of digital goods is not equivalent to physical goods but abolition of copyright does not follow from that fact nor does it follow that theft of intellectual property is not possible.

I also agree this is an important conversation to have. The whole idea around copyright is to encourage the creation of intellectual property. If there is a fundamentally better way to do it that idea should flourish but I have yet to hear a realistic alternative.

> I imagine there's been measurable harm to our society's cultural and scientific advancement because copyright forces us to recreate others' work instead of building off of it.

I would argue that the damage from having no copyright protections would be unquestionably worse than the trade offs they bring. People would of course still create, but so many works would simply not exist if there was no way to protect intellectual property.

Certainly there are minutiae of copyright that have gone too far in protections but that doesn't mean the system itself should be done away with. The modern world created this system knowing full well there were trade offs.

smichel17(10000) 5 days ago [-]

One thing I've thought about is a sort of 'crowdfunded patronage' model, where the creator of a work declares upfront how much money they want to make off of it, and releases it into the public domain once they've received that much money.

That's a ransom model. It's one of several [existing mechanisms] for funding free/libre/open (FLO) works... none of which appreciably move the needle on making a living off them.

I've spent a lot of time talking about Snowdrift.coop on HN recently so I'll skip the pitch this time and just link our [wiki], which has a suggested reading order front-and-center; I'm happy to join in follow-up discussion.

[existing mechanisms]: https://wiki.snowdrift.coop/about/existing-mechanisms#fn2

[wiki]: https://wiki.snowdrift.coop/

TheOtherHobbes(4440) 5 days ago [-]

If you steal my book, I don't get paid as much for the work I put into the book. So I am very clearly and obviously worse off.

I find it really, really strange that some people find this hard to understand.

And your 'crowdfunded patronage' model is still copyright, it's just copyright with a different revenue model.

Jasper_(3676) 5 days ago [-]

> If I steal your car, I have a car and you don't have a car. If I 'steal' your book, we both have the book. I'm better off, and you're no worse off. In an 'ideal' society, the free sharing of knowledge would be not just allowed, but encouraged.

Up until now, the value of a book has been set by the author, and reinforced by scarcity. Entire large economic models and systems have been built by tying the two together. What is supply and demand, when supply is infinite?

Basically, the author puts in some amount of thought and labor writing a book, and they get rewarded by a purchase. Capitalism is heavily based on this: 'Labor results in production, production is rewarded by consumers, because they need to part with some money to benefit for the end result'. The last item is no longer true, which has ripple effects all the way up the chain.

No wonder people want to bring back the economic models and systems of capitalism they're familiar with, by enforcing scarcity -- thousands of years of societal and economic development might be upended by new technology.

Libraries have always been a thorn in the side of capitalism, but they've usually been constrained and limited enough that they aren't a real threat. With the ease of digital copying, the seams are starting to tear much faster now.

Some want to dismantle copyright but keep the rest of the capitalist system in place, untouched, which I don't really see as realistic. I think the two are much more closely linked than you might imagine. If we're going to be overhauling the whole thing, we need to have systems in place for the huge societal impact it will have.

You fill in the rest of the dots for what new systems one might prefer over capitalism, and charting the course for what comes next, that can support people. At the very least, capitalism might just be on its way out.

pitaj(4429) 5 days ago [-]

> I imagine there's been measurable harm to our society's cultural and scientific advancement because copyright forces us to recreate others' work instead of building off of it.

Part of that harm is in creating monopolistic entities like Youtube because only big players can take on big media. If Youtube didn't have contentID they'd be sued into the ground by the publishers.

> But what does that world look like? Quite frankly, I'm not sure. Of course, we can't just abolish copyright with no replacement; it's certainly necessary to reward the creation of this work somehow.

This is, in many ways, a world in which we already live due to the proliferation of piracy. There's still plenty of monetization opportunities:

- Art commissions

- Patreon

- Ad-sponsored serial content

- Subscription models

- Live shows

xhkkffbf(10000) 5 days ago [-]

What if I just borrow your car when you're sleeping? You lose nothing.

What if I break into your house and use your TV while you're at work? You lose nothing.

What if I set up a tent on your lawn and take it down whenever you want to play frisbee? You lose nothing.

What if I grab some girl's butt at a dance club? She loses nothing.

As you can see, requiring an act to explicitly deprive someone of property is a slippery slope. There are dozens of examples of 'borrowing' that just aren't done in polite society because the owner doesn't want them to be done.

The same is even more true here. Piracy hurts the author's ability to sell books. Free copies just have that effect. Every normal human understands it. Only a few computer people like to pretend that their 'borrowing' is somehow magic.

Piracy destroys artists.

int_19h(10000) 1 day ago [-]

A simple approach is to apply the 'intellectual property' analogy consistently - and that means taxing it like we tax real estate.

You could have a short grace period - say, a couple of years - with no tax at all, to accommodate individuals and startups exploring a market etc. After that require explicit registration for continued protection, or else the work automatically falls into public domain. And anything on the registry is taxed.

Furthermore, said tax could be made progressive, to discourage copyright hoarding, and impose a natural term limit. More valuable works would have longer-lasting copyrights with such an arrangement, but there'd always be some point past which it's simply no longer profitable. No abandonware, either - if the tax isn't paid, it becomes PD.

dctoedt(447) 5 days ago [-]

You're channeling Thomas Jefferson, who himself channeled a Zoroastrian idea. Jefferson said: '... no one possesses the less because everyone possesses the whole of it. He who receives an idea from me receives [it] without lessening [me], as he who lights his [candle] at mine receives light without darkening me.' [0]

The flaw in this concept is the free-rider problem:

1. Not all new IP can be created as a side project without funding.

Suppose that I have to come up with an investment of $X million to pay for the goods and services needed to develop something.

(Even assigning existing employees to do the work is an investment, because the money used to pay them presumably could be used for something else — i.e., investing in the new thing represents opportunity cost.)

Suppose also that the resulting IP is easily copied once the new product or service is out in the open.

Without some kind of legal monopoly under IP law, copiers — who aren't burdened by my development costs — can quickly run the market price down to marginal cost.

2. A marginal-cost market price means that I can't charge enough to be able to recoup my investment while still staying competitive.

That, in turn, means that I'm unlikely to be willing (or even able) to fund the investment needed to develop the IP.

3. That's why the U.S. Constitution, article I, section 8, clause 8, allows Congress to create IP rights for limited times.

[0] https://uh.edu/engines/epi792.htm

axk(3764) 5 days ago [-]

Did You Say "Intellectual Property"? It's a Seductive Mirage

https://www.gnu.org/philosophy/not-ipr.html

bluesign(10000) 5 days ago [-]

I think another analogy can be better.

Imagine I want to fly somewhere, plane is half empty, I am going there and sitting. Don't have ticket. (Imagining no enforcement)

Should I? Eventually this plane will fly there, my effect on cost is ignorable, I will be better, airline will not be worse.

M5x7wI3CmbEem10(4414) 5 days ago [-]

weren't there unintended consequences in the pharmaceutical industry for shorter patent windows? they would modify a drug ever so slightly, but enough to patent it again. I could imagine a similar consequence for shorter copyright windows.





Historical Discussions: Apple Lightning (July 01, 2020: 596 points)

(596) Apple Lightning

596 points 4 days ago by captn3m0 in 1260th position

nyansatan.github.io | Estimated reading time – 15 minutes | comments | anchor

Apple Lightning

Created on 1.7.20

Here's my little article about (almost) everything I know about Apple Lightning and related technologies: Tristar, Hydra, HiFive, SDQ, IDBUS and etc. But first a tiny warning...

Read this article on your own risk! The information in this artcile is based on a lot of AppleInternal materials (leaked datasheets, schematics, source codes) I read in a diagonal direction. And of course on my own research too. I have to warn you, the reader, that I have never done such a research before. Thus, this write-up might use incorrect or just weird terms and turn out partially or completely wrong!

Before going deeper, let's briefly sort out the terms:

What's Lightning?

Lightning - is a digital interface used in most of the Apple's iOS devices since late 2012. Replaced the old 30-pin connector

You can see the female port pinout on the picture above and the connector pinout on the picture below: Please pay attention to the fact that in the connector, pins on both sides of connector aren't wired in exact same order. Thus, a host device have to detect orientation of a cable before doing anything else

Though it's not always applicable. Many Lightning accessories I've played with have mirrored pinouts in their connectors

What's HiFive?

HiFive - is Lightning slave, i.e. a male connector. It contains a logical element as well - that chip is known as SN2025/BQ2025

What're SDQ and IDBUS?

These 2 terms are often referred as kind of synonyms. For convinience, I'll only use term IDBUS from now on, as it seems more correct to me (and that's how this technology called in the THS7383 datasheet)

So, IDBUS - is a digital protocol used for negotiations between Tristar and HiFive. Very similar to Onewire protocol

Now we can play

Let's sniff the negotiations between Tristar and HiFive. Take a logic analyzer, a Lightning male-to-female passthrough breakout board, some accessory (normal Lightning to USB cable would fit just fine) and of course some device with Lightning port

First connect logic analyzer's channels to both ID lines of the breakout (pins 4 and 8) and connect the breakout to the device, but do not connect the accessory just yet: Right after that start sampling (any rate from 2 MHz and up should be fine). You'll see something like this: As you can see, Tristar polls each ID line by rotation - one after another. But since we didn't connect any accessory, the polling obviously fails. At some point the device will grow tired of this endless stream of failures and stop it. Meanwhile let's examine what exactly happens while polling: First, we can see a long interval (~1.1 milliseconds) when the level is just high and nothing else is happening: Apparently that time is used to charge internal HiFive's capacitor - the energy from it will be then used to power-up its internal logic chips

What happens next is far more interesting: Obviously, that's some data flowing. But how to interpret it? How to decode it? Let's virtually split it to almost the least least significant parts - to something that I call words: So basically a word is a combination of fall-rise-fall:

  • Meaningful Stage - time interval taken by this stage defines meaning of a word
  • Recovery Stage - time interval which is apparently required for processing the Meaningful Stage on recieving side and/or preparing the next word on sending stage
Here is a table of known word types with their time intervals for both stages we discussed above (all units are in microseconds):
Meaningful Recovery
Word Min Typ Max Min Typ
BREAK 12 14 16 2.5 4.5
WAKE 22 24 27 1100?
ZERO 6 7 8 3
ONE 1 1.7 2.5 8.5
ZERO with STOP* 6 7 8 16
ONE with STOP* 1 1.7 2.5 21
* - STOP is used when it's a last bit in a byte

Using the above table we can now build a simple decoder of the protocol: As you can see, the first word a host sends is BREAK - when Tristar wants to send a new request, it always starts with it. Then comes a data stage. Please pay attention to the fact that last (8th) bit of a byte has longer Recovery Stage. When a data stage is over, a host sends another BREAK. Then a slave must send a reply (after at least a 2.5 us delay - see the table). Tristar will wait for around 2.2 ms for a reply. If it's not issued in this time interval, Tristar will try to poll another ID line

Now let's examine the data stage on the example above - 0x74 0x00 0x02 0x1f:

  • 0x74 - request/response type. Always even for request, always odd for response (request type + 1)
  • 0x00 0x02 - actual data. Can be empty
  • 0x1f - CRC8 of both the request type byte and the whole data (polynomial - 0x31, initial value - 0xff)
Let's connect some accessory to our setup and see what happens. I'll use Apple's original Lightning to USB cable: And here is what appears on IDBUS after a 0x74 request: HiFive replied! And if you scroll further you'll see a lot of other request/response pairs: Some requests do not need a response though:

Interpreting IDBUS requests and responses

The most important IDBUS request is 0x74 - it is used for two purposes: to tell HiFive enable full current (in case that's supported by an accessory) and to ask it about pin configuration the cable supports and some other metadata

Not too much is known about how response 0x75's data is encoded. But some bits were available in a certain old Tristar datasheet:

First byte of 0x75 response data
7 6 5 4 3 2 1 0
ACCx Dx DATA[43:40]
ACCx configuration when ID is found on ID0
ACCx[1:0] ACC1 ACC2 HOST_RESET
00 Hi-Z (IDBUS) Hi-Z Hi-Z
01 UART1_RX UART1_TX Hi-Z
10 JTAG_DIO JTAG_CLK Hi-Z
11 Hi-Z Hi-Z HIGH
ACCx configuration when ID is found on ID1
ACCx[1:0] ACC1 ACC2 HOST_RESET
00 Hi-Z Hi-Z (IDBUS) Hi-Z
01 UART1_RX UART1_TX Hi-Z
10 JTAG_DIO JTAG_CLK Hi-Z
11 Hi-Z Hi-Z HIGH
Dx configuration when ID is found on ID0
Dx[1:0] DP1 DN1 DP2 DN2
00 Hi-Z Hi-Z Hi-Z Hi-Z
01 USB0_DP USB0_DN Hi-Z Hi-Z
10 USB0_DP USB0_DN UART1_TX UART1_RX
11 Hi-Z Hi-Z Hi-Z Hi-Z
Dx configuration when ID is found on ID1
Dx[1:0] DP1 DN1 DP2 DN2
00 Hi-Z Hi-Z Hi-Z Hi-Z
01 Hi-Z Hi-Z USB0_DP USB0_DN
10 USB0_DP USB0_DN UART1_TX UART1_RX
11 Hi-Z Hi-Z Hi-Z Hi-Z
Using the tables above let's decode our cable's ID (10 0C 00 00 00 00) with keeping in mind that ID line was found on ID0 pin: h First byte of the cable's 0x75 response data
7 6 5 4 3 2 1 0
ACCx Dx DATA[43:40]
0 0 0 1 0 0 0 0
So, ACCx is 00 meaning that ID0 pin will just stick with IDBUS, and Dx is 01 meaning that DP1/DN1 pins will be configured as USB0_DP/USB0_DN. Just what we expected from a standard USB cable

Now let's sniff something more interesting:

Accessory ID (HOSTID = 1)
DCSD 20 00 00 00 00 00
KongSWD (no Astris running) 20 02 00 00 00 00
KongSWD (with Astris running) A0 00 00 00 00 00
KanziSWD (no Astris running) 20 0E 00 00 00 00
KanziSWD (with Astris running) A0 0C 00 00 00 00
Haywire (HDMI) 0B F0 00 00 00 00
UART Charge 20 00 10 00 00 00
Lightning to 3.5 mm/EarPods with Lightning 04 F1 00 00 00 00
Here's a full (?) list of IDBUS requests provided by @spbdimka:

Tip #1: you can easily get accessory's properties including its ID using accctl: That's an Apple's internal utility shipped with NonUI/InternalUI builds. But you can easily run it on any jailbroken device

Tip #2: you can easily get cable's pin configuration with diags:

tristar -p

Please note that this command is only available on iOS 7+ diags

Tip #3: you can easily track 0x74/0x75 requests/responses generated by SWD-probes by setting debug env var to 3:

astrisctl setenv debug 3

Then, on cable's virtual COM you'll see something like this:

HOSTID

In one of the tables above you could see a mention of a thing called HOSTID. It's a 16-bit value carried in a 0x74 request. It appears that it might affect to a response HiFive will reply with. At least, if you set it to invalid value (yes, it's possible with diags), HiFive might stop working with it: Though there's an environmental variable called disableIdCheck in KongSWD/KanziSWD' firmware you can set to make your probe ignore invalid HOSTID

Important note: Kong and Kanzi do not feature HiFive as a dedicated unreprogrammable chip. Instead these accessories emulate it using microcontroller and/or FPGA unit, thus easily updatable/reprogrammable

WAKE

In the table of Accessory IDs you could see above you could notice that Kong and Kanzi send different responses depending on whether Astris (AppleInternal software designed for debugging with SWD-probes) is launched or not. If you decode those responses using the tables above, you'll find out that when Astris is not launched, a probe will act just like DCSD - USB on D1 and debug UART on D2 lines. But when the debugging software is running, ACCID lines are switched to SWD

But what if we want to launch Astris after a probe was already connected to a device? What will a cable do? How will it switch ACC lines to SWD? That's where WAKE breaks into the game! HiFive (or a device that emulates it) can initiate WAKE and IDBUS enumeration process will start again - Tristar will send 0x74 request, Kong/Kanzi will reply with new ID, Tristar will acknowledge that and route ACC lines to internal SWD lines (SoC must have Development fusing or be demoted for SWD to actually work, of course)

Power Handshakes

The last thing I'm going to cover in this chapter is Power Handshakes. That's an algorithm based on IDBUS requests/responses that kernel Tristar drivers use before allowing charging from an accessory

When a Lightning cable is just lying somewhere connected to a charger/computer, but not connected to a device, HiFive limits current on the PWR to a really small value (around 10-15 mA according to my measurements). To enable full current, 0x74 request must be issued by Tristar and processed by HiFive. For SecureROM/iBoot that's enough, but when a kernel is booted additional steps are to be made:

  1. Tristar issues 2 0x70 requests
  2. As soon as the second one is processed by HiFive and a reply is sent, it disables current at all for around 20 milliseconds
  3. After this time is elapsed, Tristar issues another 0x70 request but with 0x80 in its data. HiFive processes it and replies
  4. At this point kernel driver responsible for Tristar should allow charging
Important note: this is the part I know the least. And this is one of the parts I mostly reversed myself. Thus, be careful with this information

Few words about ESN and Tristar I2C interface

Another feature of Tristar I'd like to tell about is ESN. ESN is a little blob that Tristar stores in its EEPROM (on CBTL1610A2 and later). It can be retrieved over IDBUS using Serial Number Reader cable (or Kanzi, they are basically the same thing, except for a different USB PID and a little bit different enclosure)

In simple words, by sending this blob to ttrs.apple.com, you can get device's serial number. This mechanism is used by Apple Store/Apple Premium Reseller' staff to retrieve SN from dead devices (considering Tristar is still alive though): The things that are happening on IDBUS while retrieving ESN were already documented by @spbdimka:

Provisioning

The procedure of 'flashing' ESN to Tristar is called provisioning. Provisioning is done using diags on the device side and using EzLink on the host side and takes 3 steps

You can check provisioning status using diags:

tristar --prov_stat

...and retrieve ESN as well:

tristar --esn

By the way, diags generally has rich set of Tristar commands (available since iOS 7):

Tristar I2C

Tristar is available on I2C-bus (address 0x34 - for writing, 0x35 - for reading). That's how diags and kernel drivers communicate with it

Not much is publicly known about the registers. A lot of information about register map itself can be obtained from the leaked iBoot source code (for THS7383 (appears to be backwards compatible with CBTL1608) and CBTL1610 only), but not much about what to write to them to achieve some interesting results

Another source of knowledge is diags' Tristar module (easily extractable over SWD while it's running). For example, I did manage to reverse algorithms of reading provisioning state and ESN. I then implemented these as an addition to my iBoot payload, Lina:

I tried to reverse ESN writing algorithm as well, but failed - the mechanism was too complex for my skills. The code snippets from Lina though are available here

Tristar electric characteristics

Tristar itself is powered up from a 1.8V source. Lines used for IDBUS are 3.0V-tolerant according to my oscilloscope: So, better do not try to interact with IDBUS with 5V-tolerant devices like certain models of Arduino without a level shifter

Credits

@key2fr, @_L1ngL1ng_, @spbdimka



All Comments: [-] | anchor

Y_Y(3980) 4 days ago [-]

> The information in this artcile is based on a lot of AppleInternal materials (leaked datasheets, schematics, source codes) I read in a diagonal direction.

What does 'diagonal direction' mean here?

leipert(4424) 4 days ago [-]

In German we say „einen Text Überfliegen" - fly over a Text. Funnily a „Überflieger" is a person who excels at something, maybe with low effort.

pashky(10000) 4 days ago [-]

Seems like a literal translation of Russian idiom meaning "skimmed through the text".

bkanber(3576) 4 days ago [-]

I took it to mean 'unofficially and as an outsider'.

vbezhenar(3942) 4 days ago [-]

In literal Russian translation reading something in a diagonal direction (читать по диагонали) means reading quickly, without digging into it. I think proper English translation would be 'to skim through'.

cpa(10000) 4 days ago [-]

It means in a cursory manner.

edit: funny to see all that we all replied at the same time. Just to add to the conversation, 'diagonal reading' is not just a Russian idiom, it also exists in French :)

jchw(4438) 4 days ago [-]

Interesting.

> To enable full current, 0x74 request must be issued by Tristar and processed by HiFive. For SecureROM/iBoot that's enough

I wonder if that means, in a pinch, you could turn off the device to use an uncertified charger that iOS would block.

t413(10000) 4 days ago [-]

Actually, yes. At least anecdotally– I _had_ to do this when my iPhone 5's lightning port was damaged.

When traveling overseas a cheap charger died, partially frying my lightning port. The phone completely refused to charge and I grew more and more desperate as the battery slowly drained over the next day before eventually dying. Now trying again to charge the dead phone had a surprising result: it actually charged enough to boot up! Ridiculously: it then stopped charging once booted and drained again in about 2 minutes.

The solution I developed was to plug it in _then_ power it off and it would charge (slowly) while completely powered down. Removing power for even a moment would boot the phone and stop charging.

I'd absolutely guess that, in a pinch, you could charge your modern iPhone with a non-working cable if you did the same procedure: plug in, power off, let charge while off.

vxNsr(3187) 4 days ago [-]

Unlikely, even if you shut the device down when you attach a charger it goes into either a semi-on state or just turns on entirely. Even if you shut it down while it's plugged in.

specialist(10000) 4 days ago [-]

I love Lightning, long wished it'd conquer all, and expected to hate USB-C.

I kinda like USB-C.

I now half expect Apple's future mobile devices to not have any ports, will just use wireless charging.

Apple AirPods are already pretty amazing. I haven't used any plugin headphones 6+ months. Not even my beloved Shure; which sound fantastic, but hot damn I hate cables.

lostlogin(4302) 4 days ago [-]

> I now half expect Apple's future mobile devices to not have any ports, will just use wireless charging.

Wonder where my pocket lint will accumulate if that happens.

gumby(1487) 4 days ago [-]

I love that the protocol uses the string "BREAK". Old teletypes, and in particular the iconic ASR-33 had a key marked "BREAK" that was not a character; it simply held the line open. This could be interpreted by the person on the other end as a sign to reset what was going on and start again, or really anything that might have been pre-agreed.

With the advent of computing, and connecting teletypes to computers, it was often used to trigger an NMI. Multics and the rainbow books cited it as a way to make sure you were talking to the "real computer and not some program impersonating your computer (since it wasn't an actual character there was no way for a normal program to even see it much less generate it. The terminal controller/channel controller could see it though and notify the monitor (what we call the "kernel" these days).

TickleSteve(10000) 4 days ago [-]

UARTs and serial protocols are still in very widespread use. A large proportion of all embedded devices use them, which is arguably the majority of all computing systems sold.

What you're implying is historical is still current.

max937(10000) 4 days ago [-]

This cable is clearly superior to USB-C. Why can't they make USB-C easier to plug in and reversible?

Dahoon(10000) 4 days ago [-]

Is this a joke? USB-C is reversible.

bkanber(3576) 4 days ago [-]

Are you perhaps referring to mini USB? USB-C is already reversible.

mmastrac(109) 4 days ago [-]

First of all, Apple was a contributing member to the USB-C group. [1]

Second of all, it is reversible (?)

It's inferior to USB-C in at least two ways:

1. The part that wears out is inside the phone in Lightning and inside the cable in USB-C

2. Lightning is for the most part USB 2.0 capable (though there is a newer USB 3.0 version - unsure how many devices/accessories support that).

[1] https://9to5mac.com/2015/03/14/apple-invent-usb-type-c/

vxNsr(3187) 4 days ago [-]

They(Apple) designed both. They gave the worse one to the public. I wish they had the forethought to realize that usb-c would become ubiquitous, and people would be calling for it to replace lightning. Now it's just too late. They will need to eventually adopt the worse connector just bec it will make sense internally for all Apple devices to use the same charger.

miles(345) 4 days ago [-]

> Why can't they make USB-C easier to plug in and reversible?

USB-C is reversible and easy to plug in:

https://www.cnet.com/news/what-is-usb-type-c-phones/

userbinator(710) 4 days ago [-]

I bet most of the information in this article was already known by others, mostly in the repair as well as the 3rd-party accessories businesses, and either wasn't publicised or was not in English. I still remember the announcement in a Chinese BBS a long time ago when the first company in Shenzhen cracked the protocol --- a few weeks after Apple started using Lightning.

Apple seems to like using 1-wire buses; here's another application of one: http://www.righto.com/2013/06/teardown-and-exploration-of-ma...

saagarjha(10000) 4 days ago [-]

Right, this seems to be mostly reverse-engineered personally. I am sure that the people replicating debug cables have a fairly good idea of what's going on inside of them.

mindfulhack(4329) 4 days ago [-]

You're not wrong about your information, but interestingly your criticism parallels that commonly levelled against Apple itself - that Apple's innovation was 'was already [achieved] by others'. However, it wasn't yet presented in a nice, easy to consume package all in one place - which is sometimes just as significant as the raw engineering itself.

It's testament to the noteworthiness of both Apple and this article that here we are talking about it all - about efforts cobbled together from other parts.

jyriand(4321) 4 days ago [-]

I wonder how many iPhone's have been thrown away because it's "not charging anymore". I have had two occasions where my iPhone was not charging and i went to buy a new one, luckily both times shop assistants cleaned the socket, pulled out all the dust balls etc and it was charging again.

oxymoron(10000) 4 days ago [-]

I carry a tooth pick for this, but it's annoying. I'm in the process of switching out my main charging stations for Qi so I don't have to bother with it...

exabrial(4212) 4 days ago [-]

We'll probably never see USB-C from Apple because of the intense effort that went into the NIH Syndrome to come up with this connector

jjoonathan(10000) 4 days ago [-]

The mac I'm typing this on doesn't just have USB-C, it only has USB-C.

madeofpalk(4103) 4 days ago [-]

???

This is strange because... one of the big complaints about Apple's Mac computers is they only have USB-C. They ship Macs and iPads with USB-C.

Apple was also a significant contributing member to the USB-C spec/working group.

zapzupnz(10000) 4 days ago [-]

The entire MacBook line and the iPad Pro would like a word with you.

shreddit(10000) 4 days ago [-]

Well, the ipad pro and some MacBooks have USB Type C...

Dylan16807(10000) 4 days ago [-]

Weird, I'm not sure why so many people think an overuse of USB-C on Apple's computers disproves your accusation of NIH Syndrome for iPhones/iPads. To the point of downvoting you, even!

thebruce87m(10000) 4 days ago [-]

Well the alternative at the time was USB 3 Micro B. If you haven't seen that connector then I suggest you google it for a laugh.

Personally I'm glad Apple picked a robust, reversible connector and stuck with it.

crazygringo(3985) 4 days ago [-]

I'm curious -- what's the reason for something seemingly as simple as a Lightning cable to have chips inside of it?

For most of computing history, I was generally under the impression that cables were 'dumb' -- each pin connects to a wire which connects to a pin on the other end. There's fancy bundling and twisting and whatever involved, but it's still ultimately just conductive wires.

When did cables start getting chips, and why? Did Apple start it or somebody else? Is it solely to try to prevent third-party manufacturers? Is it for the cable to announce to the port that it supports certain specifications of power or data? (But why would that require a chip instead of just some kind of 'dumb' extra pin that has some basic electrical property that can be read?) Is it something else?

I mean I understand why certain dongles have chips, because they're connecting between sets of pins that aren't 1-1 or even in the same data format at all. Or why the same might be true for USB-C to Lightning.

But for cables to go from 'dumb' to 'smart' seems like it kind of breaks all expectations of what a cable even does, and therefore how a consumer will even know what to purchase -- which, of course, has famously been a HUGE issue with USB-C.

Would it be better for us to go back to dumb cables without chips? Or are there good reasons why this is the future, where at some point we'll expect all computer cables to have them?

mmastrac(109) 4 days ago [-]

Lightning added chips that weren't necessary IMO. You don't need smart control over the low voltages/power that go over a lightning cable. Lightning chips were likely more about control.

USB-C _does_ require e-marked cables in certain cases where passing large amounts of power over a thin cable could be dangerous. I don't believe these chips are proprietary nor expensive - just marking silicon that replaced the old resistor system that was somewhat brittle and often wrong.

liminalsunset(10000) 4 days ago [-]

Lightning cables have exposed contacts - While at 5V, this does not pose a significant shock risk, if full current were available, brushing the connector against metal could result in sparks and damage to the connector or power supply. Additionally, if the connector gets wet, the DC voltage will result in electrolytic corrosion. This still does happen even with the limited current provided before the handshake, just slower. (Don't!!!) try charging a phone with official cables and a power bank in the rain or letting a plugged in lighting cord's lightning end sit in some soup

Cables with electronics in them are useful for protections like this, and to facilitate safe interoperability of multiple voltage and current sources and sinks like on USB Type-C. The question becomes, is the protocol implemented in a simple, open consumer friendly way or is it implemented with other nasty antifeatures.

leecb(10000) 4 days ago [-]

The chip's responses are how the iPhone knows what device / cable type is connected to the port- at least that's what I understand from this.

jojobas(10000) 4 days ago [-]

How would you maintain such a margin on the cables if they could be 'dumb'?

userbinator(710) 4 days ago [-]

Is it solely to try to prevent third-party manufacturers? Is it for the cable to announce to the port that it supports certain specifications of power or data? (But why would that require a chip instead of just some kind of 'dumb' extra pin that has some basic electrical property that can be read?) Is it something else?

My theory is that it started as the former and then they realised it would seem less suspicious and more friendly to also use it for the latter.

nikanj(4207) 4 days ago [-]

In a nutshell, the answer is that the Lightning cables just seem simple, but they're not.

PinguTS(4448) 4 days ago [-]

In a nutshell: EMI (Electromagnetic Interference) is the main reason why cables got smart in the first place.

Many people in IT think, that data is just 1s and 0s. But in reality we are living in a physical world and as such some form of power is used to transmit those 1s and 0s. This power may is electrical power in terms of voltages with defined generated current or in terms of current with defined voltages (20mA communication heavily seen in industrial equipment). If you have a slow communication link, EMI and other environmental thinks like temperature is not much of an issue. But as faster you like to communicate you need to cope with those things. The electrical contacts in the connectors doesn't help. So people started to put the transceivers into the cables. Those transceivers with additional electronics like ringing suppression circuitry and other 'magic' are then best fitted for the desired cables for the desired environments. Then you started to makes those cables smart like to detect the capabilities of the cables like is it really possible to talk at 10Gbps or is it more a cheaper type where 100Mbps is the max. As more smarter those cables get people get creative to put other things in it like 'hey, the cable could become a converter as well'.

person_of_color(4432) 4 days ago [-]

Could Apple come after this guy since he has access to leaked IP?

bkanber(3576) 4 days ago [-]

Usually you can only go after the leaker or anyone republishing the IP. Simply having access to it isn't a crime. Perhaps they could issue some sort of takedown notice for the schematics.

sigwinch28(10000) 4 days ago [-]

Even though he might have not done anything illegal, of course Apple might come after him.

egwor(10000) 4 days ago [-]

Does anyone know what the black box is that he's connecting to in the picture? Is that the logic analyser or just some other accessory?

floatingatoll(3888) 4 days ago [-]

'Don't fry if you can't re-buy'.

It seems necessary to point out here that if you fry your iOS device's circuitry experimenting with this stuff, when they capture the device to analyze what happened, you will likely end up paying for those repairs.

wyxuan(4434) 4 days ago [-]

iOS is pretty stringent on power capacity coming through the wires. You'd have to make your own jailbreak tweak (or modify a current one) to fry your device with a cable

dokem(10000) 4 days ago [-]

I fried the audio in my iPod Touch 2nd gen in high school trying to build an external battery charger. I took it back to BestBuy and just said it stopped working. He updated the software, checked the water damage marker, then gave me a new one.

mobilio(4439) 4 days ago [-]

This explain why you can't charge iPhone with 'cheap cables'.

wyxuan(4434) 4 days ago [-]

its possible to bypass the 'unauthorized cable' with mitsuha reloaded tweak (if your phone is jailbroken)

TheSpiceIsLife(4288) 4 days ago [-]

I exclusively buy ~AU$5 cables from China for my iPhone 7 and SE before it.

They work fine, as do the slightly more expensive ones from the discount stores around these parts.

JadeNB(4446) 4 days ago [-]

This is way over my head, though fascinating, but I wanted to see if I could understand what part explains it. Is it the 'Power Handshakes' part?

peruvian(3972) 4 days ago [-]

The only third party brand I trust is Anker.

ksk(10000) 4 days ago [-]

Ironically, in our household anyway, there hasn't been an Apple cable that hasn't frayed or deteriorated (fire hazard) to the point where a new one was required. I now only use the third-party braided kind and those have held up quite well. When I was using Apple computers, their charging cables were no different. Maybe they've improved since.

crazygringo(3985) 4 days ago [-]

Why would a manufacturer even make a 'cheap cable' that didn't charge an iPhone?

Most people barely ever use data over a Lightning cable any more (pretty much only for dongles). Charging's pretty much the only thing most people need.

What cheap cables have you seen that don't charge? This is the first I've heard of it.





Historical Discussions: The Wrong Abstraction (2016) (July 20, 2018: 717 points)
The wrong abstraction (October 06, 2017: 3 points)
The Wrong Abstraction (February 15, 2017: 3 points)
The Wrong Abstraction (December 31, 2017: 2 points)
The Wrong Abstraction (2016) (January 12, 2020: 1 points)
The Wrong Abstraction (February 15, 2017: 1 points)

(543) The Wrong Abstraction (2016)

543 points about 9 hours ago by mkchoi212 in 4403rd position

www.sandimetz.com | Estimated reading time – 7 minutes | comments | anchor

_I originally wrote the following for my Chainline Newsletter, but I continue to get tweets about this idea, so I'm re-publishing the article here on my blog. This version has been lightly edited._


I've been thinking about the consequences of the 'wrong abstraction.' My RailsConf 2014 'all the little things' talk included a section where I asserted:

duplication is far cheaper than the wrong abstraction

And in the summary, I went on to advise:

prefer duplication over the wrong abstraction

This small section of a much bigger talk invoked a surprisingly strong reaction. A few folks suggested that I had lost my mind, but many more expressed sentiments along the lines of:

This, a million times this! '@BonzoESC: 'Duplication is far cheaper than the wrong abstraction' @sandimetz @rbonales pic.twitter.com/3qMI0waqWb'

— 41 shades of blue (@pims) March 7, 2014

The strength of the reaction made me realize just how widespread and intractable the 'wrong abstraction' problem is. I started asking questions and came to see the following pattern:

  1. Programmer A sees duplication.

  2. Programmer A extracts duplication and gives it a name.

    This creates a new abstraction. It could be a new method, or perhaps even a new class.

  3. Programmer A replaces the duplication with the new abstraction.

    Ah, the code is perfect. Programmer A trots happily away.

  4. Time passes.

  5. A new requirement appears for which the current abstraction is almost perfect.

  6. Programmer B gets tasked to implement this requirement.

    Programmer B feels honor-bound to retain the existing abstraction, but since isn't exactly the same for every case, they alter the code to take a parameter, and then add logic to conditionally do the right thing based on the value of that parameter.

    What was once a universal abstraction now behaves differently for different cases.

  7. Another new requirement arrives. Programmer X. Another additional parameter. Another new conditional. Loop until code becomes incomprehensible.

  8. You appear in the story about here, and your life takes a dramatic turn for the worse.

Existing code exerts a powerful influence. Its very presence argues that it is both correct and necessary. We know that code represents effort expended, and we are very motivated to preserve the value of this effort. And, unfortunately, the sad truth is that the more complicated and incomprehensible the code, i.e. the deeper the investment in creating it, the more we feel pressure to retain it (the 'sunk cost fallacy'). It's as if our unconscious tell us 'Goodness, that's so confusing, it must have taken ages to get right. Surely it's really, really important. It would be a sin to let all that effort go to waste.'

When you appear in this story in step 8 above, this pressure may compel you to proceed forward, that is, to implement the new requirement by changing the existing code. Attempting to do so, however, is brutal. The code no longer represents a single, common abstraction, but has instead become a condition-laden procedure which interleaves a number of vaguely associated ideas. It is hard to understand and easy to break.

If you find yourself in this situation, resist being driven by sunk costs. When dealing with the wrong abstraction, the fastest way forward is back. Do the following:

  1. Re-introduce duplication by inlining the abstracted code back into every caller.
  2. Within each caller, use the parameters being passed to determine the subset of the inlined code that this specific caller executes.
  3. Delete the bits that aren't needed for this particular caller.

This removes both the abstraction and the conditionals, and reduces each caller to only the code it needs. When you rewind decisions in this way, it's common to find that although each caller ostensibly invoked a shared abstraction, the code they were running was fairly unique. Once you completely remove the old abstraction you can start anew, re-isolating duplication and re-extracting abstractions.

I've seen problems where folks were trying valiantly to move forward with the wrong abstraction, but having very little success. Adding new features was incredibly hard, and each success further complicated the code, which made adding the next feature even harder. When they altered their point of view from 'I must preserve our investment in this code' to 'This code made sense for a while, but perhaps we've learned all we can from it,' and gave themselves permission to re-think their abstractions in light of current requirements, everything got easier. Once they inlined the code, the path forward became obvious, and adding new features become faster and easier.

The moral of this story? Don't get trapped by the sunk cost fallacy. If you find yourself passing parameters and adding conditional paths through shared code, the abstraction is incorrect. It may have been right to begin with, but that day has passed. Once an abstraction is proved wrong the best strategy is to re-introduce duplication and let it show you what's right. Although it occasionally makes sense to accumulate a few conditionals to gain insight into what's going on, you'll suffer less pain if you abandon the wrong abstraction sooner rather than later.

When the abstraction is wrong, the fastest way forward is back. This is not retreat, it's advance in a better direction. Do it. You'll improve your own life, and the lives of all who follow.

Updated News:

Public POOD course in May, 2018 in delightful North Carolina

My next public Practical Object-Oriented Design course will be held in Durham, NC on May 2-4, 2018. Yup, it's time for another POODNC . This is your chance to spend three days with like-minded peers. Join us, and change how you think about objects.

Tickets are on sale now. Get yours before they're gone!

99 Bottles of OOP Book

99 Bottles of OOP is complete, and version 1.0.1 is now available. The book is co-authorized by Katrina Owen, and was years in the painful and painstaking making. Learn more about it, read an extended sample, peruse independent reviews, or buy it now.




All Comments: [-] | anchor

sheeshkebab(10000) about 9 hours ago [-]

I'm not sure why this is #1... but since it is, both of these - duplication and wrong abstractions - are otherwise known as technical debt.

dasil003(4167) about 9 hours ago [-]

Not necessarily. Technical debt is when you do something quick and dirty to get a feature out in the short-term knowing that it won't be maintainable, scalable, etc, but you do it anyway with the expectation that you'll fix it later. Some duplication and wrong abstractions are caused by this, but definitely not all.

random3(10000) about 8 hours ago [-]

This is so true, but so shallow too. I think the big mistake is to treat the code as 'the main thing' when in reality it's just a model (a golem) mimicking some 'other thing'

We're missing an entire set of code characterizations. Yes we have a 'pattern language' but there's not much to characterize it structurally wrt 'code distance' from one part of the code to the other (e.g. in call stack depth as well as in breadth).

And again all of this needs to happen wrt the 'abstraction' not the code itself. Having 10 methods 90% duplicated in a single file with 10% pecent difference is many times better than trying to abstract it.

Having the same 'unit conversion' function duplicated in 3 parts of the code can be disastrous.

These two examples are very easy to see and understand, but in reality you're always in a continuous state in between. And 'code smells' like passing too many parameters or doing 'blast radius' for certain code changes are only watching for side-effects of a missing 'code theory'. An interesting book on the topic is 'Your code as a crime scene'.

The bottom line is we're trying to fix these problems over and over again without having a good understanding of what the real problem is and this leads to too many rules too easy to misinterpret unless you are already a 'senior artist'

ijidak(4441) about 8 hours ago [-]

> Having the same 'unit conversion' function duplicated in 3 parts of the code can be disastrous.

This.

I feel like it's really about cognitive load to remember and recognize the differences.

Duplication in 3 distant files, places a heavy load on the developer to:

1. Discover the duplication 2. To grasp the reason for the differences in the 3 different locations. 3. Remember these things

Whereas when the duplication is in the SAME file, #1, #2, and #3 can become very manageable cognitively.

Now the question changes to..

Is the cognitive load of dealing with the different special cases in a single de-duplicated method GREATER than simply leaving them in separate methods?

Often the answer is duplication WITHIN a file is less of a cognitive load.

Whereas duplication ACROSS files is a heavy cognitive load.

Minimizing cognitive load minimizes mistakes. And minimizes developer fatigue. Thus boosting productivity.

At least, that's my development philosophy, even though I've never seen it in a design pattern or a book.

It just seems to make sense.

bcrosby95(10000) about 8 hours ago [-]

I find it interesting that comments on these articles mainly discuss 1 aspect about it. But rarely this part:

> Don't get trapped by the sunk cost fallacy.

In my experience, yes, programmers are hesitant to throw out an abstraction. Why not work to change this, rather than telling people not to abstract?

ben509(10000) about 7 hours ago [-]

I don't think it's a sunk cost fallacy. I think the hesitation is more for social reasons, often not wanting to do a big pull request that's going to be scrutinized.

Pxtl(10000) about 9 hours ago [-]

Every Line Of Business codebase I've worked on has been the worst 'there I fixed it' copypasta spaghetti, and has never made it to the point where 'maybe we shouldn't add a parameter to this existing, cleanly abstracted method to handle this new similar-but-distinct use-case' was anywhere near my radar for abstraction.

I would love to have developers where my problem was 'maybe you piggybacked on existing code too much, in this case you should've split out your own function'.

misja111(4314) about 8 hours ago [-]

Every failed IT project that I have worked on in the last 20 years (except those where the cause was non-technical such as bad planning/ bad requirements), failed because it used too many layers of abstraction.

sagichmal(3876) about 8 hours ago [-]

I have had exactly and overwhelmingly the opposite experience. I wonder if it's a function of our fields, or what...

mlthoughts2018(2501) about 9 hours ago [-]

It's been the exact opposite for me. The spaghetti code has always come from poorly conceived abstractions and the massive problem of inverting an API to reimplement functionality through the API that should be extensible within the API (but fails to be because of poor choices in abstraction or abstracting prematurely).

Later on that spaghetti code gets labeled as lacking abstraction, similar to what you are saying, despite the actual problem being too much abstraction and poorly designed abstraction that became load bearing in a way where everyone decides that living with API inversion as a reality is the lesser evil and figures they'll probably quit the company and move on to greener pastures before it becomes their headache to deal with.

https://en.m.wikipedia.org/wiki/Abstraction_inversion

captainmuon(4254) about 8 hours ago [-]

So much this. I've encountered many codebases (in science and in tech) where the coder did not even use basic abstractions. In one case there was a lot of

    plot('graph1')
    plot('graph2')
    ....
    plot('graph100')
because somebody didn't know how to create strings at runtime in C++. Another codebase did complex vector calculations in components, I was able to reduce a 500 lines function to 50 lines (including comments, and with bugs fixed).

I can sympathize with this a bit, I started programming with BASIC - you could not return structs, you could not use indirect variables (no pointers/references)... but at least you had the FOR loop :-P

People get often called out for over abstracting (rightly so), but I've rarely seen somebody critisized for copypasta or for overly stupid code. Probably because we're too accidentially afraid to imply somebody can't code.

nick-garfield(4409) about 9 hours ago [-]

wow, just reading that term 'line of business' makes me anxious. I used to work on a global payments platform that supported 'multiple LOBs', and it was a nightmare of ifs and switch statements all the way down. The situation was made more difficult by the fact that our org couldn't standardize the LOBs into a common enum.

mrfredward(10000) about 9 hours ago [-]

The business codebase I'm working on now was written by OOP crazy people who thought inheritance was the solution to every line of duplicated code. When they hit roadblocks, they filled the base class with things like if(this.GetType() == typeof(DerivedClass1)){...

I would do anything to have the duplication instead.

grey-area(212) about 9 hours ago [-]

In contrast, every junior developer I've ever worked with has wanted to abstract too early and often, and been slow to recognise that abstraction has costs too (often far higher over time than is initially obvious).

There are costs to copying code, and costs to abstraction, and there's a balance somewhere in between where the most resilient and flexible code lives. The costs of both are paid later, which makes it very hard to judge when starting out where that balance lies, and hard to assign blame later on when problems manifest. Was it too little abstraction, or too much, or the wrong abstraction?

Note that the article claims that duplication is cheaper than the wrong abstraction. The problem is not abstraction in itself, but that abstraction is very hard to get right and is better done after code has been written and used.

dynamite-ready(4250) about 8 hours ago [-]

Nothing I hate more than seeing two files or more, sharing 90% of the same code. No matter what justification one attempts to use, there's a mistake somewhere in the design / development process.

I can see a case for what the OP is saying, but I feel it should always be seen as a temporary measure.

cjfd(10000) about 7 hours ago [-]

If there is one single article about programming that I hate it is this one. It is completely the wrong message. One should instead be very eager to eliminate duplication. To avoid the pitfalls that the article notes one should create abstractions that are the minimal ones required to remove the duplication to avoid over-engineering. Also one should keep improving the abstractions. That way one can turn the abstraction that turned out to be wrong into the right one. It is the attitude of constant improvement that will make one succeed as opposed to the attitude of fear of changing something that this article seems to encourage. When one does things one learns. When one is afraid to try things everything will just calcify until it is no longer possible to add any new features. What one does need to make the refactoring work is automated tests.

Ensorceled(10000) about 5 hours ago [-]

In 30 years, I can count on the fingers of one hand the number of times I've encountered projects that were in trouble because there was copy/pasted code everywhere and the team was not abstracting out of fear of breaking the existing code.

What I have encountered is dozens of projects that had essentially ground to a halt because of numerous deeply, and incorrectly, abstracted systems, modules and libraries.

Correcting projects in this state has almost always been refactoring into fewer abstractions; less complex, more cohesive and less coupling.

adamkl(4426) about 8 hours ago [-]

Sandi mentions this during a talk she gave on refactoring a few years ago. [0]

It's a great little video for showing junior developers how a messy bit of code can be cleaned up with a few well chosen OOP patterns (and a set of unit tests to cover your ass).

[0] https://youtu.be/8bZh5LMaSmE

voodoologic(10000) about 6 hours ago [-]

I'm very partial to this talk about proper abstraction (and not just for junior developers): https://www.youtube.com/watch?v=OMPfEXIlTVE

goto11(10000) about 9 hours ago [-]

I'm skeptical because it is really easy to un-share code by copying it into multiple places but it is very hard to unify duplicated code. So I prefer to err on the side of sharing.

But yes, you should be ready to change sharing into duplication if you realize the code is just 'accidentally similar' and need to evolve in separate directions.

In practice I have seen a lot more pain due to duplicate code compared to the issue of over-abstracting code, because the latter is much easier to fix.

joeframbach(10000) about 9 hours ago [-]

On the other hand, it's really difficult to know who is using that shared code. If you make an innocuous change in a shared method, it could affect someone else you don't know.

kolinko(3515) about 9 hours ago [-]

Depends on a specific codebase? I found exact opposite to be true - very hard to reuse code that was abstracted too soon, and abstracting copy&paste the right way is actually easier if you have it in multiple cases and can see how it was used.

sagichmal(3876) about 8 hours ago [-]

> it is really easy to un-share code by copying it into multiple places but it is very hard to unify duplicated code

Code that already exists has a gravity, a presumption of correctness. That presumption is very difficult to overcome, especially for programmers new to the codebase. An abstraction you think of as temporary will be, to those who come after you, simply the way things are done; breaking it apart and re-forming it is, for them, fraught with risk. It's good to keep this in mind as you make commits.

gorgoiler(4352) about 9 hours ago [-]

Brilliant insight. Always remember: (1) make it work, (2) make it right, (3) make it fast. 80% of projects get scrapped in between (1) and (2) because you end up realizing you wanted something completely different anyway.

nicoburns(10000) about 9 hours ago [-]

On my projects code doesn't make it into the main branch until it gets to at least (2).

willcipriano(10000) about 9 hours ago [-]

> (1) make it work, (2) make it right, (3) make it fast.

I've always disagreed with this. In my view you should make it a habit to write optimized code. This isn't agonizing over minor implementation details but keeping in mind the time complexity of whatever you are writing and working towards a optimal solution from the start. You should know what abstractions in your language are expensive and avoid them. You should know roughly the purpose of a database table you create and add the indexes that make sense even if you don't intend to use them right away. You should know that thousands of method lookups in a tight loop will be slow. You should have a feel for 'this is a problem someone else probably solved, is there a optimal implementation I can find somewhere?'. You should know when you use a value often and cache it to start with. Over time the gap between writing unoptimized and mostly optimized code gets smaller and smaller just like practice improves any skill.

ridaj(10000) about 9 hours ago [-]
arendtio(10000) about 7 hours ago [-]

I find that first comment particularly insightful.

However, I am not sure about the order of state and coupling. To me it seems to depend on the language, as for functional languages, avoiding state is king and in object oriented environments, coupling could be a more important factor.

kolinko(3515) about 9 hours ago [-]

I wish this article was available two years ago when I tried to explain this to a bunch of juniors working for me...

nnutter(4061) about 9 hours ago [-]

" Posted on January 20, 2016 by Sandi Metz."

pierrebai(4434) about 8 hours ago [-]

Counter: Refactoring is far, far, far cheaper than duplication or wrong abstraction.

Duplication means you lose the wisdom that was gained when the abstraction was written. It means that any bug or weird cases will now only be fixed in one place and stay incorrect for all the places you duplicated the code.

About the rule of three: I personally extract functions for single-use cases all the time. The goal is to make the caller be as close to pseudo-code as possible. Then if a slightly different case comes up, I will write the slightly different case as another function right next to the original one. Otherwise, the fact that you have multiple similar cases will be lost.

fiddlerwoaroof(10000) about 8 hours ago [-]

Yeah, the rule of three is misleading: having a name for three lines of code that do "one thing" is almost always a win and nothing prevents a future developer from either inlining that function, if it was a bad idea, or duplicating and modifying the function.

twirlock(10000) about 4 hours ago [-]

Yes, everyone on the planet understands why duplication is not ideal. Literally nobody entered the discussion not knowing that. You clearly don't grasp what abstraction is fyi.

jonahx(4349) about 2 hours ago [-]

Counter-counter:

Refactoring is by far the most expensive and error prone activity in programming. It can also be one of the most valuable. But unless it's trivial, it's the most mentally arduous and time-consuming work you do as a programmer.

adrianmonk(10000) about 7 hours ago [-]

Two questions (genuine, not rhetorical):

(1) How much of this is because it's actually hard to back out of the wrong abstraction and pivot to the correct one, and how much of it is other causes?

The article hints at this with, 'Programmer B feels honor-bound to retain the existing abstraction.' Why do they feel this way, and is the feeling legitimate? Do they lack the deep understanding to make the change, or are they not rewarded for it, or are they unwilling to take ownership, or is it some other reason? I could see it going either way, but the point is to understand whether you're really stuck with that abstraction or not.

(2) How much of the wrong abstraction is because people lack up front information to be able to know what the right abstraction is, and how much of it is because choosing good abstractions (in general and specifically ones that are resilient in the face of changing requirements) is a skill that takes work/time/experience/etc. to develop?

If it's due to being unable to predict the future, then it makes sense to avoid abstractions. If it's due to not being as good as you could be at creating abstractions, then maybe improving your ability to do so would allow a third option: instead of choosing between duplication and a bad abstraction, maybe you can choose a good abstraction.

zbentley(3049) about 3 hours ago [-]

> Why do they feel this way, and is the feeling legitimate?

In my experience, it's because the amount of diff (red or green) in a change request is--consciously or subconsciously--correlated with risk.

Even though we killed SLoC as a productivity metric years ago, the idea that 'change/risk is proportional to diff size' is still pervasive.

I'm totally into YAGNI/'code volume is liability' school of thought. But equating change volume with liability is a subtly different and very harmful pattern.

Adding a single conditional inside your typical 1200 line mixed-concern business-critical horrorshow function may assume a much greater liability (liability as in bug risk and liability as in risk/difficulty of future changes) than e.g. deleting a bunch of unused branches, or doing a function-extraction refactor pass. Standard 'change one thing at a time' good engineering practices still apply of course.

nfw2(10000) about 6 hours ago [-]

1.) I think political and interpersonal issues can play a role here. People are often hesitant to suggest other people's code needs to be rewritten. This is especially true if an abstraction is heavily-used by the organization. If there are many stakeholders using the abstraction, the motivation behind the refactor (ie the perceived defects), would likely need to be communicated widely to justify the effort the refactor requires.

Tainnor(10000) about 8 hours ago [-]

I feel some people here are misunderstanding the blog post.

Sandi Metz IMHO doesn't claim that the problem occurs at step 2 or 3. She doesn't claim that it's wrong to introduce abstraction when there is duplication.

What she is saying instead is that the problem occurs from step 6 onwards: when you find yourself wanting to reuse an abstraction that, regardless of whether it made sense in the first place or not, has outlived its usefulness.

I think this is in agreement with other points that she often makes, about being bold, but methodical about refactorings.

The whole discussion about 'you should never abstract away code before you see the third duplication' has little to do with the article, and I'm also really not sure it's good advice.

dave_sid(4412) about 1 hour ago [-]

You are correct that the problem lies at point 6. However a problem only exists if programmer B decides that it is acceptable to keep adding conditional logic to the method. Wrong. This is just a case of programmer B not knowing how to refactor properly. There really is nothing else to it. Yes remove duplication. Then if later on, that duplication then requires conditional logic then refactor to replace the conditional logic with polymorphism. Both these steps are clearly defined on Martin Fowler's Refactoring book and just need to be applied when the time comes.

I think the article has the premise that once you remove duplication you have committed to using that abstraction and need to shoehorn all future changes into that function. That's a ridiculous notion. You can simply refactor it again if the abstraction no longer makes sense.

BoiledCabbage(10000) about 7 hours ago [-]

> What she is saying instead is that the problem occurs from step 6 onwards: when you find yourself wanting to reuse an abstraction that, regardless of whether it made sense in the first place or not, has outlived its usefulness.

You're 100% correct in this. And what's even more amazing to me is that even after you explicitly calling this out, the majority of people replying to you (and presumably have read the article) still think the problem is between 2 & 3.

The argument she is making is not 'don't make abstractions until you're 100% certain they are correct'. She is essentially saying make abstractions where appropriate. Some of these abstractions will be wrong. When you start seeing yourself making certain behaviors it's probably because it's the wrong abstraction, so back it out and refactor.

Ultimately that abstraction seemed right based on the info known at the time it was created, now that you know more don't try to cling to it because it was already made. Be ok with backing it out and refactoring.

kwhitefoot(4329) about 7 hours ago [-]

> Sandi Metz IMHO doesn't claim that the problem occurs at step 2 or 3.

But the headline does.

I had to read quite a long way down the page to discover that all she is advocating is what i have always done: deduplicate when practical, undo the duplication when new requirements make it incorrect and push the unique parts into the callers.

tarsinge(10000) about 7 hours ago [-]

That's not what I get from the article. The problem does indeed occurs at step 2 and 3: leave duplication alone and don't introduce abstraction if you are not sure about future requirements.

barrkel(3364) about 4 hours ago [-]

I think you generally shouldn't create an abstraction until you have at least three uses for it.

That's very generally. You might want to create abstractions before then, but be prepared that they will be wrong, and don't invest in e.g. lots of unit tests, because when you break the abstraction you'll throw away that work. Some unit tests yes, but more in semi-integration tests that verify the stack sandwiching the under-proven abstraction.

pkulak(4406) about 7 hours ago [-]

Not to take this on a huge tangent, but I really _do_ think it's good advice. Unrolling complicated abstractions is a lot of work. Keeping two pieces of nearly identical code in sync is work too, but I've never found it all that onerous. But there's obviously a continuum; on one side it's obvious that it's a shared concept, and on the other, code just happens to be similar almost by chance, and not for much longer. But lately duplication has been turned into a code smell to be linted out, causing a lot of people to get rid of all of it, at all cost.

qes(10000) about 7 hours ago [-]

> you should never abstract away code before you see the third duplication' has little to do with the article, and I'm also really not sure it's good advice

Absolutes like that are rarely good advice.

majormajor(10000) about 3 hours ago [-]

I'm not sure it matters about applying 'prefer duplication over the wrong abstraction' at step 3 or step 6 nearly as much as applying that advice at some point.

I often consider 'is this abstraction going to be prone to misuse' (regardless of if it's the second, third, fourth... copy) and try to head it off with either strict typing or comments or internal visibility - to try to do step 3 without opening up as big of a door to step 6, but the important thing is less when to do stuff like this but just to try to avoid things reaching step 7.

foobiekr(10000) about 7 hours ago [-]

I think it's fair to say that abstractions should have to prove themselves as a necessity and that we make things abstract way too early. Most really good abstractions in an app fall out of well-written code to solve a specific problem.

In day to day life as an engineer, I find that we have very few _enduring_ abstractions - there are very deep ones, like the concept of streams, things like filesystems and related ideas, the concept of a virtual machine in the process sense, and so on - and a lot of faddish abstractions that have a pretty wide blast radius when they start to go wrong. A lot of the good ones (networking has a _lot_ of these, such as the abstractions above and below the model of an interface in professionally-written network device code) are focused on layering.

dpc_pw(4324) about 2 hours ago [-]

We humans just can't help ourselves, but to invent mental shortcuts. Making a judgment 'is this really a good abstraction or am I just mindlessly deduplicating code' is context-dependent, nuanced and requires some mental effort - much more work than 'do I have it repeating 2 or 3 times already' which is mindless and mechanical.

brandonmenc(4020) about 9 hours ago [-]

Junior programmers duplicate everything.

Intermediate programmers try to abstract away absolutely every line that occurs more than once.

Expert programmers know when to abstract and when to just let it be and duplicate.

leafboi(10000) about 8 hours ago [-]

The master never duplicates and all his abstractions are intuitive, readable and flexible.

leto_ii(10000) about 8 hours ago [-]

As I gain more and more experience (I would now call myself more or less a mid-level developer), I find that the distinction that matters is not abstraction vs duplication, but the one between developer mindsets.

I have many times met/worked with people who think the main task of the developer is to 'get shit done'. Regardless of their level of experience, these developers will churn out code and close tickets quite fast, with very little regard for abstraction, design, code reuse etc.

Conversely, the approach that I feel more and more is the correct one is to treat development as primarily a mental task. Something that you first think about for a while and try to design a little. The actual typing will in this case be a secondary activity. Of course, this doesn't mean you shouldn't iterate on your design if during execution problems come up. Just that the 'thinking' part should come before the 'doing'.

My feeling is that with this second approach the abstraction/duplication trade-off will not matter so much anymore. With enough experience you will figure out what you can duplicate and what you can design. And when you design you will develop an understanding of how far you should go.

Approaching development as a task of simple execution I think inevitably leads to illegible spaghetti down the line.

Tainnor(10000) about 5 hours ago [-]

I agree that many issues with bad code could really be avoided by first thinking about the solution a bit, of which the code is just an expression.

I'm not advocating weeks of architecture astronauting without code feedback - because practical considerations (e.g. the compiler can't deal with this kind of code due to some limitations) matter - but some people seem overeager to just start writing some code 'and see what happens'.

jpswade(2778) about 9 hours ago [-]

You can't plan for what you don't know.

This is why I like the 'Rule of three'[1]. Only once you've done it three times will you truly begin to understand what the abstraction might need to look like.

1. https://wade.be/2019/12/10/rule-of-three.html

nbardy(4422) about 7 hours ago [-]

The rule of three helped me get get over my initial abstraction issues, but I leaned much more towards a rule of 5 or 6. Around three you finally find an abstraction, but around six uses there is a good chance it breaks down. Making an abstraction saves you from having to make the same change to the code you copied multiple times. But the cost of repeating yourself is so low. With good keyboard mechanic repeating a change in four to five place take just a bit longer than doing it once since most of the upfront cost is in deciding on the correct change. It does feel a bit like drudgery, but it's also very freeing to not think about abstractions and just make progress at all costs. It's strategy can bite you if you don't take the time to look back and make a refactor later, but I find the approach of churn out code and letting the patterns emerge then restructuring with hindsight much more fruitful than pausing frequently to think about it abstractions. They are really two different mindsets and best left for different sessions of work.

geophile(3300) about 8 hours ago [-]

Exactly. With experience, you learn not to abstract too soon.

ed312(10000) about 8 hours ago [-]

Any advice on teaching this to junior engineers? Seems like folks with 3-5 years of experience keep trying to not only over-abstract but also keep re-inventing the wheel with abstractions (vs looking for existing libraries).

goto11(10000) about 7 hours ago [-]

I dislike any programming rule which includes a number.

The issue is whether sections of similar code implement the same idea or just happen to be accidentally similar. The number of instances does not really matter. If you have 100 lines of code which are almost the same two places in the program, then you should unify sooner rather than later, before they are allowed to diverge.

seanalltogether(2037) about 9 hours ago [-]

This quote from John Carmack speaks very succinctly to the problems that many abstractions in a code base can cause, and it's a constant reminder for me when building out business logic.

> 'A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in.'

https://www.gamasutra.com/view/news/169296/Indepth_Functiona...

hackinthebochs(10000) about 8 hours ago [-]

But abstractions reduce possible state and allows you to specify that state in obvious ways, e.g. on function parameters. Do not underestimate the power of functional boundaries.

hesdeadjim(10000) about 8 hours ago [-]

This is one reason I love working in the Unity ECS framework. Your data is public and state can't hide. Your systems are still free to contain a plethora of bugs, but they are easier to track down due to the functional nature of a system.

In the regular Unity OOP land, developers inevitably sprinkle state everywhere. Coroutines are by far one of the worst offenders. Good luck seeing the current executing state of your game when it's hidden in local variables inside a persistent function body...

jonnycat(10000) about 8 hours ago [-]

Reading that article and the context of the quote, it appears that Carmack is using that statement to extol the benefits of functional programming styles, not commenting on abstraction.

scrozart(10000) about 8 hours ago [-]

DRY gets abused regularly in my experience. It doesn't stop at method/class abstractions either; I've seen entire microservices & plugins developed to ensure each app doesn't have that one chunk of auth code, for instance, even though they each may have subtly different requirements (those extra params again). The logical end to this sort of thing is infinitely flexible/generic multipurpose code, when the solution is really, probably increased specificity. DRY is probably the lowest-hanging fruit for practices/patterns, and I think this leads to a disproportionate focus on it.

hesdeadjim(10000) about 7 hours ago [-]

It's also easy compared to solving new problems, so it can be an emotionally safe way of feeling productive. Failure is difficult to measure until the abstraction falls flat on its face months later, at which point it can be chalked up to the demons of "changing requirements".

avodonosov(4232) about 4 hours ago [-]

> they alter the code to take a parameter, and then add logic to conditionally do the right thing based on the value of that parameter

But that's a textbook example of bad code, competent coders don't do this.

Update: for example see Thinking Forth chapter 'Factoring Techniques', around the tip 'Don't pass control flags downward.'. Page 174 in the onscreen PDF downloadable from sourceforge.

And there is no need for duplication. The bigger function can be split into several parts so that instead of one call with flag everyone calls needed set of smaller functions.

dragonwriter(4447) about 3 hours ago [-]

Competent coders do suboptimal things all the time, especially when there is delivery pressure; competent doesn't mean infallible or perfect.

There's also not a clear boundary between what is a single appropriate abstraction and two (or N) distinct but superficially related concepts.

zbentley(3049) about 2 hours ago [-]

> that's a textbook example of bad code, competent coders don't do this.

That's reductive and dismissive.

There's a ton of subtlety in even defining the terms for that 'best practice'. What counts as a control flag versus a necessary choice that must be made by callers? Are you still passing control flags if you combine them into a settings object? What if you use a builder pattern to configure flags before invoking the business logic--is that better/worse/the same? What if you capture settings inside a closure and pass that around as a callback? How far 'downward' is too far? How far is not far enough (e.g. all callers are inlining every decision point)?

The answer to all of those is, of course, 'it depends on a lot of things'.

And that's before you even get into the reality (which a sibling comment pointed out) that even if we grant that this is inherently bad code, that doesn't imply anything about the competence of the coder--some folks aren't put in positions where they can do a good job.

Unrelated aside: Thinking Forth is an excellent book! Easy to jump into/out of in a 'bite size' way, applicable to all sorts of programming, not just Forth programming.

nfw2(10000) about 2 hours ago [-]

When considering whether some abstraction is 'right' or 'wrong', another important thing to consider is how cleanly the abstraction fits into a mental model of how the program works. Good abstractions provide value outside of removing duplication. They help us reason about a program by providing compression of logical concepts.

Consider some helper function: 'convertSnakeToCamelCase.' This abstraction would take a string, do some operations on it, and return another string. It is easy to understand what the input and output is without having to think about these operations. This abstraction provides a benefit for anyone having to think about the program because it reduces the amount of concepts the reader has to parse from N (where N is the number of operations) to 1. This is helpful because people have limited mental bandwidth and can only reason with a finite number of concepts at any given time.

Consider a different helper function: "processDataPayload." This function takes data in some arbitrary complex shape and returns data in some arbitrary complex shape. The abstraction effectively communicates nothing to the reader, and it is actively unhelpful because it forces that person to follow a reference, remember all the details of what that function does, and substitute those details into the original function.

Trying to find the conceptual boundaries that make the program easiest to reason about IMO is more of an art than a science and difficult to govern with hard and fast rules.

jasonhansel(3435) 19 minutes ago [-]

Agreed. I also think it's important to create abstractions that provide guarantees and/or maintain invariants. That way, your abstractions actually help you be more confident that your code is correct.

The point of abstraction isn't per se to reduce duplication--it's to make your code more straightforward and to make errors