Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

August 22, 2019 06:37



Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: I Miss Microsoft Encarta (August 19, 2019: 1175 points)
I Miss Microsoft Encarta (August 18, 2019: 3 points)

(1182) I Miss Microsoft Encarta

1182 points 3 days ago by zdw in 46th position

www.hanselman.com | Estimated reading time – 3 minutes | comments | anchor

Microsoft Encarta came out in 1993 and was one of the first CD-ROMs I had. It stopped shipping in 2009 on DVD. I recently found a disk and was impressed that it installed just perfectly on my latest Window 10 machine and runs nicely.

Encarta existed in an interesting place between the rise of the internet and computer's ability to deal with (at the time) massive amounts of data. CD-ROMs could bring us 700 MEGABYTES which was unbelievable when compared to the 1.44MB (or even 120KB) floppy disks we were used to. The idea that Encarta was so large that it was 5 CD-ROMs (!) was staggering, even though that's just a few gigs today. Even a $5 USB stick could hold Encarta - twice!

My kids can't possibly intellectualize the scale that data exists in today. We could barely believe that a whole bookshelf of Encyclopedias was now in our pockets. I spent hours and hours just wandering around random articles in Encarta. The scope of knowledge was overwhelming, but accessible. But it was contained - it was bounded. Today, my kids just assume that the sum of all human knowledge is available with a single search or a 'hey Alexa' so the world's mysteries are less mysterious and they become bored by the Paradox of Choice.

In a world of 4k streaming video, global wireless, and high-speed everything, there's really no analog to the feeling we got watching the Moon Landing as a video in Encarta - short of watching it live on TV in 1969! For most of us, this was the first time we'd ever seen full-motion video on-demand on a computer in any sort of fidelity - and these are mostly 320x240 or smaller videos!

A generation of us grew up hearing MLK's 'I have a dream' speech inside Microsoft Encarta!

Remember the Encarta 'So, you wanna play some Basketball' Video?

Amazed by Google Earth? You never saw the globe in Encarta.

You'll be perhaps surprised to hear that the Encarta Timeline works even today on across THREE 4k monitors at nearly 10,000 pixels across! This was a product that was written over 10 years ago and could never have conceived of that many pixels. It works great!

Most folks at Microsoft don't realize that Encarta exists and is used TODAY all over the developing world on disconnected or occasionally connected computers. (Perhaps Microsoft could make the final version of Encarta available for a free final download so that we might avoid downloading illegal or malware infested versions?)

What are your fond memories of Encarta? If you're not of the Encarta generation, what's your impression of it? Had you heard or thought of it?


Sponsor: Develop Xamarin applications without difficulty with the latest JetBrains Rider: Xcode integration, JetBrains Xamarin SDK, and manage the required SDKs for Android development, all right from the IDE. Get it today!




All Comments: [-] | anchor

groundlogic(4014) 3 days ago [-]

Obviously this is pure nostalgia. Wikipedia + Google is superior in pretty much all imaginable ways. And available to people 'in the developing world'. (I don't really get why Scott included that part. The number of people who have access to a PC with a Microsoft Encarta CD-ROM and also don't have web access via their phones.. my guess is that the total number is about 17, globally.)

That said, nostalgia is fun sometimes.

antognini(3664) 3 days ago [-]

> Wikipedia + Google is superior in pretty much all imaginable ways.

Ah, but does Wikipedia + Google have MindMaze?

NikolaNovak(4197) 3 days ago [-]

I'm sorry, this is just incorrect by somebody who either hadn't fully used Encarta / Compton's, or doesn't remember them.

I have been looking for 5+ years for a tablet or PC 'Atlas' which can easily, with a click of button, show me graphs, or colour regions, based on specified criteria such as political types; economy; climate; productivity; and other interesting stats.

Wikipedia articles are horribly inconsistent about format and details they use for different geographic regions. They are even worse at showing overviews, especially across categories and regions. It takes me hours to gather info I'd get in Encarta in a few clicks, at which point I give up. Google Maps doesn't have any of these markings / mappings easily accessible; and neither is trivially accessible offline.

This is not to say Google Maps and Wikipedia aren't AWESOME - because they are; and I wouldn't dream of replacing them with Encarta for 99% of things. But there are significant use cases which have been lost and not replaced with anything of equal ease of use.

FWIW: I'd pay $50+ right now for a good off-line atlas. All I can find in app stores when I search for 'Atlas' is scans of maps :S

com2kid(4144) 3 days ago [-]

The content on Wikipedia far exceeds what Encarta had available of course, but Encarta was made during the height of the multimedia boom and features rich interactive media and educational narratives.

That entire time period is full of lost multimedia gems. From cooking CDs that had vetted recipes taught by professional chefs, to the Microsoft Wine Guide![1]

Sure now days all of that information is available on the web, but people have to sort through 90% garbage to find the good stuff. The number of recipes on the web put out by people who just don't even know how to cook is astounding.

There is something to be said for high quality curated experiences, and that is what those multimedia CD-ROMs offered.

[1]https://www.youtube.com/watch?v=SzpNT8wAD3A

ghaff(3482) 3 days ago [-]

Yeah, as the article alludes to, Encarta was a product of the period after CDs came out but before there was much of a WWW, before most people had an Internet connection, and what home connections existed were mostly dial-up modems.

As a result, it was hard not to be in a bit of awe at how much information of various types you could suddenly have at home in the form of little silvery disks.

momokoko(10000) 3 days ago [-]

I would strongly disagree in this case. Wikipedia can be very opinionated and holds bias towards fringe information that has a niche passionate base of followers.

Google provides very shallow info and bites that give rise to information that is clever as opposed to well thought out.

I do not want to infer that they are bad resources, but I would argue that they are significantly lacking in the department of balanced, curated and well thought out information.

asdfman123(4208) 3 days ago [-]

For a kid growing up, Encarta was far superior to anything Wikipedia offers today.

Sure, Wikipedia wins in sheer volume but that's it. Encarta was a well curated collection of knowledge that was consistent and perfectly suited for its audience.

More is not always better.

shanselman(3779) 3 days ago [-]

I included the developing world stuff because it's true. I've lived all over the African continent and no, not everyone is connected always. Sometimes there's just an old PC in a room, disconnected. My mother in law taught in lower Gweru and they have no internet. What little digital work is done is moved on USB sticks. Encarta is used all over Cuba, Iraq, and elsewhere.

jszymborski(4192) 3 days ago [-]

Undoubtedly there is a healthy dose of nostalgia, however there's a lot to learn here.

Wikipedia is unbeatable for breadth and depth, but there's a lot to be said for curation.

partiallypro(4154) 3 days ago [-]

No offense, but I don't think you ever used Encarta, at least not much.

Encarta was much more interactive, even including 3D walk arounds of historical sites & quizzes.

wpietri(3332) 3 days ago [-]

Yeah, 'I really miss the era where we were so deprived that now-small things seemed amazing' is a bit of a weird take.

I mean, I have plenty of my own tech nostalgia, like the way at 300 baud you could hear the bits and practically whistle them. Happy memories! But I try never to confuse that with 'my childhood was a historical peak'. Because it wasn't particularly.

You know plenty of people were just as nostalgic for their encyclopedia volumes and thought kids had it too easy with this CD-ROM nonsense. And I'm sure that people complained that the first encyclopedias made learning too easy, that kids wouldn't value what they didn't work for. And this is a tradition of 'the kids are too soft' that goes back at least to Socrates: '[Writing things down] will introduce forgetfulness into the soul of those who learn it: they will not practice using their memory because they will put their trust in writing, which is external and depends on signs that belong to others, instead of trying to remember from the inside, completely on their own.'

liability(10000) 3 days ago [-]

> 'Wikipedia + Google is superior in pretty much all imaginable ways'

Excluding grammar and style maybe. A lot of Wikipedia, including many articles on important subjects, are painful to read. Either because they were written by committee or simply by somebody with poor language skills (sometimes even worse than my own!)

rtkwe(10000) 3 days ago [-]

Not purely. Things like the timeline shown near the bottom don't really exist as far as I can tell on Wikipedia. Also the many writers few editors leads to articles feeling rather disjointed at times with information being repeated pointlessly in different ways in different sections sometimes, there's something to be said for the edited entries of yore in that regard.

I think it'd be really neat and useful for wikipedia to add things like that. A filterable searchable timeline to say 'show me all the wars fought in or involving country X' or a map with layers showing different information or events.

quickthrower2(1306) 2 days ago [-]

I cam here to say similar, although I'd phrase it as 'Encarta was amazing in the 90's, but with Wikipedia you get a much richer and in depth encyclopedia, albeit without the interactive/multimedia type interface that kids might prefer'.

I'd also say that a monthly dump of Wikipedia to DVD/BluRay/USB would meet the offline requirements. Or as you say mobile phone internet access (for low bandwidth ... is WAP still a thing?)

Spooky23(3637) 2 days ago [-]

I'd disagree. Encarta had different, almost magazine-like ways to present information. Google has too many blind alleys these days, and Wikipedia is very inconsistent.

Wikipedia is great, until you need to explain to children how to weigh the credibility of the information provided.

cosmodisk(10000) 3 days ago [-]

Disagree.Your argument is like why buy Financial Times when all the info is out there for free. Wikipedia,while a brilliant invention on its own, is not, and probably never will be a substitute for Encyclopedia type of sources, including Encarta. Subjects like politics, history, nutrition and medicine are very sensitive ones and there's tons of resources out there that tend to either massage the data,or bend the reality all together,which renders them absolutely unreliable.

GordonS(615) 3 days ago [-]

I was going to submit this earlier today, but I thought 'nah, nobody else on HN is going to be old enough to remember Encarta' :)

I remember when our family first got Encarta, and I spent hour after hour reading and marvelling about how all this information could be so easily accessible! I think we had a 14k dial-up Internet connection at the time, and the web was barely getting started - I really had no idea how things would turn out!

curiousgal(2853) 3 days ago [-]

I am 24 and very much remember it! Simpler days!

52-6F-62(3593) 3 days ago [-]

Oh I remember it well and fondly!

There was all of one or two internet providers in our county where I grew up at that time. Maybe one. We couldn't afford it yet but my parents bought Encarta because it seemed like a wholesome tract for computer use.

I spent countless hours on there and used it for a ton of homework. IIRC my parents used it all the time as well. They likely still have the disks kicking around someplace.

asdfman123(4208) 3 days ago [-]

Are you kidding me? Encarta was the bomb. As a kid, in the days before the computer was an endless source of information and distraction, I would just browse Encarta and learn random stuff.

Once some kid in my third grade class asked why race cars made noise they do when they passed, the teacher didn't give a satisfactory answer, and I was able to explain the Doppler effect to them based on animation I watched on Encarta.

Good stuff.

NikolaNovak(4197) 3 days ago [-]

Oh no, I've been looking for 'modern Encarta' for years, in particular something that would easily show me different regions coloured by statistics and criteria - the ability to pick part of the earth, and then colour selected regions by economy, or productivity, or politics, or whatever, was phenomenal. Tablets _should_ be such a natural device for an educational geography product; but with free Google Maps + Wikipedia, nobody seems to be trying (at least I cannot find anything useful in my once-yearly attempts).

throwaway156503(10000) 3 days ago [-]

Consider sharing this sort of content anyway! Even for younger audiences, it's really cool to be exposed to new-to-you things.

gnulinux(3467) 2 days ago [-]

22 here and I remember Encarta very fondly. I did not use it very often as back then paperback encyclopedias were still a 'thing' and Wikipedia was pretty ok too; but I definitely reading about random physicists in Encarta. It was fun!

vanderZwan(3069) 3 days ago [-]

Are you kidding me? The only way hearing the opening screen again[0] could evoke more warm fuzzy nostalgia with me is if at the end it would crossfade into Baba Yetu from Civ IV's main menu screen[1].

[0] https://www.youtube.com/watch?v=0fRX4R6MY4A

[1] https://www.youtube.com/watch?v=5e0Qelqp-Cc

dtech(3753) 2 days ago [-]

Given that Encarta was the bomb in the late 90's and early 00's and the 25-35 age group is well represented on HN I'd say Encarta i prime HN nostalgia material.

jancsika(10000) 3 days ago [-]

> In a world of 4k streaming video, global wireless, and high-speed everything, there's really no analog to the feeling we got watching the Moon Landing as a video in Encarta - short of watching it live on TV in the 1969! For most of us, this was the first time we'd ever seen full-motion video on-demand on a computer in any sort of fidelity - and these are mostly 320x240 or smaller videos!

I remember watching that video. I was dazzled by the novelty of it and immediately tried searching for all videos and playing all I could find on the CD-ROM.

Just to compare I perused the Wikipedia 'moon landing' page to compare. At first I was surprised that I couldn't find this video. Then I realized there's a separate entry specifically for the Apollo 11 mission. It links to the following page:

https://apolloinrealtime.org/11/

... which as a whole HTML5 interface to go back and experience the launch in realtime.

I'm going to watch it later, but I'll ask ahead of time-- is the author really stating that the feeling of watching that single video on Encarta was more enthralling than what I'll experience with that HTML5 interface?

Edit: wording

Crinus(10000) 2 days ago [-]

> I'm going to watch it later, but I'll ask ahead of time-- is the author really stating that the feeling of watching that single video on Encarta was more enthralling than what I'll experience with that HTML5 interface?

It is also very possible that the author didn't knew about the page you linked, which is another 'issue' with Internet vs Encarta: you can find way more information on the Internet at way more depth, but... you need to find it first. Encarta was comparatively very little information and depth but everything it has is by design easily discoverable.

mopsi(10000) 2 days ago [-]

> I'm going to watch it later, but I'll ask ahead of time-- is the author really stating that the feeling of watching that single video on Encarta was more enthralling than what I'll experience with that HTML5 interface?

Encarta and Apollo in Real Time are not opposites, but the same: interactive experiences.

In contrast, Wikipedia is very static. I don't recall ever seeing an interactive visualization on Wikipedia like this animated and highlighted/annotated jet engine: https://s2.smu.edu/propulsion/Pages/variations.htm It's much better at getting information across than text or static images, but harder to put together in standardized text editors/languages like markup, so somewhere along the way we lost the colorful interactive animations that Encarta was known for.

ryacko(10000) 2 days ago [-]

It is a combination of novelty and uniqueness.

Novel because it was the first time the author saw the moon landing.

Unique because we are bombarded with videos all the time.

Balgair(2073) 2 days ago [-]

Mom cried.

She used to be a librarian. Then she was a geographer. The old school kind, the one that actually drew actual maps. The librarian skills really came in handy when drawing real maps. Everything had to be right. People could get lost in the wilderness and be using your map, or countries could be drawing international borders based on your map, war could break out over your mistakes. The printers and early dot-matrix printers put her whole department out of business though. That was fine, my siblings and I came along about then anyway.

I was too young to remember, but I think it was Encarta 95. I remember standing there next to Mom as she fired up the new program, my sister sat on her lap. She wasn't very tech savy, still isn't. We helped her out a bit with the mouse and clicking into everything.

We looked up East Germany first thing. Mom wanted to make sure Encarta was correct, right, true. We knew East Germany wasn't a thing anymore, even though a lot of the maps in school and in our children's books were saying it still was a thing. It took a while back then for things to percolate. Mom wanted to make sure this thing was as up to date as possible.

She typed it in, it took us to an index page, Mom clicked on the top listing (I think). We read a bit about East Germany. It had an end date. Mom was happy. This resource will do for her family. Then there was a bit of text that was underlined. Mom hovered over the underlined text. It said to click on the underlined text to go to that page. My sister squirmed a bit.

Mom clicked and then the internet came crashing into our house like an asteroid or an artillery shell or an earthquake; the floor was left behind. Everything changed in an instant.

It took us, automatically, to the linked page. I think I was about unified Germany. But it did it instantly. Boom, there you were. Reading.

Mom cried.

It was at that moment that Mom understood. No more card catalogs. No more having to hunt the shelves. No more having to look in the index for the page numbers, jamming your fingers in the back while you flipped through the tome. No more walking and sitting back down. No more having to ask strangers to watch your bags while you went diving into the stacks. No more checked out books you were looking for.

Just. Click. And. Read.

I asked her what was wrong. My sister was not happy with Mom crying. Mom was a bit flabberghasted. She told me about all the effort that went into just getting the information, all the time you lost, all the excitment that drained. And how that was no more. How it was so easy to read and to learn, now. The world was going to be so good for her children, for me, for my sister and my other siblings.

Mom cried in joy.

wazoox(3755) 2 days ago [-]

In 1993 I participated a contest with my girlfriend. We needed to find lots of factoids in literature, for instance we had a phrase from some obscure book and needed to find what book it belonged too; then a longer block of text from Proust, but which book? In fact the passage (about Venice, its tiled roofs) is present at two places in 'la Recherche', but with a couple of words changed. Another thing to find was the mystery component of a mint julep as per Mark Twain.

We spent something like two weeks at the library, thumbing through hundreds of books (all of Mark Twain, all of Borges, all of Proust, and some others). All of this would be a 20 minutes search nowadays.

dredmorbius(204) 2 days ago [-]

... and even in 1995, this was a 50-year-old dream, finally realised:

https://www.theatlantic.com/magazine/archive/1945/07/as-we-m...

hk__2(2515) 2 days ago [-]

One thing that commercial encyclopedias like Encarta can have that Wikipedia can't is non-free content. Say you can buy the rights for 1000s of images from some source; you can use them in your private encyclopedia, but Wikipedia's medias must be uploaded on Wikimedia Commons and for this must be under a CC-like license or in the public domain.

kragen(10000) 2 days ago [-]

This is indeed an enormous problem, and a very significant reason we need to shorten copyright terms. When copyright terms were retroactively lengthened here in Argentina, we had to delete huge swaths of historical footage and photography from Wikipedia. Into the memory hole!

Wikimedia Commons is more relaxed about licensing than Wikipedia is for text — because most of the images on Wikipedia and allied projects are not and cannot be created specifically for Wikipedia, we have to tolerate a wider spread of licensing terms. However, the images do need to be licensed for use on Wikipedia, which is already a pretty permissive license.

See my comment at https://news.ycombinator.com/item?id=20741101 for more social context.

atonse(10000) 3 days ago [-]

Not Encarta but I remember a CD ROM we got about the San Diego Zoo with our Packard Bell computer in 1994.

Boy, my brother and I learned too much about Aardvarks (because they were the first item on the list).

daveslash(10000) 2 days ago [-]

I had access to Encarta on my grandparents' computer when it came bundled with their Windows 95 computer, but in my home I had Undersea Adventure, which came bundled with our Win 3.1 machine. Man, I learned a lot about sharks and nudibranchs.

ergothus(4209) 3 days ago [-]

I'm either old enough or out-of-touch enough that I never used Encarta. Reading the comments here, I see a lot of mention of the interactive elements and video.

I generally prefer text - it's easier to search, and faster to consume. But I don't assume that my preferences are always best, so...

Why was Encarta so good? Why is it good enough that we should wish we had it today? Is there some analog today that could let me 'get it'? Because currently I live in a world where more and more info is moving to podcasts and videos and I hate it. Clearly I'm missing something, so help me out?

bonestamp2(10000) 3 days ago [-]

> I generally prefer text

Encarta had text. It was mostly text. But sometimes text isn't the best medium for conveying some information. For example, maybe there's an article about lizards. You can describe the way a lizard moves, but it makes much more sense to have a video embedded that can show you what that movement looks like with all of its nuances that would be difficult to describe and understand in text.

So, think of wikipedia, but curated by professional library scientists and verified subject matter experts with first source video and audio clips (and photos) embedded where helpful.

ses1984(10000) 3 days ago [-]

It also had a bunch of text that was easy to search. Imagine replacing a shelf full of encyclopedias (that cost a lot) with a single cd, AND it had some videos.

shortformblog(3068) 3 days ago [-]

You have to remember that this was during an era when the internet was not as common, so this was competing with physical books, not digital media. And the result was much more immersive than reading a book, and the internet was not yet mature enough to be an effective format for reading hypertext.

Additionally, consider that much the audience for it was people in grade school. If you were in the seventh grade in 1995, you were the perfect age to really engage with that kind of approach—which seemed awesome compared to a boring textbook.

I think in many ways, our information consumption has gotten much more dense over time. But in 1995, people would approach this like they would a nice, long magazine article. Efficiency was not the goal: It was something that hit all the senses.

ahje(10000) 2 days ago [-]

> Why was Encarta so good?

Simply put: It was accessible, interactive and enabled users to find information at a pace that wasn't really achievable through other means.

I first used Encarta when I was in the 6th grade, in order to find information for an essay I was writing. The local library had just gotten a new PC with a CD-ROM drive, and the librarian suggested I use Encarta as it was brand new, while any books they had on the subject would be rather old and outdated.

I got down behind the PC, launched Encarta and I was simply blown away. This was my first real multi-media experience and while computers were becoming mainstream in the early 90's, I simply hadn't seen anything like it before. Sure, an Amiga could play music and simple videos, and a fast 486 could decode MPEG video, but hard drives back then were typically less than 100 MB and the amount of storage allowed by CD-ROM's allowed Encarta to be an (until then) unparalleled experience.

Remember, until then, you'd have to use card indexes in order to find the right book about the subject, then look for the book and hope nobody else had borrowed it. With Encarta, it was as simple as typing in a search term and finding the results. Instantly. Internet wasn't really a thing back then - I'd heard about it, it existed, but it didn't reach the general populace until around '94-'95, and there simply wasn't anything comparable until Wikipedia came along.

joblessjunkie(10000) 2 days ago [-]

Encarta was part of my first job out of university. I worked in Microsoft's Multimedia Division, tasked with creating the first video and audio drivers for Windows. IIRC the BMP, WAV, and AVI file formats all came from this team at about this time.

In the summer of '91, I was tasked with assembling a 386 PC with an early CD-ROM drive to demo an early build of Encarta at a trade show. It was a lot of effort to find a combination of hardware and drivers that would work reliably together.

Encarta itself (at least at that time) was written as a Word document. Hyperlinks were defined using footnotes, and animation and audio placeholders were defined with custom OLE objects. The whole thing got exported as RTF and fed to a compiler, which created the runtime data structures optimized for CD-ROM access and that also built a full-text search index. The compilation was very slow and required huge amounts of RAM.

Around this time, the team had some awareness of HTML as one of many emerging hypertext markup languages, but the internet was still a few years away and no one knew what format would 'win' for hypertext. In any case, there were no tools or browsers at all, so we had to build everything -- including our own search engine for the CD-ROM.

SuoDuanDao(10000) 2 days ago [-]

Hey, thanks for your work! I got a CD-ROM holding Encarta '95 for Christmas when I was in grade school and I loved it - spent a lot of time on that program over the years, and it really helped develop my love of learning.

akgoel(10000) 2 days ago [-]

This basically sounds like WinHelp, which was a compiled format of RTF, prior to the availability of HTML.

fallensatan(10000) 2 days ago [-]

I just want to thank you for your work on Encarta. It was the first _real software_ I interacted with in my teen years; it fascinated me a lot back then and I believe it's part of the reason I'm in software industry. Thank you.

elboru(4175) 2 days ago [-]

This kind of comment is the exact reason why I keep coming back to HN. It's great to read stories of real people who have worked in all kinds of amazing and inspiring projects, thanks for sharing your experience with us, it's always nice to read about technical details of legendary software.

mixmastamyk(3492) 2 days ago [-]

> In the summer of '91, I was tasked with assembling a 386 PC

Interesting, as the 486 came out in late '89. Seems you'd like a little more oomph, but perhaps they were too expensive at the time.

numismatex(10000) about 1 hour ago [-]

Thanks for your work. I pirated it, I can't remember when, after release via a warez FTP dump, and a later version via DCC on IRC and it took forevvverrrrr to download but it was so very much worth it. I was born in '81, so having a somewhat recent encyclopedia was so very much worth the time and effort. I couldn't afford a new set of encyclopedias, and the ones we had at home were very old, with very small print. You're partly responsible for some of the beginnings of my self-education -- to you and your team, thanks.

Sorry I couldn't pay for it at the time, and didn't think to write an e-mail (yeah, I was a preteen using Waffle shell at 9, and used WFW and W 3.11 with Trumpet Winsock through my teen years, along with a shell account later). I had access, but not much of a frame of reference. Your product started giving me that frame of reference.

hollerith(3309) 1 day ago [-]

>the internet was still a few years away

Internet history fairy here to make sure people know that the internet has been in continuous operation since 1969 [1]. (And DARPA started funding research for the express purpose of creating a global computer network in 1960.)

What was still a few years away in '91 is awareness or interest in the internet by a significant fraction of consumers (or by Bill Gates for that matter).

[1]: Some people would deny the name 'internet' to any network prior to the introduction of the internet protocol suite in 1982, but it was the same user-visible services (email, ftp, telnet, netnews, etc) running on both the earlier network that began operation in 1969 and the later network that some want to reserve the name 'internet' for. In other words, the switch in 1983 from NCP to TCP/IP was mostly transparent to users.

cryptoz(238) 2 days ago [-]

I miss Encarta, but most of all I miss Mindmaze.

timwaagh(10000) 2 days ago [-]

Never heard of it. Care to enlighten me?

Causality1(10000) 3 days ago [-]

There was a certain method of presenting information that just doesn't seem to exist anymore in the modern world. The by-design weaving together of text, images, video, and audio when giving information about a certain topic. Sure, wikipedia articles have one or two images, maybe an audio clip, and plenty of links to other text articles, but nothing like an encyclopedia or other informational software of the 90s. You'd have software about, say, dinosaurs, and there'd be big diagrams where every word was a link to a descriptive media. It feels like we've shrunk. You can watch a Youtube video but it's just a video; you can't pause it on a picture of a T-Rex and click on its mouth to learn about T-Rex teeth. The modern web forever has you finding any piece of information you need starting from zero, and most of the time you can't summon the energy to do that for everything you might find interesting.

Interactive multimedia is dead and I think today's children are worse off for it.

mopsi(10000) 2 days ago [-]

Video of the dino browsing experience: https://www.youtube.com/watch?v=dgOZUPUGuwY

fsiefken(10000) 3 days ago [-]

There also was the Encyclopedia Brittanica, not sure how it compares to Encarta. I remember getting it on DVD or CD and choosing it instead of Encarta. No idea where it went. I'm not sure if the information is better or more complete then an offline Wikipedia dump combined with the Stanford Encyclopedia of Philosophy.

potta_coffee(4203) 2 days ago [-]

Brittanica was the king of encyclopedias at that time, as far as I can remember. I had a set of those, and I remember comparing and finding other sets to be less good. I was pretty young but I spent a TON of time with them. I don't have experience with the Brittanica CDs but my guess is the content was of similar quality.

sombremesa(10000) 2 days ago [-]

Reading this thread makes it clear that there was (and perhaps still is) one market where Encarta beat modern Wikipedia hands down - children.

leshokunin(4183) 2 days ago [-]

That's a perfect summary. So many apps for kids are just designed to sell them entertainment as micro transactions. This was a useful and informative piece of software, made accessible. I know Wikipedia has a simple mode, but it's not the same.

ChuckMcM(619) 3 days ago [-]

I miss it as well, as the author points out it isn't large and is easily contained on a thumbnail flash drive (several times over!).

I've started creating an offline reference library[1] because I see that search engines will be over whelmed with crap and frankly I don't want to have to find cell service to ask my handheld device to locate a factoid.

If you've read that now more than 50% of Google desktop searches don't result in a click, it is because Google is getting better at 'one boxing' the answer so you don't need to click through to read it.

[1] That project started by digitizing my referenced by less often used text books. Which lead to me digitizing text books for others as well. After Google's win on digitizing books is fair use for air cover much of the work applies NLP and a bit of machine learning to pull out facts which are not, in themselves copyrightable.

[2] 'Google Wins Copyright Suit' --- https://www.washingtonpost.com/news/the-switch/wp/2016/04/18...

floren(3817) 2 days ago [-]

What technique are you using to digitize these books? I'd be interested in digitizing a few books I have.

dman(3533) 3 days ago [-]

How do I get a peek into this world according to Chuck slice of cyberspace?

shortformblog(3068) 3 days ago [-]

This sounds amazing. I know it's a personal project, but you should seriously share it with more people at some point.

leemailll(3962) 3 days ago [-]

I miss it and I also miss a real encyclopedia collection. They both offer a thing Wikipedia always lacks: consistency, in writing style, in topic depth, in error rate. I still remember the weekend in 90s when I had a copy of Encarta and threw the whole weekend to click through topics after topics, and amazed at things it offered. It's so much fun.

ryacko(10000) 2 days ago [-]

Have you looked at Citizendium? It is the closest thing there is to a community curated encyclopedia.

Namrog84(10000) 2 days ago [-]

>in error rate

I dont have source at moment. But I remember reading a few years ago that wikipedia had fewer errors than most all encyclopedias on average.

There is definitely something to be said about consistency and clustering but on average I believe wikipedia is still the best in fewest errors.

com2kid(4144) 3 days ago [-]

Encyclopedia's are interesting.

They had to pick and choose what to go into depth on, but when they wanted to go into depth, they would hire the best writers in the field to explain everything really well. Issac Asimov penned more than one encyclopedia article, as did other science fiction authors.

I honestly learned most of my fundamental science knowledge from old 1950s and 1960s science fact books. Asimov in particular has multiple books that explain science really well, down to the level of nuclear fusion and fission, biochemistry, and a lot of other subjects, and I had a few other books in the same vein that were all really old, and really good.

I remember one book that was biographies of famous scientists, it went into depth about Marie Curry and her contributions to science. Reading that ~age 10 certainly impacted my views on gender equality and helped me believe that potential is not limited by gender.

Wikipedia, for all the great things about it, won't really do that. There isn't a Wikipedia page I can point literally anyone[2] at and say 'read this and you'll understand the fundamentals of radiation and atomic physics'.

I am sure such resources are out there on the web, but the odds of stumbling upon those resources at random is less than the odds of coming across a good science book at random in a thrift store.

That said there are some awesome YT science channels who have worked to fill this gap in quite well, but video sources are different than written sources, with each having their strength. (I read that short 10 or so page biography of Marie Curry probably a half dozen times, not going to do that with a YT video!)

[2] Where anyone is defined as 'age 10 and up, with no mathematical background, who wants to be entertained while they read.'

abhinavsharma(4033) 3 days ago [-]

The period of time before the infiniteness of the internet but after CD-ROMs could pack lots of information was fascinating to grow up in. You had lots of curiosity but still limited access to information so you deeply explored stuff.

Add to that, something like Encarta was curated by opinionated editors (whether or not you agree with them is different from the fact that the nature of content is different when it isn't trying to find consensus amongst absolutely everyone) and did not survive based on advertising as most internet content does today.

That set of factors made for a very different world of reading than today.

karthikb(10000) 3 days ago [-]

>You had lots of curiosity but still limited access to information so you deeply explored stuff.

I was about to write this myself. Wikipedia has so much information that it's easy to surf page to page, link out, etc, and end up having skimmed 50 articles and still not understand the thing you came to look at.

cortesoft(10000) 3 days ago [-]

This makes me think about when I used to go to the MacWorld convention in San Francisco as a kid in the early 90's. My mom would take me out of school every year to go (just for the expo hall, we didn't have enough money to actually go to the conference).

I was just overwhelmed by all the different software on offer, and would load up my bag with free samples and spec sheets for software I had no legitimate need for as a 10 year old, but I ate it all up. I would ride the BART home after, just pouring over my loot and being so excited to try the demos at home.

I went a few times after the internet took off, and it just wasn't the same. Who needs demos from an expo hall, when you can just download everything? There was nothing new to see at the hall, everything could be seen from your home just by visiting a url.

I love the world with the internet, but there certainly has been something lost.

rchaud(10000) 3 days ago [-]

I had Encarta '97 as an 11yo, purchased at a Malaysian street vendor market for probably all of 3 bucks. It felt incredible to have that much knowledge available for almost no money. And it wasn't just boring text like the paper versions, there were videos too!

There was point in time where that truly felt like the future. Considering that we're now in the age of autoplaying, monetized 'content' video nearly everywhere you go, it was nice to have only 'primary source' video, aka stuff like the moon landing, or the MLK speech, instead of some nobody warbling on about its importance while splicing in stills of said video because they don't want to be in violation of the copyright.

The concept of finite, structured and bounded content is also extremely powerful in this era of overwhelming data, much of which rehashed and warmed over by various publications not to mention utterly infested with ads.

The concept of spending hours down a Wikipedia rabbit hole is not new, and it's one of the reasons I donate annually. But it first started with Encarta, and it's offline-only format has grown to be only more valuable as time goes on.

sanbor(3499) 2 days ago [-]

Pretty much same experience but in Argentina. I got Encarta 97 or 98 for 5 pesos, 5 dollars at that time, when I was 11 years old. Later I read about a Visual Basic course in a newspaper so I figured out that the way to build software was using this Visual Basic software. So I went to the same place where I bought Encarta and got Visual Basic 5, which was my first programming language.

aksss(10000) 2 days ago [-]

Pretty sure the hours-long rabbit hole thing didn't start with Encarta :D But you still would have been considered a geek of the highest order doing it with Encarta. Wiki first popularized the experience.

agumonkey(961) 3 days ago [-]

Encarta 97 is the first Microsoft product I ever bought. But I hassled my father to pay for the whole 99$.

chicagofan98(10000) 3 days ago [-]

What I remember most about Encarta is how impossible the 'Mind Maze' game was as a kid

dTal(4007) 2 days ago [-]

Oh god I can still hear the music

theandrewbailey(2080) 2 days ago [-]

If there was something like Mind Maze today, I might check it out.

kfrzcode(3967) 3 days ago [-]

Oh man, many many many hours

V-2(4141) 2 days ago [-]

Here in Poland Encarta is probably best remembered for its ridiculous entry on Poland, depicting it in a rather outdated manner, to put it mildly. It caused quite an uproar back in the day, and went sort of viral.

See http://of19.internetdsl.pl/bzdury/encarta/ (in Polish, but includes the screenshots in English, which are self-explanatory)

asdf21(10000) 2 days ago [-]

Seems about right.

oneplane(10000) 2 days ago [-]

We never really used Encarta, and I don't really know anyone who ever did. Right in the middle of the CD-ROMs are big and DSL just starting to become available period we got a few educational CDs that did more specialised things like only the history and current state of geographical information, or just mathematics, or just historical events of one country. No full encyclopedia was really digitally popular as the schools and libraries all had plenty of physical copies and people around to help use/find them.

It probably didn't help that it was a Microsoft-only deal as most of the stuff was an odd mix Netware-driven PC's without Optical drives (we had ZIP!) or library computers that were locked in a box so you couldn't really put a CD in them. And everything else was an Apple Mac, so you could put the CD in but you got nothing out of it.

We did have something else, I don't remember the name of it, but it was like a network-CD-drive-server device that worked on Windows 95, 98 and 2000, as well as Macs, and we had a cross-platform (flash/shockwave based AFAIK) viewer thing that was probably an encyclopaedia but it was never named or marketed as such. It didn't have videos but it did have the same information as the paper versions.

Most people who did end up owning (some version of) Encarta got a demo or companion version with some other product, much like the Microsoft Works package you sometimes got with systems, but right around that time Wikipedia was already gaining popularity.

kalleboo(3941) 2 days ago [-]

I never used Encarta - in my world it was all Grolier's Multimedia Encyclopedia

paulddraper(4011) 2 days ago [-]

I used Encarta heavily in 6-8th grades.

Internet searches were of dubious quality, and libraries were inconvenient.

Encarta was the first stop for looking something up.

shortformblog(3068) 3 days ago [-]

Encarta is a great example of Microsoft winning a market, and after it was defeated, letting it fade out. Wikipedia in some ways beat it, but the interactivity elements of Encarta make it something that you kind of wish had a more direct role in the modern day.

I wrote about this topic a few years ago (https://tedium.co/2017/07/13/who-killed-the-encyclopedia/), and it led a former CEO of Encyclopaedia Britannica to speak up (I added some of his comments to the piece). There was a lot of back-room wheeling and dealing around digital encyclopedias during that era, much of it centered around Encarta—Microsoft acquired a lot of publishers during that period and it effectively disrupted most of the rest out of existence.

But even considering that, Encarta was special. I think Microsoft had the right idea—it was the killer app for CD-ROMs—though it turned out that the internet would only make it a temporary success story.

It didn't have to be like that, honestly. Imagine what might have happened if, for example, Microsoft worked more closely with the Wikimedia Foundation on the highly visual treatments the company was known for with Encarta. The nonprofit ownership was a good move, but the fact that MS seemed to cede the market entirely, especially so soon after disrupting the whole thing, was unfortunate.

nickthegreek(10000) 3 days ago [-]

I remember getting Encarta 95 back when it first came out (I was 14 at the time) and the interactive elements and audio clips blew my mind. It changed how I thought about information and got me that much more invested in learning technology.

Literally a week after getting Encarta a door to door encyclopedia salesmen came to our house and I showed him why I did not need his books. You could see the look of terror in his eyes.

NeedMoreTea(3299) 2 days ago [-]

We have an old encyclopedia set - not Britannica though, from the 1910s or 1920s. I think it was my grandfather's originally, bought for father and uncles. Perhaps the generation earlier.

Whilst it's not especially useful in today's world, it's a fascinating relic, especially on trades and skills that have declined or died out. On medicine, zoology and other sciences, it's occasionally surprising how we used to think. I'd periodically pick it up and read a few entries at random, and often disappear down a rabbit hole. At some point it stopped being obsolete and pointless to become fascinating history.

That's one thing Encarta could have been had it continued - a valuable snapshot of how thought and knowledge evolved. Always updated Wikipedia can and will never be that as sadly you can't snapshot a given date, and digging through edit histories is horribly unfriendly. We may have lost something there.

ksec(1707) 2 days ago [-]

I see it as a Free Vs Paid Model, Wiki is Free, and it is good enough for most use cases. Encarta had a lot of licensing, and I doubt any of that will ever be free, but the quality of those articles inside Encarta or Britannica in many ways are still way ahead of Wiki.

tzs(3281) 2 days ago [-]

> I think Microsoft had the right idea—it was the killer app for CD-ROMs—though it turned out that the internet would only make it a temporary success story.

It was a killer app for CD-ROM, but for the title of the killer app I'd say DeLorme's 'Street Atlas USA' had at least an equal claim.

davidgerard(400) 2 days ago [-]

> It didn't have to be like that, honestly. Imagine what might have happened if, for example, Microsoft worked more closely with the Wikimedia Foundation on the highly visual treatments the company was known for with Encarta. The nonprofit ownership was a good move, but the fact that MS seemed to cede the market entirely, especially so soon after disrupting the whole thing, was unfortunate.

We actually asked them about this, IIRC - it turns out a lot of that stuff was licensed from third parties, and they weren't free to release it under a free content license, and didn't have the resources to sort out what was what.

duxup(3952) 2 days ago [-]

It's funny to think of it but CD-ROMs ... really were where all the multimedia was.

Outside professionals, not many people stored videos or audio on their PC in the early day, so all the multimedia type stuff, and to some extent including video games ... was all centered around the CD-ROM.

I used to gobble up all the super cheap 'multimedia collection' type disks available at the time.

To some extent what made computers the multimedia place to be was the good old CD-ROM, well before the internet took it over.

joblessjunkie(10000) about 17 hours ago [-]

I appreciated this article. I agree that 'Microsoft was never in it for the history.'

Not to sound cynical, but I believe MSFT wasn't trying to sell encyclopedias, they were trying to sell CD-ROM drives.

The upcoming Windows 93 (whoops, we slipped a little, make that 95) was going to require a billion floppy disks for distribution. Retail products were shipping in large boxes with comically tall stacks of expensive floppy disks.

CD-ROMs, on the other hand, are so cheap to distribute that AOL mailed them out as unsolicited junk mail.

It was worth sinking some money into a bunch of high-quality CD-ROM titles to get those drives standard on all PCs. Once that task was accomplished....

scarface74(3847) 2 days ago [-]

Encarta is a great example of Microsoft winning a market, and after it was defeated, letting it fade out.

Encarta lasted from 1991-2009.

The iPod was introduced in 2001, saw sells start to fall off a cliff around 2009 and were all discontinued in 2017 - except for the iPod in name only iPod Touch. No one would say that the iPod was a failure. Encarta helped start the multimedia PC boom that helped make home computers more ubiquitous just like the iPod was Apple's introduction to a wider audience.

anthony_doan(10000) 2 days ago [-]

> Encarta is a great example of Microsoft winning a market, and after it was defeated, letting it fade out.

Felt the same way with IE until Mozilla came along after Netscape died.

m1sta_(10000) 2 days ago [-]

Is Encarta open source? Microsoft might be open to allowing it to be integrated into Wikipedia.

hugh4life(4164) 3 days ago [-]

Wikipedia could do a better job integrating media.

I remember Encarta being pretty good about having video files showing the outlines of wars and battles... at least for the Vietnam War which is what I remember most. I prefer watching a video outline of wars and battles on youtube rather than dig right into a wall of text on wikipedia.

rtkwe(10000) 3 days ago [-]

Also things like the globe or the timeline mentioned in the article would be great to add to wikipedia. Being able to quickly pull up a timeline of battles mentioned on wikipedia that took place or involved a certain country could be neat. Similar things on the globe could illustrate a lot of things.

WorldMaker(10000) 2 days ago [-]

Yes. Part of the problem is the CC license restriction of Wikipedia/Wikimedia. A lot of primary sources such as the ones Encarta included were often proprietary licensed and copyrighted products. According to Wikipedia's annual donation bars they (supposedly) spend enough on hosting costs every year that spending money on licensing media content is (probably) out of the question for them.

I almost wish for a sort of intellectual property 'eminent domain' promoting more things like famous news footage into the public domain where they fit an 'almanac' need. Though I admit creating any sort of such program would be really hard to do, as the boundaries would be murky and it potentially would impact the revenue of companies and groups that generally need such revenue to survive (PBS, for example).

situational87(3849) 3 days ago [-]

I still remember using one of the earliest Encarta versions and seeing a tiny postage stamp sized video animate and play some half broken wav file for the first time. I remember it hitting me like a baseball bat: we can do VIDEOS on computers now! This will change everything! TV is going to die because there are no ads on computers!

RandomTisk(4202) 2 days ago [-]

Encarta still has the best video/animation on Quantum physics I've seen: https://www.youtube.com/watch?v=cQnhqqISBSA

athrun(10000) 2 days ago [-]

Thank you for sharing this video! It certainly clicked for me.

Wowfunhappy(10000) 2 days ago [-]

Does anyone know how easy it is to extract just the text / html out of an Encarta CD/DVD? If it's easy, I'll go out and buy one right now.

For no particularly good reason, I have this urge to create a local txt or basic html version of one of the major encyclopedias (something a bit more concise than the full Wikipedia). I love the idea that I could keep a summary of most human knowledge in a 400mb plain text file on my hard drive.

kragen(10000) 2 days ago [-]

You should check out my comment at https://news.ycombinator.com/item?id=20741101 for a summary of the current state of some of the existing projects that do some of this, but also check out the Wikipedia Vital Articles lists. Typical 'vital' articles are large, about 100K, so in 400MB you could manage about 4000 articles, which is most of the ones in the fourth-level Vital Articles list.

mongol(10000) 2 days ago [-]

I bet it is an Access database inside. But probably some copy protection too

galonk(3834) 3 days ago [-]

It's not reflected in OP's screenshot, but there was a period where Encarta (and a companion product called Microsoft Bookshelf) used a very flat, Swiss-design aesthetic that was catnip to graphic-designer-wanna-be kid me. You can draw a direct line from that era through Microsoft's Neptune UI experiments down to the Metro design language.

contextfree(4171) 2 days ago [-]

Here's a talk by Bill Flora who was a lead designer for many of the products along that line, from Encarta through to Windows Phone: https://vimeo.com/56764845

leoxvi(10000) 2 days ago [-]

I am having fun developing an alternative interface for Wikipedia called 'Explore':

https://wikischool.org/explore

https://wikischool.org/explore/Ludwig%20Wittgenstein?l=en&t=... (read an article chronologically)

The larger goal is to make self-study more effective and enjoyable. Desktop browser recommended for now, as the mobile UX needs improvement.

bearbin(3909) 2 days ago [-]

I had a short browse and I was really impressed by how much nicer to read you have managed to make wikipedia. Normally I much prefer static HTML to whatever mess the latest javascript framework manages to conjure up but what you've done really seems to work.

The text actually fills the screen rather than just hiding in the gutter on the left, and even though you've removed most styling cues around links and actions, it's done in a way that it's still obvious what everything does, so the main content can be clearer.

The dynamic sidebar table of contents is also particularly effective - on desktop, everybody has a widescreen monitor now and it's disappointing now few sites take advantage of it.

My only two critical points are your choice of typeface, which is awful and significantly degrades the experience (browsing with fonts turned off is a great improvement), and the speed - not only does clicking a link take several seconds to do something but there isn't really any indication whether things are happening or not.

mohn(10000) 2 days ago [-]

Nice! I like this idea and look forward to seeing it develop further.

A bug to report: inspired by comments elsewhere in this thread, I searched for 'radiation', and your interface is rendering the bulleted list about different kinds of radiation above the actual first paragraph (which ends with 'This includes:' to introduce the list).

Tokkemon(10000) 3 days ago [-]

One of my favorite CD-ROMs from my childhood was an Encarta offshoot called Microsoft Exploropedia, World of Nature. Basically Encarta for natural sciences but tailored for kids. That game was the bomb.

pmyteh(10000) 3 days ago [-]

There is an entire book on the creation of Explorapedia, I sing the body electronic: A year with Microsoft on the multimedia frontier by Fred Moody (my copy is ISBN 0-340-64927-5). It's fascinating for its insights not only on the messiness of the design process, but also the process and deadline constraints that the team were working under (which feel like a general rather than a specific set of lessons).

withinrafael(10000) 2 days ago [-]

It was also bundled with the Microsoft EasyBall, the giant mouse with a yellow ball [1].

[1] https://www.microsoft.com/buxtoncollection/detail.aspx?id=22...

grawprog(3849) 2 days ago [-]

I used to love encarta. The thing I remember most though were the entries on musical instruments from around the world. I remember being blown away by the different types of instruments and used to spent lots of time just clicking the interactive map and reading about them and listening to the different clips they had. That was where I first learned about and heard the difference between all the different types of bag pipes they have in different countries. Up until then, I only knew about the Scottish ones.

dx7tnt(10000) 2 days ago [-]

Holy moly I'd forgotten about that. Endless hours of clicking in the school library.

kragen(10000) 2 days ago [-]

Writing this from an occasionally-connected hand computer in the "developing world", don't use Encarta. Use Kiwix. I have a 45000-article slice of the English Wikipedia here in 6 GB, with nearly all the pictures, and even animations and some video and audio, such as the radio broadcast of the Hindenburg crash. To give you an idea of its breadth, articles I've consulted recently include 'anal sex', 'rectum', 'convolution', 'ferret', 'Dorothy Parker', and 'principal component analysis'. (This probably paints an unjustifiably interesting picture of my hobbies.) The whole English Wikipedia, 4 million articles, is 35 GB without pictures.

(Encarta was 32000 articles and, as pointed out in https://news.ycombinator.com/item?id=20741028, ridiculously biased on some subjects, in precisely the way that Wikipedia scrupulously avoids.)

The biggest problem I have with Kiwix is that, on Android, its built-in downloader for ZIM files is flaky, and I haven't figured out how to load ZIM files I downloaded in some other way yet, such as on a MicroSD card. The non-Android version works fine with such sideloaded ZIM files. (I also miss the interlanguage links; I have a few languages of Wikipedia on here, and even when I don't it's often useful to find out things like the Guarani word for 'ferret'. And of course I don't have a local Google Search.)

Kiwix also has ZIM files for things like Wikisource (7.2 GB for English), Project Gutenberg (54GB for English), and Stack Overflow and other Stack Exchange sites. The English Stack Overflow is 134 gigabytes.

As for the globe, a much better option than Encarta is OsmAnd~. My country is under 0.6 GB, including most of the public transit routes, down to the bus-stop level. Bonus: lat/long links in Kiwix open in OsmAnd~. Minus: OsmAnd~ doesn't have any way to load satellite imagery, even the Blue Marble MODIS dataset.

Both Kiwix and OsmAnd~ are available on F-Droid, so you know they're not malware.

But none of this is interactivity. Watch Alan Kay's talks where he talks about kids building dynamical models in Etoys to understand gravitational acceleration. Much more interesting than the Wikipedia article on convolution by itself: I have IPython, Numpy, and SciPy on my netbook, so I can create signals and kernels, convolve them, and look at the results. I can compare my own implementation of convolution to the library implementation. I can do principal component analysis with the LAPACK functions for it. That's interactivity, and it's a far more powerful learning tool than mere hypertext. And I don't have anything like it for Android. Is there anything?

An interesting thing to note about all of these — Wikipedia, OpenStreetMap, IPython, and LAPACK — is that none of them are products of capitalism. They coexist with capitalism, but they are all products of a non-capitalist form of production, sometimes called 'commons-based peer production', which evidently outperforms capitalist production in certain areas just as capitalist production outperformed feudal agriculture. The crucial question for the future of humanity, I think, is how we can extend commons-based peer production to new spheres, and the biggest obstacle seems to be artificial scarcity produced by government-granted monopolies — patent, copyright, and so on.

lstamour(3897) 2 days ago [-]

While I appreciate the recommendations, the language/tone is a bit over the top. Nobody's seriously suggesting you should dump offline slices of Google Maps, or Open Street Map (OsmAnd) in order to use an outdated virtual atlas. It's more like we're reminiscing on how National Geographic (and others) used to actually publish these very large world atlases, and how for it's time, CD-ROM-delivered encyclopedias for the world's primary platform of the day (Windows) were revolutionary.

chrisco255(4148) 2 days ago [-]

Don't pretend like Wikipedia doesn't depend on donations from primarily capitalist economy citizens. Just because an organization is non-profit does not mean it's anti-capitalist, or commons-based. It is still a private organization run by private individuals. They just don't distribute profits to shareholders and their business model is donation-based, just as churches have been for centuries.

I agree that there's a lot of promise for open source resources like Linux, Wikipedia, and Bitcoin, but I see that there's also a great trade-off between Cathedral vs. Bazaar style production.

sureste(10000) 3 days ago [-]

The thing about Encarta to me is that it felt so much more interactive than what we have today.

Nowadays we read a Wikipedia article or watch a Youtube video on a topic but never interact with the content. It's all very passive. I was a child during the heyday of Encarta and I remember I had it and a few other specialized encyclopedias (one about animals, one about sea life, one about dinosaurs!!) and they were all interactive. I miss this about computers. Not all things have to be a game or passive which is what I see nowadays when a child has a screen in front of their face.

cpach(2810) 2 days ago [-]

I never mess interactivity when browsing Wikipedia. I love that WP is text-based, so that I can read it in my own preferred tempo. I rarely watch videos since to me they always feel too slow compared to reading text. I understand on an intellectual level that Youtube is a great platform that provide lots of value and entertainment to lots of people, but I never feel that greatness since it's not something that I personally appreciate. Oh well, to each their own :)

kragen(10000) 2 days ago [-]

What kinds of interactivity are you referring to? I didn't have Encarta. My comment at https://news.ycombinator.com/item?id=20741101 is largely about how interactivity is missing from Wikipedia articles and Kiwix.

blueboo(10000) 2 days ago [-]

People fondly remember using Encarta as children. It was a child's encyclopedia for children.

> In a world of 4k streaming video, global wireless, and high-speed everything, there's really no analog to the feeling we got watching the Moon Landing as a video in Encarta - short of watching it live on TV in the 1969!

This is just absurd nostalgia. Even back in the author's good old days, kids also watched this stuff on VHS and LaserDisc. These days, kids watch SpaceX landing reusable rockets live on their phones.

What was special was being able to freely browse it. Wikipedia crushes it in every imaginable respect, from being more readable, to having more rich media.

It's fine to cherish your memories of Encarta...but kids have it better now and experience the same things more richly. Sorry, dad!

smacktoward(42) 2 days ago [-]

> These days, kids watch SpaceX landing reusable rockets live on their phones.

These days kids type 'space' into the YouTube app on their phone and end up watching videos telling them that the earth is actually flat.

bluedino(2219) 3 days ago [-]

When I was a kid, my grandmother brought home a cardboard box with a 4-5 mis-matched volumes of encyclopedias in it. Always a bookworm, I could read an article about any subject in them and be content for hours.

You can imagine my joy when, omewhere around 1991 or 1992, we got a new computer in the library at school. It was a dedicated station for the encyclopedia, and it was even set up in the same location as the printed ones were.

My faded memory recalls it being Compton's Encyclopedia. It ran on an 286 IBM PS/2 (the all-in-one unit), and it had a mouse, external CD-ROM, and a laser printer. The rumor was that it cost $10,000. It wasn't fast, it probably took closed to a minute to search for, and then retrieve an article.

TransAMrit(10000) 2 days ago [-]

You're close! It was Compton's Interactive Encyclopedia, which I started with as well. When I later saw Encarta, I thought Compton's was a superior product.

meerita(3607) 2 days ago [-]

The problem with products like Encarta was bias and the speed of updating the mistakes, misinformation, etc. I had Encarta, and Internet wasn't an option back then, I had it in a CD. But today it would be nice to have some product that competes with the vastness of Wikipedia.

jonknee(1253) 2 days ago [-]

It competed against very expensive and bulky sets of books, so the update speed was more than acceptable.

penguinlinux(4129) 2 days ago [-]

off topic: If you have a raspberry pi zero w and a large SD card you can download kiwix an offline wikipedia reader that has a built in webserver. It is amazing how you can download wikidedia full or subsets of it and have it running out of a raspi w zero. I am working on a walkthrough on how to do it so I will be posting an article soon.

dingaling(4065) 2 days ago [-]

You don't need an RPi, kiwix is available for desktop OS as well as phones.

Why carry around another device instead of the computer in your pocket?

smacktoward(42) 3 days ago [-]

I would also add that I miss Microsoft Cinemania (https://en.wikipedia.org/wiki/Microsoft_Cinemania), which was to IMDB what Encarta was to Wikipedia.

The sad bit is that in some ways Cinemania in 1997 was superior to IMDB as it is today. IMDB is more comprehensive, but Cinemania was better designed and more fun to just browse through.

turc1656(10000) 2 days ago [-]

I thought I was going to be the first to mention Cinemania but it seems you beat me to it! It was a major influence on how I found great/classic movies to watch. It was really informative and just overall a pretty great piece of software for the time. I distinctly remember discovering Silence of the Lambs on Cinemania and they had a clip of the famous '...ate his liver with some fava beans and a nice chianti' scene and one or two other clips from that film and I knew I had to watch the movie. Really interesting product, for sure.

WorldMaker(10000) 2 days ago [-]

Yeah, I remember some of the historic reviews from Leonard Maltin and Roger Ebert in it being incredibly informative. Particularly where they had good video excerpts of the reviews from their respective syndicated television review segments. (Remember when TV News used to have regular movie reviews? It's almost strange how in the 24/7 news channel era we've lost more types of segments than we've gained.)

mikepurvis(4201) 2 days ago [-]

I came here looking for someone to have mentioned Cinemania— it's crazy to me that Cinemania '97 is now 22 years old. Even a classic like The Matrix is too new to have been part of it.

But yeah, it was a terrific resource; if nothing else it solidified Roger Ebert for young me as a source of truth on movies. :)

tasogare(10000) 2 days ago [-]

I have the one on musical instruments: https://archive.org/details/microsoft-musical-instruments. Spend hours in it even if I wasn't that interested by music.





Historical Discussions: Information operations directed at Hong Kong (August 19, 2019: 1106 points)

(1108) Information operations directed at Hong Kong

1108 points 3 days ago by whatok in 612th position

blog.twitter.com | Estimated reading time – 2 minutes | comments | anchor

We are disclosing a significant state-backed information operation focused on the situation in Hong Kong, specifically the protest movement and their calls for political change.

What we are disclosing This disclosure consists of 936 accounts originating from within the People's Republic of China (PRC). Overall, these accounts were deliberately and specifically attempting to sow political discord in Hong Kong, including undermining the legitimacy and political positions of the protest movement on the ground. Based on our intensive investigations, we have reliable evidence to support that this is a coordinated state-backed operation. Specifically, we identified large clusters of accounts behaving in a coordinated manner to amplify messages related to the Hong Kong protests.

As Twitter is blocked in PRC, many of these accounts accessed Twitter using VPNs. However, some accounts accessed Twitter from specific unblocked IP addresses originating in mainland China. The accounts we are sharing today represent the most active portions of this campaign; a larger, spammy network of approximately 200,000 accounts — many created following our initial suspensions — were proactively suspended before they were substantially active on the service.

All the accounts have been suspended for a range of violations of our platform manipulation policies, which we define as:

  • Spam
  • Coordinated activity
  • Fake accounts
  • Attributed activity
  • Ban evasion

Examples of violative content

This Tweet is unavailable

Next steps Covert, manipulative behaviors have no place on our service — they violate the fundamental principles on which our company is built. As we have said before, it is clear that information operations and coordinated inauthentic behavior will not cease. These deceptive strategies have been around for far longer than Twitter has existed. They adapt and change as the geopolitical terrain evolves worldwide and as new technologies emerge. For our part, we are committed to understanding and combating how bad-faith actors use our services.

Today we are adding archives containing complete Tweet and user information for the 936 accounts we've disclosed to our archive of information operations — the largest of its kind in the industry.

We will continue to be vigilant, learning from this network and proactively enforcing our policies to serve the public conversation. We hope that by being transparent and open we will empower further learning and public understanding of these nefarious tactics. You can access relevant information on these account sets below.

August 2019, set 1:

August 2019, set 2:

Note: We have divided these archives into two parts to better enable review of the intersecting networks of accounts which comprise this campaign. We have evidence to indicate that both sets are associated with the same entity.

This Tweet is unavailable




All Comments: [-] | anchor

tmux314(10000) 2 days ago [-]

Good on Twitter and Facebook.

On top of blocking thousands of websites (which includes Facebook, Google, Twitter) China's government employs thousands of government employees just to purge even the most mild criticism of the CCP on Weibo [1]. They also employ tens of thousands to export their propaganda overseas, using sock puppet accounts to push their worldview[2]. And their worldview is fiercely anti-democratic.

The Internet cannot remain free if we allow governments to use their power to control narratives and suppress the truth. US-based Social media companies are not ideal judges, but at least they publish their methodology and allow public criticism of their platforms.

[1] https://en.wikipedia.org/wiki/Sina_Weibo#Censorship [2] https://en.wikipedia.org/wiki/50_Cent_Party

khawkins(10000) 2 days ago [-]

>The Internet cannot remain free if we allow governments to use their power to control narratives and suppress the truth.

I wish more people applied this same line of reasoning towards the US-based social media companies themselves.

If we hold true that Twitter's influence is so powerful that world superpowers are gaming it effectively to control narratives, then why aren't people more outraged at the near constant censoring of alternative voices in the West? Why are we cheering the use of corporate power when it's wielded against an ideological opponent in the East, but when it's wielded against ideological opponents in the West we hear a chorus of 'it's a private company, it can do what it wants'. Ultimately, we're just rejecting communist state-power with corporatist private-power.

I can agree that the latter is preferable, but I wish people would see the parallels.

kr4(4005) 2 days ago [-]

China's all-weather partner, Pakistan, has been doing similar information warfare against India regarding Kashmir after abrogation of article 370 on Aug5.

'We have found that just after the annulment of Article 370, more than 1,500 bots (fake profiles) from Pakistan surfaced on social media and started trending anti-India narrative. In these tweets, they are also urging United Nations to intervene,' said Tarun Vig, co-founder of Innefu Labs, a cybersecurity firm. He added on August 5 alone, more than 1,000 Twitter accounts were created from Pakistan which was discussing India and Kashmir. Bot profiles were sharing the same tweets over and over again.

0: https://www.msn.com/en-in/news/newsindia/pakistan-bots-wage-...

Some fake videos: https://www.altnews.in/pak-minister-shares-edited-clip-to-fa... https://www.indiatoday.in/fact-check/story/kashmir-massacre-... https://www.republicworld.com/india-news/general-news/isi-ba...

Some twitter action: https://www.hindustantimes.com/india-news/check-our-transpar...

mycall(10000) 2 days ago [-]

Most of those 'sock puppet' accounts are not hard to detect and bucket.

trilila(10000) 2 days ago [-]

Soon there will be comments from those against facebook and twitter moderating such content because "we are loosing freedoms in the west".

monster99(10000) 2 days ago [-]

>The Internet cannot remain free

Uhh have you not been paying attention for the last 20 years? We went from ownign our software and devices to not owning them as the tech illiterate masses got internet and smartphones. Google apps, app stores, steam, mmo's... those are all NOT FREE internet.

They are not there for your benefit when you have to login and your software is controlled by a remote party, you are monitored and controlled.

People in the west seem blind to the fact that they are basically corporate slaves, when a cursory examination of the facts is that they are amongst the most enslaved indoctrinated people in history.

The US is a lawless empire and has been it's founding, if you don't think western governments aren't interested in clamping down on the internet further, just view this by former national security advisor of the United states, Zbigniew Brezinski:

https://www.youtube.com/watch?v=n7ZyJw_cHJY

samstave(3768) 2 days ago [-]

You know whats going to be an interesting dev in the next ten years:

Musk's Skynet!

How will that be regulated. What device signals and routing can be blocked from that.

Imagine a GoPro which is perm-linked up to that constellation and a cloud storage platform that it streams its blobs to. And a not youtube corrupt POS viewing platform.

Get these devices that talk to the constellation and stream continuously and cant be censored or blocked.

We will then realize true information freedom....

C'mon Musk. Don't fuck it up.

ryacko(10000) 2 days ago [-]

US-based social media companies publish more methodology about how they remove Chinese accounts as opposed to how they remove American accounts. It becomes even more absurd considering that Twitter's Trust and Safety Council contains no civil libertarians or anything resembling balanced interests.

kypro(4149) 2 days ago [-]

> The Internet cannot remain free if we allow governments to use their power to control narratives and suppress the truth. US-based Social media companies are not ideal judges, but at least they publish their methodology.

Would it be too cynical to suggest the only reason they're doing this is because they don't operate in mainland China? Do you think they would write a report like this if they found the US government trying to push a pro-democracy narrative into China instead?

This puts them on the right side of US regulators where as exposing a US propaganda campaign, not so much. It's not about truth, or what's morally right, it's about money.

zawerf(3296) 2 days ago [-]

There are pretty fun examples of how people try to dodge censorship.

For example the Hong Kong billionaire Li Kha Shing managed to post a two page newspaper ad which contains a hidden message:

https://i.imgur.com/7IJLn3g.jpg

Translation: https://www.reddit.com/r/news/comments/csg9yz/hong_kong_bill...

woah(3733) 2 days ago [-]

Even here on Hacker News, a week or so ago I saw someone being chided for "breaking the HN guidelines" by calling out a sock puppet. When I looked at the comment history of the account doing the chiding, all of its comments were on China related articles, taking a pro-China view.

daveheq(10000) 2 days ago [-]

The US mainstream media is bought and paid for by corporations, who don't have the public good in mind, but maximizing profit at the expense of all else including reporting on corruption, oligarchy, and environmental abuse.

The government should not control the media, but neither should advertisers.

revel(10000) 2 days ago [-]

The internet is already not free for much of the world. It's depressing to type this but China successfully created a parallel version of the internet that is anything but free. That model got exported and now you see the same approach being taken across the world. The direction we seem to be heading in is one where the world's collective internet systems are effectively Balkanized.

wybiral(3633) 2 days ago [-]

Some of the accounts were so obvious that I accidentally found them just reading tweets in Hong Kong related hashtags.

There were random-looking screen names with stolen images for avatars tweeting almost verbatim the same messages:

https://twitter.com/davywtf/status/1160959728626929671

alephnan(10000) 2 days ago [-]

I immediately thought to lookup Twitter's company about section. Under Values:

> We believe in free expression and think every voice has the power to impact the world.

Edit: I didn't realize I appeared to be supporting the trolls. No, I meant it's good to know Twitter would support the Hong Kong protestors.

djohnston(4084) 2 days ago [-]

authentic voice, not paid gov't troll armies.

Shinobi881(10000) 2 days ago [-]

But yeah "every voice". May have to start explicitly excluding government propaganda and/or "troll armies".

Bhilai(2997) 2 days ago [-]

I am curious to know how do companies like Facebook control this kind of activity on WhatsApp where there is end-end encryption. I see a lot of propaganda on Whats App being circulated everyday and people keep forwarding such messages into various groups.

nobodyshere(10000) 2 days ago [-]

Can you seriously trust WhatsApp's security now that it belongs to Facebook?

the_watcher(3977) 2 days ago [-]

It's hard, and it's an explicit tradeoff made when deciding to prioritize secure communication. One thing Facebook did with WhatsApp was limit group size and monitoring forwarding behavior (which can be done via metadata without needing to know the content), but it's an imperfect solution at best.

FabHK(3948) 2 days ago [-]

I don't see how you can control it.

Which makes it very worrying that lots of people get their 'news' from social media now.

kryogen1c(10000) 2 days ago [-]

This shit is goddamn terrifying. Since when are american corporations the arbiters of accurate, international, and political, information?

We just crossed the rubicon. Public, proactive action against a relatively hostile foreign power by a non-governmental entity. If this isnt proof positive we need to either bust these monopolies or completely overhaul our view of the power megacompanies wield, idk what is.

ganzuul(10000) 2 days ago [-]

The view is being overhauled, slowly. I scantly believed myself the school presentation I did on Menwith Hill in 1999. My classmates' blank stares were removed by degrees from any understanding of the subject. Today children in school here are being taught to be suspicious of what they read on the internet, and not bully each other through the medium, yet said institution is not willing to relinquish their illusion of authority and demand that the kids think for themselves even though what remains is such a small step.

The TV series Max Headroom explored the subject over 30 years ago in quite some detail. It too was way ahead of its time. It's idea of TV viewership democracy is almost benign, compared to the reality of manufactured public opinion that we are waking up to.

Another easily accessible but rage-teasing production is a three-part documentary called The Century of The Self. I recommend it warmly.

JumpCrisscross(45) 2 days ago [-]

> Since when are american corporations the arbiters of accurate, international, and political, information?

They're not and that's not what this story is about.

These accounts were banned for violating Twitter's platform manipulation policy by spamming, co-ordinating activity across multiple accounts, creating fake accounts, mis-attributing their activity and evading bans.

An individual from Hong Kong or China is free to post exactly what these accounts tweeted.

jeremynixon(1810) 2 days ago [-]

From FB:

'We're taking down these Pages, Groups and accounts based on their behavior, not the content they posted.'

https://newsroom.fb.com/news/2019/08/removing-cib-china/

jokoon(4195) 2 days ago [-]

Are you saying that the CIA is not trying to help the protests?

I'm not trying to say that the CIA is involved, but I would be surprised to learn that the US is not trying to get involved to amplify the noise the protests are making.

With Trump and his trade war, don't you think he would not ask the CIA to try to damage the reputation of chinese government by using protests?

I don't like the PRC and their method, but I'm just curious to ask what the US is doing.

yyhhsj0521(10000) 2 days ago [-]

Who says anything about CIA?

Please don't bring whataboutism into the discussion.

meerita(3607) 2 days ago [-]

> Covert, manipulative behaviors have no place on our service

While I celebrate this from them to kill manipulation, this sentence can be applied to their own bias towards conservative people who get banned and progressive don't, even saying worse things or using the service on massive organization to take down accounts. If both parties do harm, they should be take down. But both Twitter and Facebook do enormous efforts to shutdown voices. Including Google and Youtube.

laughinghan(3583) 2 days ago [-]

> their own bias towards conservative people who get banned and progressive don't

Interesting, what leads you to believe this is the case?

I've heard of this happening; I've also heard of the exact opposite happening, e.g. this post which cites examples of specific people and pieces of content that were banned as well as the specific dates and times and durations of the bans: https://medium.com/@thedididelgado/mark-zuckerberg-hates-bla...

I'm sure it is true that there are conservatives who've been banned and self-proclaimed 'progressives' who've done as bad or worse things who haven't. Hopefully you also agree that the examples in that post, of progressives being banned while racists who should have been banned but weren't, probably really did happen.

Due to scale, it wouldn't surprise me that all of those examples could be true without there being any systemic bias on their service. What leads you to believe that such a bias exists?

From what I've read, there's actually some reason to believe the exact opposite bias exists: https://www.propublica.org/article/facebook-hate-speech-cens...

Again, I think all those examples are probably true. But I think they could be consistent with a bias in any direction, due to scale.

groundlogic(4014) 2 days ago [-]

I don't think HN is immune to these kinds of CCP ops.

vkou(10000) 2 days ago [-]

The problem is that calling someone a paid, or unpaid shill on HN is unproductive.

There's a world of difference (But little way for a third party to tell apart) between:

A paid shill.

An unpaid troll.

Someone who genuinely believes what they are posting, because they do not have the same information you do.

Someone who genuinely believes what they are posting, despite having the same information as you, because they prioritise different bits of that information differently. That is called a 'political opinion'.

For reference, I have been called a CCP shill at least once. I have never been to China, I have no affiliation with the CCP, I am not trolling, and I have not received a penny for posting nonsense here.

sdinsn(10000) 2 days ago [-]

The mods here state that HN is immune. Anyone that calls out potential hostile information operations is threatened by the mods.

kelnos(3832) 2 days ago [-]

I doubt HN is even on the CCP's radar. Not only are we a low-value target, we're much harder to subvert.

mortenjorck(1080) 2 days ago [-]

The elephant left standing in the room is not the spam and bot accounts, but the official, state-actor accounts such as Xinhua News that pay for promoted tweets carrying the same message.

Granted this is likely beyond the scope of Twitter's safety team, but this completely sidesteps the issues raised here: https://techcrunch.com/2019/08/19/twitter-is-blocked-in-chin...

martey(2079) 2 days ago [-]

See https://blog.twitter.com/en_us/topics/company/2019/advertisi..., which was posted to the Twitter blog at approximately the same time as this post.

ktln2(10000) 2 days ago [-]

Yes basically this is a state sponsored misinformation campaign.

theirkf(10000) 2 days ago [-]

So this has become the norm now. Social media products who's primary utility is to share information between people are now being used by governments to sow discord.

Shinobi881(10000) 2 days ago [-]

Nothing new except the medium.

novok(10000) 2 days ago [-]

Your basically complaining that paper can be used to write whatever you want on it, by you and by people you don't like.

colechristensen(10000) 2 days ago [-]

They're just yet another business selling user's attention. That's not new. The utility was never users sharing with each other, it was always users providing minable information.

All of that information sharing just makes the sold attention more valuable to the buyers.

countryqt30(3872) 2 days ago [-]

Chinese people don't even have the most basic human rights. That's insane.

acros(10000) 2 days ago [-]

Yes.But they also have different understanding in human rights. Most chinese think death penalty is necessary ; Or in public security, privacy right should make a concession. Westerns may think it's ridiculous.

b_tterc_p(10000) 2 days ago [-]

Maybe one day our cyberpunk corporate overlords will be able to fine governments for this kind of shit.

crispyambulance(3868) 2 days ago [-]

Ain't gonna happen.

In reality, our cyberpunk overlords will fall over themselves to provide the best level of service to the highest paying autocratic government customers.

flying_sheep(3586) 2 days ago [-]

I was living in HK for 30 years. What I want to say is that we all need to prepare for a new form of propaganda, which is partially based on partial fact but with twisted explanation. They are getting more logical, therefore more persuasive to middle-classes. If we do not have freedom or human rights in our minds, it will be very hard to deny those propaganda.

mef(414) 2 days ago [-]

Historically the most effective propaganda has been based partially on fact. Outright fabrication doesn't work as well as taking an accepted fact and twisting what came before or what will come after.

guerrilla(10000) 2 days ago [-]

.

smt88(4192) 1 day ago [-]

Instead of redacting your comment, you should admit that you were wrong and learned something. I've never seen someone do this on HN before, and it's annoying to see it.

gtirloni(2269) 2 days ago [-]

You should read the linked article.

hker(3801) 2 days ago [-]

For those who want to understand the bigger picture of this propaganda campaign, check out this allegedly leaked instruction for how China controls media (picture in Chinese [1], transcribed in Chinese [2], google translated to English [3]).

[1]: https://pincong.rocks/article/3572

[2]: https://pastebin.com/anMtU0Lz

[3]: https://translate.google.com/translate?hl=en&sl=auto&tl=en&u...

LilBytes(10000) 2 days ago [-]

[3] - what a read.

'5. Video images of excessive use of force by police from outside the country must be promptly blocked and deleted at the first time, and it is absolutely forbidden to spread on WeChat and any social media.'

ryanchankh(10000) 2 days ago [-]

HongKonger here. I have some friends in China posting similar anti-protest posts on WeChat social media. It's like the news they read has a completely different story than what it's being told in legitimate new sources. The problem of fake news does become very apparent, and I hope people in China can eventually gain awareness or at least start to question the validity of their news sources.

ospider(4094) 2 days ago [-]

Native Chinese here. Hacker news have been a great place to learn new things to me for 5 years. But the political views on Hacker news are somewhat naive to me. It seems that the Chinese Government is always evil and wrong, but why haven't the government collapsed after so many years if there were no people supporting them?

People in China, at least those millions people who are able to cross the Great Firewall, know that democracy is generally good, but they also know that a strong central government can also be useful for certain circumstances. Most westerners and HongKongers on Hacker news have a very extreme political view, you just believe 'democracy is good'(TM), protesting against the evil Chinese government is good. But can you take a closer look at what is really happening in HK and then decide what you believe?

BTW, I'm neither pro-protester nor pro-police, I think the protest is a result of economic regression in HK. You could also check my comment and posting history to see that I'm not a 五毛党.

Aperocky(10000) 2 days ago [-]

It's not necessarily fake, just very selective. When all you hear/see are videos of molotov cocktail, beating the reporter up for having a pro-police tshirt, and british/american flag waving (All of which did happen). You'll have a pretty different image of the protestors.

I have to say this selection bias exist on the other end as well, perhaps not as blatant as mainland, where the girl gotten her eye shot or the triad beating people wasn't shown at all.

mycall(10000) 2 days ago [-]

Disinformation is very popular right now. It is happening everywhere. Humans have much power with social media, it is getting abused. The good news is that Facebook and Twitter are finding ways to slow it down. Maybe that technology can be imported into WeChat somehow.

rjf72(4028) 2 days ago [-]

Interesting. Are the images shared from this article suggesting that protesters are destroying property and engaging in other such behavior fake or somehow being taken out of context?

NicoJuicy(369) 2 days ago [-]

I think those that don't agree, just don't post it. Being scared of the state and their score.

Not sure though.

But since 996, I think most of them are pretty aware what is happening abroad ( sometimes)

ma2rten(2829) 2 days ago [-]

I think it's not just about censorship. Many mainland Chinese abroad who have access to unfiltered media still support Beijing.

Chinese people generally value stability and harmony over individual rights. So, why would mainland Chinese people support violent protesters who protest for Hong Kong having more individual rights than the mainland does?

echelon(4022) 2 days ago [-]

It's been awhile since I've been over there. How easy is it for those in mainland China to get access to banned tv shows and films? I remember lots of DVD content being sold on the street, but I never came across anything that was in direct opposition to the politics of the regime.

What if Netflix (or some other production company) produced a compelling show with an incredibly critical narrative about the PRC and its leadership? Would citizens in mainland China be able to access it through roundabout means? Would you be able to sow the seeds of democracy? Would they even be interested?

The last time I was on campus in Shanghai and Beijing, half of those I spoke to were critical of the party and did not believe China would catch up to the West. The other half were total ideological zealots and made me know that my government was inferior.

Given my vantage point, I believe the citizenry is at a precipice, but there is no inciting moment to tip the scale. Life in China is comfortable for the burgeoning middle class, and nobody would want to rock that boat.

In any case, I think an unfettered political drama about the PRC would be fascinating to watch.

throwaway-43958(10000) 2 days ago [-]

I have a Chinese friend who went to an Ivy League University, married an American, and currently works for a FAANG company and lives in the Bay Area. Yet they publicly support the PRC Government on Facebook and vehemently attack anyone who express support of Hong Kongers.

People believe in what they choose to believe.

kevin_b_er(10000) 2 days ago [-]

When the fake news is generated and controlled by a government entity, I think the better term is 'propaganda'.

ggg3(10000) 2 days ago [-]

same in brazil and crimea and before that arabspring.

the obvious fake content on youtube and facebook was reported and removed, but all it did was hide the traces of the content that flooded private whatsapp groups.

sp332(313) 2 days ago [-]

This doesn't really explain the money they took from Xinhua News to promote certain tweets that lied about the protests being violent or unwanted by the majority of HK residents. https://twitter.com/pinboard/status/1162711159000055808 https://www.businessinsider.com/twitter-is-running-paid-ads-...

bouncycastle(10000) 2 days ago [-]

One of the strangest things I seen so far was not online but on the street in Sydney, Australia. There was an 'anti-democracy protest' https://www.smh.com.au/world/asia/anti-hong-kong-protest-in-...

It looks people who oppose Hong Kongers really exist and truly hold on to their beliefs.

ajdlinux(1949) 2 days ago [-]

On the street in several cities in Australia so far.

diNgUrAndI(10000) 2 days ago [-]

Why not? Even within HK there are thousands pro-police anti-protest supporters.

https://www.hongkongfp.com/2019/08/17/oppose-violence-save-h...

Our opinions are shaped by what media are fed into our brains. I saw my friends divided into two camps because of the different media articles they read.

How do you make sure you are not biased and only read what you believe? Rad more and find the opposing articles!

diNgUrAndI(10000) 2 days ago [-]

What about 'pro-democracy' activists using twitter or fb smearing the police with their own version of truth?

It seems like twitter judges which version of truth to allow based on whether twitter censorship team likes the arguments or not.

dqpb(4030) 2 days ago [-]

It might seem like that if you don't read the article.

If you did read the article, you would see this:

> All the accounts have been suspended for a range of violations of our platform manipulation policies, which we define as:

- Spam

- Coordinated activity

- Fake accounts

- Attributed activity

- Ban evasion

woutr_be(10000) 2 days ago [-]

I think it depends, pro-China camp uses bots to share fake news, or out of context articles. While pro-Democracy are just individuals sharing posts where the truth is bend in their favour.

I do find it surprising that the pro-Democracy camp is allowed to share the same post / image, but when pro-China does it, they're all marked as bots or '50cents'. It's their way to delegitimise the posts.

ktln2(10000) 2 days ago [-]

China has been doing this for a while, not only on social media, but traditional media / institutions too. It's about time to get serious about this problem.

https://www.hoover.org/news/china-exerting-sharp-power-influ...

prewett(4201) 2 days ago [-]

It's inherent in all communist states (that have existed). There is a document called 'Chapter NN', where NN is some number, I think in the teens, from eastern Europe that describes the dynamic. Unfortunately I have no idea how to search for it, so if someone could post a link I'd be grateful. If I recall correctly, the problem is that the State claims to represent the people, but in fact it does not represent the people. Thus, it must lie in everything it says, otherwise it could not claim to represent the people.

xt00(10000) 2 days ago [-]

Was in China when the first protest happened and the news coverage was mostly saying things like that the people are clueless in Hong Kong and don't understand the law about extradition. They had various experts all saying similar things and they would also point to random weibo posts about totally not on the payroll "people" saying they want their city back from the protesters.. I mean it's one thing if China'S central government wants Hong Kong to be a certain way but dressing it up as "lots of ordinary folks agree with them" is pretty disingenuous and it seems very fair for any platform to call the government out on it—-especially non-Chinese owned platforms.

derefr(3674) 2 days ago [-]

> but dressing it up as "lots of ordinary folks agree with them" is pretty disingenuous

I wouldn't say it's disingenuous... given that there's nothing legally preventing mainland-Chinese from choosing to live in HK, and then being quoted as people who want 'their' city back.

Odd to think, but HK is now experiencing a real Red Scare, the likes of which the US only ever imagined in the depths of paranoid delusion. There are real (CCP) Communists, living amongst the HK citizenry, nudging HK's democracy (such as it is) toward their agenda by appearing to stand for 'public sentiment'!

rolltiide(10000) 2 days ago [-]

> was mostly saying things like that the people are clueless in Hong Kong and don't understand the law about extradition

It is important to understand there is some truth in the statement, while it neglects the symbolism that Hong Kong citizens have rallied behind.

It is no longer important, but the possibility of extradition to mainland China was a very small part of the bill, which updated a long list of countries that extradition would be possible to. It also lowered the number of crimes that were extraditable.

So on its face it was a meaningful updating of an old law and mostly kept the existing process which was full of appeals and approvals of various departments to actually extradite someone, but the mere inclusion of one particular country allowed everyone to channel their disdain for crossing an ideological line.

So when the first protest happened, China's cherry picking of the issue is not at all inaccurate. Coupled with the understanding that all the disdain comes towards them, from their own autonomous region, it is pretty embarrassing.

Gabriel_Martin(3831) 2 days ago [-]

I wonder what other sites are seeing this, Reddit and Instagram are pretty obvious Quora perhaps?

markdown(3399) 2 days ago [-]

/r/sino has been working overtime of late, and sending their minions all over reddit to push their message.

throwawaysea(10000) 2 days ago [-]

What makes coordinated activity from government entities different from brigading performed in coordinated ways by non-government people with political or activist agendas? Isn't that also manipulation? Or is it that Twitter just has its own political and economic agenda and their TOS/rules/blog posts are self-justification?

jancsika(10000) 2 days ago [-]

A government from a wealthy nation can surreptitiously force Twitter off the internet indefinitely by [choose your favorite attack here].

A government from a wealthy nation could also send law enforcement into Twitter HQ at any time and force Twitter to shut down another government's propaganda campaign.

In conclusion, 'the ability to start WWIII' is an important, obvious, and relevant capability that 'non-government people' do not possess.

woutr_be(10000) 2 days ago [-]

In my opinion; the main difference here is that the government entities used bots or paid people to post propaganda. While the pro-democracy non-government camp used their own account to share their views. This I'm okay with.

But a good example on how not to do it is the /r/hongkong subreddit, it's filled with pro-democracy accounts who will delegitimise any posts that goes against the general sentiment. It's impossible to have a constructive discussion there without being marked as a troll, or a pro-china bot. That to me is just the same as spreading propaganda and preventing free speech. That's not ok.

JumpCrisscross(45) 2 days ago [-]

What can technically-savvy Americans with moderate resources on hand do to help the protesters?

lucb1e(2123) 2 days ago [-]

(Or from any other place that is not HK, for that matter.)

KerrickStaley(3780) 2 days ago [-]

Do what you can to make democracy stronger at home. In the long run, what will help Hong Kong and other places that are at crossroads on the path to fully democratic rule is to have strong examples of thriving democracies that they can look to.

29_29(4123) 2 days ago [-]

I was recently in Beijing and noticed when posting on Instagram many bot accounts posting 'Shame on Hong Kong' in Chinese.

This is a Facebook problem too. If you work at facebook please help!

29_29(4123) 2 days ago [-]

You could probably still view these using any geotag of a major Chinese city.

bertil(10000) 2 days ago [-]

Flag all the accounts that seem suspicious. The more signal, the easier for Threat Intelligence to identify them.

r1b(4191) 2 days ago [-]

Would twitter, fb et al ever expose a Western intelligence operation with this tone?

See https://en.m.wikipedia.org/wiki/Manufacturing_Consent

Edit: In other words, is this post itself not an "information operation"?

Schnitz(10000) 2 days ago [-]

A foreign state actor is using a platform maintained by a US-based company to spread anti-democratic propaganda that goes against the values of the US. They got blocked and called out. Sounds like a no-brainer to me, especially after the 2016 election.

AnthonyWnC(10000) 2 days ago [-]

Let me guess, all pro-China tweeks are blocked but anti-China are not blocked. Keep toeing the party line Twitter.

interpenetrate(10000) 2 days ago [-]

you gotta love the language: 'coordinated state-backed operations' are banned. such careful language. what does that make USAID, who is undoubtedly responsible for shaping this discourse? perhaps an 'uncoordinated' state-backed operation? and what does that make twitter?

it's not enough to call everyone here 'dumb' or 'sheeps,' because all of the middle-class americans on this wretched forum are specifically acting in their self-interest by helping (in whatever pathetic way they can) to topple china. just because they won't admit to this outright and instead take advantage of careful language ('democracy' etc.) doesn't make them dumb. historically it has made them a real threat and that it is exactly why china has to issue such tight command over this conflict (and the 'uyghur problems', and 'free tibet', and 'tank man', and ...). luckily american hegemony is waning and the front against china will fail. just sit, wait and enjoy the shit show.

ThinkBeat(10000) 2 days ago [-]

An analysis of information operations that caused and more importantly sustained the protests would be equally interesting.

I am 90% certain that foreign intelligence services or service are actively involved in the riots.

Probably funding it at least partially and inciting the crowd. Probably have assets in the group of protesters to direct their activity.

I have written before that China has been far more lenient than the US would have been under similar circumstances.

The moment someone had broken into congress they would most likely get shot, at least enough to stop them.

Shutting down an airport would mean spending quite a bit of time at a 'detainment camp'. Often in warehouses created by law enforcement to hold a large number of people outside of regular courts and jails (See the DNC in Denver).

The protesters are, for whatever reason, hell-bent on forcing China to act in a brutal manner. Any regime would.

mythrwy(10000) 2 days ago [-]

While that possibly might be, there are legitimate grievances when you see that many people in the street.

I'd say this is almost certainly not instigated by foreign intelligence services and I'm skeptical they would risk trying to run it. (they probably aren't too broken up about it happening though).

envy2(3347) 2 days ago [-]

Facebook made a similar announcement today: https://newsroom.fb.com/news/2019/08/removing-cib-china/

doubleunplussed(10000) 2 days ago [-]

Wow, those posts. They are so unsubtle.

avip(4065) 2 days ago [-]

My YT overlord lately decided I should watch Jackie Chan commentates about the situation in Hong-Kong [0]. I was very surprised to see the English comments almost unanimously support China. Would have expected the opposite sentiment.

[edit: stupid me, the uploader is CGTN, a well-known state propoganda channel]

[0] https://www.youtube.com/watch?v=M7rIg49I0yI

mytailorisrich(10000) 2 days ago [-]

Jackie Chan is entitled to his opinions.

During those protests there have been rioting and ransacking of the local Parliament with some expressing views that HK was not, or should not be, China. This can genuinely trigger feelings of patriotism and outrage.

The lastest protest seems to have been peaceful and to have respected the authorities' restrictions as much as possible so it may be that they have understood that using violence was playing into the hands of the other side.

kweks(4195) 2 days ago [-]

Jackie Chan has a very pro-China stance. In his autobiography, he mentions that he is proud to be Chinese, and makes several associations to Chinese culture, seemingly over Hong Kong.

I found it to be a curious position, seeing as he is adored by Hong Kongers, and always seemed to identify as / with HK.

apo(882) 2 days ago [-]

> We are disclosing a significant state-backed information operation focused on the situation in Hong Kong, specifically the protest movement and their calls for political change.

Two points:

1. The post presents no evidence that this is a 'state-backed' operation.

2. Censorship seems like a wonderful tool when applied to your opponents. That support dries up pretty quickly when the censor targets you.

ddlsmurf(10000) 2 days ago [-]

Is there another actor than the state in China that can get unfiltered mainland IPs ?

rjf72(4028) 2 days ago [-]

This seems like a very slippery slope.

Recently Jackie Chan and Liu Yifei are a couple of higher profile individuals have expressed support for the mainland or condemnation of the protesters' riotous behavior. Jackie Chan is a Hong Kong national - Lieu Yifei is Chinese American. The reason that this is relevant is because we live in the day and age where holding the wrong opinion is increasingly frequently grounds for getting black listed or worse. They feel strongly enough that they're willing to risk an immense amount, just to share their views and values.

They they feel so strongly about this issue is reasonable evidence that many others also likely feel strongly about this issue, and not necessarily in support of the protesters. Consequently, there are undoubtedly large numbers of people expressing genuine negative views against the protesters. How many of these individuals ended up getting banned? It's twitter. A huge chunk of the userbase posts like bots. There's not exactly a whole lot of insightful discussion going on with 140 characters + memes. When you get into political talking points where real people are simply repeating the same talking points at one another, this gets even more extreme.

kevin_b_er(10000) 2 days ago [-]

Jackie Chan has, for a good few years now, become negative toward HK and Taiwan and highly nationalistic for Beijing.

The CPC has orchestrated media personalities to also produce similar/identical statements of pre-prepared condemnation. Chan's words are just another angle of CPC's propaganda and disinformation campaigns.

sephamorr(10000) 2 days ago [-]

I just looked through Jackie Chan's movies for the past few years, and they're almost exclusively produced by mainland Chinese film companies. I understand he's a Hong Kong national, but it seems that he is toeing the line for the movie industry he's working for. Considering how the Chinese have cracked down on Cathay Pacific (forcing them to fire employees who have participated in the protests), I doubt a mainland movie star could get away with saying anything else.

threeseed(10000) 2 days ago [-]

Sure some people may support the Chinese government position. That has nothing to do with Twitter banning coordinated propaganda campaigns.

And China has the concept of ambassadors which are paid influencers there to assist in these campaigns. So I wouldn't blindly assume every famous actor has an authentic position.

kyledrake(1298) 2 days ago [-]

> Recently Jackie Chan and Liu Yifei are a couple of higher profile individuals have expressed support for the mainland

Yeah I wouldn't want to get arrested for 'tax fraud' either, especially if I was listed in the Panama Papers like Chan was.

But granted, it could also just be the actual crank opinion of the guy from a cheesy fighting movie about crooked cops in LA or something and why should I care? HK's government gives special powers to the wealthy and entitled and that's a big part of the problem with their democracy. Obviously the wealthy and entitled are by-and-large going to advocate for maintaining that status quo, just like they do here for our less codified version.

AnthonyWnC(10000) 2 days ago [-]

The Western media basically surpassed any news of violence in the protests, barely mention any anti-protest rally in Hong Kong. What a surprise then that you have bunch of ignorant people who eat this up like a bunch of sheep.

chibg10(3914) 2 days ago [-]

This isn't true at all, at least in the sources I was reading (NYT, WSJ). Entire paragraphs were devoted to protester violence in nearly every article. This weekend's protest was even described as peaceful in contrast to the previous weeks' protest.





Historical Discussions: How to Build Good Software (August 19, 2019: 981 points)
How to Build Good Software (August 16, 2019: 20 points)
How to Build Good Software (August 18, 2019: 4 points)

(996) How to Build Good Software

996 points 3 days ago by jingwen in 3752nd position

www.csc.gov.sg | Estimated reading time – 21 minutes | comments | anchor

Why Bad Software Happens to Good People

Bad software is one of the few things in the world you cannot solve with money. Billion dollar airlines have flight search apps that are often inferior to those built by groups of students. Established taxi companies the world over have terrible booking apps despite the threat they face from ride-sharing services. And painful corporate IT systems are usually projects with massive budgets, built over the course of many years. Whatever the cause of bad software is, it does not seem to be a lack of funding.

Surprisingly, the root cause of bad software has less to do with specific engineering choices, and more to do with how development projects are managed. The worst software projects often proceed in a very particular way:

The project owners start out wanting to build a specific solution and never explicitly identify the problem they are trying to solve. They then gather a long list of requirements from a large group of stakeholders. This list is then handed off to a correspondingly large external development team, who get to work building this highly customised piece of software from scratch. Once all the requirements are met, everyone celebrates as the system is launched and the project is declared complete.


The root cause of bad software has less to do with specific engineering choices, and more to do with how development projects are managed.

However, though the system technically meets specifications, severe issues are found when it is put in the hands of actual users. It is slow, confusing, and filled with subtle bugs that make using it an exercise in frustration. Unfortunately, by this time the external development team has been dismissed and there are no resources left over to make the necessary fixes. By the time a new project can be initiated years later, all knowledge of what caused these problems has left the organisation and the cycle starts over again.

The right coding language, system architecture, or interface design will vary wildly from project to project. But there are characteristics particular to software that consistently cause traditional management practices to fail, while allowing small startups to succeed with a shoestring budget:

• Reusing good software is easy; it is what allows you to build good things quickly; • Software is limited not by the amount of resources put into building it, but by how complex it can get before it breaks down; and • The main value in software is not the code produced, but the knowledge accumulated by the people who produced it.

Understanding these characteristics may not guarantee good outcomes, but it does help clarify why so many projects produce bad outcomes. Furthermore, these lead to some core operating principles that can dramatically improve the chances of success:

1. Start as simple as possible; 2. Seek out problems and iterate; and 3. Hire the best engineers you can.

While there are many subtler factors to consider, these principles form a foundation that lets you get started building good software.

Reusing Software Lets You Build Good Things Quickly

Software is easy to copy. At a mechanical level, lines of code can literally be copied and pasted onto another computer. More generally, the internet is full of tutorials on how to build different kinds of systems using ready-made code modules that are available online. Modern software is almost never developed from scratch. Even the most innovative applications are built using existing software that has been combined and modified to achieve a new result.

The biggest source of reusable code modules is the open source community. Open source software is software in which code is freely published for anyone to see and use. Many of the largest contributors to the open source community are giant tech companies. If you want to use a state-of-the-art planet scalable database as Facebook does, just download the code for Cassandra that they open sourced in 2008. If you want to try out Google's cutting-edge machine learning for yourself, download the TensorFlow system published in 2015. Using open source code does not just make your application development faster, it gives you access to technology that is far more sophisticated than anything you could have developed yourself. For the most popular open source code, it is even more secure as there are many more people paying attention and fixing vulnerabilities. This is the reason digital technology has made such rapid progress: even the newest engineers can build upon the most advanced tools our profession has to offer.

The advent of cloud services has taken reusability even further, offering the full use of even proprietary systems for just a subscription fee. Need a simple website? Just configure one in a few clicks using a website building service like Squarespace or Wix. A database? Subscribe to a virtual one from Amazon Web Services or Microsoft Azure. Cloud services allow developers to benefit from specialisation; the service provider handles the setup, maintenance, and continued development of a reliable, high-quality piece of software that is used by all its subscribers. This allows software developers to stop wasting time on solved problems and instead focus on delivering actual value.

You cannot make technological progress if all your time is spent on rebuilding existing technology. Software engineering is about building automated systems, and one of the first things that gets automated away is routine software engineering work. The point is to understand what the right systems to reuse are, how to customise them to fit your unique requirements, and fixing novel problems discovered along the way.


Software engineering is about building automated systems, and one of the first things that gets automated away is routine software engineering work.

Software Is Limited by Complexity

How useful a piece of software can be is usually limited by its complexity rather than the amount of resources invested in building it.

IT systems are often full of features but are still hated by users because of how confusing they become. In contrast, highly ranked mobile apps tend to be lauded for their simplicity and intuitiveness. Learning to use software is hard. Beyond a point, new features actually make things worse for users because the accumulated complexity starts to become overwhelming. For example, after serving as the hub of Apple's media ecosystem for almost 20 years, iTunes was split into three different apps (for music, podcasts, and TV shows) this year, as its features had grown too complex for one app to handle. From a usability perspective, the limit is not how many features can be implemented, but rather what can fit into a simple intuitive interface.

Even ignoring usability, engineering progress slows to a halt once a project becomes too complex. Each new line of code added to an application has a chance of interacting with every other line. The bigger an application's codebase, the more bugs are introduced whenever a new feature is built. Eventually, the rate of work created from new bugs cancels out the rate of work done from feature development. This is known as "technical debt" and is the main challenge in professional software development. It is the reason why many large IT systems have issues that go unfixed for years. Adding more engineers to the project just adds to the chaos: they start running faster in place as the codebase keels over from its own weight.


Building good software involves alternating cycles of expanding and reducing complexity.

In such cases, the only way forward is to take a step back to rationalise and simplify the codebase. The system architecture can be redesigned to limit unexpected interactions. Non-critical features can be removed even if they have already been built. Automated tools can be deployed to check for bugs and badly written code. Bill Gates once said "Measuring programming progress by lines of code is like measuring aircraft building progress by weight". Human minds can only handle a finite amount of complexity, so how sophisticated a software system can get depends on how efficiently this complexity budget is used.

Building good software involves alternating cycles of expanding and reducing complexity. As new features are developed, disorder naturally accumulates in the system. When this messiness starts to cause problems, progress is suspended to spend time cleaning up. This two-step process is necessary because there is no such thing as platonically good engineering: it depends on your needs and the practical problems you encounter. Even a simple user interface such as Google's search bar contains a massive amount of complexity under the surface that cannot be perfected in a single iteration. The challenge is managing this cycle, letting it get messy enough to make meaningful progress, but not letting it get so complicated that it becomes overwhelming.


There is no such thing as platonically good engineering: it depends on your needs and the practical problems you encounter.

Software Is about Developing Knowledge More than Writing Code

In software development, most ideas are bad; this is not anyone's fault. It is just that the number of possible ideas is so large that any particular idea is probably not going to work, even if it was chosen very carefully and intelligently. To make progress, you need to start with a bunch of bad ideas, discard the worst, and evolve the most promising ones. Apple, a paragon of visionary design, goes through dozens of prototypes before landing on a final product. The final product may be deceptively simple; it is the intricate knowledge of why this particular solution was chosen over its alternatives that allows it to be good.

This knowledge continues to be important even after the product is built. If a new team takes over the code for an unfamiliar piece of software, the software will soon start to degrade. Operating systems will update, business requirements will change, and security problems will be discovered that need to be fixed. Handling these subtle errors is often harder than building the software in the first place, since it requires intimate knowledge of the system's architecture and design principles.

In the short term, an unfamiliar development team can address these problems with stopgap fixes. Over time though, new bugs accumulate due to the makeshift nature of the additional code. User interfaces become confusing due to mismatched design paradigms, and system complexity increases as a whole. Software should be treated not as a static product, but as a living manifestation of the development team's collective understanding.


Software should be treated not as a static product, but as a living manifestation of the development team's collective understanding.

This is why relying on external vendors for your core software development is difficult. You may get a running system and its code, but the invaluable knowledge of how it is built and what design choices were made leaves your organisation. This is also why handing a system over to new vendors for "maintenance" often causes problems. Even if the system is very well documented, some knowledge is lost every time a new team takes over. Over the years, the system becomes a patchwork of code from many different authors. It becomes harder and harder to keep running; eventually, there is no one left who truly understands how it works.

For your software to keep working well in the long term, it is important to have your staff learning alongside the external help to retain critical engineering knowledge in your organisation.

3 Principles for Good Software Development

1. Start as Simple as Possible

Projects that set out to be a "one-stop shop" for a particular domain are often doomed. The reasoning seems sensible enough: What better way to ensure your app solves people's problems than by having it address as many as possible? After all, this works for physical stores such as supermarkets. The difference is that while it is relatively easy to add a new item for sale once a physical store is set up, an app with twice as many features is more than twice as hard to build and much harder to use.

Building good software requires focus: starting with the simplest solution that could solve the problem. A well-made but simplistic app never has problems adding necessary features. But a big IT system that does a lot of things poorly is usually impossible to simplify and fix. Even successful "do it all" apps like WeChat, Grab, and Facebook started out with very specific functionality and only expanded after they had secured their place. Software projects rarely fail because they are too small; they fail because they get too big.


Software projects rarely fail because they are too small; they fail because they get too big.

Unfortunately, keeping a project focused is very hard in practice: just gathering the requirements from all stakeholders already creates a huge list of features.

One way to manage this bloat is by using a priority list. Requirements are all still gathered, but each are tagged according to whether they are absolutely critical features, high-value additions, or nice-to-haves. This creates a much lower-tension planning process because features no longer need to be explicitly excluded. Stakeholders can then more sanely discuss which features are the most important, without worrying about something being left out of the project. This approach also makes explicit the trade-offs of having more features. Stakeholders who want to increase the priority for a feature have to also consider what features they are willing to deprioritise. Teams can start on the most critical objectives, working their way down the list as time and resources allow.

We followed a similar process for all our most successful apps. Form.gov.sg started out as a manual Outlook Macro that took us six hours to set up for our first user but today has processed about a million public submissions. Data.gov.sg started out as a direct copy of an open source project and has since grown to over 300,000 monthly visits. Parking.sg had a massive list of almost 200 possible features that we never got around to building but still has over 1.1 million users today. These systems are well received not in spite of their simplicity but because of it.

2. Seek Out Problems and Iterate

In truth, modern software is so complicated and changes so rapidly that no amount of planning will eliminate all shortcomings. Like writing a good paper, awkward early drafts are necessary to get a feel of what the final paper should be. To build good software, you need to first build bad software, then actively seek out problems to improve on your solution.

This starts with something as simple as talking to the actual people you are trying to help. The goal is to understand the root problem you want to solve and avoid jumping to a solution based just on preconceived biases. When we first started on Parking.sg, our hypothesis was that enforcement officers found it frustrating to have to keep doing the mental calculations regarding paper coupons. However, after spending just one afternoon with an experienced officer, we discovered that doing these calculations was actually quite simple for someone doing it professionally. That single conversation saved us months of potentially wasted effort and let us refocus our project on helping drivers instead.

Beware of bureaucratic goals masquerading as problem statements. "Drivers feel frustrated when dealing with parking coupons" is a problem. "We need to build an app for drivers as part of our Ministry Family Digitisation Plans" is not. "Users are annoyed at how hard it is to find information on government websites" is a problem. "As part of the Digital Government Blueprint, we need to rebuild our websites to conform to the new design service standards" is not. If our end goal is to make citizens' lives better, we need to explicitly acknowledge the things that are making their lives worse.

Having a clear problem statement lets you experimentally test the viability of different solutions that are too hard to determine theoretically. Talking to a chatbot may not be any easier than navigating a website, and users may not want to install yet another app on their phones no matter how secure it makes the country. With software, apparently obvious solutions often have fatal flaws that do not show up until they are put to use. The aim is not yet to build the final product, but to first identify these problems as quickly and as cheaply as possible. Non-functional mock-ups to test interface designs. Semi-functional mock-ups to try different features. Prototype code, written hastily, could help garner feedback more quickly. Anything created at this stage should be treated as disposable. The desired output of this process is not the code written, but a clearer understanding of what the right thing to build is.


Beware of bureaucratic goals masquerading as problem statements. If our end goal is to make citizens' lives better, we need to explicitly acknowledge the things that are making their lives worse.

With a good understanding of the right solution, you can start work on building the actual product. You stop exploring new ideas and narrow down to identifying problems with your particular implementation. Begin with a small number of testers who will quickly spot the obvious bugs that need to be fixed. As problems are addressed, you can increasingly open up to a larger pool who will find more esoteric issues.

Most people only give feedback once. If you start by launching to a large audience, everyone will give you the same obvious feedback and you'll have nowhere to go from there. Even the best product ideas built by the best engineers will start out with significant issues. The aim is to repeatedly refine the output, sanding down rough edges until a good product emerges.

Even after all this iteration, after launch is when problems with a product matter the most. A problem that happens only 0.1% of the time may not get noticed during testing. But once you have a million users, every day the problem goes unresolved is a thousand more angry people you have to deal with. You need to fix problems caused by new mobile devices, network outages, or security attacks before they cause substantial harm to your users. With Parking.sg we built a series of secondary systems that continuously check the main system for any discrepancies in payments, duplicate parking sessions, and application crashes. Building up an "immune system" over time lets you avoid being overwhelmed as new issues inevitably come up.

Overall, the approach is to use these different feedback loops to efficiently identify problems. Small feedback loops allow for quick and easy correction but miss out on broader issues. Large feedback loops catch broader issues but are slow and expensive. You want to use both, resolving as much as possible with tight loops while still having wide loops to catch unexpected errors. Building software is not about avoiding failure; it is about strategically failing as fast as possible to get the information you need to build something good.

3. Hire the Best Engineers You Can

The key to having good engineering is having good engineers. Google, Facebook, Amazon, Netflix, and Microsoft all run a dizzying number of the largest technology systems in the world, yet, they famously have some of the most selective interview processes while still competing fiercely to recruit the strongest candidates. There is a reason that the salaries for even fresh graduates have gone up so much as these companies have grown, and it is not because they enjoy giving away money.

Both Steve Jobs and Mark Zuckerberg have said that the best engineers are at least 10 times more productive than an average engineer. This is not because good engineers write code 10 times faster. It is because they make better decisions that save 10 times the work.

A good engineer has a better grasp of existing software they can reuse, thus minimising the parts of the system they have to build from scratch. They have a better grasp of engineering tools, automating away most of the routine aspects of their own job. Automation also means freeing up humans to work on solving unexpected errors, which the best engineers are disproportionately better at. Good engineers themselves design systems that are more robust and easier to understand by others. This has a multiplier effect, letting their colleagues build upon their work much more quickly and reliably. Overall, good engineers are so much more effective not because they produce a lot more code, but because the decisions they make save you from work you did not know could be avoided.

This also means that small teams of the best engineers can often build things faster than even very large teams of average engineers. They make good use of available open source code and sophisticated cloud services, and offload mundane tasks onto automated testing and other tools, so they can focus on the creative problem-solving aspects of the job. They rapidly test different ideas with users by prioritising key features and cutting out unimportant work. This is the central thesis of the classic book "The Mythical Man-Month"1: in general, adding more software engineers does not make a project go faster, it only makes it grow bigger.


Building software is not about avoiding failure; it is about strategically failing as fast as possible to get the information you need to build something good.

Smaller teams of good engineers will also create fewer bugs and security problems than larger teams of average engineers. Similar to writing an essay, the more authors there are, the more coding styles, assumptions, and quirks there are to reconcile in the final composite product, exposing a greater surface area for potential issues to arise. In contrast, a system built by a smaller team of good engineers will be more concise, coherent, and better understood by its creators. You cannot have security without simplicity, and simplicity is rarely the result of large-scale collaborations.

The more collaborative an engineering effort, the better the engineers need to be. Problems in an engineer's code affect not just his work but that of his colleagues as well. In large projects, bad engineers end up creating more work for one another, as errors and poor design choices snowball to create massive issues. Big projects need to be built on solid reliable code modules in an efficient design with very clear assumptions laid out. The better your engineers, the bigger your system can get before it collapses under its own weight. This is why the most successful tech companies insist on the best talent despite their massive size. The hard limit to system complexity is not the quantity of engineering effort, but its quality.

Conclusion

Good software development starts with building a clear understanding of the problem you want to solve. This lets you test many possible solutions and converge on a good approach. Development is accelerated by reusing the right open source code and cloud services, granting immediate access to established software systems and sophisticated new technology. The development cycle alternates between exploration and consolidation, quickly and messily progressing on new ideas, then focusing and simplifying to keep the complexity manageable. As the project moves forward, it gets tested with successively larger groups of people to eliminate increasingly uncommon problems. Launching is when the real work ramps up for a good development team: layers of automated systems should be built to handle issues quickly and prevent harm to actual users. Ultimately, while there are infinite intricacies to software development, understanding this process provides a basis to tackle the complexities of how to build good software.


ABOUT THE AUTHOR

Li Hongyi leads a team of engineers, designers, and product managers who build technology for the public good. Projects they have worked on include Parking.sg—an app to replace parking coupons, Form.gov.sg—a web app for building online government forms in minutes, and Data.gov.sg—the government's open data repository. Prior to joining the public sector, Hongyi worked at Google on the distributed databases and image search teams. In his free time, he works on personal projects like typographing.com and chatlet.com.


NOTE

  1. Frederick P. Brooks, Jr., The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition, 2nd ed. (Boston, MA: Addison-Wesley Longman, 1995).

Back to Ethos homepage




All Comments: [-] | anchor

smacktoward(42) 3 days ago [-]

Write code. Not too much. Mostly test-covered.

ssijak(3446) 3 days ago [-]

I like this Pollan reference

hnick(10000) 3 days ago [-]

This is great. So many quotable quotes. If only we could make it required reading for our clients!

This one struck me, because as soon as I read it I knew it was true yet had never considered it:

> Most people only give feedback once. If you start by launching to a large audience, everyone will give you the same obvious feedback and you'll have nowhere to go from there.

I've been on both sides of that fence and it rings true.

joes223(10000) 3 days ago [-]

Anonymous feedback (like really anonymous) is the answer. people can't give real feedback and be nice at the same time.

mychael(4187) 3 days ago [-]

This reads like conventional wisdom. No one is going to argue against 'Hire the Best Engineers You Can' and 'Start as Simple as Possible'.

ikll(10000) 3 days ago [-]

If only.

Industry this days is more about headcount than quality itself. Why hire two good engineers when you can have three mediocre ones for the same price?

On simplicity, common wisdom these days dictate that we should use bloated kitchen-sink backend MVC frameworks that generate dozens of directories after `init`, because supposedly nobody knows how to use routers. Frontend compiler pipelines are orders of magnitude more complex than the reactive frameworks themselves, because IE11. And even deployment now requires a different team or expensive paid services from the get go. We're definitely not seeking simplicity.

The second point is also something that most developers and managers would balk at: 'To build good software, you need to first build bad software, then actively seek out problems to improve on your solution'. Very similar to the Fred Brooks 'throw one away' advice that no one ever followed.

kitsune_(4029) 3 days ago [-]

Except for the 10x myth a fairly good article.

dajonker(10000) 3 days ago [-]

I disagree with that, if you describe it as stated in the article: 'Overall, good engineers are so much more effective not because they produce a lot more code, but because the decisions they make save you from work you did not know could be avoided.'

I've seen plenty of poor decisions that cause 10x the work, and end up with something 10x less maintainable.

commandlinefan(10000) 3 days ago [-]

How many more decades are we going to have to spend learning this lesson before we learn it?

diafygi(1306) 3 days ago [-]

There's a saying in most other fields of engineering (civil, chemical, mechanical, etc.): 'Regulations are written in blood.' A whole lot of bridges collapsed and a whole lot of people died before strong requirements were put in place.

It seems we are on the path to repeat history with software engineering, what with how software and the internet is being developed with such little regard for public safety and long term consequences.

Unfortunately, it appears that the 'free love' phase of software engineering is coming to an end, as society now relies more and more on software and major tech players for life and safety. It's starting to get real for software engineering.

Luckily, other engineering fields have been here before, so this sort of transition shouldn't be anything new.

Relevant Tom Scott video: https://www.youtube.com/watch?v=LZM9YdO_QKk

nerdponx(3618) 3 days ago [-]

As many as they need to start teaching it in business schools?

stygiansonic(3261) 3 days ago [-]

Some good tidbits from the government perspective on software development:

"Beware of bureaucratic goals masquerading as problem statements. "Drivers feel frustrated when dealing with parking coupons" is a problem. "We need to build an app for drivers as part of our Ministry Family Digitisation Plans" is not. "Users are annoyed at how hard it is to find information on government websites" is a problem. "As part of the Digital Government Blueprint, we need to rebuild our websites to conform to the new design service standards" is not. If our end goal is to make citizens' lives better, we need to explicitly acknowledge the things that are making their lives worse."

dehrmann(10000) 3 days ago [-]

This also very much reads like something from Singapore.

steventhedev(3785) 3 days ago [-]

Regarding 'Seek Out Problems and Iterate', it's a bit of an understatement how important this is. I've invested a lot of time helping my coworkers understand the distinction between tasks and problems. The end goal being only tracking problems in the ticketing system. It's not easy to do this and it takes constant effort, but it pays off very quickly. I've yet to see a real 'problem' ticket stay unresolved for a long time, whereas 'task' tickets tend to stay around until they're either irrelevant or they get closed after getting kicked between a few people.

A good example of this is:

- Add worker thread for X to offload Y

When the actual problem is more along the lines of:

- Latency spikes on Tuesdays at 3pm in main thread

Which may be caused by a cronjob kicking off and hogging disk IO for a few minutes.

A good rule of thumb I've found is that task tickets tend to have exactly one way of solving them, whereas problem tickets can be solved in many ways.

Dumblydorr(10000) 3 days ago [-]

Can you explain the 3pm on Tuesdays issue? My sister works for LLS and she said their servers get very slow at a precise time every Tuesday. Not saying it's the same bug, but what was the solution in your specific case?

axilmar(10000) 3 days ago [-]

The initial proposition of the article, that software is bad because it follows the lifecycle 'gather requirements - write software - deliver it' is simply wrong. There are huge projects in specialized domains that are delivered on time and on budget and use this approach.

The problem is lack of knowledge. The successful projects mentioned above did not have a lack of knowledge, and so they were finished successfully.

When there is a lack of knowledge, then it makes sense to use the iterative approach...as knowledge is slowly gathered, the software gets improved. As with all things in life!

fbr(10000) 3 days ago [-]

Yes, the lack of knowledge is definitively one of the issue.

But starting a 'gather requirements - write software - deliver it' lifecycle because you are confident that you have all the knowledge is one as well.

koevet(2289) 3 days ago [-]

The article lists the characteristics of a good engineer:

  * has a better grasp of existing software they can reuse
  * (has) a better grasp of engineering tools, automating away most of the routine aspects of their own job
  * design systems that are more robust and easier to understand by others
  * the decisions they make save you from work you did not know could be avoided
I obviously concord with the analysis (not sure about the 10X myth). It also states that:

  * Google, Facebook, Amazon, Netflix, and Microsoft all run a dizzying number of the largest technology systems in the world, yet, they famously have some of the most selective interview processes
This sounds a bit like a paradox to me. Given the current state of 'selective interview processes' (algo riddles, whiteboard coding, etc.), none of the above traits can be easily evaluated in a candidate during an interview. On the other hand, these companies do hire stellar engineers: the technological supremacy of FAANG is irrefutable.
unicornmama(10000) 3 days ago [-]

Former Googler here.

Google views picking new engineers like picking quality construction metals. In the end, the machine melts you down and hammers you into a pristine cog.

joes223(10000) 3 days ago [-]

It's not a paradox because the statement that the interview processes don't evaluate candidates is false. It's proven that this particular interview format has a very high correlation with the future candidate's performance.

hnick(10000) 3 days ago [-]

The way I interpreted that last comment was as a counterpoint to the idea that large companies necessarily end up hiring many mediocre employees because the talent pool simply isn't deep enough to stack the deck. Instead of just being happy with what they can get, the big tech companies make it a real challenge to be hired.

jrumbut(10000) 3 days ago [-]

I thought 'what a bold title, if someone's figured it out we can just close HN' and upon reading, hey it's not far off.

The following is a wonderful point I have hardly ever heard said directly:

'The main value in software is not the code produced, but the knowledge accumulated by the people who produced it.'

unityByFreedom(4199) 3 days ago [-]

weird, I've heard it said frequently for decades in various forms,

'value your knowledge workers'

'your employees are your most valuable asset'

Some companies don't treat employees well, and some employees at good companies feel they are not treated well enough

If the above quotes do not strike a chord with you, you might just be a software engineer who thinks you're more important than non-SEs.

lifeisstillgood(1594) 3 days ago [-]

It's not that they have the knowledge but that the knowledge is now encoded in software and available to anyone else - software shares knowledge without the users having to learn it (for example having to learn which five systems need to have their names enters in order to pay their parking fine

einpoklum(10000) 3 days ago [-]

> Software has characteristics that make it hard to build with traditional management techniques

Perhaps some software development techniques would work though...

> The main value in software is not the code produced, but the knowledge accumulated by the people who produced it.

Those people go on to work on other things or for other organization. So, while that statement might have some truth to it, it's still the case that the code has to be useful, robust, and able to impart knowledge to those who read it (and the documentation).

> Start as Simple as Possible

That's a solid suggestion to many (most?) software projects; but - if your goal is to write something comprehensive and flexible, you may need to replace it with:

'Start by simplifying your implementation objectives as much as possible'

and it's even sometimes the case that you want to sort of do the opposite, i.e.

'Start as complex as possible, leading you to immediately avoid the complex specifics in favor of a powerful generalization, which is simpler'.

AnimalMuppet(3730) 3 days ago [-]

> > Software has characteristics that make it hard to build with traditional management techniques

> Perhaps some software development techniques would work though...

As you go up the management chain, you usually run into some layer where people are traditional managers, who want to run a software project like a traditional project. And behold, you're at this problem. Saying 'software development techniques would work' is useless unless you can get those managers to change. And when you get them to change, the problem moves up one layer.

driverdan(1460) 3 days ago [-]

#1 web rule: Don't require JS to read an article.

This site is an empty page without JS.

davidperrenoud(10000) 3 days ago [-]

Strangely, all the HTML elements are there but the opacity of <body> stays at 0 without JavaScript.

9nGQluzmnq3M(3580) 3 days ago [-]

The article appears to be written by Singaporean prime minister Lee Hsien Loong's son, Li Hongyi.

http://theindependent.sg/li-hongyi-singapore-has-a-lot-of-pr...

hnick(10000) 3 days ago [-]

Now I'm wondering why the children romanized their surname as Li not Lee.

I came across this article: https://mothership.sg/2015/03/lee-hsien-yang-reveals-the-sto...

> I have taught my children never to mention or flaunt their relationship to their grandfather, that they needed to make their own way in the world only on their own merits and industry.

wenc(4172) 3 days ago [-]

Also, his brother Li Haoyi wrote Ammonite, a well-known Scala REPL.

soup10(10000) 3 days ago [-]

>The better your engineers, the bigger your system can get before it collapses under its own weight. This is why the most successful tech companies insist on the best talent despite their massive size.

Translation: the successful tech companies have so much poorly documented legacy enterprise spaghetti code and tooling that they need the best talent they can get just to make sense of it and maintain it

abacadaba(10000) 3 days ago [-]

Alternate translation: Bad devs are worse than no devs and all your competent devs will spend most of their time dealing with the former's crappy code until they quit. (my code is of course perfect and free of all technical debt)

crimsonalucard(10000) 3 days ago [-]

Nice Post. But everyone needs to understand something. Even if you follow these principles to the letter T, you can still produce very bad software. In fact you can also find many cases where people did the exact opposite of what this guy said and still produced great software. I'm sure many people can name examples of software that just came together out of blind luck.

Why?

Because there is no formal definition for what is bad or good software. Nobody knows exactly why software gets bad or why software gets good or what it even exactly is... It's like predicting the weather. The interacting variables form a movement so complex that it is somewhat impossible to predict with 100% accuracy.

What you're reading from this guy is the classic anecdotal post of design opinions that you literally can get from thousands of other websites. I'm seriously tired of reading this stuff year over year rehashing the same BS over and over again, yet still seeing most software inevitably become bloated and harder to work with over time.

What I want to see is a formal theory of software design and by formal I mean mathematically formal. A axiomatic theory that tells me definitively the consequences of a certain design. An algorithm that when applied to a formal model produces a better model.

We have ways to formally prove a program 100% correct negating the need for unit tests, but do we have a formal theory on how to modularize code and design things so that they are future proof and remain flexible and understandable to future programmers? No we don't. Can we develop such a theory? I think it's possible.

carapace(2866) 3 days ago [-]

So you're not talking about 'formal methods'? https://en.wikipedia.org/wiki/Formal_methods

The Applied Category Theory folks have some very interesting stuff, like Categorical Query Language.

https://www.appliedcategorytheory.org/

https://www.categoricaldata.net/

But it sounds to me what you mean is more like if 'Pattern Language' was symbolic and rigorous, eh?

al_form2000(10000) 3 days ago [-]

We know a great deal about dynamics,kinematics, thermodynamics and generally the physics that governs car components, yet we are a long way from an algorithm that applied to a car will produce a better car. My guess is that doing that for software is as hard, if not harder.

Also the sentence 'algorithms that applied to algorithms produce a better model' has a strong smell of halting problem, at least to this nose.

iEchoic(4059) 3 days ago [-]

> Reusing software lets you build good things quickly

It also introduces unknown amounts of debt and increases the likelihood that you'll end up with intractable performance/quality/velocity problems that can only be solved by re-writing large portions of your codebase.

This can be a dangerous cultural value when it's not presented with caution, which it isn't here. I think it's best to present it alongside Joel Spoelsky's classic advice: 'If it's a core business function — do it yourself, no matter what'.

https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

jeffmcmahan(4060) 3 days ago [-]

I agree completely. Dependencies solve problems but not for free - bugs, security issues, versioning headaches, performance problems, compatibility gotchas, churn, &c. I live by the following:

(1) If a problem can be exhaustively specified in a formally well-defined way (mathematical logic), it will be wise to adopt a mature implementation - if it exists.

(2) If a problem can't be so specified, all implementations will be incomplete and will contain trade-offs. I have to address these problems myself to ensure that limits and trade-offs suit as well as possible what the business needs. If I can.

So, (1) says I shouldn't parse my own JSON. (2) says I should avoid the vast majority of what shows up in other people's dependency trees.

anderspitman(1741) 1 day ago [-]

Yeah I think the OP article is great but doesn't pay enough respect to the costs of dependencies.

yowlingcat(10000) 3 days ago [-]

Great article. I liked this quote:

```

The best advice I can offer:

If it's a core business function — do it yourself, no matter what.

Pick your core business competencies and goals, and do those in house. If you're a software company, writing excellent code is how you're going to succeed. Go ahead and outsource the company cafeteria and the CD-ROM duplication. If you're a pharmaceutical company, write software for drug research, but don't write your own accounting package. If you're a web accounting service, write your own accounting package, but don't try to create your own magazine ads. If you have customers, never outsource customer service.

```

This all rings true in my experience. You should write the software that's critical to your core business competency yourself, because the maintenance cost is worth paying if you can achieve better software. But if it's not a core competency and your business isn't directly going to benefit from having best in class vs good enough, then it may be worth outsourcing.

andy_ppp(4102) 3 days ago [-]

I've personally been thinking about this for some time and wondering if in the real world this looks like building as much as possible at the database level and treating your DB as a state machine for your app, aiming to disallow whole classes of errors and communicating the design of the business logic at the SQL functions/triggers/data layer, separate from the API, Services, Programming Language, and Frontend layer(s).

This means that instead of lots of issues with business logic being separate from the data the business logic and data sit together and prevent your system from getting into bad states.

Thinking about this, maybe I just stole this thought from Derek Sivers: https://sivers.org/pg

dajonker(10000) 3 days ago [-]

Yes, the data model is probably the most important aspect of your application, it defines the relations and constraints. With a good data model, you don't need to write a lot of code to deal with it. Having lots of code that deals with weird situations in the database means your data model needs some serious consideration.

A database in my opinion is not a good place to write business logic with functions and triggers, since there is lack of tooling that would make development and debugging easy. Let the database do what it does well, which is storing and querying data.

nneonneo(10000) 3 days ago [-]

I found this to be an incredibly accessible and easy to read guide for software development. It's a very short read - just a few minutes - but it's full of practical examples and written in a way that speaks to non-engineers (like bureaucrats). If you are a non-technical person handling software stuff, this article should definitely be high on the reading list.

The author seems like an unknown in the software development world, but they're one of the managers for Singapore's fairly successful digital government initiative. So it does feel safe to say they have some experience.

angelsl(10000) 3 days ago [-]

Li Hongyi is the son of Singapore PM Lee Hsien Loong, as well as a deputy director in GovTech Singapore (the Government Technology Agency). (He's also an MIT CS grad, and a past Googler.)

I suppose he wrote this for other people in the Singapore civil service.

akersten(10000) 3 days ago [-]

One of the principles the article highlights is that additional features make a software complex and therefore more likely to fail. This is true, but I'd argue it's not for the reason the article claims.

The claim is:

> Stakeholders who want to increase the priority for a feature have to also consider what features they are willing to deprioritise. Teams can start on the most critical objectives, working their way down the list as time and resources allow.

In other words, the argument is 'competing priorities in a large-scale project make it more likely to fail, because stakeholders can't figure out which ones to do first.' Actually, in this very paragraph, the author glosses over the real issue: 'Teams can start on the most critical objectives, working their way down the list' - treating development as an assembly line input-to-output process.

I argue that it's not time constraints that complex programs bad, but instead the mere act of thinking that throwing more developers at the work will make it any better. Treating the application as a 'todo list' rather than a clockwork of engineering makes a huge difference in the quality of the work. When developers are given a list of customer-facing features to achieve, more often than not the code winds up a giant ball of if-statements and special cases.

So yes, I do agree that complex software is worse and more prone to failure than simple software - but not for the reason that there's 'too much to do' or that prioritizing is hard. Complex software sucks because it's requirement-driven, instead of crafted by loving hands. No one takes the time to understand the rest of the team's architecture or frameworks when just throwing in another special case takes a tenth of the time.

Mertax(10000) 3 days ago [-]

I've also seen the failures in requirement driven software. When engineers receive unfiltered customer requests as requirements or tasks they tend to focus simply on getting that functionality into the software. Most times not understanding the job the customer is trying to get done.

There are different personalities of engineers, those who thrive on explicit requirements and can accomplish difficult engineering tasks when they are given clear requirements. But those engineers should only be given those requirements once the job that the customer is trying to get done is clearly understood. Some engineers have the ability to find creative solutions, that customers or product managers can't see, when they are provided with problems and jobs rather than requirements and tasks.

Managers would be wise to distinguish between the type of engineers they are managing and play to their strengths. Whatever type you have, understanding the job the end user is trying to get done must occur, preferably by an engineer that's capable of articulating that, if needed, to team members as technical requirements.

JohnBooty(10000) 3 days ago [-]

    I argue that it's not time constraints that complex programs bad, 
    but instead the mere act of thinking that throwing more developers 
    at the work will make it any better. 
The bit about throwing more developers is true, but really does not follow from anything else you or the author is talking about.

    Treating the application as a 'todo list' rather than a clockwork 
    of engineering makes a huge difference in the quality of the work. 
    When developers are given a list of customer-facing features to achieve, 
    more often than not the code winds up a giant ball of if-statements 
    and special cases.
Admittedly, this is often the case when doing feature-driven development.

But it absolutely does not need to be the case.

If you treat engineers as interchangeable cogs who only need to know about one story at a time, and never tell them about the medium- and long-term goals of the business and the application? Then yes. Then you get an awful code base with tons of if-then crap.

However, it doesn't need to be this way. If you give engineers visibility into (and some level of empowerment with regard to) those longer-term goals, they can build something more robust that will allow them to deliver features and avoid building a rickety craphouse of special cases.

I have experienced both scenarios many times.

kilburn(2728) 3 days ago [-]

> In other words, the argument is 'competing priorities in a large-scale project make it more likely to fail, because stakeholders can't figure out which ones to do first.'

This is a misinterpretation of the article's claim. The article very explicitly begins by saying that the best recipee to increase a project's chances to success is to:

> 1. Start as simple as possible;

> 2. Seek out problems and iterate;

The priority part reads to me as a way to determine which features are critical (and hence part of the as simple as possible set) and which ones are not (and hence you should not build 'yet'). The underlying vibe being that these other features should probably never get implemented because once the critical ones get built and the software is put to use you will actually find other critical fearures that solve actual problems found through usage.

That is, only when you find that one of the initially non-critical features has become a hindrance for users actually using your software you should seek to implement it.

I really think this would be a better way to build software, just as much as I think that you will have a very very hard time getting any management on board with it...

ensiferum(3972) 3 days ago [-]

'2. Seek out problems and iterate;'

This is bad advice. It's like saying 'go into a bar and start picking up fights'.

If some part of the software has problems, runs slow or has bugs but nobody is complaining, then there's no problem. Why waste time improving it?

Almost 100% of the time when you solve a problem you just create new problems of different kind in turn.

Be lazy. The less code you write the better off you are.

Silhouette(4155) 3 days ago [-]

If some part of the software has problems, runs slow or has bugs but nobody is complaining, then there's no problem.

This depends very much on context. To pick an extreme example, if you're writing the control software for a nuclear weapon and you know you have a bug that might cause it to activate unintentionally if you eat a banana while it's raining outside, I think we can reasonably agree that this is still a problem even if so far you have always chosen an apple for lunch on wet days.

mr_tristan(10000) 3 days ago [-]

This struck a chord with me: 'Software Is about Developing Knowledge More than Writing Code'

I've experienced more issues caused by management passing around tasks between teams and never paying attention to knowledge and knowledge transfer.

What's amazing, is that in over 18 years as a software engineer, I've seen this so many times. Teams will function well, then the institution tries to change. Often they will try to open up the 'innovation' by throwing money at R&D, basically trying to add bodies in order to grow. Then you have tons of teams, and communication becomes very challenging, so then they grow some kind of 'task management' layer. Management that never understands who actually _knows_ something, just tracks how much 'theoretical bandwidth' they have and a wishlist of features to create. And then the crapware really starts flowing. And then I get bored and move on to the next place.

hising(2554) 3 days ago [-]

> 'Then you have tons of teams, and communication becomes very challenging'

communicationChannels = nrOfTeams(nrOfTeams-1)/2

More people should read The Mythical Man-month

atoav(10000) 3 days ago [-]

So knowledge about:

- the problem you are trying to solve

- how you could solve it

- how you actually did solve it

- which solutions come with which flaes and merits

clumsysmurf(561) 3 days ago [-]

> 'Software Is about Developing Knowledge More than Writing Code'

The company I work for uses Scrum. They consider the User Stories + the code to be everything you need. I struggle with this, but my manager says they don't want to get tied up doing documentation 'because it goes out of date'. Beside, they are being Agile which 'prefers working code over comprehensive documentation'.

I am wondering what other companies do to capture this 'distilled knowledge'. The backend services I rely on are undocumented beside some paltry swagger that leaves much to be desired. The front end has no product-level 'spec', if you want to rebuild the thing from scratch. There isn't even a data dictionary, so everyone calls the same thing by different terms (in code, and conversation).

There are just user stories (thousands) and code.

Does anyone have any suggestions on how to fix this?

kgwxd(2461) 3 days ago [-]

The first place I worked as a software dev had an owner that would explain everything about the business and the problems to me in very good detail. He would just stop by my desk whenever he thought of something he thought might be good for me to know. Eventually understanding the business became just as interesting as the coding. These days, I hate getting a task without knowing the business side of things or not being able to discuss it directly with the person that does.

mmatants(10000) 3 days ago [-]

Apparently, Peter Naur (the N in BNF) wrote this up nicely back in 1985 in 'Programming as Theory Building': http://pages.cs.wisc.edu/~remzi/Naur.pdf

JamesBarney(4202) 3 days ago [-]

Haha yeah the number of times I've gotten 'Hey someone estimated a task would take 40 hrs on a project you've never seen with libraries you've never touched. mind knocking that out this week?' is astounding.

cntlzw(10000) 3 days ago [-]

+ 1 Yes, things really changed for me once I started to ask why we implement things. Tried to understand the manager/customer/stakeholder what is their domain? What kind of issue they want to solve? What is the business case we are working on?

I know as a software developer you don't want to do that. More fun refactoring code than dealing with management. More fun writing that piece of SQL than sitting in a meeting. Easier to whine about missing specifications than to understand the big picture.

Once I stepped back from coding and looked at the software from a birds eye view, I had actually a much easier time programming features than before. More knowledge, less writing code.

drderidder(4159) 3 days ago [-]

I think this is also known as the fallacy of the fungible engineer / myth of the interchangeable programmer.

_pmf_(10000) 3 days ago [-]

> This struck a chord with me: 'Software Is about Developing Knowledge More than Writing Code'

Managers are very unhappy when I tell them of all the knowledge I've developed.

civicsquid(4191) 3 days ago [-]

One of my professors condensed that point into something I thought was clever: 'Software engineering is the distilling of ambiguity'.

I think about that whenever I get frustrated about a vague spec or lack of details. It's the job!

ak39(4050) 3 days ago [-]

Building good software requires mainly achieving two things:

1. Making sure what you build is what was really requested (correct), and

2. Making sure what you've built doesn't have a higher running 'cost' than the thing it replaced (either manual process or old automated solutions).

Everything else, IME, is ancillary. Performance, choice of platform, frameworks, methodology to build, maintainability etc are sub-objectives and should never be prioritized over the first two objectives. I have worked on many projects where the team focussed mostly on the 'how to build' parts and have inevitably dropped the ball on the 'what' to build of the projects. Result: failure.

Sauce: personal experience with several years of different projects (n = 1; episodes = 20+ projects that have gone live and have remained live versus 100+ projects lying by the wayside).

Writing software is not easy.

0x445442(10000) 3 days ago [-]

I agree with you but what I've noticed is for all the large projects I've worked on it was impossible to get an official answer as to whether or not the whole endeavor had a positive ROI. In fact, with a little back of the napkin math and some knowledge of the project's resource allocation it was obvious in most cases there would not ever be a positive ROI.

tony(10000) 3 days ago [-]

Nice post! Agreed on keeping the initial stuff simple as possible.

In python, I typically follow a pattern of keeping stuff in __name__ == '__main__' block and running it directly, then splitting to functions with basics args/kwargs, and finally classes. I divide into functions based on testability btw. Which is another win, since functional tests are great to assert against and cover/fuzz with pytest.mark.parameterize [1]

If the content of this post interested you: Code Complete: A Practical Handbook of Software Construction by Steve McConnell would make good further reading.

Aside: If the domain .gov.sg caught your eye: https://en.wikipedia.org/wiki/Civil_Service_College_Singapor...

[1] https://docs.pytest.org/en/latest/parametrize.html

ptx(3822) 1 day ago [-]

I prefer putting the main code into a 'main' function (called from the __name__ == '__main__' block) fairly early, since otherwise the functions you extract might accidentally keep relying on global variables.

siempreb(10000) 3 days ago [-]

> 3. Hire the best engineers you can.

This is where most companies fail. Yes, they do want the best developers, but for the budget of an average junior/medior dev.

For some reason most companies/managers I worked for do not understand the financial impact of a not so good developer. Or the other way around; they fail to value the best developers and are unable recognize them.

I've worked for plenty companies where they let mediocre dev's build a big app from scratch (including the architecture), in an Agile self managed team.. These are the codebases that always need to be rewritten entirely because they have become an unmanageble buggy mess of bad ideas and wrong solutions.

jshowa3(10000) 3 days ago [-]

I've worked for companies where supposed senior devs write a massive amount of code without even the slightest indepth thought because they think they know everything.

Then the project turns out to be months late, even though I called the timeline of the project virtually unfeasible, and we have to go back and make several changes that could've been caught early on with a better strategy.

The problem with hiring the 'best' engineers is as follows:

1. Nobody can ever tell you what the best means. People just throw 10x around without any explanation.

2. Most people in the world are average. You simply don't have enough of the best people to handle the work load, even if they're 10x average. So much existing software and new problems exist that it's nigh impossible to have the best everywhere.

3. Many of the best people are able to write really good code, but they consider it so easy that they often write code that they think is correct and it gets put in production. Since they're loners, they often don't do the necessary leg work either because of their own arrogance, or because the company hasn't clearly defined its processes and the developer can't even reach this goal despite numerous efforts. So management just believes the code is correct without any verification.

4. Many average developers support the best ones by taking needed work away from them through comparative advantage. Just because X employee is awesome at Y task, doesn't mean he meets the highest utility by doing Y task all the time. Especially when there are conflicting priorities.

5. The best engineers aren't going to be working at a small company in most cases. They also aren't likely to be paid well outside a large company either. The article sites Google, Facebook, and all the large tech companies and their supposed stringent interview process as a reason. But these companies have written terrible software (Google+, AMP pages) and become ethically compromised easily. Plus their interview process is often so outside the daily work flow because it involves answering algorithm questions, that it often makes no sense. Even worse, it teaches people to do katas instead of build actual projects. Project based interviews make much more sense.

6. Rewriting code bases is one of the worst things you can do and is what caused Netscapes downfall. Companies with supposedly the best engineers (ie. Netscape), can't even do it well.

So while hiring the best engineers is an awesome goal. It isn't feasible in a lot of cases.

I admit I have some bias as I consider myself pretty average. But I do a lot of crap on the side that '10x devs' don't even hear about because they're working on something more urgent. Does that mean I'm worthless?

lelima(3982) 3 days ago [-]

>'3. Hire the best engineers you can.'

If every single company wants that, where is he space to grow and learn from mistakes?

Maybe I'm wrong but I think those 'mediocre dev's' learned a lot building a big app from scratch, solving bugs and refactoring.

acd(4113) 3 days ago [-]

Keep it simple software should be open source. Government software often has similar demands as other countries. Share and reuse.

Reusing good modules and software will make the software work.

Kiss engineering still works keep it simple stupid. Make it as simple as possible. Simple software and systems are easy to maintain and understand.

Use modules as these can be swapped out.

Use proven boring technology such as SQL and JSON. Boring tech has been tried by others and generally works well.

tester344(10000) 3 days ago [-]

>Government software often has similar demands as other countries.

What makes you think so?





Historical Discussions: Japanese anime studio Khara moving to Blender (August 16, 2019: 935 points)

(936) Japanese anime studio Khara moving to Blender

936 points 6 days ago by robin_reala in 31st position

www.blender.org | Estimated reading time – 11 minutes | comments | anchor

Original article first released on Engadget Japan

"We finally got a 3D creation tool like paper and pencil" The reason why Khara, Inc. started moving to Blender.

Khara, Inc. is known as Hideaki Anno's motion picture planning and production company. They are currently working on "EVANGELION:3.0+1.0", film to be released in June 2020.

Khara announced on July 30th, 2019:

It was titled "Supporting Blender Development Fund", announcing that they support Blender Foundation on developing the open source 3D creation software "Blender".

Of course, it was not only a story of funding.

Khara and Anime/CG production company "Project Studio Q, Inc." are preparing to switch their primary 3D CG tools to Blender. Blender will be used for some parts of "EVANGELION:3.0+1.0" they are currently working on.

Why is Khara moving to Blender?

"Tool cost" challenges to Anime production.

3D CG is essential to modern Anime productions. Not only the so called "animation films" such as those from Pixar and Dreamworks, but it's now common for 3D CG to be used in many Anime films. 3D CG and hand-drawing make a great combination that improves productivity and quality of Anime.

Khara is a company focusing on a "hybrid of 3D CG and hand-drawing". They are currently working on "EVANGELION:3.0+1.0" with the same policy. They also founded "Project Studio Q, Inc." in 2017, a company focusing mainly on the movie production and the training of Anime artists as the joint venture with Dwango Co., Ltd. and Aso College Group.

Khara has been using Autodesk "3ds Max" as their primary tool so far. "EVANGELION:3.0+1.0" production is mainly done with 3ds Max. They are now starting to switch from 3ds Max to Blender. Usually the reason being "due to differences such as quality and functionalities", but Khara's reason is different.

Hiroyasu Kobayashi, General Manager of Digital Dpt. and Director of the Board of Khara and President of Studio Q, and Daisuke Onitsuka, CGI Director of Digital Dpt. of Khara and General Manager of Production Dpt. of Studio Q, told about their situation.

"Simply, the project scale got bigger than what was possible with 3ds Max." – Hiroyasu Kobayashi

Khara Director and General Manager of Digital Department and President of Project Studio Q. Mr. Hiroyasu Kobayashi.

"We need more production resources from outside, not only from Studio Q. We need cooperative work with friend companies for our production. However, many of those companies are small or middle-sized, so if we stick to 3ds Max it will cause higher management costs." – Daisuke Onitsuka

You may need a bit further explanation about this.

3dx Max is offered by Autodesk on subscription fee basis, this fee is expensive. According to Autodesk's website, an annual fee for a single user is JPY 254,880. It is expensive still when they offer discounts for multiple users and years. A large company can absorb this cost with a number of users as it has large revenue. But it turns out to be difficult to recoup those costs of 3ds Max for all users in a company of 20-30 people size.

Different tools are used depending on the studios and works in Anime production. Productivity decreases unless they use the same tools and assets (materials such as models, animation data, etc.). Due to this, they tend to work with companies that use the same tools.

3ds Max is an excellent tool and one of the industry standards, but in some cases it causes a "lack of artists" due to its high costs.

In the past, Khara thought of moving to "Maya" which has as many users as 3ds Max in the industry. "We worked with data prepared for both 3ds Max and Maya, for Maya users." said Onitsuka. But resulted in a duplication of work resources to have the same data twice. Now, Blender attracted their attention.

Khara Digital Department CGI Director and Project Studio Q Production Department General Manager Mr. Daisuke Onizuka.

Onitsuka: "As a matter of course, we can produce with the same quality and schedule to maximize the efficiency of our limited budget. Blender is an open source software and available for free. Also, it has almost the same functionalities with other software, so it was undoubtly attractive to us. While we still have the challenge whether a new partner company can use Blender or not, but at least, cost-wise is much simpler, so we are proposing them to use Blender as we use it."

No lack of functionalities are blocking the switch. In fact, "Grease Pencil" was key to make the move.

Of course they can not switch the tools if a new tool is lacking functionalities for production. Blender's reputation has been "for beginners" or "for students" because it was free software. However, the latest version has more functionalities and the difference between Blender and other tools is getting smaller.

Khara had started thinking of the use of Maya 3 years ago, and also about Blender right after that.

Kobayashi: "We have been watching on it and got the feeling that we can use it as our primary software. Discussing with Onitsuka, we decided that we may be better to use Blender as our primary tool than to use various software in the future."

Takumi Shigyo at production dpt. of Studio Q, has been using Blender also for private use. He is a top "Blender wizard" at Khara and Studio Q. Artists like him helped Khara's decision to move to Blender. Shigyo spoke about Blender as follows.

Project Studio Q Production Department. Mr. Takumi Shigyo.

"I feel the latest Blender 2.8 is intentionally 'filling the gap' with 3ds Max to make those users feel at home when coming to Blender. I think the learning curve should be no problem." – Takumi Shigyo

Grease Pencil

And there was a key functionality that brought Khara to Blender as production studio of "2D Anime works". It was "Grease Pencil".

You can draw lines in 3D space directly with a pen, and not only by making 3D models but you can also draw extra details in the animations themselves.

Onitsuka: "As for our work, we create an image in 3D, and modify it into each frame as '2D Anime'. Even a chief animation director directly modifies each frame manually. We also even add shadows with tools like After Effect afterwards. For example, if we wanted a more sharp elbow in a character, we had to modify the 3D model and insert custom rigs (skeleton) into them. This was time consuming. But Blender enables us to draw with Grease Pencil after the animation is finished. It may not be the way for photo-real CG but it's a good way for 2D Anime. This helps reducing rigging costs."

"More animators who are not from 3D are choosing Blender as a 'steppin-up' tool not only because it is free software but because of Grease Pencil. This is an important factor for our decision." – Hiroyasu Kobayashi

Studio Q is also a facility for training new animators. They are hosting a contest for creators "Award:Q", more and more candidates are using Blender nowadays.

Shigyo: "We are getting more artists that started by using Blender in Studio Q. We are also seeing more high quality works by Blender users from high school students in Award:Q. I expect these new generations to be the majority working at studios in the future."

Cooperation with Blender Foundation to be a leading user in the Japanese Anime industry.

As mentioned, Khara uses Blender, not only the software itself but also helps the Blender Foundation by joining the Development Fund as a corporate member.

Masahiro "Rocky" Iwami, Production Manager of Digital Dpt. of Khara and General Manager of Overseas Dpt. of Studio Q, who is the primary contact person with the Foundation, said "We had no special relationship with Blender Foundation before this".

Iwami: "We had the contact information of Foundation chairman Mr. Ton Roosendaal, on their website. We got in touch with him from there. It was lucky that the name of 'Evangelion' attracted his attention. He kindly replied back to us immediately. We were glad to know that he thought the news that Japanese Anime companies using Blender was important for both of us. We became the first, leading player in Japanese Anime industry. As a corporate member of the Blender Development Fund, we are expecting our proposals may be closer to the development team. We would like to get closer with the Blender developers community."

One of the advantages of Blender is being open source, anyone can develop and add their own functionalities to Blender. Regarding this, Kobayashi said "Development details are to come".

Onitsuka: "Please understand that we are keeping a good relationship with Autodesk. Not only with Autodesk but also with other companies, we keep requesting improvements. However, it takes time for those improvements to make it to proprietary software. We expect faster improvements by using open source software."

The full switch will take place after "EVANGELION:3.0+1.0", but Blender is already used in it.

Some people may think "EVANGELION:3.0+1.0" can be made by Blender from the beginning. But pipelines do not switch that fast in reality.

Kobayashi: "Preliminary testing has been done already. We are now at the stage to create some cuts actually with Blender as 'on live testing'. However, not all the cuts can be done by Blender yet. But we think we can move out from our current stressful situation if we place Blender into our work flows. It has enough potential 'to replace existing cuts'."

Onitsuka: "We will be working on 'EVANGELION:3.0+1.0' until June 2020, so all the process cannot be changed before that. But we think we can say that 'we use Blender from this time onwards' for the next project. We would like to use and test Blender aggressively in our current 'EVA' for the ongoing year."

Kobayashi: "There are currently some areas where Blender cannot take care of our needs, but we can solve it with the combination with Unity. Unity is usually enough to cover 3ds Max and Maya as well. Unity can be a bridge among environments."

Onitsuka: "Hand-draw animators can start drawing only with paper and pencil. Just like that, 3D CG animators can start creating only with Blender at the same level cost with 'paper and pencil'. We already started speaking to partner companies 'let's use Blender together'. We are just starting now but high quality work Blender can be the best proof that our choice is the right one. What shows our value is the pictures we make, not the techniques and technologies. We want to share the know-hows with studios so that the Blender community will grow through those."

Translation provided by Masahiro "Rocky" Iwami.


Announcement in Khara website

Blender開発基金への賛同について https://t.co/mU4V9oGqC1

— 株式会社カラー (@khara_inc) July 30, 2019

発表の通り、弊社でもBlender開発基金へ賛同いたしましたことをご報告します。 https://t.co/ucIcjcYwWB

— プロジェクトスタジオQ (@Project_StudioQ) July 30, 2019




All Comments: [-] | anchor

ramigb(4056) 6 days ago [-]

I am sorry to be the guy who does that, but do you know what does Khara mean in Arabic? this brings back memories of the app that went successful in Brazil because the name meant something 'naughty'.

ripdog(10000) 6 days ago [-]

Khara is the name of the Japanese anime studio, not the tool. The tool is called Blender.

I'm not sure why Khara would care what their name means in Arabic.

flyingfences(10000) 6 days ago [-]

> The name khara comes from the Greek word χαρά, meaning joy.

parsimo2010(3461) 6 days ago [-]

Funny aside: Khara means 'shit' in Arabic. I don't think it affects Studio Khara's market that much but they'll have to change their name if they want to sell Evangelion in the Middle East.

arayh(2315) 6 days ago [-]

Ironically, it would seem that their name Khara was derived from a greek word (χαρά) meaning 'joy'.

faitswulff(3383) 5 days ago [-]

> Kobayashi: "There are currently some areas where Blender cannot take care of our needs, but we can solve it with the combination with Unity. Unity is usually enough to cover 3ds Max and Maya as well. Unity can be a bridge among environments."

Does anyone know how they're using Unity? I'd be curious to hear more about gamedev tools being used in animation.

SpliffnCola(10000) 4 days ago [-]

Star Wars: Rogue One used Unreal Engine for a few parts where they could get the same fidelity to Renderman output.

https://www.youtube.com/watch?v=pnigQTOig8k

boyadjian(10000) 6 days ago [-]

I love Blender, but knowing how to use it is a real profession. So many functionalities ... Even reading a book on Blender is not enough.

mdtusz(10000) 6 days ago [-]

'I love $PROFESSIONAL_SOFTWARE_TOOL, but knowing how to use it is a real profession. So many functionalities ... Even reading a book on $PROFESSIONAL_SOFTWARE_TOOL is not enough.'

In our current age of endless SaaS tools that bill themselves as being made for professional use, it's somewhat frustrating encountering this attitude that things are too complex or 'overkill' to be worth learning.

I understand that isn't the intention of your comment, but it's something I've struggled with at my work where co-workers seem to have no interest in learning new and powerful tools if it might take them more than a day to master.

ur-whale(4163) 5 days ago [-]

> knowing how to use it is a real profession

Actually ... with things like Google, youtube and stackoverflow, that's not really true anymore.

These days, whenever I want to do something non-obvious with blender, I always find an online tutorial.

Grazester(3117) 6 days ago [-]

I felt the same way about 3DS MAx when I discovered it in the 90's. I then realized that I didn't have to know how to use every feature in it but just what I needed/wanted to do. That at the time was simple model creation and keyframe aninmation

graphicsRat(10000) 6 days ago [-]

Blender would give Maya a real run for its money if it had an API.

ur-whale(4163) 5 days ago [-]

Not sure why you think it doesn't have an API ... The entire thing is accessible via Python. If that's not an API, I'm not sure what you mean.

mogpt(10000) 6 days ago [-]

Some friends of mine recently launched the pilot episode of their independent series, Culturesport, all done completely in Blender. https://youtu.be/UEcNEO3fVF8

They've been proselytizing Blender for years now, seems like the tide is rising.

lioeters(3990) 5 days ago [-]

Thanks for sharing this pilot episode made with Blender.

Woo, what a trip! Great use of colors and sounds.

I recognized some Scheme code scrolling by in the futuristic synthesizer.

justinclift(3630) 6 days ago [-]

Just watched it. That was brilliant. :)

Must have taken tonnes of effort.

yowlingcat(10000) 6 days ago [-]

Wow, what a coincidence! I saw a live screening of this in NYC -- nice to have a permanent way to see it again when I please.

mhh__(4160) 6 days ago [-]

Amazing lighting although one of the characters looks exactly former British prime minister John Major which broke the immersion

KuhlMensch(4206) 6 days ago [-]

That was pretty mind-blowing.

crimsonalucard(10000) 6 days ago [-]

Anybody from the industry know how much better is the closed source modelling software like lightwave, 3DS, Maya, Softimage compared to Blender. I know they are all different but is blender a peer a step above or a step below?

Stevvo(10000) 5 days ago [-]

I think it's hard to say which is better, each software has quirks that will drive you insane.

In 3ds Max; simple things like selection are broken. You double click to select an object and its children in the Hierarchy (contrary to single click in every other software ever made), the problem is, this is laggy; you get a loading spinner when selecting objects!

In blender despite the UI refresh in 2.8, the interface is still horrible. There are hundreds duplicate menu entries, and many functions without any menu entry at all, accessible only by keyboard shortcut.

The only software I've use that doesn't drive me mad is Cinema4D, but it's missing some features from other softwares.

ralusek(4191) 6 days ago [-]

Blender 2.8 is on par in most areas, better in many. For example, the sculpting tools available right inside of your standard modeling package rival dedicated sculpting programs like ZBrush, and you never have to leave Blender.

Likewise, the Cycles render engine is built into the package, without the need to go to VRay or Octane. The texturing pipeline in the modern PBR era make most rendering environments much more of an apples-to-apples comparison.

There are other features like particle systems and cloth simulators that are bonuses to have inside of the 3d package, although the best in the business like Houdini are as available to Blender users as they would be to Max/Maya.

ur-whale(4163) 5 days ago [-]

If I was a large VFX studio in 2019, I would not spend a cent on proprietary stuff anymore.

With an OpenSource base like Blender, anything you need to add to your pipeline can be grafted on top of blender by a small team of coders, and you are guaranteed that it'll keep on working for the next 10 years.

preommr(10000) 5 days ago [-]

lightwave, Softimage are pretty much dead.

3ds max and maya are both owned by autodesk. So the former is better for architectural stuff and product design, the latter for modeling, rigging, animation etc. Blender is fairly close to both. Especially with useful addons. A lot of major players customize things like their rigging so in that sense blender is far from being able to remove maya's position. Objectively, I would say blender is even slightly better at things like modeling. Blender's animation tools are good, rendering... is in some ways really good with the new eevee engine. Cycles is horribly slow.

I would say the real standouts are probably Houdini and Substance3d with kind of zbrush. Houdini is just a step up in simulations and particle effects. Its insane how good it is at the things it does especially with its procedural workflow. Blender's materials and painting system is there, but even with tweaking its only somewhat as good as designer+painter let alone the extensions that allow to export materials to unity/unreal. Even though the painting is primitive, the material node system is fairly powerful if used with textures instead of procedural generating everything. So depending on the workflow its still pretty powerful and can achieve the same results.

I mentioned zbrush for sculpting, and it just has a way higher ceiling. I don't know if most people will reach it, but zbrush can handle way more polygons and is just a way better tool for sculpting since that's what its main purpose is. Blender's sculpting can still be used to sculpt complicated things, but it probably has to be planned a little bit better by separating the mesh up. It has things like dynotopo to add details and the brush engine can do a lot of the things zbrush can do so its possible to make things. But you quickly have to start using textures and normals to simulate details whereas I've seen people just sculpt those details directly in zbrush.

tl;dr Blender is a jack of all trades and covers a lot of bases fairly well. Its good now, but it will be an absolute beast once the kinks get ironed out with eevee and the painting + sculpting tools get improved.

snek(4130) 5 days ago [-]

having the option between free maya on an educational license and blender, i still chose blender. before 2.8 the ui in blender wasn't fantastic, but it's always been better in terms of features (at least as long as i've been doing 3d), and the community is super friendly.

ttflee(3720) 6 days ago [-]

Perhaps a better title should be phrased like that the studio producing Rebuild of Evangelion is moving to blender.

0xcde4c3db(10000) 6 days ago [-]

I hope somebody is still actually producing it. The final episode has been 'in production' for almost 7 years now.

Smithalicious(4205) 5 days ago [-]

Should it? I don't like the trend of referring to companies/organisations/individuals as 'those who made X thing'. People aren't defined by one specific thing they did and also I think people would generally do well to be somewhat more aware of who actually makes the things they like anyways.

ngngngng(3857) 6 days ago [-]

What an amazing story of human cooperation. One on hand, everyone wants to make boatloads of money, on the other hand, sometimes we realize that if we all pitch in to a shared resource, we can make something better and cheaper than otherwise possible.

There are so many things I wish humanity could apply this to, glad to see it working in practice!

MrLeap(10000) 5 days ago [-]

Pretty sure blender started out as the product of a company that went out of business in the 90's, for whatever reason, they decided to open source the product instead of let it die.

I'm not countermanding you. Blender is awesome, the volunteers who make it perpetually better are awesome.

VikingCoder(4150) 5 days ago [-]

The one that kills me is textbooks. I mean... WTF.

devoply(10000) 6 days ago [-]

All things are made this way... Not because of money. It's just that in order to make them often you need money before the fact, and people with money get ownership and credit. The workers never make boat loads of money or get acknowledged for actually making it happen.

swebs(3917) 6 days ago [-]

>Not only with Autodesk but also with other companies, we keep requesting improvements. However, it takes time for those improvements to make it to proprietary software. We expect faster improvements by using open source software.

That's pretty cool. I wonder what kind of anime-focused features will come to the next version of Blender as a result.

samstave(3768) 6 days ago [-]

Dude, I was an EXPERT in Autocad from 1992 on....

I was the fastest ever to complete the ACAD design challenge (3 - hour test, finished it in 33 minutes)

I was the only person to have ever completed the autocad design interview test at the first architectural firm I worked at.... (8 minutes)

I came is second in the US CAD Olympics (only second because I skimmed the notes and it also required me to draw a bolt and I didnt notice that req in the directions)

---

I love autodesk --- but I hate their arrogance. They have lightened up a bit with 360 and such....

But I am rooting for Blender.

The amount of amazing talent around the world that will blow shit up is immense.

We will see some guy from freaking thailand or something just come out with something unbelievable.

I was asked to design a rendering factory in asia for lucas film some years ago... (I was the designer for the Lucas Presidio Complex's Data Center) -- their comment was 'US 3D designers are over-paid pre-madonnas and complain too much'

---

So, I hope blender kicks some fucking ass and that we see some amazing shit come out from a completely unknown person as of yet.

hex12648430(10000) 6 days ago [-]

>I wonder what kind of anime-focused features will come to the next version of Blender as a result.

Work is currently being done on the LANPR branch as part of the GSoC. This branch is meant to replace the Freestyle system which is currently used for line rendering. [0]

Otherwise we already have the 'Shader to RGB' node which is extremely useful to create NPR shaders.

There's also an add-on which can be used to deform a mesh based on greasepencil strokes, I assume 2D animators would love this. [1]

If you're interested in blender NPR news I highly recommend following the BNPR Show on youtube. [2]

[0]: https://wiki.blender.org/wiki/User:Yiming/GSoC2019/Updates

[1]: https://www.youtube.com/watch?v=w3nmEjum8kg

[2]: https://www.youtube.com/watch?v=38ETYyzrTzM&list=PLSMetEtY22...

microcolonel(4188) 6 days ago [-]

2.8 already has the vastly more useful version of grease pencil, and the demo the foundation commissioned was amazing. I love how seamlessly they mix the 3D in with the direct illustration. It shows what skilled animators can do with a tool like Blender.

https://youtu.be/pKmSdY56VtY

dan_quixote(10000) 6 days ago [-]

> We expect faster improvements by using open source software.

As a long-time consumer and contributor to open-source, I would caution one to temper their expectations when it comes to velocity. Glaring problems (like CVEs) get immediate attention, but features are another story - remember that open-source projects are a democracy and the loudest guy in the room often gets their way...

nailer(428) 6 days ago [-]

That's awesome. Combine this with Epic's recent investment and it looks like the modelling community may be mobilising behind Blender as an alternative to Autodesk tools.

nguoi(10000) 6 days ago [-]

That's such an exciting sentiment to hear, that open source could beat the economic incentives of propriety software. It's so far from what you would have heard in 1999.

roboyoshi(10000) 6 days ago [-]

As someone who worked in a similar section (animation), this is great news. Autodesk tools (especially Maya) are way too expensive in what they deliver. I'm glad many big studios realized this in that they either build their own tools (Pixar/Dreamworks) or they adapt Open Source (Blender) and help improving it.

There is still other tooling (Cinema4D, ZBrush) but those are actually working.. Maya has failed me so many times, I cannot recommend it in any way other than it is still (sadly) used very commonly among studios. And artists do not like to learn new toolings, that I also know from experience.. edit: grammar

dagmx(4020) 6 days ago [-]

Autodesk did recently release Maya and Max Indie which is roughly $300 for the first year. Pricing after that is ambiguous right now unfortunately.

agumonkey(961) 6 days ago [-]

where did maya fail you ?

I love this thing to bits, so .. basically blind to its defects, tell me

dahart(3778) 6 days ago [-]

I'm quite excited about Blender's growing popularity.

But, FWIW, I used to animate effects at PDI/DreamWorks, and we were using Maya. Today I think it's Houdini. In my time, modelers were using Maya as well. We had an in-house character animation tool mainly because the character rigs were custom, evaluated with an in-house scripting language, and Maya couldn't evaluate them.

While costs are always scrutinized, the decision to use Maya had very little to do with the price of a seat of Maya. It was just a tool that many people knew already (which is a huge factor in cost, BTW) and it also did most of the things we needed to do.

All the in-house tools were constantly scrutinized for cost, and usually the question was not whether we could reduce costs by building in-house, it was the other way around: can we reduce costs by purchasing off the shelf software. Remember that each and every developer is a 6-figure salary, which can buy a lot of seats of Maya. Note this applies to contributing to Blender too... it's very expensive to contribute to Blender, so it really has to be appreciated when it happens!

pfranz(10000) 6 days ago [-]

> I'm glad many big studios realized this in that they either build their own tools (Pixar/Dreamworks) or they adapt Open Source (Blender) and help improving it.

Places like Pixar/DreamWorks developed proprietary tools because they didn't exist when they started. Disney/Pixar heavily uses Maya. Disney was one of the first places to use Maya before it was released in the late 90s. Like many other places they heavily modify it. DreamWorks uses a lot of Maya and Houdini, but have proprietary rigging, animation, and renderer.

I haven't seen any of those places contribute much to Open Source unless they're driving development.

I haven't used Blender much at all. Apps like Maya, Houdini, and Nuke have a lot of under the hood architecture necessary to use in large studios that I hope Blender has, too. Photoshop, for example, is very difficult to build that kind of pipeline around.

FreeFull(10000) 6 days ago [-]

Sculpting in Blender is already pretty good, but there is also a work-in-progress sculpt branch that should make it even more of a competitor to ZBrush, once it's merged in. https://code.blender.org/2019/06/sculpt-mode-features-branch...

santaclaus(1992) 6 days ago [-]

> Autodesk tools (especially Maya) are way too expensive in what they deliver. I'm glad many big studios realized this in that they either build their own tools (Pixar/Dreamworks) or they adapt Open Source (Blender) and help improving it.

Cost isn't the issue, it's mainly the fact that Autodesk effectively put Maya on life support. Pixar is paying hundreds of software engineers in the Bay to develop their proprietary animation tools, which is most definitely not cheaper than a few hundred Maya seats per show.

fortran77(4033) 6 days ago [-]

Blender, despite its 'odd' UI, is the best open source GUI software out there. I don't use any other GUI open source software -- I pay for Microsoft Office and Adobe Applications -- but I find Blender to be useful, productive, and complete. It's solid, and has a very good community. Unlike other open source 'alternatives' to commercial software (like, say Gimp or Open Office), it's actually _better_ in many ways, and not just FUD or useful to casual users.

hesdeadjim(10000) 6 days ago [-]

Version 2.8 addressed the odd UI issue didn't it? I think they have a fully supported "industry standard" keybinding mode now.

andybak(2022) 6 days ago [-]

> 3ds Max is an excellent tool and one of the industry standards, but in some cases it causes a "lack of artists" due to its high costs.

A liberal non-commercial offering from Autodesk would at least help. Houdini realized this. Unity and Unreal are both essentially free to learners and hobbyists.

You need a pool of talent. And it needs continually refreshing.

mschuster91(3260) 6 days ago [-]

Adobe, back a decade ago, was quite liberal on that front - the stuff was easily crack-able, keygens whever one would look. As a result, whenever I ask colleagues with Photoshop skills how they acquired them, the answer always is 'with 14 and a Photoshop crack'.

Clever move to not stand in the way of cracks for amateurs too much until they introduced an affordable (!) subscription.

asutekku(10000) 6 days ago [-]

A "public secret" is that autodesk pretty much gives free non-commercial student licenses to anyone, no relation to an actual education institution needed. Advertising it better would definitely help them I think:

nineteen999(4201) 6 days ago [-]

I think the two subreddits (/r/Maya and /r/blender) for those products really illustrate the point further.

Blender usage exploded around 2.6 with the first UI revamp, and it seems to have swelled even more as 2.8 approached.

ethelward(3986) 6 days ago [-]

> A liberal non-commercial offering from Autodesk would at least help

Granted, my 3D experience is limited to making some custom planes for Flight Simulator 10 years ago; but is that not what Gmax is/was?

void_nill(4072) 6 days ago [-]

> Unity and Unreal

Unity and Unreal are Game Engines, not 3D Software. Yes, It's possible to use U and U for 3D modelling, but nobody with a chopchop brain do it. If you work with U and U you create the model in Blender/What ever and import it to the engines for the programming backend.

orbital-decay(10000) 6 days ago [-]

>Not only with Autodesk but also with other companies, we keep requesting improvements. However, it takes time for those improvements to make it to proprietary software. We expect faster improvements by using open source software.

This is ironic considering that along with Softimage, Maya is the history of CGI, written in software. Most of the features and paradigms we see as classic today have been developed in TAV/TDI Explore/PowerAnimator/Maya and accompanying software either by request of leading animation studios, or by the studios themselves. However, they slowed the cycle down significantly even before they've been bought by Autodesk.

HelloNurse(10000) 4 days ago [-]

Before it was sold to Autodesk, I mean, invaded by barbarians, the Roman Empire was the history of European civilization, written i roads and cities, then it 'slowed the cycle down significantly' and began shrinking.

tylerl(10000) 6 days ago [-]

Probably the most compelling feature of Blender being open source with a significant community is that it's the one 3d tool that Autodesk can't simply buy out.

sprafa(4057) 5 days ago [-]

I like this comment too much. I pretty much steer clear from Autodesk ever since they detonated softimage.

rpastuszak(4203) 6 days ago [-]

I needed 3D software to create a bunch of models for a RealityKit demo and the only usable version of Maya was $200 per month (Maya LT doesn't support plugins).

I gave Blender another shot (Blender 2.8 to be specific). The new UI is just fantastic! And it's actually usable with a touchbar-only setup.

I wish GIMP progressed so much—*nix environment + Adobe CC are the two main reasons I'm sticking to MacOS.

bparsons(4068) 6 days ago [-]

Creative cloud runs wonderfully on a cheap gaming PC.

VMG(3294) 6 days ago [-]

I hope Blender will include more GIMP features in the future

xemdetia(10000) 6 days ago [-]

I've given up on GIMP in a sense, Krita is the closest to what I actually want it to be but it's still missing some key stuff for me when it comes to vector selections and font work (but font work on GIMP was also less than great). It's much closer to what I've wanted GIMP to evolve into, so I've been crossing my fingers there.

lukaszkups(4193) 5 days ago [-]

I've made a switch almost year ago from Linux (and OSX earlier) to Windows - thanks to WSL it's like being still on *nix system but with running painlessly .exe applications ;)

jarsin(2817) 6 days ago [-]

Autodesk is responding to all this blender news by releasing full versions of Maya and 3dsMax for $250/yr for indies. It's the full versions not the LT versions.

I think too little too late.

https://area.autodesk.com/maya-indie/

Majestic121(10000) 6 days ago [-]

'Currently, this pilot offering is only available in Australia, Canada, New Zealand, the United Kingdom and the United States. We are currently piloting indie versions of 3ds Max and Maya so that recent graduates, freelancers, hobbyists and those starting out in the industry have a more affordable option to use our tools. We hope that this pilot performs as expected and can be expanded to other countries in the future.'

No thanks

philtar(4126) 6 days ago [-]

If you don't turn off the auto-renewal, it actually charges you for a full year of NORMAL priced Maya.

hajile(4188) 6 days ago [-]

> Your annual gross revenue from design work must be less than USD$100,000/year

$100K gross....

That basically means you are self-employed and undercharging, you are going broke, or you are a startup using some creative accounting.





Historical Discussions: Cloudflare S-1 (August 15, 2019: 915 points)

(915) Cloudflare S-1

915 points 7 days ago by tpw212 in 3303rd position

www.sec.gov | Estimated reading time – 4 minutes | comments | anchor

subject to differing interpretations, and may be inconsistent among jurisdictions, or conflict with other rules. These data protection and privacy-related laws and regulations are evolving and may result in ever-increasing regulatory and public scrutiny and escalating levels of enforcement and sanctions. For example, the EU's General Data Protection Regulation (the GDPR), which became fully effective on May 25, 2018, imposes more stringent data protection requirements than previously effective EU data protection law and provides for penalties for noncompliance of up to the greater of 20 million or four percent of worldwide annual revenues. Additionally, Brexit has created additional uncertainty with regard to the regulation of data protection in the United Kingdom. In particular, although the United Kingdom has enacted a Data Protection Act that is designed to be consistent with the GDPR, it is unclear how data transfers to and from the United Kingdom will be regulated.

We are also subject to the terms of our privacy policies and contractual obligations to third parties related to privacy, data protection, and information security. We strive to comply with applicable laws, regulations, policies, and other legal obligations relating to privacy, data protection, and information security to the extent possible. However, the regulatory framework for privacy and data protection worldwide is, and is likely to remain, uncertain for the foreseeable future, and it is possible that these or other actual or alleged obligations may be interpreted and applied in a manner that is inconsistent from one jurisdiction to another and may conflict with other rules or our practices.

We also expect that there will continue to be new laws, regulations, and industry standards concerning privacy, data protection, and information security proposed and enacted in various jurisdictions. For example, in the United States, various laws and regulations apply to the collection, processing, disclosure and security of certain types of data, including the Electronic Communications Privacy Act, the Computer Fraud and Abuse Act, the Health Insurance Portability and Accountability Act of 1996, the Gramm-Leach-Bliley Act, and state laws relating to privacy and data security, including, the California Consumer Privacy Act (the CCPA), that will, among other things, require covered companies to provide new disclosures to California consumers and afford such consumers new abilities to opt-out of certain sales of personal information when the CCPA goes into effect on January 1, 2020. The CCPA was amended on September 23, 2018, and it remains unclear whether any further modifications will be made to this legislation or how it will be interpreted. We cannot yet predict the impact of the CCPA on our business or operations, but it may require us to modify our data processing practices and policies and to incur substantial costs and expenses in an effort to comply.

Any failure or perceived failure by us to comply with our privacy policies, our privacy-related obligations to customers or other third parties, applicable laws or regulations, or any of our other legal obligations relating to privacy, data protection, or information security may result in governmental investigations or enforcement actions, litigation, claims, or public statements against us by consumer advocacy groups or others and could result in significant liability or cause our customers to lose trust in us, which could cause them to cease or reduce use of our products and otherwise have an adverse effect on our reputation and business. Furthermore, the costs of compliance with, and other burdens imposed by, the laws, regulations, and policies that are applicable to the businesses of our customers may limit the adoption and use of, and reduce the overall demand for, our products.

Additionally, if third parties we work with, such as sub-processors, vendors, or developers, violate applicable laws or regulations, contractual obligations, or our policies—or if it is perceived that such violations have occurred—such actual or perceived violations may also have an adverse effect on our business. Further, any significant change to applicable laws, regulations, or industry practices regarding the collection, use, retention, security, disclosure, or other processing of users' content, or regarding the manner in which the express or implied consent of users for the collection, use, retention, disclosure, or other processing of such content is obtained, could increase our costs and require us to modify our network, products, and features, possibly in a material manner, which we may be unable to complete, and may limit our ability to store and process customer data or develop new products and features.

43




All Comments: [-] | anchor

zemo(3660) 7 days ago [-]

'we may maintain relationships with customers that others find hostile, offensive, or inappropriate' these dudes would be neutral in the face of Sauron.

SanchoPanda(10000) 7 days ago [-]

They discuss this very thoughtfully in the blog post from when they kicked the daily stormer to the curb.

https://blog.cloudflare.com/why-we-terminated-daily-stormer/

tinbucket(4131) 7 days ago [-]

I adore, and extensively use, Cloudflare's free services for a couple of hobby projects. I hope they don't substantially change them post-IPO.

zaroth(2914) 7 days ago [-]

But tinbucket, I'm pretty sure you just loudly proclaimed you would be happy to pay at least $5/month!

superkuh(4005) 7 days ago [-]

Now they'll have even more incentives and pressures for censorship.

leetrout(4055) 7 days ago [-]

That's an interesting opinion. What about being publicly traded makes you think they'll me more incentivized to censor?

Facebook's performance suggests that playing fast and loose is preferable

greatjack613(4193) 7 days ago [-]

Another unicorn strikes. Is it just me, or have all of these companies decided to do IPOs around the same time? I mean uber, lyft, we work, and now cloudflare. I wonder who's on the menu tommorow

dsl(1980) 7 days ago [-]

If you started your filings in December of last year, right about now is when your public disclosures hit.

Why December? Because if you made less than a billion dollars in calendar year 2018 and filed in that year, you can carry over your 'small business' (I don't remember the exact name) status to 2019. Under the JOBS act, qualifying businesses aren't required to disclose executive compensation and a bunch of other stuff.

heavyheavy(10000) 7 days ago [-]

Maybe trying to get ahead of the impending recession?

the_duke(3506) 7 days ago [-]

Maybe because the economy has been doing very well and a eventual downturn is inevitable, which makes this a good time for an IPO?

ahartmetz(10000) 7 days ago [-]

I also thought 'emergency IPO before they won't be able to get a good price for years' for all of these.

frostyj(4177) 7 days ago [-]

Don't even compare Uber Lyft WE with Cloudflare..

koolba(664) 7 days ago [-]

Here's the meat and potatoes:

> We have experienced significant growth, with our revenue increasing from $84.8 million in 2016 to $134.9 million in 2017 and to $192.7 million in 2018, increases of 59% and 43%, respectively. As we continue to invest in our business, we have incurred net losses of $17.3 million, $10.7 million, and $87.2 million for 2016, 2017, and 2018, respectively. For the six months ended June 30, 2018 and 2019, our revenue increased from $87.1 million to $129.2 million, an increase of 48%, and we incurred net losses of $32.5 million and $36.8 million, respectively.

Compared to the numbers we've seen for recent tech IPOs, being on track to lose about $72M for 2019 seems reasonable.

Also, haven't read through it all but I'm curious how strong this clause will be for future control:

> The dual-class structure of our common stock will have the effect of concentrating voting control with those stockholders who held our capital stock prior to the completion of this offering, and it may depress the trading price of our Class A common stock.

zaroth(2914) 7 days ago [-]

What the heck happened to their OpEx in 2018?!

They went from growing at 60% YoY in 2017 with an 8% Operating Loss, to growing 43% YoY with an Operating Loss of 44%.

To put it another way, in 2018 they grew their revenue 40% (+$58m) and grew expenses 100% (+$115m).

I mean, apparently they raised a huge round and felt the need to increase spending massively. And then saw slowing growth as a response?

Was there a massive boost in R&D output? Entering entirely new product lines? My uneducated impression is that they offer largely the same product this year as they did last year.

The only thing I can think of is that they went from "scrappy startup" to "bloated unicorn" status?

erikig(4082) 7 days ago [-]

Both CloudFlare and Zoom are pretty great products and as much as I applaud their team's IPOs, I'm worried that the vagaries of being a public company will result in the GOOG syndrome. That's where a company with a great product is forced to expand every quarter to the point that it becomes a frankenstein-ish hulking version of itself.

sieabahlpark(10000) 6 days ago [-]

Zoom is great especially the malware side of it

whatshisface(10000) 7 days ago [-]

Why not just pay a dividend? That's what stocks are supposed to be, shares of ownership in legitimately valuable enterprises with return on investment.

groundlogic(4014) 7 days ago [-]

Finally an IPO I want to buy into. Technical fundamentals matter.

Edit: I mean that in the software/service way, not the economical way, in case there was any confusion.

dsl(1980) 7 days ago [-]

> Technical fundamentals matter.

Centralization of the entire internet behind a single entity is a _bad_ thing for people who care about technical fundamentals.

PatrolX(3409) 7 days ago [-]

Ditto, I'm excited about this one.

buildzr(10000) 7 days ago [-]

I'm sure it's just in my head, but it feels like the first tech IPO in several years where there actually seems to be a viable business model.

fastbeef(4168) 7 days ago [-]

I mentioned it in the We S-1 thread, but it really weirds me out that the S-1 is the first chance the public has a chance to see a company's financial status.

umeshunni(3453) 7 days ago [-]

Yeah, that's why it's the first step in becoming a 'public' company.

matthewowen(4062) 7 days ago [-]

why would the public have a chance to see the financial status of a privately held company?

alternatively, would you like to share your financial status with the rest of us?

cremp(10000) 7 days ago [-]

> We have a history of net losses and may not be able to achieve or sustain profitability in the future.

I never got this... Why go public if you literally say you won't make money?

Who would buy stock in something that will not return a positive?

Are they starving for money and hope some public trading gets them a boost?

Edit: It seems to be following the same pattern as the other tech IPOs recently, just to cash out the early investors.

perspective1(4189) 6 days ago [-]

'Why go public if you literally say you won't make money?' Because you can get rewarded heavily for it.

They have $88 million in working capital as of June 30, 2019 and are on track to lose another $40 million before the end of the year, so they need more capital. With $330 million carrying value of preferred stock, I think it's go public, somehow issue 100s of millions of dollars in debt or die.

tyingq(4160) 7 days ago [-]

Well, revenue of ~$130M, net loss of ~$37M. They aren't bleeding money like Uber or WeWork. You can imagine many ways they could close that gap, and in fact, the ratio has improved over the last few years.

khuey(3920) 7 days ago [-]

They're not saying they won't make money, they're saying they 'may not be able to' make money.

tiernano(3686) 7 days ago [-]

> Who would buy stock in something that will not return a positive?

cough uber... cough

meesles(4139) 7 days ago [-]

This comment is on every S1 thread. And every time someone mentions that it is essentially standard copy for all S1s, because you can never guarantee anything on the market.

SanchoPanda(10000) 7 days ago [-]

This is a risk report, and such statements represent a dire case outcome. It is the goal of everyone involved to make sure this doesn't happen, though its there because it could.

throw03172019(4199) 7 days ago [-]

Is this just a risk factor that the SEC applauds or forces?

PatrolX(3409) 7 days ago [-]

It's legal SOP for IPO's, to protect themselves in the event of lawsuits.

It doesn't 'literally say' they won't make money, that's you twisting it that way.

hprotagonist(2451) 7 days ago [-]

this is a normal sentence in every S-1 you care to name.

EXTREMELY SERIOUS WARNING (printed on a separate page, in red letters on a yellow background): Unless you are as smart as Johann Karl Friedrich Gauss, savvy as a half blind Calcutta bootblack, tough as General William Tecumseh Sherman, rich as the Queen of England, emotionally resilient as a Red Sox fan, and as generally able to take care of yourself as the average nuclear missile submarine commander, you should never have been allowed near this document. Please dispose of it as you would any piece of high level radioactive waste and then arrange with a qualified surgeon to amputate your arms at the elbows and gouge your eyes from their sockets. This warning is necessary because once, a hundred years ago, a little old lady in Kentucky put a hundred dollars into a dry goods company which went belly up and only returned her ninety nine dollars. Ever since then the government has been on our asses. If you ignore this warning, read on at your peril you are dead certain to lose everything you've got and live out your final decades beating back waves of termites in a Mississippi Delta leper colony. Still reading? Great. Now that we've scared off the lightweights, let's get down to business.

EXECUTIVE SUMMARY: We will raise [some money], then [do some stuff] and increase shareholder value. Want details? Read on.

the_duke(3506) 7 days ago [-]

The free tier of Cloudflare has been really amazing for small to medium websites.

It gives you a basically unlimited CDN, SSL, HTTP/2, DDos protection and fast DNS servers with almost 0 effort... for free ...

Thus increasing the stability and speed of many web properties that otherwise wouldn't pay for it.

I'm afraid this could easily change once the company is public and is under the pressures that come along with it, which is a shame.

(I also try to get them paid business when I can by nudging companies to use them)

louis-paul(2284) 7 days ago [-]

They list it as a key element of their business model:

> Free customer base—Free customers are an important part of our business. These customers sign up for our service through our self-serve portal and are typically individual developers, early stage startups, hobbyists, and other users. Our free customers create scale, serve as efficient brand marketing, and help us attract developers, customers, and potential employees. These free customers expose us to diverse traffic, threats, and problems, often allowing us to see potential security, performance, and reliability issues at the earliest stage. This knowledge allows us to improve our products and deliver more effective solutions to our paid customers. In addition, the added scale and diversity of this traffic makes us valuable to a diverse set of global ISPs, improving the breadth and economic terms of our interconnections, bandwidth costs, and co-location expenses. Finally, the enthusiastic engagement of our free customer base represents a "virtual quality assurance" function that allows us to maintain a high rate of product innovation, while ensuring products are extensively tested in real world environments before they are deployed to enterprise customers.

(page 80)

OrgNet(3095) 7 days ago [-]

Cloudflare is getting too big so I'm all for anything that could possibly make it smaller

JorgeGT(3882) 6 days ago [-]

You forgot the API! I have my own dynamic DNS with a raspberry and the cloudflare API for instance, it's very useful.

i_cant_speel(10000) 6 days ago [-]

I'd imagine there is significant value in capturing small customers with a free product who end up growing into their enterprise products.

davchana(10000) 6 days ago [-]

This. I have about 20+ hobbyish domains total, & 13 of them are with free Cloudflare Plan, & 3 of them are with their Domain Registrar side. I am completely fine with their free tier & paying for Domain Reg/Renewal. But just in case if in future i need to paygor DNS hosting, I would fallback to regular domain registar's free dns.

JohnAtCC(10000) 7 days ago [-]

How so? If the service is WORTH paying for, then it SHOULD be paid for. The world of service, if proven magnetizable, should always be monetized. #FreeMarkets

saagarjha(10000) 7 days ago [-]

Completely unrelated: the font that they chose to use in their marketing material looks a lot like San Francisco. I didn't know you could use it for promotional material that wasn't software for Apple's platforms.

mantap(10000) 7 days ago [-]

I could be wrong here but my understanding is that the letter forms of a font are not subject to copyright, only the font file itself. This is why Microsoft was able to create Arial which is a virtual clone of Helvetica without it being a derivative work.

craze3(4049) 7 days ago [-]

This is perfect timing considering they made headlines all of last week for banning that one site from their platform. But Google Trends seems to reveal that the banning incident was only 1/4 of the search volume they got for their big service outage on July 2nd:

https://trends.google.com/trends/explore?date=today%203-m&ge...

I wonder how the IPO news will compare in terms of global search volume

judge2020(4183) 7 days ago [-]

This is noted in their risk factors:

> Following the events in Charlottesville, Virginia, we terminated the account of The Daily Stormer. Similarly, following the events in El Paso, Texas, we terminated the account of 8chan. We received significant adverse feedback for these decisions from those concerned about our ability to pass judgment on our customers and the users of our platform, or to censor them by limiting their access to our products, and we are aware of potential customers who decided not to subscribe to our products because of this.

Within the next week we'll see the trends that represent how much nouse their IPO made (IPO news is generally slower moving): https://trends.google.com/trends/explore?date=today%201-m&ge...

Side note: you can see that an outage makes about 4x more noise than PR about DDOS protecting a website like 8chan: https://trends.google.com/trends/explore?date=2019-06-25%202...

zaroth(2914) 7 days ago [-]

If you think they banned sites for PR cover against the moral high ground before they went public, just wait till you see them once they are public!

poidos(10000) 6 days ago [-]

(I don't know a lot about IPOs...) When would I (i.e. a 'normal' stock buyer) be able to buy some shares of Cloudflare?

trazire(10000) 6 days ago [-]

It's unclear, but as we've seen in normal IPOs of this calibre (e.g. Slack, Pintrest, Fastly), probably next month.

jakarta(933) 7 days ago [-]

why would the cohort expansion rate be so much lower than Fastly's? https://www.sec.gov/Archives/edgar/data/1477333/000119312519... ^ cloudflare data on p. 80 shows a rate of 10-15% expansion

https://www.sec.gov/Archives/edgar/data/1517413/000119312519... ^fastly data on p. 66 shows cohort expansions that are much faster

judge2020(4183) 7 days ago [-]

This is a shot in the dark, but maybe the fact that there's a free tier is a deterrent for (old-school) established enterprises?

yosefzeev(10000) 6 days ago [-]

So cloudflare basically fucks over 8chan over some skimpy mass shooting evidence, and now has a public offering as a corporate company? Get rid of free speech as internet police and sell stock and make millions?

8bitchemistry(10000) 6 days ago [-]

The Free Speech Clause of the First Amendment prohibits only governmental, not private, abridgment of speech. [1]

A CDN is hardly the Internet Police.

[1] https://www.supremecourt.gov/opinions/18pdf/17-1702_h315.pdf

greyskull(4108) 6 days ago [-]

Is this satire?

cm2187(3368) 7 days ago [-]

It's odd to have these IPOs in the middle of August, in the middle of a super volatile market. I can't help thinking that there must be a sense of urgency, based on the perception that this is the last opportunity before a bear market and much more modest (realistic?) valuations.

big_chungus(4177) 6 days ago [-]

This is just about when kids go back to school. You see fewer IPOs over the summer because the bankers are all off in houses in the Hamptons or Nantucket. Your underwriters may not want to IPO until you get back, and there will be fewer people to buy your shares.

dsl(1980) 7 days ago [-]

> It's odd to have these IPOs in the middle of August

Read my other comment further down. It is from doing your private SEC filings in December to use a loophole to avoid disclosing a bunch of information.

These companies didn't just wake up in the morning and plan to IPO, it a good 18 months of planning that end with a public filing.

rapsey(10000) 7 days ago [-]

Or last resort because private money has dried up.

jrockway(3414) 7 days ago [-]

Am I reading this right when it looks likes they spend twice as much money on sales as engineering?

There was also a very strange spike in stock compensation in 2018, $27,000,000 compared to $2,755,000 in the previous year and $1,849,000 in the next. What happened there?

nacs(3873) 7 days ago [-]

Stock compensation spiking a year before their IPO doesn't seem unusual. It's likely an employee incentive.

marmada(10000) 6 days ago [-]

Are any people in agreement with me that Cloudflare's emphasis on neutrality is actually a bad thing?

I see several comments admiring how Cloudflare is willing to serve alt-right/nationalist/racist content in the name of avoiding censorship.

I never understood why this is a good thing. The argument seems to be that not-censoring things is valuable because who knows when Cloudflare censors something that is actually valuable/not racist or nationalist.

Considering tech companies tend to censor right-wing extremist content, I consider the probability of that happening to be very low.

ksec(1707) 6 days ago [-]

I quite like the way how Cloudflare has managed things so far. It is like Apple being the GateKeeper, they have the power, but they wont use it unless they absolutely have too.

If they started to actively banning and censoring content then at some point they are highly likely to corupt themselves and misuse that power.

sexy_seedbox(3695) 6 days ago [-]

Fun fact: Matthew Prince had to borrow money from his mother when Cloudflare was just starting out.

They've come a long way.

https://twitter.com/eastdakota/status/1129929178382446592

yumraj(3550) 6 days ago [-]

No offense, but borrowing from a parent is not really borrowing, at least in most cultures.

ignoramous(3599) 7 days ago [-]

Cloudflare has excellent offerings in the networking space, their tech stack seems solid. The customer base seems loyal: Once you're a customer, free or paid, there's very little reason to move out. Their revenue has been growing 43% and 59% for past 2 years: The business side of things seem to be kicking along at a good pace. They seemed to have figured out a perfect balance at offering services to freelancers, to small to medium businesses, to enterprises, to Fortune 500s. The diversity is astounding.

A few acquisitions and the next thing you know they are in the video-streaming business, in the ISP business, in the mobile carrier business as 5G rolls in and what not: The possibilities are endless? Esp as the world becomes more and more connected (IoT and proliferation of smartphones) and business move online, security and speed at scale are going to be of paramount importance and Cloudflare is primed to seize that market, imo

From running a honeypot network to winning techcrunch disrupt to this. What an amazing journey. One the few tech IPOs I'm genuinely excited about.

Congratulations eastdakota et al.

the_duke(3506) 7 days ago [-]

They already are in the video streaming business. ;)

https://www.cloudflare.com/products/cloudflare-stream/

qes(10000) 6 days ago [-]

> Once you're a customer, free or paid, there's very little reason to move out.

I find their sales tactics and pricing strategies aggressive and unfriendly.

peterhadlaw(4189) 6 days ago [-]

Man what if the government just sporadically decided to shut down Cloudflare one morning based on a feeling. That'd be unfortunate.

knd775(10000) 6 days ago [-]

How would 'the government' go about doing that?

martin1975(4155) 6 days ago [-]

It's too bad that Matthew Prince consistently violates 1A rights by hiding behind the 'we're a private company' excuse... other than that, they're poised to succeed.

owenmarshall(10000) 6 days ago [-]

> 1A rights

> private company

...?

billpg(1666) 7 days ago [-]

> We have applied to list the Class A common stock on the New York Stock Exchange under the symbol "NET."

Pardon?

666lumberjack(4141) 7 days ago [-]

Looks like CFR and CLD are already taken, to be fair

mxuribe(4008) 7 days ago [-]

Yeah, i would have tried CFLR, or something like that...unless, they have other ambitions...?

mcintyre1994(4201) 7 days ago [-]

I guess someone beat them to CLOUD?

asatterfield54(10000) 7 days ago [-]

Probably a good one to hop into for a quick gain due to hype and usage, but long-term the price will fall.

Will continue to monitor, but looking like it will be the standard 7/11 IPO strategy - get in and immediately start the sell off if price drops 7% day-to-day or 11% consecutively, then short to price where losses are matched.

quickthrower2(1306) 6 days ago [-]

How is that better than just putting 11% of your investment on black or red?

rainyMammoth(4200) 7 days ago [-]

How can you get into this IPO at the listing price ?

vechagup(10000) 6 days ago [-]

You generally need to be an institutional investor or a preferred customer of one of the brokers that is issued shares by the IPO underwriter. Here's a good general writeup https://money.usnews.com/investing/stock-market-news/article... and Fidelity's eligibility requirements https://www.fidelity.com/customer-service/how-to-participate....

Note that even if you meet the eligibility requirements, if the IPO is sufficiently popular you may still not get a chance to participate, as the institutions and people in line before you may have already bought all the shares.

mathieutd(3719) 6 days ago [-]

Call your broker. They might have shares available.

kd3(10000) 7 days ago [-]

Here's what Cloudflare's CEO wrote on their blog:

'In the two years since the Daily Stormer what we have done to try and solve the Internet's deeper problem is engage with law enforcement and civil society organizations to try and find solutions. Among other things, that resulted in us cooperating around monitoring potential hate sites on our network and notifying law enforcement when there was content that contained an indication of potential violence. We will continue to work within the legal process to share information when we can to hopefully prevent horrific acts of violence.'

Monitoring?? Sounds a lot like surveillance and spying to me. This is why they want to MITM the entire Internet. All encryption becomes useless. Breaking privacy and security just like Facebook. They even co-opted Wireguard recently. If you understand and see where this is going, stay away from them.

zemo(3660) 7 days ago [-]

> All encryption becomes useless.

if you're gonna conspiracy theorize at least go the whole mile. They provide SSL termination so the encryption is useful to them because it gives them a monopoly on the surveillance apparatus. Not useless at all, it guarantees their position in the ecosystem.

brundolf(2468) 7 days ago [-]

They specifically talk about violence. Intent to incite violence isn't protected under the first amendment: https://en.wikipedia.org/wiki/First_Amendment_to_the_United_...

rossdavidh(4106) 6 days ago [-]

While I agree there is a risk, I think a lot of their free customers would either not be using SSL at all, or else would find it too complex to remain independent and just have a Facebook page instead, if Cloudflare didn't make that kind of thing easier. Neither seems like it would be better. Not to mention things like being able to resist a DDOS from anyone who dislikes you.

Sure, there may be some people using Cloudflare free services who would otherwise be doing it themselves, but I think by far the greater majority of their non-paying users are folks too small and/or non-technical to maintain their own site otherwise.

kd3(10000) 7 days ago [-]

No wonder lots of it is free. You are the product, including your users.

rvz(4065) 7 days ago [-]

> Today, our network spans 193 cities in over 90 countries and interconnects with over 8,000 networks globally, including major ISPs, public cloud providers, SaaS services, and enterprises. We estimate that we operate within 100 milliseconds of 98% of the Internet-connected population in the developed world, and 93% of the Internet-connected population globally (for context, the blink of an eye is 300-400 milliseconds). We intend to continue expanding our network to better serve our customers globally and enable new types of applications, while relentlessly driving down our unit costs.

This right here is a real tech IPO that has a strong portfolio behind it with a viable business model too and with an impact that affects millions of websites. Throughout this year it is just loss-making companies floating everywhere on the markets here and have boarded the hype-train into the red.

I hope CloudFlare with these numbers here don't board the wrong train here.

soup10(10000) 6 days ago [-]

Only issue is that they don't have that much revenue compared to the amount of interest there will be in their stock so its going to turn into a speculation vehicle.

867-5309(10000) 6 days ago [-]

300-400ms for blinking? is that during a stroke?? I see they went with the top search engine result Quora - not the best source for facts - and the second result states a tenth of a second, which is probably more accurate. we get it, your service is quite fast, as fast as blinking

closeparen(4093) 6 days ago [-]

Since Cloudflare serves the tech industry, it's also pretty vulnerable to a downturn in the tech industry. The hype-train companies are its customers.

bonestamp2(10000) 6 days ago [-]

I agree with all of that, but the other important thing (at least to me as an investor) is their 51% growth over two years (with 77% gross margin). You can have the best product, but if you're not actually growing and profitable then as much as I like the company or the product I'll put my investment dollars elsewhere.

quartus(4201) 6 days ago [-]

>Throughout this year it is just loss-making companies floating everywhere on the markets here and have boarded the hype-train into the red.

Cloudflare is a loss-making company too, and their losses are growing. From the S-1: 'we have incurred net losses of $17.3 million, $10.7 million, and $87.2 million for 2016, 2017, and 2018, respectively'

ksec(1707) 6 days ago [-]

>We estimate that we operate within 100 milliseconds of 98% of the Internet-connected population in the developed world,

Am I missing something here? For Reference; A one way New York to Hong Kong connection takes roughly 110ms.

paggle(10000) 7 days ago [-]

The blink of an eye is not 300ms.

agandy(3996) 7 days ago [-]

I would contend that Zoom is also a good example of this.





Historical Discussions: Tasty Seaweed Reduces Cows' Methane Emissions by 99% (August 16, 2019: 865 points)

(865) Tasty Seaweed Reduces Cows' Methane Emissions by 99%

865 points 6 days ago by RickJWagner in 2665th position

www.goodnewsnetwork.org | Estimated reading time – 3 minutes | comments | anchor

A puffy pink seaweed that can stop cows from burping out methane is being primed for mass farming by Australian researchers.

The particular seaweed species, called Asparagopsis, grows prolifically off the Queensland Coast, and was the only seaweed found to have the effect in a study five years ago led by CSIRO. Even a small amount of the seaweed in a cow's diet was shown to reduce the animal's gases by 99%.

Associate Professor Nick Paul, who is the leader of the Seaweed Research Group at the University of the Sunshine Coast (USC), said that if Australia could grow enough of the seaweed for every cow in the nation, the country could cut its greenhouse gas emissions by 10%.

"Seaweed is something that cows are known to eat. They will actually wander down to the beach and have a bit of a nibble," Dr. Paul said.

RELATED: Student Treks to Yellowstone and Finds Bacteria That Eats Pollution and 'Breathes' Electricity

"When added to cow feed at less than 2% of the dry matter, this particular seaweed completely knocks out methane production. It contains chemicals that reduce the microbes in the cows' stomachs that cause them to burp when they eat grass."

The USC team is working at the Bribie Island Research Centre in Moreton Bay to learn more about how to grow the seaweed species, with the goal of informing a scale-up of production that could supplement cow feed on a national—and even global scale.

Photo by USC

"This seaweed has caused a lot of global interest and people around the world are working to make sure the cows are healthy, the beef and the milk are good quality," Dr. Paul said.

"That's all happening right now. But the one missing step, the big thing that is going to make sure this works at a global scale, is to make sure we can produce the seaweed sustainably.

LOOK: Trees Growing Out of Buildings Could Help Heal China's Air Pollution Problem

"If we're able to work out how to scale up the seaweed to such a level to that can feed all of the cows and the sheep and the goats around the world, then it's going to have a huge impact on the climate; it's going to address a whole lot of carbon-neutral agendas that different countries have; and it's ultimately going to save us all billions of dollars," he concluded.

This article was reprinted from the University of the Sunshine Coast.

(WATCH the intriguing interview with Dr. Paul below)

Feed Your Friends Some Positivity By Sharing The Good News To Social Media...




All Comments: [-] | anchor

Hitton(10000) 6 days ago [-]

That's cool, but let's not forget that methane stays in atmosphere only for 12 years. That means that if methane production stays constant, it doesn't actually increase amount of methane in atmosphere. Therefore I think that reducing production of greenhouse gases with longer longevity should be preferred.

danmaz74(2448) 6 days ago [-]

You don't need to prefer one or the other. We should do both.

dillonmckay(10000) 6 days ago [-]

Do grass-fed cows produce less methane than corn-fed cows?

wysifnwyg(10000) 6 days ago [-]

Yes.

whenanother(10000) 6 days ago [-]

how about all that antibiotics they are being fed as well? this could be a man made problem.

xmichael999(10000) 6 days ago [-]

Ok so strange question, does this have an effect on humans as well? Do regular sushi eaters fart less? (;

WhompingWindows(4199) 6 days ago [-]

The seaweed is a specific kind of red seaweed that isn't used in sushi. The seaweed contains compounds which tamp down specific microbe levels. I doubt the exact same effect is occurring in sushi eaters, though a similar one hypothetically could be.

goodalgae(10000) 6 days ago [-]

My company is currently working to grow Asparagopsis Taxiformis, here in the US, for this purpose.

I agree with comments that we have big issues to tackle in regards to methane sinks failing, such as permafrost, and there are things to be said about people moving en masse to plant based diets – but, the future will always be a mix of tomorrow and today. Many people will still eat beef, and we need to ensure we reduce the negative effects of raising cattle as much as possible. We need a giant, mixed bag of tricks working in unison to slow the warming we have caused over the last 100 years.

Luckily, current research is showing that – not only does AT nearly eliminate methane from the bovine digestive process entirely when supplementing between half and two percent of daily diet – we are seeing early indications in research of some amazing additional benefits – improved milk production, increased immunity, improved food conversion ratio (meaning you can feed cattle less and have them pack on more protein). Studies are not finalized, but what I have heard of them sounds very, very good.

My background is in tech (founded and exited a TechStars company and have had a career in software and hardware startups) and my colleagues are in marine biology and phycology (algae science). My hope is that by approaching this problem from the perspective of a fast-iterating startup, we can get this product to market faster than the traditional routes of academic timelines and federal funding sources this industry is used to.

If you are an investor, beef serving restaurant owner, or in business related to cattle raising and finishing and would like to find out more, please email us at [email protected] And follow our newsletter for updates at https://goodalgae.com.

Cheers!

tr3ndyBEAR(10000) 6 days ago [-]

If AT can do this for bovines, I wonder what the health benefits for us could be

wwweston(4198) 6 days ago [-]

> early indications in research of some amazing additional benefits – improved milk production, increased immunity, improved food conversion ratio

What are the chances the gains here might be on the order where we could stop dosing cattle with antibiotics?

muzakthings(4092) 6 days ago [-]

this is super cool. do you think the primary demand driver for your sales will be 1) gov't mandates, or 2) cattle farmers wanting to differentiate their meat to eco-conscious customers / retailers? just curious about how you see the market.

leipert(4177) 6 days ago [-]

What is the feasibility of this, deploying it on a large scale?

I did some quick calculations on a napkin.

Apparently there are between 1 billion and 1.5 billion cows [0][1]. Let's just say 1.5 Billion.

In New Zealand the beef (dry cow) vs dairy cow ratio is almost 1:1 [2], but I assume that countries like India have more dairy cows, so I just assumed that 60% percent of cows are dairy cows.

Apparently dry cows consume around 25 lb (11 kilogram) dry matter per day [3], milking cows 25kg dry matter [4].

Let's just assume that 1% of the dry matter needs to be replaced with the algae, which is half of what the article mentions. That would mean, that we need:

(1500 Million * 0.6 * 0.01 * 25kg) + (1500 Million * 0.4 * 0.01 * 11kg) = 291 Million kilogram = 291.000 metric tons of algae per day

Just the logistics alone is major feat to pull off, put on a yearly scale the weight of these algae equals half of the trash produced in the US per year [5].

[0]: https://beef2live.com/story-world-cattle-inventory-ranking-c...

[1]: https://www.drovers.com/article/world-cattle-inventory-ranki...

[2]: https://teara.govt.nz/files/1_183_Beef_Ratios_0.pdf

[3]: http://www.thebeefsite.com/articles/3154/how-much-forage-doe...

[4]: https://albertamilk.com/ask-dairy-farmer/how-much-feed-does-...

[5]: https://www.wolframalpha.com/input/?i=291*365+million+kilogr...

masonic(2419) 6 days ago [-]

BTW, I find the wording on your landing page to be a bit awkward and the font used to be a difficult read as well.

The '+' directive in the email address: do you use business Gmail, or does another email system use it similarly?

2_listerine_pls(4051) 6 days ago [-]

What's the mechanism behind methane reduction?

deevolution(10000) 6 days ago [-]

This would be a tremendous benefit if it can be deployed at scale, which I'm sure it eventually could. HOWEVER, This solution fails to address the excessive amount of water consumption required to raise these cows. fresh, clean water is already becoming a major global crisis. The simplest and most effective solution to ensuring our survival as a species remains switching to a plant-based diet. Too many people adopt a cow-first mentality when we should be adopting a humanity-first mentality. The future is Vegan!

m0zg(10000) 6 days ago [-]

Seems like just growing seaweed won't be enough though. You'd have to grow it _in close proximity to the cows_, otherwise there are other greenhouse gases that will be emitted before the cows can eat it. It's also unclear to me what this means in terms of water consumption, and therefore sustainability. I'd be curious to hear how (if at all) these issues could be addressed.

stevespang(10000) 6 days ago [-]

Microbes in the cows stomach develop resistance to the seaweed compounds rapidly.

ptah(10000) 6 days ago [-]

where does the methane go instead? there must be some trade off somewhere?

JulianMorrison(3944) 6 days ago [-]

> not only does AT nearly eliminate methane from the bovine digestive process entirely when supplementing between half and two percent of daily diet – we are seeing early indications in research of some amazing additional benefits – improved milk production, increased immunity, improved food conversion ratio

Does this not rather imply that the cows are nutritionally deficient of something that this seaweed provides? Perhaps something in their ancestral environment?

legohead(4203) 6 days ago [-]

but can we address the water consumption?

baroffoos(10000) 6 days ago [-]

If it can be grown in salt water like the name 'seaweed' suggests then there is no issue with water consumption.

PopeDotNinja(4165) 6 days ago [-]

My inner child was disappointed to learn that cow farts are not the major source of bovine methane emissions. But saying cow burps is stilly pretty funny.

https://climate.nasa.gov/faq/33/which-is-a-bigger-methane-so...

barking(3510) 6 days ago [-]

I'm disappointed too! I was going to ask if this might work for us humans.

chiefalchemist(4027) 6 days ago [-]

Let's not forget:

- Producing cows is resource-intensive. Water and feed could be used more effectively elsewhere.

- Consumption of red meat and dairy is generally unhealthy.

- Runoff from cow 'pastures' is effectively pollution.

Encouraging less use of cows because of their impact on climate change was a win on a number of levels. Not any more.

toper-centage(10000) 6 days ago [-]

Not any more? How come?

whiddershins(2556) 6 days ago [-]

I really think it is irresponsible to keep repeating dietary hypothesis as fact.

The notion that red meat is bad for you does not have compelling evidence, based all my years of obsessively researching this ... and with the help of equally obsessive and more qualified friends.

As far as I can tell, how a diet makes you feel and presents visually is the closest thing we have to a reliable indicator of the health effects of that diet.

From that standpoint, many many people get amazing results from a diet high in red meat.

I cannot urge people enough to maintain skepticism with regards to "scientific" dietary advice, and try to do what makes them feel healthier and more energetic.

stronglikedan(10000) 6 days ago [-]

> Consumption of red meat and dairy is generally unhealthy.

That could be said about anything. Everything is generally unhealthy if done in excess. Moderate amounts of red meat and dairy is fine (and delicious).

asteli(10000) 6 days ago [-]

Question from the peanut gallery:

The gut bacteria in a cow digest components of the animal's food preferentially and excrete methane. If you knock those bacteria out, does the stuff the bacteria would have eaten get excreted by the cow? If so, do the organisms that digest it external to the cow emit greenhouse gases?

stinos(3982) 6 days ago [-]

I also wonder what effect this whole thing has on the cow. IIRC in humans there are even links between (lack of/bad) gut bacteria and certain diseases. So it seems at least worth it figuring ouf how this affects cows, imo.

mklarmann(4148) 6 days ago [-]

Actually by knocking out the bacteria, they can see a slight uptake (5%) in feed efficiency of the cows. So part of it is put into fat and muscle.

sc4les(10000) 6 days ago [-]

But not water usage, land usage, suffering for sentient beings, transport of feed, waste of human food for feeding etc etc

rimliu(10000) 6 days ago [-]

I am pretty sure that growing rice requires land, water, and transport. And I do not think cows eat human food nor humans would be very pleased to eat what cows eat.

mklarmann(4148) 6 days ago [-]

There is a similar product made from garlic, that is encapsulated so it just doesn't wash out with the cows saliva, that has a similar effect. You need about 50kg of it per cow, and it kills the methane producing bacteria. Yet the issue (which they no not fully communicate publicly, but one of the lead scientists told me in private), I guess is the same as with the seaweed. It is not all about reducing methane. We need to do a CO2eq calculations, which includes nitrous oxide at a much higher factor.

And there is much more of this GHG coming out of the treated cow. Which almost cancel the methan reduction effect.

DennisP(3463) 6 days ago [-]

Does the garlic cause methane reduction by the same mechanism as the seaweed?

darawk(10000) 6 days ago [-]

I feel like I read this same story once a year or so for the past 5-6 years. Why hasn't this been commercialized yet?

sdinsn(10000) 6 days ago [-]

Hard to scale

moultano(3067) 6 days ago [-]

No commercial incentive to do so. If we had any effective climate regulation, you'd see these things rapidly put into practice.

axaxs(10000) 6 days ago [-]

There's just not enough of it, nor an efficient way to get it to every animal.

toomanybeersies(4200) 6 days ago [-]

There's an outfit in New Zealand that sells seaweed as cattle feed, not sure if it's the same stuff though.

reportgunner(10000) 6 days ago [-]

I'm missing an 'of something' after the 99%. It might the case that 99% in the title is actually 0.0001% of all cows in the world. It does not make sense to use % without knowing what is the total, the number carries no information whatsoever.

crayola(4128) 6 days ago [-]

I interpret this as, 'for an individual cow given a diet of that seaweed'. I don't think there's an issue there.

cmroanirgo(4128) 6 days ago [-]

From the article:

> Even a small amount of the seaweed in a cow's diet was shown to reduce the animal's gases by 99%.

and

> When added to cow feed at less than 2% of the dry matter, this particular seaweed completely knocks out methane production. It contains chemicals that reduce the microbes in the cows' stomachs that cause them to burp when they eat grass

The 99% is relating to the gases emitted by a cow eating this seaweed vs not eating the seaweed.

> if Australia could grow enough of the seaweed for every cow in the nation, the country could cut its greenhouse gas emissions by 10%.

hispanic(2615) 6 days ago [-]

Man - cows eating seaweed is no more natural than cows eating corn. We need to stop bastardizing natural systems for our own comfort and convenience.

coryfklein(2715) 6 days ago [-]

Cows allowed near the ocean will naturally graze on seaweed.

RosanaAnaDana(10000) 6 days ago [-]

I mean, you know there are no 'cows' in the wild. We made cows from aurochs.

alphakappa(3620) 6 days ago [-]

There are many planet saving discoveries like these. There's also a catch, which is why the planet hasn't been saved yet.

> "If we're able to work out how to scale up the seaweed to such a level to that can feed all of the cows and the sheep and the goats around the world, then it's going to have a huge impact on the climate;

And there you go. That's the hard part that you cannot just hand-wave away.

You also have to wonder what the impact of growing all this seaweed (or harvesting it) to feed the planet's bovine population will be. In the end, the problem is due to the massive scale of human consumption on this planet. The mitigation might be just as bad when you consider the scale at which you have to produce the 'better' solution.

JMTQp8lwXL(10000) 6 days ago [-]

From an environmental perspective, it's easier to directly consume fruits, vegetables and nuts yourself, rather than have an animal do the same, then eat the animal. That doesn't address the human appetite for animal proteins, but there is beyond meat.

tempestn(1468) 6 days ago [-]

True that that's a non-trivial problem, but seaweed farming is actually something that we could massively scale up. With vertical farms, a ton (metaphorically speaking; more than that literally) of food can be grown in a small surface area of ocean. Would be much more efficient for feeding livestock than relying on farmland that could be growing human food. (Although as an aside I'm personally in favour of a decent amount of natural grazing space and time for farmed animals, so I'd like to see a balance.)

stephenr(4018) 6 days ago [-]

It's simple really. After global warming causes sea levels to rise, the cows will be underwater anyway, they can just find their own seaweed to eat.

Problem solved!

belorn(4160) 6 days ago [-]

Scaling is a pretty common objection in this kind of articles but I have a hard time understanding the reasoning for it. Scaling is always an issue regardless of what the new technology is so the question is then if there is specific reasons to doubt that scaling can not be solved.

To compare two different tactics to reduce global warming, electric cars and seaweed for cows. It sound as a much easier job to grow seaweed compared to the electrification of the transport sector by scaling up electric engines and battery production, and we (as a human species) are pretty sure the later will happen regardless because we can't continue to burn fossil fuels.

whatshisface(10000) 6 days ago [-]

>In the end, the problem is due to the massive scale of human consumption on this planet.

The technology required to alter human nature to consume less or alternatively the technology required to win the deadliest war ever fought is going to be a lot harder to develop than it would be to grow a lot of seaweed.

bvinc(10000) 6 days ago [-]

If there was a profit motive, and it wasn't free to emit greenhouse gasses into the atmosphere, I bet we would figure it out.

vtrips(10000) 6 days ago [-]

I would rather we focus on cutting down the rest of the 90% of emissions than the natural emissions which have been occurring since before the industrial revolution messed it up.

There is a fine balance in nature and maybe cows are the wrong place to experiment.

mc32(4141) 6 days ago [-]

It's a big of cosmic irony that cows will be eating what so called sea-cows evolved to eat by way of their natural environment.

The obviously this messes with cows' gut bacteria. Wonder if it has any significant effects on cows or is it better all around?

hannob(2089) 6 days ago [-]

Skeptical science remark:

This whole 'if cows eat seaweed that'll vastly reduce their carbon footprint' is based on a single study with 12 cows.

If you take into consideration that especially small, never replicated studies often don't hold up to scrunity the next thing to do is not to ask how to produce the seaweed. The next reasonable step would be to try to replicate this study in a larger sample (and probably also with a larger variety of cow breeds).

solveit(10000) 6 days ago [-]

You're absolutely right, but the effect size is pretty damn impressive and I expect this to at least partially replicate.

strainer(4199) 6 days ago [-]

'is based on a single study with 12 cows.'

Its not - most often 'skeptical science remarks' although popular, are poorly informed and misleading.

A quick scholar search shows numerous studies broadly agreeing with the one featured here, including the seaweeds effects on sheep digestion and microbial and metabolic details:

https://scholar.google.com/scholar?q=asparagopsis+methane

sieabahlpark(10000) 6 days ago [-]

But the confirmation bias is so easy! It's also green! Why do you hate the environment? /s

avip(4065) 6 days ago [-]

Even more skeptical remark:

Effects of nutrition on livestock methane production have been studied for over 20 years now. We have methane reduction results from both in-vivo and in-vitro research. What varies is the numbers (99%, 50%, 15%?). Replication is always welcome but I believe we have enough sound science to move into more industry/engineering efforts.

dedestruction(10000) 6 days ago [-]

Further food for thought: the lower the statistical power of a study which meets a given significance threshold, the more exaggerated the effect size will be.

So our default posture to evidence like this, even assuming we find it broadly credible, should be that the real-world effect is very likely to be smaller than this, and very unlikely to be larger.

dsfyu404ed(4076) 6 days ago [-]

Yup. There's plenty of boutique dairy farms within easy driving distance of rich colleges in the northeast (some of whom already have existing relationships with said colleges). I'm sure anyone who seriously wants to replicate the study would be able to find a farmer that will work with then to get an n value of a few hundred. It's not like you have to go somewhere remote to study cow farts.

ianai(4208) 6 days ago [-]

Anecdotally, taking in a little chlorophyll per day dramatically reduces how much gas I produce. So in my experience, it's at least possible the effect may be large. Worthy of more research.

SamuelAdams(3856) 6 days ago [-]

I agree, we need to slowly study other groups of cows to see if this is a viable solution.

I'm reading Factfulness by Hans Rosling at the moment. He had a great point about 'single numbers' - the jist was to always be skeptical of just a single number or metric.

Here's an example [1]. During WWII, soliders were dying in their own vomit because they were laying prone on the battlefield, in sick beds, etc and left unattended. Doctors tried using the Recovery Position [2] and saw a decrease in those types of deaths by about 99% (this is in the book).

Sounds like it should work for other types of people, right? First aid organizations quickly updated their 'best recommendations' for all types of people: sick, drunk, young, old. This position was the best possible position for increased airway access, thereby decreasing preventable patient deaths.

It wasn't until the 1980's that this idea was challenged. A few health organizations noticed an increase in infant deaths. Hong Kong's group was the first to investigate this [3]. Other groups joined in and realized that infants, when turned on their stomach, do not have the strength to tip their heads after they vomit. As a result, infant deaths increase when they are placed into the Recovery Position.

The recommendation was quickly updated in the late 1980's / early 1990's. Now the recovery position is recommended for people ages 5 and up.

Between 1940 - 1985, there was a larger number of infant deaths because one group of well-meaning people took some knowledge they gained for one group and quickly applied it to all other groups.

I think this study (cows reducing methane based on diet) is a great idea. But we need to be cautious about applying this quickly to all cows, else we could kill half the cow population and cause the cost of beef to skyrocket.

[1]: https://www.gapminder.org/factfulness-book/notes/, Page 163.

[2]: https://en.wikipedia.org/wiki/Recovery_position

[3]: https://www.ncbi.nlm.nih.gov/pubmed/2866397?dopt=Abstract

mklarmann(4148) 6 days ago [-]

The study misses out to report on the increased production of N2O. Which really does not help.

rmason(64) 6 days ago [-]

Research takes time. There are researchers trying to grow the stuff on both US coasts. This articled details some of what is being done. They hope to test some of seaweed on cows at UC Davis soon.

https://phys.org/news/2019-05-cultivate-seaweed-slashes-gree...

philipkglass(3848) 6 days ago [-]

For several months, Smith has been experimenting in her lab with cultivating the seaweed to, among other things, maximize concentrations of bromoform—the compound that blocks the production of methane in cows, sheep, goats and other ruminant animals.

'This is the sporophyte,' she said at her lab in La Jolla, holding one of a dozen flasks filled with the red algae, dancing in aerated seawater. 'In this case, the bubbles are not as vigorous, allowing these puff balls to get bigger.

'By just manipulating nitrogen and phosphorous, we have already seen that we can double the concentrations of bromoform in just a week,' she added.

As commenters have noted previously, it seems like using synthetic bromoform would be cheaper and more scalable. The rest of the seaweed appears to be incidental to the methane suppression. But maybe there are regulations about introducing new cattle feed additives that you can sidestep by mixing whole algae into the food. Or maybe they're already developing this with consumer branding/marketing in mind; 'seaweed' sounds better than 'bromoform.'

blue_devil(3781) 6 days ago [-]

More of a philosophical perspective here: The eco-systems on Earth have evolved over a long time. Without bias against scientific progress, why do we think we can 'solve' this without going down a slippery slope of creating another problem? When is comes to complex systems that we don't completely understand, maybe the simplest solution is to just restore the system, that is 'give up' on cows as human feed.

rkul(10000) 6 days ago [-]

Your argument implies that cows who did not eat seaweed survived and therefore we see cows not eating seaweed now.

Since cows cannot live underwater and seaweed does not grow outside water, these 'two worlds' could not meet and the outcome of their interaction is unknown. It might work, it might not, however, evolution was not here yet.

52-6F-62(3593) 6 days ago [-]

I don't think it's as simple as you're making it out to be—not universally, anyway. Diet and nutrition are also complex systems—moreso for some of us in the population. I've written this out a number of times before so I'm just going to link my last comment on the subject.

I feel the need to speak up for people with similar conditions for whom simply cutting out a crucial part of our diet is not resolved.

https://news.ycombinator.com/item?id=20635612

eemil(10000) 6 days ago [-]

That's obviously the most elegant, best solution. But the time for such is long past. It's clear that humanity will not stop consuming, and expecting endless growth. At least for a few generations. In the short term, we need any win we can get, even if there are risks.

rubber_duck(10000) 6 days ago [-]

Restoring what system - nature is not some harmonious place where everything just works and people came and ruined it - there were plenty of extinction events prior to humans, evolution can lead to dead ends and get stuck at local minima. It doesn't optimize for a lot and collects garbage allong the way (eg. eye optic nerve blind spot)

Man made climate change is showing us how vulnerable we are to changes in the Earth ecosystem - instead of putting our heads in the sand and hoping everything goes back to the way it was while we pray to the weather gods we could actually figure out ways to engineer our environment and become more resilient to such changes.

If we are so sensitive to climate change what would happen in another Chicxulub impactor scenario - another global extinction ? Not to mention much more probable global disasters like disease outbreaks.

vallode(10000) 6 days ago [-]

Personally I think this is a pessimistic point of view. The reason we think we can 'solve' these issues is because we have been doing a good job of using tools to help us along the way of human history.

If we do as you propose, where do we stop? What makes us think we can 'solve' anything? Computing, timber, fire, clean water. The truth is that humans are a part of the eco-system, and the system will evolve with us. I support slowing down certain things, so we do not hit a dead-end, since I am vegan myself. Giving up will not solve anything, especially since we have a hell of a lot of clean up to do.

sak5sk(10000) 6 days ago [-]

Is this even relevant anymore? I mean, with permafrost thawing and triggering sudden soil collapse - forming lakes and bubbling up even more methane... to what degree do cow methane emissions even matter when looking at the big picture? I tried googling for some numbers but didn't find any concrete credible sources other than 'most methane comes from permafrost thawing'

irjustin(10000) 6 days ago [-]

Seriously? https://timeforchange.org/are-cows-cause-of-global-warming-m...

It wasn't even that hard to find one article. It might not be perfect, but you can clearly keep going.

And yes, it's one facet. We won't solve our problems just by fixing this, but your attitude definitely won't.

lazyguy(10000) 6 days ago [-]

This sounds extremely unhealthy for the cows.

nkozyra(10000) 6 days ago [-]

Why? Do we assume seaweed lacks the cellulose and minerals various grass an land weed varieties do?

triceratops(10000) 6 days ago [-]

Nothing about the industrial farming system is healthy for the cows. The feedlots, antibiotics, living in filth, eventual transportation to a slaughterhouse etc.

irjustin(10000) 6 days ago [-]

where did you get this? i haven't read anything towards this end.

Accacin(10000) 6 days ago [-]

Disclaimer: I'm plant-based, and of course biased.

What would it take people to realise that eating meat is just not good for the environment and rather than going through all these 'fixes', just stop eating meat?

We're getting to the point where we're going to have to sacrifice some 'comforts' IMO.

redka(3967) 6 days ago [-]

Do you think it's been proven enough to be sure that plant based diet is better in terms of GHGE than meat based diet? There are also health concerns that come along with cutting meat entirely from ones diet. Would you say it's easy to hit all required protein's targets while eating only plant foods?

siempreb(10000) 6 days ago [-]

> just stop eating meat?

For most people that's simply not an option. But, if all people were eating just a little less meat, that would be an incredible improvement already.

I believe we better fight down the meat problem with small steps, especially by creating more delicious plant based food for the mainstream, creating an alternative that is really hard to resist because it tastes so good.

Avamander(10000) 6 days ago [-]

> We're getting to the point where we're going to have to sacrifice some 'comforts' IMO.

Why can't those comforts we 'sacrifice' be cruises, gasoline/diesel cars and all advertising? My point being that saying 'we're going to have to make some sacrifices' and pointing at someone else never works unless the person pointing has a lot of power to enforce it.

tunesmith(4201) 6 days ago [-]

I saw some math once... not sure how correct it is, but it was something like given the co2-equivalent emissions of methane, even if all cows simply disappeared tomorrow, it would still only take care of about 5% of our global GHG emissions. So if that's true it wouldn't really be a gamechanger.

Still worth doing though (the seaweed, not the mass slaughtering)! And if everyone plants 350 trees then maybe we'll get somewhere.

jerrysievert(4007) 6 days ago [-]

> it would still only take care of about 5% of our global GHG emissions

have you never been 'nickel and dimed'?

FeepingCreature(10000) 6 days ago [-]

5% of a global problem is huge.

darkerside(10000) 6 days ago [-]

Cows may account for a small amount of total emissions, but they are responsible for almost half the methane released on the planet. Methane is a potent but relatively shortlived GHG. So, removing all cows would be a sizable but short term fix.

Toine(10000) 6 days ago [-]

I'm not sure your number is correct. It's more like 20%.

esotericn(4203) 6 days ago [-]

Some variant of this same comment is on every single thread about this here.

We need to tackle all of the 'only 5%' issues.

HillRat(3762) 6 days ago [-]

5% doesn't solve the problem, but it makes it more tractable. Enough 5% (or 4%, 3%, 2%...) solutions that integrate well into the existing economy, and you get GHG emissions into a range where the complex, expensive, and disruptive (in the traditional sense of the term) game-changing technologies become reasonable. At some point we will end up diverting a sizable percentage of global productivity to either scrubbing CO2 or dealing with the effects of our failure to prevent or capture source emissions, so those small reductions, implemented as quickly as possible, can add up to significant downline savings.

triceratops(10000) 6 days ago [-]

You just need 5 of these 5% solutions to cut the problem by a quarter. Many mickles make a muckle, as George Washington was fond of saying.

It's also good because a wide variety of experts from different fields can attack the problem from different angles. We won't be talent-constrained.

dclowd9901(4155) 6 days ago [-]

Did that include the carbon used to grow their feed, harvest their water and maintain their lots, as well as all the carbon used by the people to service them?

bacon_waffle(4151) 6 days ago [-]

The EPA says [1] 9% of the US's GHG emissions are from agriculture, so there's an upper limit. However, the more relevant consideration is how much more GHG comes from ruminants compared to other protein sources [2].

[1] https://www.epa.gov/ghgemissions/sources-greenhouse-gas-emis...

[2] https://www.businessinsider.com.au/the-top-10-foods-with-the...





Historical Discussions: WeWTF (August 17, 2019: 824 points)

(826) WeWTF

826 points 5 days ago by braythwayt in 995th position

www.profgalloway.com | Estimated reading time – 9 minutes | comments | anchor

Really? Really?

I've started nine firms and I'm, generously, 3-4-2 (win-lose-tie). In retrospect, and I think about this a lot, the only reliable forward-looking indicator of our firm's success or failure was ... timing. Specifically, the part of the economic cycle at founding. The firms we started in recessions had an easier time finding talent, controlling costs, and getting immediate feedback about if this thing worked as clients/consumers held their purse strings closed. Then, armed with a battle-tested value proposition, as the recession ended, we enjoyed the afterburner of confidence to spend more and try new things. #disco.

In frothy markets, it's easy to enter into a consensual hallucination, with investors and markets, that you're creating value. And it's easy to wallpaper over the shortcomings of the business with a bull market's halcyon: cheap capital. WeWork has brought new meaning to the word wallpaper. This is more reminiscent of the cheap marbled panelling you'd find in Mike Brady's home office — panelling whose mucilaginous coating will dissipate at the first whiff of a recession, revealing a family of raccoons or the mummified corpses of drug mules.

The features of seventies sitcom panelling:

Cult

WeWork's prospectus has a dedication (no joke): 'We dedicate this to the power of We — greater than any one of us, but inside each of us.' Pretty sure Jim Jones had t-shirts printed up with this inspiring missive. Speaking of idolatry, 'Adam' (as in Neumann) is mentioned 169 times, vs. an average of 25 mentions for founder/CEOs in other unicorn prospectuses. Uber's CEO, Dara Khosrowshahi, is mentioned 29 times in their prospectus. Granted, 'Adam' is super dreamy, in sort of an Argentinian polo player way (he's Israeli). But he's not 6x dreamier than Dara, who has a whole "Omar Sharif, if he went to Brown" thing going on. But I digress. We's mission is 'to elevate the world's consciousness.' Maybe, but it's clear the mission of the prospectus is to dampen our consciousness ahead of the sh*tshow that is 'The Story of Us: We.'

Nomenclature

Find the hottest sector, and if you don't have the insight, IP, genius, capital, code, skills, human capital, or a clue, then just borrow the words. SAAS firms trade at a multiple of revenues (yay), vs. real estate firms, which trade at a multiple of EBITDA (boo). So, We isn't a real estate firm renting desks, it's a Space as a Service (SAAS) firm. I know, use the word 'technology' over and over, despite having little R&D and computers and stuff, and voilà ... we're Salesforce.

Today I froze water and used this technology to reconfigure the environment encapsulating my Zacapa and Coke. So, I'm Bill Gates. Better yet, today I began calling my wife Gisele, which I'm pretty sure means I'm the starting QB for the Pats.

At WeWTF, you're not a guest, but a member. Member has a more "recurring revenue" sound to it. So, I plan to be a member tomorrow night at the Marriott in Boston, where I will then get membership to the TD Center so I can watch a 21-year-old Canadian (Shawn Mendes) with my 8-year-old son — also a member of the Marriott and TD Center, for tomorrow at least.

Invented Metrics

GAAP accounting standards got you down? No problema at WeWTF. We has begun reporting 'Community-based EBITDA,' profitability before the BITDA, but is also taking out expenses, including real-estate, that comprise the bulk of cost required to deliver the service. A more honest description of the metric would be 'EBEE, Earnings Before Everything Else.' As someone who follows stocks and goes on TV to pretend I have any idea which direction a given stock is going, I'd like to suggest a few metrics to provide insight into We:

  • EBG, Earnings Before Gluten
  • EBBG, Earnings Before the Big Dawg (tennis balls, pig's ears, etc.)
  • EBEPW, Earnings Before Equal Pay for Women

Red Flags

My goddaughter informed me she's dating a club promoter, a red flag. Occasionally, red flags marry each other, the Biebs and Hailey Baldwin — what could go wrong? So now, imagine red flags the dimensions of Kansas. Buckle up:

— Adam Neumann has sold $700 million in stock. As a founder, I've sold shares into a secondary offering to get some liquidity and diversify holdings. Ok, I get it. But 3/4 of a billion dollars? This is 700 million red flags that spell words on the field of a football field at halftime: 'Get me the hell out of this stock, but YOU should buy some.'

— Gross margins are a pretty decent proxy for how good or bad a business is. And this is a sh**ty business:

— Adam has several family members working in the business who make "less than $200,000.'

— The ownership structure chart is similar to a hieroglyphic on a cave wall about the survival of the species: Harvest the crops when the sun is high in the horizon, do not venture over the hills, hostile tribes live there, and ... don't buy this stock. The corporate governance structure of WeWTF makes Chinese firms look American, pre–big tech.

— The related party section of this prospectus reads like the Trump administration. Adam owns 10 buildings, several that he leased to WeWTF at a handsome profit. Adam also owned the rights to the 'We' trademark, which the firm decided they must own and paid the founder/CEO $5.9 million for the rights. The rights to a name nearly identical to the name of the firm where he's the founder/CEO and largest shareholder.

YOU. CAN'T. MAKE. THIS. SH*T. UP.

— Mismatched durations. The founder of Kohlberg Capital, Jim Kohlberg (total gangster), taught me investment firms go out of business because of 'mismatched durations.' It's about raising money short (customers who can stop buying your product service soon/tomorrow) and investing money long (10-year leases). WeWTF is an especially risky business going into a recession, when the ability to variabilize costs is limited, but revenue decline is unlimited. WeWTF has $47 billion in long-term obligations (leases) and will do $3 billion in revenue this year. What could go wrong?

There are other businesses like this (real estate, Hertz), and they are good businesses. Businesses that trade at, I don't know, 0.5 to 2x revenues. However, WeWTF is claiming it's not in this neighborhood, or even the same planet. So, let's talk valuation.

Insane. Seriously loco. Ok, let's assume WeWTF is onto something, better than peer IWG or Hertz. But is this firm, trading at 26x revenues, superior to Amazon, which trades at 4x revenues? There appears to be no scale effects, as losses have kept pace with revenue growth. There is little pricing power, as they are still a mole on the elephant of commercial real estate. There is no defensible IP, no technology, no regulatory moats, no network effects, and no flywheel effect (the ancillary businesses are stupid, just stupid).

The last round $47 billion 'valuation' is an illusion. SoftBank invested at this valuation with a 'pref,' meaning their money is the first money out, limiting the downside. The suckers, idiots, CNBC viewers, great Americans, and people trying to feel young again who buy on the first trade — or after — don't have this downside protection. Similar to the DJIA, last-round private valuations are harmful metrics that create the illusion of prosperity. The bankers (JPM and Goldman) stand to register $122 million in fees flinging feces at retail investors visiting the unicorn zoo. Any equity analyst who endorses this stock above a $10 billion valuation is lying, stupid, or both.

Adam's wife is Gwyneth Paltrow's cousin, meaning Adam is two degrees removed from Goop, an assault on humanity.

Ms. Neumann created controversy when she went on CNBC and said: "A big part of being a woman is to help men [like Adam] manifest their calling in life."

Ok, fine ... whatever works for you and Adam. But it's not retail investors' role to help Adam realize his calling — he should feel pretty manifested with $700 million. The panelling is compelling and cool, but it's beginning to curl and the substance behind the wood veneer stinks. I mean, stinks.

Life is so rich,

P.S. I've started doing short video hot takes on Twitter:

To see past or upcoming ones, search "from:@profgalloway #hottake" on Twitter.

And we're back on YouTube:

Subscribe, bitches!




All Comments: [-] | anchor

dlgeek(2822) 5 days ago [-]

> Adam also owned the rights to the 'We' trademark, which the firm decided they must own and paid the founder/CEO $5.9 million for the rights. The rights to a name nearly identical to the name of the firm where he's the founder/CEO and largest shareholder.

Really? How is this sort of shenanigan allowed?

n-exploit(3958) 5 days ago [-]

The Tragedy of the Commons 2.0TM

marcinzm(10000) 5 days ago [-]

Adam controls the majority of voting right to the company and the board (made up of VCs) doesn't want a messy legal fight since they stand to lose money if We tanks. So a moderate amount of graft is tolerated since the alternative is less desirable.

JumpCrisscross(45) 5 days ago [-]

> How is this sort of shenanigan allowed?

Investors gave Adam virtually unchecked power over their capital.

navigatesol(10000) 5 days ago [-]

This guy has become a multi-hundred-millionaire by renting out office space under a business model that appears to make no financial sense. Because 'tech company'. It's utterly bonkers.

juliusmusseau(3640) 4 days ago [-]

This is a classic due-diligence item during early funding rounds - up there with 'does the company actually own its domain name or do one of the founders own it'?

I suspect either due-diligence caught it and decided to live with it after some negotiation, or they missed it - oops!

DannyBee(3266) 4 days ago [-]

It usually is not. But delaware

ggg3(10000) 5 days ago [-]

lol. let alone trademark 'we'. guess I should trademark 'you' and call google about youtube.

completelylegit(10000) 4 days ago [-]

I get that the author doesn't like WeWork, which is fine. But the throw-away comment that there is some tertiary connection to Goop seems unnecessarily personal.

Goop is a bunch of nonsense but I'm not sure why its relevant to why WeWork is nonsense.

digianarchist(3978) 3 days ago [-]

I took that as a poor attempt at humor. I don't think the author actually implied a connection.

panabee(3035) 4 days ago [-]

bull hypothesis: the office space industry is very large but fragmented. wework becomes the central hub in a worldwide network of tenants and landlords.

bear hypothesis: wework is an overhyped-regus with a frail capital structure, no scale or network effects, and dangerous liabilities. it will likely collapse during a recession.

questions for investors:

1. how much flexibility/escapability does wework have regarding leases and liabilities? in particular, how much control do they have over cash flows to avoid disaster during a recession?

2. how much does it cost and how long does it take for enterprises like cisco and ibm to open new offices: (1) with wework; (2) with regus or another wework competitor; and (3) without any outside assistance? how about for companies with 100-1000 employees?

3. what is the current breakdown, lifetime value, and retention rate for wework members among solos, startups (2-99 employees), mid-sized companies (100-1000 employees), and enterprises (1000+ employees)?

4. cashing out $700mm seems like a red flag if you believe the company is worth much more. what percentage of equity does this represent for the founder? is cashing this much out normal for large IPOs? by comparison, how much did zuckerberg, hastings, and bezos cash out at IPO?

question for others: what are the first questions you would research on wework?

thadjo(4201) 4 days ago [-]

A lot hinges on what would actually happen to WeWork during a recession. I think there's some possibility that when a recession hits, many companies may find WeWork (and coworking generally) more attractive as they downsize. I'm also curious about whether WeWork has leverage to renegotiate leases, since commercial real estate (which is already not doing well) will weaken in a recession.

panabee(3035) about 2 hours ago [-]

5. what are the unit economics for the average wework location, where revenue is projected at 10%/25%/50%/80% occupancy?

mola(10000) 5 days ago [-]

I don't understand why all analysis of the craziness that is The We Company is filled with opaque language and colorful mind bending terms. The company is a fraud, it should be explained with facts, and clear simple language. Everything else is just muddying the water.

avip(4065) 4 days ago [-]

It seems irresponsible to use the term fraud. You need strong evidence to accuse someone in a criminal offence.

Fraud is Theranos. We seems more of a BetterPlace - delusional, charismatic founder + too much money - sound business model.

lancesells(10000) 4 days ago [-]

I'm not sure there's fraud here. It's more akin to voting for Trump to 'Make America Great'. It's purely a pr / sales-pitch and if people decide to invest they may lose their money or they may not. If I believe the S-1 it pretty clearly lays out the shadiness and ridiculousness of the company without being fraudulent.

baby(2323) 4 days ago [-]

I'm genuinely curious, mind you I haven't read much about Wework.

So, I can understand how a company like Theranos is fraud, because they are lying and have no product.

But, I've used wework successfully, I know a lot of people using wework successfully. They have a product that works and that people pay for. How can such a company be a fraud?

endorphone(3000) 4 days ago [-]

A lot of the behaviors of the founder seem outrageously fishy, however they are legitimately filling a rapidly growing need. At this very moment I have the WeWork Toronto locations page open as I contemplate biting the bullet on a private office there just to get out of the home office occasionally.

logicallee(4119) 5 days ago [-]

>it should be explained with facts, and clear simple language.

since it looks like you've spent some time on this, would you care to take a shot?

gumby(2217) 4 days ago [-]

Because it's so obviously a fraud that to say so doesn't add anything to the conversation.

On the other hand the fraud is wrapped in the most delicious of absurd flfftery that it's a pleasure to try to ape it.

simplecomplex(4190) 5 days ago [-]

How is it fraud? Investors make idiotic investments. Nobody is being forced to buy shares of WeWork, and all the financials are there. It's not like WeWork is lying about financials.

paultopia(4212) 4 days ago [-]

I know almost nothing about finance, and the only things I had to look up in the linked post were 'EBITDA,' 'DJIA' (which turns out to just be an acronym for the Dow Jones) and 'flywheel effect.' Seems pretty plain to me.

JumpCrisscross(45) 4 days ago [-]

> The company is a fraud

This is extreme. Adam's self-dealing is abundantly disclosed. There is no evidence he is acting in bad faith. (Versus being deluded himself.)

Lots of businesses leverage paper-thin margins. Lots of businesses borrow short and lend long. Lots of businesses, particularly real estate businesses, feature self-dealing and family control galore. American corporate law goes out of its way to avoid criminalising stupidity.

There is a chance WeWork attains enterprise lock-in sufficient to let it weather a storm. There is a chance it expands cross-selling to bring its books into the black. These chances are slim. But they're clearly disclosed.

mark_l_watson(2717) 4 days ago [-]

Well, I live in the mountains in Central Arizona, so no WeWork offices near me (Phoenix and Los Vegas are the nearest). I would be tempted, even though I am mostly retired, to use a shared office space. I have a great home office but I still like to go to the library, or to a coffee shop, to write or work.

It just seems like WeWork should charge a little more to nail down profitability.

insulanus(10000) 4 days ago [-]

Oh, there's nothing wrong with shared office spaces. It just seems unreasonable to thing they are going to multiple investor money 100x.

It seems like the valuation should be part commodity (the cost of the services provided), and part location (proportional to your local office real estate market). And maybe your size compared to your next largest competitor.

panopticon(10000) 4 days ago [-]

For what it's worth, there are non-WeWork coworking spaces all over. I'm not sure if you're near Flagstaff, but there are several in the area.

I think this business model will continue to survive regardless of WeWork's success or failure here.

methodover(4002) 4 days ago [-]

I think I might be the only person on HN optimistic about WeWork.

Remote work is on the rise here in the US. Startups and entrepreneurship continue to grow. Finding traditional office space sucks and is a massive waste of money, and probably always will. Co-working spaces make sense, and it's baffling to me that it wasn't the default way to work in 2010, when I started programming, let alone 2019.

I feel like if somehow fast food hadn't been invented til now, and MacDonalds just got started, we'd be reading pessimistic threads about how it's a real estate business masquerading as a food business and is destined for collapse.

But like, it actually does work, as a business, to provide some necessary service (office space, food) at scale with a big central marketing apparatus. The fundamentals here seem sensible to me.

philwelch(3822) 4 days ago [-]

If McDonalds lost money on their restaurants and their executives profited from multiple conflicts of interest, McDonalds wouldn't have become a successful business.

pat87(10000) 4 days ago [-]

The professor that wrote the analysis distinctly says WeWork as a business has value (worth maybe $2b). The issue is with the valuation of the company that is 26 times the revenue, and priced as a tech company even though it has no tech. It's a real-estate company (these trade at x3 revenue).

arbuge(2898) 4 days ago [-]

WeWork != Coworking

Coworking will grow and prosper. WeWork, I don't know. I don't know how long any company can survive the egregious investor-fleecing WeWork's founder seems to wantonly engage in. Perhaps if a clear stop is put to that, and if every other part of the business plan goes flawlessly, then it has a future.

But coworking is about alot more than WeWork, and the industry will do fine whatever WeWork's shenanigans imply for the latter's future. There are currently around 20,000+ coworking spaces worldwide, of which less than 5% are WeWork's:

http://www.deskmag.com/en/2019-state-of-coworking-spaces-2-m...

https://www.wework.com/locations

beager(10000) 4 days ago [-]

I think most people here think the value of WeWork is definitely not $0, but believe it's well well well south of their last private valuation, and certainly well below what they'll try to dump it on the market at.

schmrz(10000) 4 days ago [-]

Only if by fundamentals you mean the fundamental idea behind coworking spaces. Sure, that makes sense. But the WeWork company fundamentals are out of whack. Nobody is arguing that coworking spaces are a bad idea. WeWork is a company that turns 2$ into 1$.

vmurthy(4167) 5 days ago [-]

The most interesting part of this article is the term 'Community Adjusted EBITDA'. It's not a term invented by the Prof but something used by WeWork. I remember Warren Buffett/Charlie Munger calling EBIDTA as bulls* earnings [1] but community adjusted EBITDA just smells like someone in Enron got hired by the We company recently. It almost feels like We IPO is the bellwether for an upcoming crash :-(

[1] https://www.forbes.com/sites/brentbeshore/2014/11/13/ebitda-...

dvtrn(10000) 5 days ago [-]

Twice this week I've heard of EBITDA, the last conversation while taking an outdoor break with a coworker bemoaning some head scratching strategy choices from management where I work.

Thanks for the explainer link.

csours(3989) 4 days ago [-]

> 'More than its cash-burning ways, WeWork's IPO will test investor tolerance for made-up accounting metrics. You might recall "Community Adjusted EBITDA," the gauge WeWork devised to measure net income before not only interest, taxes, depreciation, and amortization, but also "building- and community-level operating expenses," a category that includes rent and tenancy expenses, utilities, internet, the salaries of building staff, and the cost of building amenities (which WeWork has described as "our largest category of expenses").' [0]

Hahaha. I think its pretty fair to say that this is not a Generally Accepted Accounting Practice measurement. (non-GAAP)

I've only visited a We Work location once, but it really felt like a VC funded startup - particularly the amenities. Without the amenities, it would have been just another depressing office.

0 - https://qz.com/1685919/wework-ipo-community-adjusted-ebitda-...

blunte(4141) 4 days ago [-]

The current system was ripe for a fantastical display of absurdity, and maybe this is it. It's startingly familiar given the current US political situation.

All this reminds me of Blackadder 2, Ink and Incapability. https://en.wikipedia.org/wiki/Ink_and_Incapability

Blackadder finds himself in a situation that quickly becomes increasingly unbelievable. Finally he recognizes he is in a dream, and he wakes up in disappointment. I'm still waiting to wake up.

hyperdunc(10000) 4 days ago [-]

Compared to the relative stability of the 1990s and early 2000s, the current political and economic situation does seem somewhat absurd.

But history is littered with turbulent times, and compared to those times the present is a pleasant dream. Whether it turns nightmarish remains to be seen.

csours(3989) 4 days ago [-]

If you have Prime, here you go: https://www.amazon.com/gp/product/B07CJZLF3S?camp=1789&creat...

It's also on Hulu

rolltiide(10000) 5 days ago [-]

Has anyone seen good coverage of WeWork as an investment?

All my favorite finance meme accounts are tearing this to shreds too.

The only people I've heard be excited are the WeWork employees, and you can practically never get the employees to have a counter view on this even if they don't get stock. I dont mean in like "I dont want to rock the boat" way, but more in an common ignorance of personal finance let alone investing way. They also arent really the investing class, so far they dont really understand the memes aside from the comments like "is this company run by Congress?" Not WeWork specific they just arent investors or financial professionals.

What do the "members" think? Is there any "yes I cant wait to align my interests with my favorite coworking space by investing in their future growth" sentiment? Im just not around those people so I dont know

WSJ and Bloomberg have been ripping on this for over a year, and now with the S-1 out Hackernews and finmeme accounts are ripping it too

Think they'll retract the S-1 and stay private?

skewart(3803) 4 days ago [-]

I work out of a WeWork location sometimes. I don't like the product and only work out of WeWorks because I need to for my job. I would never choose to be a 'member' on my own.

Here's why I'm not a fan: The interiors are cheesey, badly designed, and generally seem poorly built (things are janky and break way too easily). It's often hard to find a good place to work in the common areas - pop music is blasting in the main common areas and the few phone booths and quiet corners are often full (people sometimes just work in phone booths, which is annoying). The fact that you have to pay to use conference rooms (at least with my company's contract) and you have to pay for granola bars and other snacks feels kinda nickel-and-dimey. (There is free coffee, and free beer but during the workday I'm not usually looking to drink a pint.) Also there's a theme of forced happiness everywhere (the mugs all say 'do what you love' or 'always half full'), which feels at best like vapid corporate fluff and at worst kinda cult-like, but either way it's mildly off-putting. Finally, they often invite salespeople from different companies to set up tables in the common area to sell/advertise random stuff, which isn't that big a deal but monetizing their tenants' attention during their work day kinda seems at odds with 'building a community' and 'elevating consciousness' and all.

I like the concept of a global coworking space network. I'm just not a fan of WeWork's implementation.

My bet is that in ten years WeWork will end up a bit like Groupon is today - still going, but far from the world-changing force they were once hyped to be. Of course, a lot of weird stuff can happen and maybe they'll end up dominating the worlwide office market, or maybe they'll flame out spectacularly in a couple of years.

streetcat1(10000) 4 days ago [-]

if all the news are bearish, it would probably pop up.

perlgeek(2647) 4 days ago [-]

> Think they'll retract the S-1 and stay private?

And get their next cash infusion from where, exactly? If they manage to find even more private investors, that might be an option, but would probably hurt their image more than the negative press around the S-1.

Or try to become profitable really quickly?

40acres(3633) 5 days ago [-]

Check out the Odd Lots podcast episode: The Bullish Case for WeWork. Personally, I don't see it -- but capital is so cheap and WW brand so strong that they may be able to pull a rabbit out in a downturn.

projektfu(4189) 5 days ago [-]

The problem they have is that you can't save this business by clapping louder. Either it starts generating profits or it'll go out of business. I am just personally surprised that Adam Neumann isn't getting the kind of negativity Elizabeth Holmes got. Perhaps after it goes down, who knows, but his activity has been sufficiently shady that it should be drawing more fire.

goatinaboat(10000) 5 days ago [-]

Think they'll retract the S-1 and stay private?

It's not clear what that would mean for SoftBank but it can't be good. All VC funds are looking for an exit, Vision Fund is no different.

adrianN(3580) 5 days ago [-]

I work in a Wework office and wouldn't touch Wework as an investment with a ten mile pole. It's just an office. A bit more expensive than other coworking spaces. There literally nothing special about it, except that it never turned a profit.

I like that they at least pretend to care about the environment. I'd like it better if they actually did things to lower the energy use of their buildings.

philwelch(3822) 5 days ago [-]

It's likely that the company itself is weirdly cult-like and actively manages out any employees who don't drink the kool-aid. The vegan thing is a small hint to this.

JSavageReal(10000) 4 days ago [-]

Loved this article. Entertaining and informative (as someone not very informed on WeWork).

> Ms. Neumann created controversy when she went on CNBC and said: "A big part of being a woman is to help men [like Adam] manifest their calling in life."

In what kind of twisted society does a statement like that create 'controversy'? I'm a man, and I've always felt that in my relationship it is my duty to bring out the best of my woman and enable her to reach her full potential. Are my fellow men outraged as well? Only in America does an innocuous statement like that generate controversy (unless of course this is just a couple random tweets that the media is trying to turn into a big controversy to generate ad revenue).

rossdavidh(4106) 4 days ago [-]

Of all the controversies regarding WeWork, this is one of the smaller ones, really.

wallflower(36) 5 days ago [-]

Can anyone knowledgeable talk about what happens in a severe business downturn with We? They have most likely setup separate LLCs for each building's 10-15 yr lease. Do the LLCs isolate We enough so that LLCs can declare bankruptcy and the building owners or banks or creditors don't go after the mothership We?

JumpCrisscross(45) 4 days ago [-]

> Can anyone knowledgeable talk about what happens in a severe business downturn with We?

No. This is a known unknown.

On one hand, contracts get cancelled and WeWork defaults. On the other hand, WeWork's aggregated buying power lets it renegotiate leases, passing the pain to landlords.

eaenki(4199) 4 days ago [-]

Just short it.

joelx(3739) 4 days ago [-]

I thought this would be a good idea on that super hot hamburger chain a few years ago. I then learned that short interest changes based on short demand for that stock. I was putting 120% interest per year to short the stock. Be extremely careful with shorting, Warren Buffett himself says he won't touch it.

DoreenMichele(206) 4 days ago [-]

I really enjoyed the acerbic and colorful style of this article, but wouldn't have enjoyed it nearly so much if it were all style and no substance. He's clearly knowledgeable, though I know the style of writing isn't the usual fare for HN.

He's correct that really good businesses tend to start during recessions. Starting a business during a Roaring Twenties style economy when you could literally support yourself with a ridiculous idea like selling pet rocks will not test your business model and help you find its weaknesses. A la some African saying to the effect that 'Calm seas don't produce skilled sailors.'

I learned a lot and I am a bit envious of his acerbic wit. I feel like that's more acceptable from a man, but sarcastic contempt was pretty much my default as a teen and I sometimes think I've become a bit too PC over the years.

tfha(10000) 4 days ago [-]

We've all become too PC and the stains of that culture are bursting out the sides in unexpected and colorful ways. It's oppressive to be PC and we haven't acknowledged that side of it well enough. (Oppressive to the person constantly censoring themselves in the name of PC)

pdonis(3957) 4 days ago [-]

> I really enjoyed the acerbic and colorful style of this article

I did too, and have started looking at some of his other recent posts. I think they will well repay the effort. For example, this from the one on 'Mueller and Night Invasions':

'As a kid, I would digest my stomach waiting in the living room, after school, for my mom to get home so I'd tell her I had lost another $33 jacket. My 8-year-old spent $44 for an in-app Princess Celestia Pony and, when confronted, beamed with pride, as spending money is a new skill he's mastered, similar to math or (not) feeding the dog. It took just one generation for spending money to evolve from a crime to a competence.'

JumpCrisscross(45) 4 days ago [-]

He had a terrific YouTube channel that's worth watching through.

https://m.youtube.com/user/l2thinktank

simonebrunozzi(492) 4 days ago [-]

> I've become a bit too PC over the years.

What's 'PC' ? Did you mean PC as in 'PC vs Mac' boring/conformist vs cool?

buboard(3474) 4 days ago [-]

he was more caustic than acerbic

TAForObvReasons(3090) 5 days ago [-]

For a contrasting view, consider this tweet from @patio11 https://twitter.com/patio11/status/1161796809741627392

> In 2025, every Fortune 500 company will have 10k+ remote workers, and every purchasing department will approve a reimbursement for WeWork with no questions asked.

The actual bull case for WeWork is similar to Uber: there may be a massive trend, and WeWork is positioning itself to be the winner. With Uber it's the push to self-driving cars, and with WeWork it's the corporate move away from massive campuses and office buildings. In this new landscape, WeWork has extremely strong branding, experience, and a valuation that suggests it is the biggest player in the space.

mstade(4189) 5 days ago [-]

This line of reasoning makes sense to me. Anecdata: most of the top tier banks I've worked with or at now have 3-4 people per desk, with more and more people working almost 100% remotely. When I started working with banks about eight years ago, I was usually laughed out the room when suggesting we should allow remote working. I became a kind of pilot case for 100% remote working about four years ago, and nowadays people in the same teams can go months without meeting in person because if they aren't remote all the time they come in to the office on different days.

goatinaboat(10000) 5 days ago [-]

In 2025, every Fortune 500 company will have 10k+ remote workers

But what does that mean, exactly? I mean, imagine I'm an employee of BigBank in BigCity. My options are work from their office in the centre of town, work from my house wearing my pyjamas or... get dressed for work and commute into the centre of town, and sit in a WeWork instead and then charge it back on expenses because... reasons? I have better coffee and better wifi at my house...

philwelch(3822) 4 days ago [-]

The bull case is that there is a vaguely WeWork-shaped market opportunity, and WeWork exists. The bear case focuses on the flaws in WeWork itself. Is WeWork going to sort themselves out before someone else overcomes their market incumbency?

phonon(3430) 5 days ago [-]

All that may be true--but any F500 will (or already has) strike a heavily discounted deal with WeWork. And WeWork already has small gross margins. And WeWork is still shouldering the risk of matching heavy long terms obligations with short term revenues. And F500 companies have already been doing exactly that for decades with Regus.

dragonwriter(4199) 4 days ago [-]

If I believed patio11's 2025 projection, I'd also think by 2030 every Fortune 500 would be looking to optimize their remote-working costs by identifying their steady baseline needs in each major metro, acquiring leases to meet that dedicated capacity, and only allowing individual reimbursements for workers they don't have dedicated space available for (possibly on a day-to-day basis.)

But, I'm not convinced that his 2025—6 years from now—vision is even approximately correct.

Traster(10000) 4 days ago [-]

That's a really compelling argument for why WeWork will be a successful office space rental company. That would mean WeWork somehow loses 90% of its value and is still successful...

ohashi(2524) 4 days ago [-]

There is little network effects for office space vs taxi availability.

Second, Regus is larger (more revenue, more locations iirc). Regus is profitable.

That's just a direct coworking/office space globally company. WeWork isn't even a leader. It's just shiny bullshit.

addicted(10000) 5 days ago [-]

Real Estate is also a much bigger business than taxi cabs.

The big bet on them I suspect is that any upcoming recession will probably lead to a lot of companies wanting to reduce their office costs. They may want to switch to WeWork managing their office spaces much like tech companies want to switch to Amazon Microsoft or IBM managing their IT infrastructure.

MrBuddyCasino(2974) 5 days ago [-]

I'm afraid he is correct, BigCorps are the main tenants as far as I can see. Which is sad, because WeWork is as fake and psychopathic as its CEO.

crawftv(10000) 4 days ago [-]

It all really comes back to Trump in the end.

egdod(10000) 4 days ago [-]

Seriously. He just couldn't help himself from trying (and failing, frankly) to take a shot at Trump. In an article that has absolutely nothing to do with Trump.

The TDS is strong with this one.

SyneRyder(4145) 4 days ago [-]

I thought this was a weirdly off topic comment, until I read the full article. Towards the end it goes off on a wild tangent claiming Union Square Ventures are the reason progressives could lose the 2020 election to Trump. (?!)

A shame, because the first half of the article is great.

sergiotapia(768) 4 days ago [-]

They can't help themselves lol





Historical Discussions: OpenDrop: An Open Source AirDrop Implementation (August 19, 2019: 794 points)

(806) OpenDrop: An Open Source AirDrop Implementation

806 points 3 days ago by vsurabhi in 10000th position

github.com | Estimated reading time – 6 minutes | comments | anchor

OpenDrop: an Open Source AirDrop Implementation

OpenDrop is a command-line tool that allows sharing files between devices directly over Wi-Fi. Its unique feature is that it is protocol-compatible with Apple AirDrop which allows to share files with Apple devices running iOS and macOS. Currently (and probably also for the foreseeable future), OpenDrop only supports sending to Apple devices that are discoverable by everybody as the default contacts only mode requires Apple-signed certificates.

Disclaimer

OpenDrop is experimental software and is the result of reverse engineering efforts by the Open Wireless Link project. Therefore, it does not support all features of AirDrop or might be incompatible with future AirDrop versions. OpenDrop is not affiliated with or endorsed by Apple Inc. Use this code at your own risk.

Requirements

To achieve compatibility with Apple AirDrop, OpenDrop requires the target platform to support a specific Wi-Fi link layer. In addition, it requires Python >=3.6 as well as several libraries.

Apple Wireless Direct Link. As AirDrop exclusively runs over Apple Wireless Direct Link (AWDL), OpenDrop is only supported on macOS or on Linux systems running an open re-implementation of AWDL such as OWL.

Libraries. OpenDrop relies on current versions of OpenSSL and libarchive. macOS ships with rather old versions of the two, so you will need to install newer version, for example, via Homebrew. In any case, you will need to set the two environmental variables LIBARCHIVE and LIBCRYPTO accordingly. For example, use brew to install the libraries:

brew install libarchive [email protected]

Then set environmental variables:

export LIBARCHIVE=/usr/local/opt/libarchive/lib/libarchive.dylib
export LIBCRYPTO=/usr/local/opt/[email protected]/lib/libcrypto.dylib

Linux distributions should ship with more up-to-date versions, so this won't be necessary.

Installation

Installation of the Python package release is straight forward using pip3:

pip3 install opendrop

You can also install the current development version by first cloning this repository, and then installing it via pip3:

git clone https://github.com/seemoo-lab/opendrop.git
pip3 install ./opendrop

Usage

We briefly explain how to send and receive files using opendrop. To see all command line options, run opendrop -h.

Sending a File

Sending a file is typically a two-step procedure. You first discover devices in proximity using the find command. Stop the process once you have found the receiver.

$ opendrop find
Looking for receivers. Press enter to stop ...
Found  index 0  ID eccb2f2dcfe7  name John's iPhone
Found  index 1  ID e63138ac6ba8  name Jane's MacBook Pro

You can then send a file using

$ opendrop send -r 0 -f /path/to/some/file
Asking receiver to accept ...
Receiver accepted
Uploading file ...
Uploading has been successful

Instead of the index, you can also use ID or name. OpenDrop will try to interpret the input in the order (1) index, (2) ID, and (3) name and fail if no match was found.

Receiving Files

Receiving is much easier. Simply use the receive command. OpenDrop will accept all incoming files automatically and put received files in the current directory.

$ opendrop receive

Current Limitations/TODOs

OpenDrop is the result of a research project and, thus, has several limitations (non-exhaustive list below). I do not have the capacity to work on them myself but am happy to provide assistance if somebody else want to take them on.

  • Triggering macOS/iOS receivers via Bluetooth Low Energy. Apple devices start their AWDL interface and AirDrop server only after receiving a custom advertisement via Bluetooth LE (see USENIX paper for details). This means, that Apple AirDrop receivers may not be discovered even if they are discoverable by everyone.

  • Sender/Receiver authentication and connection state. Currently, there is no peer authentication as in Apple's AirDrop, in particular, (1) OpenDrop does not verify that the TLS certificate is signed by Apple's root and (2) that the Apple ID validation record is correct (see USENIX paper for details). In addition, OpenDrop automatically accepts any file that it receives due to a missing connection state.

  • Sending multiple files. Apple AirDrop supports sending multiple files at once, OpenDrop does not (would require adding more files to the archive, modify HTTP /Ask request, etc.).

Related Papers

  • Milan Stute, Sashank Narain, Alex Mariotto, Alexander Heinrich, David Kreitschmann, Guevara Noubir, and Matthias Hollick. A Billion Open Interfaces for Eve and Mallory: MitM, DoS, and Tracking Attacks on iOS and macOS Through Apple Wireless Direct Link. 28th USENIX Security Symposium (USENIX Security '19), August 14–16, 2019, Santa Clara, CA, USA. Link

Authors

  • Milan Stute (email, web)
  • Alexander Heinrich

License

OpenDrop is licensed under the GNU General Public License v3.0. We use a modified version of the python-zeroconf package (essentially adding rudimentary IPv6 and AWDL support) which is licensed under the GNU Lesser General Public License v2.1. Both licenses are found in the LICENSE file.




All Comments: [-] | anchor

ekwogefee(4200) 3 days ago [-]

Cross-platform local file transfer alternative with resumable file transfer:

https://feem.io

amq(3670) 3 days ago [-]

Looks great, I wonder why it doesn't have more traction.

nesadi(10000) 3 days ago [-]

This looks interesting. I'm a bit confused though. It seems like all the pricing plans are structured in a way that you're either only paying for iOS devices or for every kind of device except iOS ones. Does this mean transfers between iOS and non-iOS devices aren't supported? Or do I have to buy two plans in order to do it?

edit: Okay, on a closer look, it seems like you do need two licenses, because the iOS license can only be purchased through the iOS app. Just wish they'd provide a brief trial so I could test it.

edit2: Nevermind, apparently, if you install the apps, you can use them in a limited fashion before buying. It doesn't say anywhere what the limits are, but it says there are limits.

Fnoord(3923) 3 days ago [-]

Interestingly the article on the bottom links to a Usenix 2019 (held Aug 14 - 16) paper with the title 'A Billion Open Interfaces for Eve and Mallory: MitM, DoS, and Tracking Attacks on iOS and macOS Through Apple Wireless Direct Link'

Abstract:

'Apple Wireless Direct Link (AWDL) is a key protocol in Apple's ecosystem used by over one billion iOS and macOS devices for device-to-device communications. AWDL is a proprietary extension of the IEEE 802.11 (Wi-Fi) standard and integrates with Bluetooth Low Energy (BLE) for providing services such as Apple AirDrop. We conduct the first security and privacy analysis of AWDL and its integration with BLE. We uncover several security and privacy vulnerabilities ranging from design flaws to implementation bugs leading to a man-in-the-middle (MitM) attack enabling stealthy modification of files transmitted via AirDrop, denial-of-service (DoS) attacks preventing communication, privacy leaks that enable user identification and long-term tracking undermining MAC address randomization, and DoS attacks enabling targeted or simultaneous crashing of all neighboring devices. The flaws span across AirDrop's BLE discovery mechanism, AWDL synchronization, UI design, and Wi-Fi driver implementation. Our analysis is based on a combination of reverse engineering of protocols and code supported by analyzing patents. We provide proof-of-concept implementations and demonstrate that the attacks can be mounted using a low-cost ($20) micro:bit device and an off-the-shelf Wi-Fi card. We propose practical and effective countermeasures. While Apple was able to issue a fix for a DoS attack vulnerability after our responsible disclosure, the other security and privacy vulnerabilities require the redesign of some of their services.' [1]

I got nothing to add regarding OpenDrop other than that I love interoperability, and that I love it when FOSS enables this.

[1] https://www.usenix.org/conference/usenixsecurity19/presentat...

toxik(10000) 3 days ago [-]

I can't hep but think that these seemingly numerous security flaws are a product of proprietary software development. There is the old "many eyes" idea for software bugs, but even on a standards level. Did Apple not send out an RFC? Isn't this type of architectural level screw up exactly what you want to avoid with an RFC?

I'm glad Apple are taking the privacy issue to heart, but for every inch we've won in privacy, we lost an inch in openness and interoperability. Apple is perhaps one of the worst offenders when it comes to vendor lock-in.

I use almost only Apple products out of sheer laziness (and honestly inertia.) At least their war with Qualcomm and NVIDIA creates some competition in their respective markets...

atombender(2961) 3 days ago [-]

There's also this 2018 paper by the same authors: 'One Billion Apples' Secret Sauce: Recipe for the Apple Wireless Direct Link Ad hoc Protocol' [1]. Not sure which paper came first.

[1] https://arxiv.org/pdf/1808.03156.pdf

HNcantBtrustd(10000) 3 days ago [-]

FOSS may have downsides, for instance it's free, but doesn't have iTunes built in.

I consider both of these advantages, but some users don't. I consider the benefits of open source far exceeds the perceived benefits of Apple's Ecosystem.

Maybe as developers it's important to teach our Apple Peers the problems they perpetuate by paying for walled gardens.

andrewstuart(987) 3 days ago [-]

I'm guessing someone will take a $10 esp32 chip and put this code on it and just drop the esp32 in some hidden location and it just sending images to any open airdrop that passes by.

dvcrn(3644) 3 days ago [-]

I had it a couple times in trains that people tried airdropping me porn when I forgot to turn it off

snuxoll(10000) 3 days ago [-]

Bluetooth OBEX spam for the current decade.

perillamint(10000) 3 days ago [-]

ESP32 theoretically can provide all of the hardware requirements (WLAN monitor mode, BLE) but there is one missing part:

> Triggering macOS/iOS receivers via Bluetooth Low Energy. Apple devices start their AWDL interface and AirDrop server only after receiving a custom advertisement via Bluetooth LE (see USENIX paper for details). This means, that Apple AirDrop receivers may not be discovered even if they are discoverable by everyone.

If someone reverse engineer BLE advertisement, yes they can build such hardware.

smurfpandey(3924) 3 days ago [-]

An HTML5 alternative: https://github.com/cowbell/sharedrop

Uses WebRTC for file transfer.

theomega(4101) 3 days ago [-]

Just to make it more obvious: You don't have to self host it, there is a free (no registration) Version deployed at https://sharedrop.io

Works perfect between IOS, Android, Linux, MacOS and Windows.

epixcz(10000) 3 days ago [-]

Xiaomi, Oppo And Vivo just introduced their own version of AirDrop. https://www.forbes.com/sites/bensin/2019/08/19/xiaomi-oppo-a...

Darman(10000) 3 days ago [-]

Yeah typical of isheep, they create biased title for money and traffic.

Their pattern for non apple product 'XXX (other company) Introduces Own Version Of iPhone's AirDrop'.

And for apple their title will be like 'Apple Introduce Their Very Own Swipe Keyboard'.

dijit(3511) 3 days ago [-]

Seems like theirs is not cross-compatible with airdrop. But OpenDrop is.

bborud(3868) 3 days ago [-]

God, I wish people would stop using Python for these sorts of things.

It is an okay language, but after tracking down why it doesn't build and considering messing around in my system and making either installing older versions of libraries or messing around with symlinks I stopped and asked myself 'really? I want to spend my time fixing this?' and just deleted the entire clone of the git repos.

Python is a nice language and all, but it is not a language suitable for writing applications that you distribute. (I wish the Python core developer would devote some time to making Python less horrible for distributing applications, but after around 30 years, I don't think so).

xmichael999(10000) 3 days ago [-]

I agree, it really is to bad there isn't a better way of distributing Python. I've not had much of an issue with it running for console based pure linux stuff mind you, it tends to in my experience fall apart when you add gui elements and/or go outside of linux.

oefrha(4032) 3 days ago [-]

Gosh, I wish people stop complaining about the language choice whenever cool project X happens to be written in a pet peeve of theirs.

nitrogen(3952) 3 days ago [-]

Does Python have something analogous to RVM or Gemsets, or Bundler (Bundler can exclude incompatible dependencies at runtime)?

jtl999(10000) 3 days ago [-]

I do agree that's an issue, although I wouldn't put it as drastic as you did.

I was working on a Flask project many years ago and it didn't seem straightforward to 'vendor' dependencies with the `pip install` paradigm. Guess I was wrong.

https://medium.com/underdog-io-engineering/vendoring-python-...

There's a reason why virtualenv and similar exist. Another interesting point. Gentoo Linux (which uses Python in the Portage package manager) doesn't support installing modules as system (by default) to presumably avoid breaking the system package manager.

encoderer(3836) 3 days ago [-]

You're right and the other half of this is the hoops you jump through as a developer to target as many python releases as possible. It's insanity.

I built all of Cronitor in Python, still love it, but when we started building server agents it was not a hard decision to leave Python behind. (In this case, for Go)

rolltiide(10000) 3 days ago [-]

People use Anaconda to avoid this and get a pretty seamless experience. Lets you use multiple python configs on the same system and manages that for you.

I learned this when taking a computational finance class specifically to figure out why that and the quant/ai/ml community gravitated around python.

Python's advantages here came from about 12 years ago, when the syntax was friendly but other similar syntax friendly languages had a lot of overhead and other compromises. But these differentiators are mostly non-existent today, and Python is the only one with basically two different programming languages (py 2.x, py 3.x) operating under a single 'Python' brand.

djsumdog(1058) 3 days ago [-]

I honestly haven't run into this issue in a long time. `pip install --user` is one of your friends. Just using the official python:3 docker container is another. If you really want, you can even go back to virtualenvs.

npm was also really bad about nothing building or working a few years back. It's improved, and there are alternatives like yarn. Rust/Cargo has this issue as well (whenever I attempted to pick up some Rust; every example I found would break -- constant language changes were an issue; not sure if that's still the case).

Package management is a big problem in general, but we have solutions like the ones I've mentioned. This is a bad argument against not using Python. I honestly thing this type of application is fine in Python (you might need a privileged container if you go the docker route; wasn't sure how low level the Wi-Fi stuff it needs is).

What language do you recommend for this type of application and why?

atombender(2961) 3 days ago [-]

I don't use Python, but I've never had any issues with any distributed Python app. I have, however, had endless problems with C and C++ apps.

The C way of referring to header files and libraries on the host system invariably leads to situations where the app wants to use a specific version that your system doesn't have. And we're not necessarily talking about system libs, either. Apparently authors thought the only way to mitigate the problem was to invent Automake/Autoconf in order to sniff what your system is capable of. (The saner solution for non-system libs would be to 'vendor' your dependencies inside the app's source tree.)

Python has that pretty much solved with PIP. (Dependencies can still be a problem if a package uses the C way to link to things like Readline or OpenSSL or whatever.)

slovenlyrobot(10000) 3 days ago [-]

- Click link, blown away by impossible feat of reverse engineering proprietary system done by apparently a lone hero contributor

- Realize nothing I work on really has so much impact as this one lone hero is likely to have already achieved by releasing this code

- Open HN thread with a sense of wonderment

- Read top comment, engulfed by a wave of revulsion, remember why we can't have nice things

em3rgent0rdr(1549) 3 days ago [-]

This is great, especially considering that AirDrop is used for instance by Hong Kong protesters to bypass the great firewall [1].

[1] https://qz.com/1660460/hong-kong-protesters-use-airdrop-to-b...

est(2934) 3 days ago [-]

GFW is only implemented in the mainland China, not Hong Kong.

    China == (mainland + (Hongkong SAR + Macau SAR) + (... possibly other de jure claims))
To use Airdrop the main advantage is it can be used to spam everyone nearby.

If it can be used by protestors it also can be used by advertisers.

pashky(10000) 3 days ago [-]

And in UK it's used to send dickpics to strangers on the train... opendrop requires to set security level to everybody instead of contacts. If you forget to switch back to default you get instant reminder once you get onto public transport.

shireboy(4194) 2 days ago [-]

Let's talk for a minute about why all the Apple things aren't open. There is zero about iMessage or AirDrop that should be proprietary. The only reason I know of is vendor lock-in, and that stinks for users. It would be way more helpful to way more people if these features were ubiquitous, open, and standards-based like SMTP or IMAP is for email. We wouldn't except an Apple-only iMail, why do we accept iChat and iPhotoShare?

Terretta(1685) 2 days ago [-]

OK, I'll talk about why...

What does Apple sell? A commodity computing device running undifferentiated software? Or the experience of a holistic tool?

When someone distributes a Messenger by itself, you don't ask why it isn't open. You don't ask why the hardware device isn't open. Why should a vendor unbundle the two just to make you happy?

What if it doesn't really unbundle, what if the capabilities combine to offer most buyers something they value more than pieces parts?

lawnchair_larry(2777) 2 days ago [-]

I agree that they should be open, but there are some advantages to the current model besides vendor lock-in.

Requiring a device certificate to communicate puts a pretty high bar on the integrity of the system. Being closed also allows unrestricted innovation at a faster pace.

It's very difficult to add features once an open standard is out. Just look at the evolution of the major web browsers for example.

devtanna(10000) 3 days ago [-]

I wish we had something like AirDrop to work between android and iOS devices.

jbverschoor(3467) 3 days ago [-]

There it is

jeroenhd(4193) 3 days ago [-]

Knowing Apple, such an effort would likely be destroyed by an army of lawyers the moment you bring out an app that provides such features.

The Apple ecosystem is very closed and Apple will fight tooth and nail to keep it that way. Removing vendor lock-in would, after all, allow users to try switching to another brand without an enormous amount of hassle.

scq(10000) 3 days ago [-]

https://snapdrop.net works pretty well if you're on the same Wi-Fi network.

jumelles(2498) 3 days ago [-]

iMessage too. It's insane to me that this is still a problem.

cprecioso(4152) 3 days ago [-]

They found code for kind of that in the new Google Play Services bundles ( https://www.androidpolice.com/2019/06/30/google-fast-sharing... ). They theorize in the article that it uses the same technologies as Airdrop, so they might be compatible - though I doubt it.

lucb1e(2123) 2 days ago [-]

I use https://dro.pm a lot to share files or links. Since the links are super short (I just made https://dro.pm/a --- note that it'll expire in 12 hours), you can just give the link to someone over the phone, put them in a presentation, or just share files independent of any operating system.

I even use it even between my laptop and my phone fairly frequently, since the top suggestion on my phone keyboard in the browser is dro.pm and I just have to add a slash (long-press 'm') and a letter. It's quicker to use dropm than to open a chat with myself or send myself an email or something.

Of course this is just protected by https, and although it is source-available and the links are really gone after expiry (or when you edited it, the old contents are irretrievable), magic-wormhole is superior in that you don't have a trusted third party. For the cases where the other party doesn't have magic-wormhole installed, this might be helpful. This also alleviates the requirement of WebRTC for both parties to be online simultaneously.

newscracker(4185) 2 days ago [-]

Apple wouldn't allow a solution that's integrated within the OS, but if you want to share data between devices, AirDroid [1] is one such free service. It is available on multiple platforms, including Android, iOS, macOS, Windows and web. Use web.airdroid.com on devices/computers where you don't want to install the application.

[1]: https://www.airdroid.com/

mobilemidget(10000) 3 days ago [-]

Would it also be possible to share contacts and links to sites? Thinking of a new reception desk feature (lots of apple customers)

dsl(1980) 3 days ago [-]

I believe AirDrop defaults to 'Contacts Only' so you'd need to have every customer add your reception phone number.

badwolf(10000) 3 days ago [-]

You can absolutely do both with AirDrop (from ios/MacOS)

krtkush(4007) 3 days ago [-]

I haven't looked at the OP yet but as convenient as AirDrop is, I find its reliance on both BT and WiFi confusing. One needs both the devices to be connected to the same network to be able to able to drop stuff.

Few days back my home router broke down and I was unable to send URLs from my iPhone to Mac just because there was no common network.

I wish for AirDrop to be more like Pushbullet.

trollied(3576) 3 days ago [-]

It doesn't need a wifi network, it's direct device-to-device.

In fact, it can be much faster than copying using traditional file shares etc, as wireless routers can slow things down.

dannyw(3918) 3 days ago [-]

You just have to be within bluetooth distance; AirDrop doesn't require the same wifi network. It does require wifi to be enabled, because AirDrop creates its own wifi network side channel for the actual transferring of files.

rektide(4195) 3 days ago [-]

Balls in Android's court to deliver some kind of p2p connectivity that works beyond Android2android. Can't happen soon enough. Stop playing with yourself & start doing real computing, Android.

fulafel(3461) 3 days ago [-]

WebRTC has been around for a while.

micheljansen(3571) 3 days ago [-]

I wonder if it's more reliable than Apple's own implementation for MacOs. It used to be rock solid – and between iOS devices it still is – but between Macs I regularly have to switch both to 'Search for an older Mac' to make them see each other, with no explanation why.

shimms(3570) 3 days ago [-]

I was complaining about this with a coworker today. AirDrop 1 was flawless, since 2.0 it has been more often than not unreliable between Macs.

I'd love to know if anyone knows the technical details about what caused this regression?

DerDangDerDang(10000) 3 days ago [-]

I've found it very flaky on iOS - regularly try to transfer between my phone and my wife's sitting next to each other and it will randomly not see the other phone or take minutes to find it :(

When it works, it just works.

chapium(10000) 3 days ago [-]

I found that the macos firewall is to blame.

postcynical(10000) 3 days ago [-]

Could this technology be used to create a 'shadow' internet/network/messaging service where devices connect and communicate directly with each other. This way governments can't just block internet access or services during demonstrations.

willvarfar(476) 3 days ago [-]

A government - or even just a sufficiently motivated individual - can trivially jam all phone, wifi, bluetooth etc communication.

Or they could just 'listen in' to record the signatures of all detected phones, so they can work out and identify who is in the crowd etc.

Military systems use frequency hopping etc and would be a more difficult target. But doubtless motivated governments could hamper their use in crowds too. Here's a fun comment I just found when I googled that: https://www.quora.com/Is-noise-jamming-an-AESA-radar-possibl...

neilalexander(4063) 3 days ago [-]

For what it's worth, I have been able to get the Yggdrasil Network (https://yggdrasil-network.github.io) to peer over AWDL, allowing nearby Macs to mesh without even being connected to the same Wi-Fi network, or any network at all.

It's not perfect - there are trade offs, like how the wireless performance is reduced somewhat when AWDL is active due to channel hopping and how AWDL expects a single node to play the role of clock sync source. It's also not very well tested yet.

However, it works and in theory it allows an infrastructure-free IPv6 mesh network to be built ad-hoc.

ymolodtsov(4174) 3 days ago [-]

FireChat.

rlpb(3260) 3 days ago [-]

> this technology

XMPP over Zeroconf (https://xmpp.org/extensions/xep-0174.html) has already existed for many years and at least previously has been trivial to enable on Ubuntu (I haven't tried it recently).

mxuribe(4008) 3 days ago [-]

It is 2019, and it is quite surprising - and disappointing - that we STILL haven't universally solved the means to easily, securely, and (yes, I'll use this term again) universally share files. I wish we could share files in a peer-to-peer fashion securely without hindrance of mobile platform, nor blockage of network MiTM, etc. </sigh>

Fickry(10000) 3 days ago [-]

What do you expected from a company who even cant made a phone with simple bluetooth file sharing

dvfjsdhgfv(2539) 2 days ago [-]

Personally I've been using Nitroshare for years in my home network and it works like a charm (between Windows, Linux, and macOS). Android is also supported, but not iOS (I saw someone created an iOS app, but I doubt it works still).

cyphar(3723) 3 days ago [-]

Have you taken a look at magic-wormhole[1]? I've started using it recently and it's insanely easy to use.

It does have a centralised signalling server for key exchange between peers, but it does attempt to do peer-to-peer data transfer (only falling back to a TURN-style relay if both clients are behind NATs and aren't on the same local network). An explanation of the cryptography and design was given at PyCon 2016[2]. It also has built-in optional Tor support (though I'm not sure if it attempts to use an onion service for data transfer).

[1]: https://github.com/warner/magic-wormhole [2]: https://youtu.be/oFrTqQw0_3c

infectoid(10000) 2 days ago [-]

Isn't that what Bump was all about until Google killed it?

http://techland.time.com/2014/01/01/sadly-but-inevitably-goo...

skrowl(4046) 3 days ago [-]

https://files.google.com/ is a standard and it works on the vast majority of phone sold this year, by every manufacturer except Apple. Checks all 3 of peer-to-peer, secure, no MiTM network blockage. It's only 11MB.

Many popular file manager apps on android have peer-to-peer xfers as well, via WiFi direct, etc.

EDIT - As people are pointing out this isn't universal because it doesn't work on Apple devices or desktops / laptops, but it's as close as I can think of currently.

anderspitman(1741) 3 days ago [-]

file.pizza has mostly solved this for me. Not ideal UX, but it gets the job done.

pbhjpbhj(3981) 3 days ago [-]

Opera Unite did it (and that's about the fifth time this year I've mentioned Opera Unite in response to a comment bemoaning a lack in currently embedded internet systems/apps).

staticvoidmaine(10000) 3 days ago [-]

Rather than the google solution suggested by a sibling comment, I'd suggest a look at send.firefox.com.

mackrevinack(10000) 3 days ago [-]

the safe network is making good progress lately.you might be waiting another year or two though

andrewstuart(987) 3 days ago [-]

Is there any valid use cases for something like 'legitimate airdrop'?

Maybe things like hyper local advertisements.

azinman2(4081) 2 days ago [-]

I wouldn't consider that legitimate at all. Legitimate airdrop is its exact intended use case: p2p transfer between people who are both consenting into it.





Historical Discussions: Tech Interview Handbook (August 17, 2019: 742 points)

(750) Tech Interview Handbook

750 points 4 days ago by yangshun in 3153rd position

yangshun.github.io | | comments | anchor

'The Tech Interview Handbook played a crucial role in the success of my previous job search. The contents are carefully curated and well organized. It served as an excellent roadmap for my interview prep.

In addition to the thorough Data Structures and Algorithms section, the handbook also provides a lot of resources on other aspects of the application process that helped me see the tech interviews in a more holistic way. My favorite non-technical part was 'Questions To Ask'! I used quite a few insightful questions from there to challenge and impress my interviewers. The results were great!

With the help of Tech Interview Handbook, I was able to land offers from Google, Amazon, Uber and several other great companies. Really appreciate Yangshun and other contributors for putting out such quality content for the community. I'd wholeheartedly recommend this handbook to anyone!'




All Comments: [-] | anchor

jumelles(2498) 4 days ago [-]

> Get a proper email account with ideally your first name and last name, eg. '[email protected]' instead of '[email protected]'... Avoid emails like '[email protected]' or '[email protected][mycooldomain].com' -- because it is very prone to typo errors.

I don't think I've ever seen anyone explicitly recommend against using a custom domain for email.

sdan(3698) 4 days ago [-]

That's what I was thinking as well. If I gave you the following emails: [email protected] [email protected] [email protected] [email protected]

Which would look most professional?

yumraj(3550) 4 days ago [-]

Given the current state of tech interviews, they have more or less become like standardized tests, such as SAT, ACT, GMAT, GRE... with guides, cheat sheets and perhaps neighborhood coaching institutes on the horizon with instructors who have cleared tech interviews in FAANGs.

Are we going to see tech recruitment become more and more like college admissions where a top score in the interview is just one of the criteria and no longer sufficient to get a job?

Perhaps next is asking people to write essays regarding career, goals, why they want to work their, extra curricular etc..

risubramanian(10000) 4 days ago [-]

You joke, but I've seen several applications where I've been asked questions like 'What achievement are you proudest of?' and 'How would you contribute to the diversity of our team?'.

Also cover letters cover some of this.

Actually, I wouldn't mind a standardized test like the GRE, where a good score might actually keep your resume from getting thrown out immediately.

sthatipamala(2079) 4 days ago [-]

There's a YC company that's literally making a standardized test for programmers: https://cspa.io/

leovander(4202) 4 days ago [-]

I went to a Cal State, and I recall in the last few years Google has taken professors to their campus and pretty much preach Cracking the Code Interview. So it looks like schools are already making that their standard. I have yet to see what classes look like but I could see there being a required technical interview course you have to take in order to increase school's numbers on where their alumni work.

closeparen(4093) 4 days ago [-]

Bright high school students are vastly oversupplied compared to seats in elite colleges. Graduation rates are in the high 90s. There are many more than 5,000 kids who can handle the workload; which 5,000 you pick is arbitrary.

Engineering competence is not even slightly oversupplied compared to useful engineering work. Project failures, incompetent people, and systematically incompetent orgs are still very much alive at the most selective tech employers. There are real business needs to hire better engineers.

jackmodern(4162) 4 days ago [-]

The next step is back channel reference checks

_hardwaregeek(10000) 4 days ago [-]

Y'know, as someone who went to a very competitive public school, there were a particular brand of driven, studious kids who knew exactly how to paint along the lines, play the game, and get into the college of their choice. They understood the precise combo of grades, extracurriculars and essays to get into a 'good' school.

I definitely see some of those kids doing the same tactics to get into a 'good' company. The steps are slightly different but the philosophy is the same: grind Leetcode, participate in tech clubs and take bullshit jobs juuust long enough to get an internship, get said internship and parlay it into a return offer. Put your head down, play the game and reap the rewards.

namelosw(10000) 4 days ago [-]

I was against the algorithm tests or something more like exams before.

Until I interviewed a lot of people with my colleagues during these years. Interview processes are highly biased based on the knowledge of the interviewer's background and value, or even mood.

Some interviewers are too sloppy on interviewing, asking ill-defined questions, demanding answers they want, or just in a hurry and wanting to go back to work. I often feel bad and angry for interviewees - they spent time and patience preparing themselves carefully, then was treated very casually. It isn't fair at all.

I hate to say that, although standardized tests are bad, they are better than the most nowaday interviews.

mylons(10000) 3 days ago [-]

except there is no standardization at all.

Aperocky(10000) 3 days ago [-]

Judging from the people I work with (FAANG). I would say that this screening is successful, if someone truly awful got in, then there's still PIP. But those are relatively uncommon.

interviewthr(10000) 4 days ago [-]

How is this much different than a paid smaller version of LeetCode?

When I was prepping I got the most value out of LeetCode for solo prep, followed by mock interviews with sites like Pramp.com, Gianlo.co, and PracticeCodingInterview.com.

I don't know why tech companies don't just admit that this is all pretty much standardized at this point. Just build a standardized test, or certification, and get it over with.

nvarsj(3985) 4 days ago [-]

That's a great idea. Tech interviews are a poorly implemented version of something like actuarial exams. Let's formalise it and give people a certificate for it - Certified Software Engineer.

I'm only half joking. At least we'd only have to go through the process once.

nharada(3660) 4 days ago [-]

Some of these 'questions for the interviewer' are very good. In particular I like some of the 'tell me the negatives' ones:

> What is the most costly technical decision made early on that the company is living with now?

> What is something you wish were different about your job?

> What has been the worst technical blunder that has happened in the recent past?

cactus2093(10000) 4 days ago [-]

These are good questions, I like how they are phrased so concretely instead just asking "do you have a lot of tech debt?" They're interesting for getting to know the team and getting a piece of insight you usually wouldn't have until a few weeks into the job.

But I would be careful how you interpret these. In fact I would almost factor in these answers in the opposite way of what I think you intended. The company that admits to the worst technical issues is at least honest and self reflective. The company that doesn't admit to any serious issues might be just as bad or worse, but their strategy is to tell employees to lie about it rather than be open to addressing it.

kthejoker2(10000) 4 days ago [-]

Having hired dozens of devs, I can confidently say there is absolutely 0% chance to consistently successfully identify good developers in any reasonable amount of interviewing / assessment time period.

The best way is someone brings in an existing code portfolio and discusses it.

The second best way is someone completes multiple design and development exercises of varying complexity, constraints, and use cases.

The third best way is they complete a single exercise and provide commentary on alternative designs.

There is no fourth best way; all other approaches are essenrially stochastic and select for interviewing traits not development traits.

The actual best method I think is a 3 month probationary period which is more or less an extended interview. They're asked to contribute to existing codebases, participate in code review, go through some architecture design sessions, conduct stakeholder interviews - things that again are mostly impossible to accurately gauge in a typical candidate assessment window.

By the way a tremendous book for interviewers and hiring managers is How Judges Think by Richard Posner. A lot of it applies to hiring, and he's a great writer.

closeparen(4093) 4 days ago [-]

How often are you hiring people whose professional work is open source? How often are you hiring people for roles where their professional work will be open source?

EliRivers(10000) 4 days ago [-]

The actual best method I think is a 3 month probationary period

My workplace has a six month probationary period; I brought in a mandatory three month review after watching one group screw this up badly. At the end of the six months, the employee came in to work all happy as usual thinking everything was great, and was promptly fired.

The three month review is the point at which the employee is told that they're on track, or they're below standard. If on track, just keep going the same way and if they don't get taken aside for a specific chat in the following three months, told to assume that they're going to pass the probation period; we have definitely turned probation failures into probation successes via this mid-point review. It's also their opportunity to tell the company what the company is doing wrong; what the company is doing that will make them choose not to stay. This too has happened, and we have retained good employees by listening to them at the three month point and making changes.

If they're below standard, they're told what they need to improve and are offered help to improve, or they can just sack it now and walk (or, as happened once and once only so far, they're considered unrecoverable and we take a long hard look at how that person was hired).

The principle we subscribe to is that if the employee is surprised by the results of their probation period, that employee's team lead and by association 'the company' has really screwed up. If an employee doesn't know how they're doing after six months on the job, something has gone very wrong.

akelly(10000) 4 days ago [-]

Nobody does probationary periods because few candidates would pick a probationary period offer over a standard job offer. And it's presumed that top level talent will have multiple offers on the table, and if you don't then there's something wrong with you, so having a probationary period selects for lower quality candidates.

triceratops(10000) 4 days ago [-]

> The actual best method I think is a 3 month probationary period which is more or less an extended interview

Most states in the US have at-will employment terms. Probationary periods are common in other countries. And yet no one seems to want to do what you're suggesting.

boltzmann_brain(10000) 4 days ago [-]

Actually there is a better way assuming a candidate with experience: discuss what they did, ask questions like: 'why did you chose this solution?', 'what were the main challenges you encountered?', 'how would you change your solution in hindsight?', etc. There is no way to bullshit through this or such attempt will be quickly clearly apparent.

DavidWoof(10000) 4 days ago [-]

#1 is great, but impossible for a lot of people due to NDAs. The 3-month probationary period is indeed the only way to really evaluate devs, but I'm not so sure it's a good way to hire them. After 3 months, the mediocre devs have friends, etc., which makes it hard to get rid of them without creating a weird morale issue and massive team disruptions. And then you have to add in the limiting effect on the hiring pool (though I wonder if that really matters, I suspect good devs know who they are).

Number 3 is my favorite, but there's a trade-off: almost anything large enough to allow for multiple designs is probably too large to require all applicants to complete. But I also love having a candidate discuss their own code. I'm really curious what types of exercises you have for this kind of thing, and where it fits in the interview process?

jonotime(3725) 3 days ago [-]

I'm going a different route with interviews and wonder if anyone has input... I'm a senior backend dev with 10+ years of experience. Breadth of projects, masters in CS, open source work, team lead, top contributor, great references, etc. But I cant pass a coding interview, because I kinda freeze up during algorithm whiteboarding questions. I get requests for interviews daily and have been considering starting with a cover letter that basically says I'm not good at coding interviews, but love to talk tech (and see other positives above). Could I do a homework assignment or anything else to show my worth?

Has anyone had any luck with an approach like this?

mixmastamyk(3492) 3 days ago [-]

Put "I don't do pop quiz interviews" on top of your documents and see what happens. I do this but haven't been so blatant about it. Networking is the only other avenue.

DavidWoof(10000) 4 days ago [-]

I realize everybody's going to jump in and rant about algorithms in interviews, but I wish you'd all add something constructive as well.

I just had to conduct a round of interviews in a non-SF large US city, and it was a hellish crapshoot. Resumes are meaningless, and often re-written by recruiters to match the job anyway. Everyone has the same canned answers to the stupid behavioral questions. And as for the code, we included what we thought was a trivial nested for-loop problem and virtually nobody could even get started on it.

Is this kind of code problem too complicated in your opinion? For all I join in when complaining about irrelevant algorithmic questions, I have to admit that they at least test something, even if it's just willingness to study for the interview.

Instead of reading everybody's complaints about interviewing, I'd love to hear how you think it should be done. Because I have to admit I'm pretty much lost right now.

detaro(2186) 4 days ago [-]

> And as for the code, we included what we thought was a trivial nested for-loop problem and virtually nobody could even get started on it.

How did you run these? While it sounds like something that should be ok even on paper, you can vary comfort a lot through the medium. E.g. for the last interview I had with a substantial coding part, I think being able to do it on my personal machine made a big difference. (I obviously was told before what kind of environment I'd need to have ready)

dustfinger(3950) 4 days ago [-]

> And as for the code, we included what we thought was a trivial nested for-loop problem and virtually nobody could even get started on it.

> Is this kind of code problem too complicated in your opinion?

May you please post the problem so that we can provide you with meaningful feedback regarding said problem?

wcunning(3567) 4 days ago [-]

I'm getting up the gumption to look for a different, hopefully better job in the nearer future (in automotive control software, looking to move to AV), and I had a bit of a revelation when talking to a friend of mine who does interviewing. I cannot talk about the things in my current job that would make me a good hire for the things I want to move into. It's all hunting down an obscure bug buried in layers of technical debt and overly complicated standards, but giving any depth beyond that bare platitude requires going into things that my NDA covers. I even work in driver assist technologies, but I can't go into detail about that because I'm working on unreleased, unannounced features. That means that the only thing left is side projects or whiteboard interviews. To be fair, this is at least partly an industry problem due to long product development cycles and a culture of secrecy, but it makes a lot of the better solutions to interviewing unworkable and causes companies to drop back to whiteboard interviews. And as a candidate, my personal maximizing function is to hit the books and be ready for curly braces and logic puzzles. Sometimes there really isn't a better way.

toupeira(10000) 4 days ago [-]

At my current position for a remote job, one of the interview assignments was reviewing a pull request for a very basic example app using their tech stack, and then implementing my own suggestions during a video call (while sharing my screen), updating/running tests, discussing trade-offs etc.

I think this was a great way to not only verify coding ability, but also testing team work and communication skills.

bufferoverflow(3606) 4 days ago [-]

I have a similar experience doing interviews in NYC. We started by asking algo questions, but quickly found out that 95% of the candidates can't answer them. My boss, an ex-programmer, was surprised too, and she asked me to dumb them down, a lot. We ended up with a bunch of really trivial stuff like 'write a function to reverse a string' or its slightly harder version - 'reverse an integer'.

Could you give the exact wording of your 'nested for-loop' question?

non-entity(3990) 4 days ago [-]

I realize everybody's going to jump in and rant about algorithms in interviews, but I wish you'd all add something constructive as well.

I was like this a couple years ago. I was a self taught, 'college is a scam', 'practical experience' type guy. I now, however do see immense value in the ability to be able to work through these algorithm questions,especially if you ever want to do something besides web / app development.

danmaz74(2448) 4 days ago [-]

I use basically the same process, but use a very simple programming problem at the end - simple, but amenable to discussing optimization and edge cases. I let the candidate choose the language (or just use pseudo code) and don't care at all about the syntax.

I find that this is very useful especially when interviewing juniors who don't have many projects under their belt for the first part. It's also useful when a candidate has good verbalisation skills, but poor programming ones (which happens).

choppaface(4089) 4 days ago [-]

I agree a code test is necessary. I've seen several panels neglect to do a code test, the candidate was hired, then within a month fired because it was clear they couldn't do anything (other than, well, argue).

I've been in a panel where I was the only person who asked a code question, the candidate flunked, and then the VP of Engineering went over my complaints and hired the guy anyways. He had been a Professor of Software Engineering and had a graduate degree from Princeton. Within three weeks, the VP of Engineering had to fire him because he couldn't make it through a simple code review.

BUT the extremely negative sentiment here towards the technical interview process is very well-deserved.

Assessment of code (and the selection of problems) is most often no less subjective than any non-technical assessment. Sometimes the interviewer doing the grading is flat out wrong. Several times I've been asked the famous "given an array of stock prices, find the optimal buy and sell indices for the biggest profit." One interviewer was not aware of the linear time solution to this problem, and didn't believe me when I wrote and tried to explain it to him.

But sometimes, the interviewer doesn't even want you to do well. Once time I was interviewing with an injury that prevented me from typing efficiently. I had a doctor's note and the injury was quite conspicuous. Nevertheless, three start-ups made me solve problems by typing on a keyboard, which guaranteed an excessively long completion time. Those companies held those results against me. (An it's not like there is anybody to hold those panels accountable).

And then there are those who just don't care. I had a phone interview with Airbnb that was literally as bad as the stories on Glassdoor: the guy answered the phone in a noisy office (not a conference room), gave no introduction, then simply stated the problem and dumped me into Coderpad. I literally thought it was a prank, since I had met with people at Airbnb face-to-face prior to the call. But the recruiters confirmed the guy was a real employee.

The root problem here is there is no feedback loop back to interviewers. The candidates get "feedback," but people asking code questions, especially new grads, typically get zero assessment on how well they are doing as interviewers. What's worse is that recruiters and hiring managers both have incentives to deprive ICs of such feedback, since it would invariably make ICs more aware of opportunities outside the company.

Until the incentive structure of technical interviewing changes dramatically, we're stuck with Leetcode and hope for the best. People like Gayle Laakmann are helpful (especially when Facebook gives candidates a live hour-long session with her for free), but these people are ultimately invested in their own income and not the task of fundamentally fixing this broken process.

Jach(1951) 4 days ago [-]

Seems like your problem is basic competency? Move the 'can you even code?' question to as early in the filter stage as possible (first 'phone' screening). If you have a lot of applicants, you'll have to do some earlier filtering in the name of time (like on degrees, years experience, 'the lucky half'), but don't pretend it's fair or very accurate since both the false positive and false negative rates will be high.

I agree that some coding problem needs to be used to try and answer the question, though with the right interviewer they can answer it without seeing code. The problems you use for that don't have to be at octree-collision-detection whatever challenge, a trivial nested for-loop is fine -- fizzbuzz level is fine. Sometimes you can rely on github or strong internal referral to skip this, but watch out, and anyway it's worth giving your questions to people you're sure will do fine (you've timed at least yourself right?) for the benchmark data and because sometimes they don't do fine, perhaps since maybe your question is too much. e.g. Floyd-Warshall can be done simply with a few nested loops, still I would never give it as a problem and I'd expect nearly everyone I've worked with to flunk it given only the standard hour (which really means 45 minutes).

Some jobs only need basic competence, so you might want to extend an offer if you've been convinced of its presence. At my last job, which ended up being more technically challenging / interesting than my current job, I was hired after posting my resume to Craigslist which led to exchanging some emails and having lunch with the startup founder to talk about my past work and whether I would be useful for his most pressing work. At my current job, I've been part of on-sites where I've established 'can you even code?' is 'no'. Those were costly failures of not having that answered earlier. But we also like to believe we need more than basic competence, so rejections can still occur because of a lack of 'testing mindset' or certain 'behavioral answers'. Only once you fix your 'can you even code?' filter is it even worth considering what else you might want to justify an interview pipeline with more stages than a 'phone' screen or lunch conversation.

dehrmann(10000) 4 days ago [-]

> nested for-loop problem

Ah, dynamic programming.

winrid(10000) 4 days ago [-]

Phone screens should prevent those candidates from ever getting onsite.

You'd be surprised (or maybe not now) how many applicants for a senior frontend position can't build a progress bar for the phone screem.

jjav(10000) 4 days ago [-]

I can tell you how I do it and would certainly recommend it as the way it should be done. For some context, I've been interviewing software engineers for about 25 years in companies ranging from established multi-nationals to tiny startups in very fast headcount-growth mode. I'm in silicon valley. I can say that I've never regretted a hire I said yes to, so the method works to my satisfaction.

It'd be nice to think I have some special skill here but I really don't. This is just how interviewing was done in the 90s. To some of the younger generations, I've been told it sounds crazy.

If you send me your resume I'll actually read it, carefully. If the person described in this resume fits the background experience the role needs, you get an interview.

During the interview we'll talk about all those projects you worked on that are relevant to this role. Which parts you enjoyed the best and why? Which parts were boring and why? Which parts were the most challenging and why? What you find too easy and why? What would you have done differently? Could you have? If you were to do the same project all over how would you approach it? Other open ended conversations along these lines.

I don't ask anyone to whiteboard code, that's not part of the job so it's not part of the interview. No puzzles, no trivia-pursuit style questions.

It works great. You can't BS your way through such a conversation with a senior technical peer if you didn't actually do the work described in the resume. You just can't.

It is, however, vital that the interviewer must be a expert in the field.

patientplatypus(3861) 4 days ago [-]

This is surprisingly easy, but you have to understand what the problem is first or the solution won't be apparent. If this sounds patronizing it comes about from my frustration on being on the other side of the table too often.

What you want is very good football (soccer) players. Unfortunately you (or upper management) may not know all of the rules to soccer. You may not know the training regime that goes into winning a good soccer game - and it's a big risk spending money training for the big game on the off-season only to lose during the Big Show. So what do you do to test potential candidates? You see if you can get along with them, if they're a team player, and then see how well they play foosball.

It's perfect! There are soccer players on the field, there's a goal, it takes skill and coordination. But oh no, it turns out in the population at large really good soccer players really sort of suck at foosball. After all, they'd rather spend their time and energy playing soccer. So now they spend all their time reading up on books about foosball and what the best foosball strategies are.

You see where I'm going with this? No one uses algorithms in their day jobs. OK a few of you, but come on man, I make full stack web applications. As do most programmers. So why are you testing theory that has literally nothing to do with the job? Somehow someone thought this was a proxy for smart people, but I mean, if the guy who wrote this 'Tech Interview Handbook' was really smart wouldn't he have spent his time writing a cool program? I mean how lame is this?

If you want to hire competent engineers tell them what you're building and ask them how they feel is best for them to show they're competent. If you have a big data pipeline in Scala ask if they can construct a data pipeline example that is cool over a few days (take home) or do something similar in the office. Some people like one, some like the other. But just communicate with the people you want to hire! And if not everyone's interview process is the same then maybe that's ok.

I mean I just got a guy who sent me some automated code interview program that had a timer that counted down from an hour at the top! WHO THINKS TREATING THEIR POTENTIAL EMPLOYEES LIKE STAR WARS DRONES IS A GOOD THING? All you have to do is treat the people you want to work with like you would want to be treated and demand that they know their shit.

This. Isn't. That. Hard.

dccoolgai(3476) 4 days ago [-]

1) Resume screen 2) 30-minute coderpad/codeshare exercise on a problem/pattern you actually use/encounter during the course of your work over phone. (No inverting binary trees). Expect a 20% pass rate here. 3) Reasonable take-home problem that you've timed 2 of your own staff completing well in 50 minutes. This is where you will get the most complaints from applicants, but that's OK. Let them select themselves out of the process. Expect a 30-40% pass rate here. 4) In-person interview. At this point, you should be mostly committed to hiring the candidate. Do a couple livecoding deals, but be extremely lenient in how you interpret results. Other week, we had a candidate fail the problem, but they kept their composure and showed they knew what they were doing on the way to bombing the problem. Candidate seemed sad at the end of the interview and happily surprised when we extended an offer.

IMHO this process works fairly well and does a good job of being economical with people's time.

davidjnelson(2007) 3 days ago [-]

What works for me is to consider what work the engineer will be doing, and ask them to write code to prove they can do it.

For instance, a front end engineer can be expected to be able to write a to do list or similar app in a framework (ideally the one your team uses, but not a hard no hire if not) app with minimal googling (although that's fine as long as not excessive) in ~45 minutes.

Then you have to look at what level of experience they have. Less experience requires more mentoring generally, which may be fine depending on how much time your team budgets for that work.

Lastly, measure their body language and tone of voice to check for red flags pointing to difficult communication styles or people who treat others poorly.

If all three match, hire!

daphneokeefe(10000) 4 days ago [-]

> we included what we thought was a trivial nested for-loop problem and virtually nobody could even get started on it.

If no one can answer the question, maybe they don't understand it? Maybe there's something unclear in the way it is worded?

neetfreek(10000) 4 days ago [-]

I prefer take-home assignments. I know that they often get a bad rep and have the potential for abuse; for this reason I'd argue for implementing some or other solution in the field.

I recently had one requiring me to develop a native mobile application, which I enjoyed. It was interesting, the code is useful down the line, and if I don't land the job, it beefs up my portfolio.

Initial screening by recruiters is tough as my background's missing a degree and industry experience.

Context: self-taught, started out with game dev, tried going solo - not a runaway success. Looking to move away from the field.

wan23(4176) 4 days ago [-]

The way most fields that require some level of knowledge or ability handle this is by having industry-standard assessments that everyone takes. When doctors interview they get almost entirely behavioral questions because the hiring hospital / office merely needs to check that the doctor is a) licensed and b) certified in whatever specialty they are being hired for. They certainly never get asked whatever the medical equivalent of FizzBuzz might be.

ale22(10000) 4 days ago [-]

How much were you paying for these positions?

leovander(4202) 4 days ago [-]

Nest for loop for the technical interview? Where do I apply?

underwater(10000) 4 days ago [-]

> And as for the code, we included what we thought was a trivial nested for-loop problem and virtually nobody could even get started on it.

It's possible that you asked the question poorly, or the solution wasn't as obvious as you thought.

Designing interview questions is hard[1]. I'll test out new questions on my peers at least two or three times before putting them in front of a candidate. And many don't make the cut. If a good engineer who's relaxed can't solve it easily, then a stressed out candidate will have no hope.

[1] This is why I hate seeing candidates share specific questions online. As an interviewer you'll have to scrub a good question, and switch to something you're not as familiar with. This hurts good candidates.

momokoko(10000) 4 days ago [-]

Is there any chance you could share the problem? Changed enough to protect your identity, of course.

traverseda(3124) 4 days ago [-]

> And as for the code, we included what we thought was a trivial nested for-loop problem and virtually nobody could even get started on it.

Could it be that you're having issues communicating the problem?

rapht(4168) 4 days ago [-]

I don't work in tech (though I sometimes think about uuyýjmq

nimrody(10000) 4 days ago [-]

One company sent me (before the interview) a small technical assignment. After I had submitted the code, an interview was scheduled. The entire interview was an extended code review -- talking about trade-offs, about other potential solutions, etc.

I felt this was much better in that it was less stressful, yet allowed me to demonstrate both knowledge and design skills.

Another company did something similar but more thorough: They invite candidates for a full day of work where they try to solve a small problem. Then the code is reviewed and evaluated together. They also start the day with a 1-hour overview of their current architecture and you get to ask questions and talk about alternatives. I think this gives both sides a better chance of finding the right fit.

I realize this is not always reasonable.

alkonaut(10000) 4 days ago [-]

If you risk having a bullshitter you need some way of identifying them. The best way is through references, personal projects etc.

But if you need to recruit someone without any such credentials then you may need to do a simple coding aptitude test. Could be a code review or a simple excercise but whatever you do, don't do whiteboard coding and don't have people recite/implement memorized CS textbook algorithms. Anyone can do that and still no code.

dmoy(10000) 4 days ago [-]

> Is this kind of code problem too complicated in your opinion? For all I join in when complaining about irrelevant algorithmic

No. Just programming is actually not easy and a lot of people apply for jobs they can't do.

Plus, a nonzero number of people freeze up in any interview situation.

I once failed an interview loop because I forgot how bucket sort works.

andrewstuart(987) 4 days ago [-]

>>> how you think it should be done. Because I have to admit I'm pretty much lost right now.

- Treat recruiting in the same way as you do software development.

- Formulate a set of requirements.

- Define interview questions that give insight into whether or not the candidate meets those requirements. This is the equivalent of 'tests' in the software process.

- Specific skills with your technology stack is good, but not necessarily essential.

- Ability to discuss sophisticated software concepts, and to explain software that they have built, and how they would build out ideas given to them is good.

- Evidence that this person gets stuff done is good (ref Joel Spolsky).

Coding tests are, for the most part, garbage. Not because the test is of no value, but because you the employer probably don't evaluate the result properly.

greenyouse(4158) 4 days ago [-]

I've been through a few different types of interviews and the random algorithm style seems off to me. I recognize that there are companies like FAANGs that want a deep bench and will have expert algorithm people on-call for that one time when it's needed. Most companies should be OK to just present real problems either solved or being worked on which they would expect applicants to be able to solve.

This is more true for startups that need less know 'how to reverse a binary string' and more 'how to properly design a database with 3rd order normalization', etc. If you're presenting an interview question, it should have actual job relevance and your co-workers should be able to solve it in the same time as the candidate. If you're drilling people on non-job qualities (e.g. invert a binary tree for a web dev role...) then you should expect a large difference between audience that can pass your bad interview tests and audience which will perform well at actual job.

Not trying to complain. I think job interviews should focus more on the 99% of what you do in your job on a Tuesday. Poor interviews seem to be more gotchas and algorithm tricks to disqualify roles which have little actual use for algos + data structures. The tests might seem too easy but as a front-end engineer I would rather be with a coworker which understands the CSS box model, knows semantic markup for accessibility, and similar web things than a person which is good at creating hash tables and doubly-linked lists in JS. Leet code probably doesn't test vertical centering techniques with CSS but if you're applying for a web dev position you better know them.

dustfinger(3950) 4 days ago [-]

> Everyone has the same canned answers to the stupid behavioral questions.

If the person conducting the interview thinks the behavioral questions are stupid, then perhaps they are. In that case, don't ask 'stupid' behavioral questions.

> Resumes are meaningless, and often re-written by recruiters to match the job anyway

Was the position entry level? Students coming right out of compsci often have little to no practicable experience. They may have difficulty thinking about what to put in their resume. After one or two years of full time experience that should no longer be an issue.

> For all I join in when complaining about irrelevant algorithmic questions, I have to admit that they at least test something, even if it's just willingness to study for the interview.

Asking those 'stupid' behavioural questions and receiving the same canned answers also demonstrates a willingness to study for an interview.

The coding problem should be testing a candidate's problem solving capabilities as practicably required by the role being interviewed for. The chosen problem should reflect the types of problems that they will actually need to solve if hired. For example, you could select a small PR from one of projects being actively developed by the company. The selected PR should involve only one or two classes (assuming a language with classes) and require improvement. You can look through the history of a PR and just pull out a segment that was selected for improvement by the reviewer(s), or have the team select it for you. Then ask the candidate:

- to conduct a code review of the PR

- to improve the code

new299(3931) 4 days ago [-]

Personally I can't see an issue with very simple FizzBuzz style programming interview questions. I used to ask a simple 'count duplicate substrings' question [1]. Maybe some people consider this too hard? I never used to require exact syntax, and would have been happy with pseudo-code. Using libraries is fine etc..

I also found very few people could solve this (similar non-SF large city location). Occasionally, people who could not solve this were hired for other teams. Based on their performance, I don't think I would have been comfortable working with them.

I don't think it's unreasonable. But I'm not sure I'd use it as a screen if I was hiring now. I think I'd just have a chat and try and discuss a previous project. After that I'd move to a paid take home project (ideally representing real, useful work).

[1] Take a string, for example 'ABCCABC' and count the number of times each 3 character substring occurs. In this case the answer would be 2xABC 1xBCC 1xCCA 1xCAB.

_hardwaregeek(10000) 4 days ago [-]

Where are you posting your jobs? Are you paying a recruiter? How much? Maybe you could have interviewers rank candidates (hire; competent, but no hire; slightly incompetent; how did they get here?), then see where the majority of your 'how did they get here?' candidates come from.

watwut(10000) 4 days ago [-]

I like the for loop code thing. Imo, good hiring process for hiring programmers must involve small piece of code. Not difficult or algorithmic, but something to distinguish those who can't do anything at all.

seanmcdirmid(2200) 4 days ago [-]

Resumes and prior experience should be taken more seriously, backed by more rigorous verification of said things and real reference checking. In a way, that is what already happens for the more technical positions (someone already knows you can do the job, the interview is just a formality) and sucks for those who don't have well known enough reputations.

ummonk(4160) 4 days ago [-]

I don't think the problem lies with your interview if they can't write a simple nested for loop...

Which I think is why a lot of startups locate in the expensive Bay Area - not a lot of cities have a similar concentration of decent talent.

Aperocky(10000) 3 days ago [-]

This.

I loath writing algorithm on whiteboard, especially the 'catchy' type. But I've interviewed people who can't even write a for loop.. the amount of brain drain in the flyover country is insane.

modzu(4027) 4 days ago [-]

what is everyone smoking? youre still trying to optimize for a test and not a job.

dont give them a fizbuz. give them an example of a real problem your engineers need / are trying to solve. how do they respond? thoughts / intuitions / pseudo code. do they show knowledge of the problem space / domain?

ot if you just want a kid who can code and they pass the fizbuz but fail at the real job, what does your training/culture look like? who does that really reflect on?

it seems to me that interviewing is terribly cargo cult. the problem is real, the practises ostensibly supposed to be solutions are not.

/end rant

RijilV(3866) 4 days ago [-]

> Everyone has the same canned answers to the stupid behavioral questions.

The art of behavioral questions isn't "ask and answer" it's the follow up questions. As you astutely point out, the questions are 'stupid'. They might as well be "do you want a stick of gum?"

Next time you ask those questions do a couple of things. First keep in the forefront of your thoughts what information you're trying to get out of it and keep the candidate on track answering your data point. Do that by relentlessly asking follow up questions. When you think you have everything ask more.

As an anecdote I was being shadowed during an on-site recently. I asked some arbitrary 'dumb' behavioral question, went back and forth a bit, wasn't getting much out of it. I noticed my shadow clearly moving on to the next question in their notes and decided to keep pushing on the original question - why did you do this, what were you trying to solve, what motivated you. Turns out the candidate did all of this to generate new revenue for the company and ended up bringing in $10m a year extra at the small company there currently worked for. Loads of great data, would have never gotten there if I'd settled for the canned answer the candidate had.

Behavioral questions aren't comp-sci trivial questions, you can't just ask the behavioral equivalent to fizbuz/Fibonacci/floodfill and copy down the answer (and you should never be asking those questions either, but that's a separate rant).

Behavioral questions are stupid and to some degree that's the point. When you ask your significant other or kids "how was your day?" — guess what, that's a stupid question too. What matters is what follows from your line of interviewing.

If you want to get good at behavioral questions listen to Fresh Air and try to be like Terry Gross.

gaoshan(4196) 4 days ago [-]

For the code portion of things I stick to a set of increasingly difficult 'real world' problems. The first one should be easily answerable by any potential candidate, the last one should be too hard for most people but I'm really looking at how they problem solve and handle themselves. I've recommended people for hiring who eventually gave up on that last one (and who went on to be fantastic in their jobs).

strikelaserclaw(10000) 4 days ago [-]

For all the negativity for these type of tech interviews, they are, from what i've seen, one of the most merit based systems out there. It is either this or we need to create some sort of national developer exam. The other alternate is to get jobs at good companies, they will only look at what school you went, whether you graduated with a CS degree, what companies you worked etc... All things which do not guarantee merit.

mixmastamyk(3492) 3 days ago [-]

Merit at high pressure performance, not being a good developer.

jiux(10000) 4 days ago [-]

I once interviewed at a startup for a senior engineer position and was asked "if aliens came to Earth and asked you to go into their UFO with them, would you?"...

As much shit as we give white-boarding, I would have chosen it instead if it were an option.

glandium(3871) 4 days ago [-]

What's the point of hypothesizing on things significantly less likely than someone winning the lottery (edit: while being hit by lightning, for good measure)? In fact 'what would you do if you won the lottery' might actually give more interesting insight.

xtracto(10000) 4 days ago [-]

At one time i was asked 'why do frogs croak? I gave a series of answers like: to attract the opposite sex, gave some biological explanation of how they achieve that. But the stupid interviewer kept asking me why?

Needless to say after the interview ended i ran away from that deal.

ixtli(3906) 4 days ago [-]

What if people not being able to answer simple programming questions is a problem with the interviewers, and not the interviewees?

maehwasu(4202) 4 days ago [-]

This is one of those ideas that sounds really sophisticated, but is just wrong.

The fact is that there a tons of people out there representing themselves as programmers who actually can't perform basic tasks.

The HN population massively selects for competence, so people here have a hard time imagining what things are like from the interviewer side.

codingslave(10000) 4 days ago [-]

Two outcomes for the technical industry:

1.) Everyone is studying these problems all of the time and they finally disappear.

2.) Other outcome is a dystopian field fueled by a race to the bottom where everyone is practicing algorithms problems all of the time. If you read the blind forums, some people are completing 500-1000 leetcode problems before heading into interviews.

I'm putting my money on number 2, which is where we already are. Can only imagine what this is doing to code quality...

decebalus1(10000) 4 days ago [-]

I started leetcoding again this year because I want to jump ship and holy crap! I used to solve problems on leetcode 5 years ago (last time I switched jobs) and it was pretty laid back. Nowadays, I'm seeing dynamic programming with 3D memoization arrays like it's something normal. It all started like a way to check if someone knows how to write code or knows data structures and basic algorithms but now it's competitive programming level.

yodsanklai(4047) 4 days ago [-]

> some people are completing 500-1000 leetcode problems before heading into interviews.

Over the course of one year, I completed, classified, commented 200 leetcode problems. I also taught algorithms to third and fourth year university students not too long ago. I believe I write readable code, I know perfectly the language I'm using (at least for that purpose), I'm totally fine with complexity, and I know most methods involved in these algorithms, including more advanced algorithms such as KMP.

Yet... I failed my round of interviews at Google. After this preparation, I'm still not able to solve quickly any leetcode problem in the context of an interview. On a whiteboard, with an interviewer in my back, in a stressful situation. I need to practice more if I want to get consistent results.

So I agree with your conclusion. We are competing with a lot of people who train using the same resources. Including young graduates who have a lot of free time on their hand.

On a positive side, I'm thankful to Google for giving me a shot. Based on my resume (40+ with little experience in software industry), I'm not sure I would have been interviewed in a more traditional company, let say a bank.

To come back to my interviews, system design went very well. Algorithms quite well too but not well enough. Couldn't solve one problem, and a bit slow on an other one.

The recruiter first told me I passed, and that they were going to find a team for me and make me an offer. But they finally asked me to re-take the algorithmic interviews a few months later, to 'make a stronger point to the hiring committee'.

Never heard from them since then, about 8 months ago. Recruiter doesn't answer emails. I think she moved to a different position. Not sure what to do now?

solipsism(10000) 4 days ago [-]

Can only imagine what this is doing to code quality...

I don't get why no one seems to consider the possibility that these sorts of interviews actually do get high quality engineers in the door.

I get downvoted for raising the question every time. But isn't it possible this interview style actually works, even though it doesn't resemble real coding and even though many of us hate it?

I have yet to see any compelling argument for why I should believe these interview practices don't work. And yet the fact that so many companies, with so many resources to change things up if they felt it was in their best interest, keep interviewing this way must at least suggest the possibility that maybe it works?

triceratops(10000) 4 days ago [-]

> Can only imagine what this is doing to code quality...

Why would studying algorithms and data structures affect code quality? They would similarly be able to learn to write quality code once they're inside the company, no?

fogetti(3931) 4 days ago [-]

There is something I cannot understand and this has already become a cliche: > Technology industry is an extremely fast-moving one. Many technologies used today didn't even exist/were popular a decade ago; in 2009, mobile app development and blockchain were pretty much unheard of. Engineers constantly need to upgrade their skills to stay relevant to the demands of the job market. Engineering is a great career for passionate individuals who like to learn.

Why would anyone give this advice? Can we stop handing out this advice and encourage everyone to stand up for their rights instead? If you think about it, by proxy this gives you the following advice: as a programmer you will have no life outside of work and you are supposed to be an idiot who spends his time working and studying outside office hours even when you could spend time with your family. So yeah, go for it and suck it up you idiot.

At least that's how most employers handle this problem. And interestingly by comparison no-one tells a MBA diploma holder that it's a great career for passionate individuals who like to spend their whole life studying.

And before you think that I am against studying, that's not the case at all. If my company pays for it and I can kick back on a sofa during working hours to study I am fine with it. But I cannot see any value studying something on my own expense on my own time which will be outdated in 3 years anyway and so by definition it only benefits my employer.

closeparen(4093) 3 days ago [-]

Teachers, doctors, lawyers, professional engineers, and many others get their licenses revoked by the government if they fail to go back to college and take classes every few years.

username90(10000) 4 days ago [-]

MBA holders is a bad example since many of them happily work many hours of overtime, so I am pretty sure that they work more hours than software engineers even if you include the time it takes to study for interviews.

> It turns out that the median number of hours racked up by an MBA in his or her first year of employment is a whopping 54 hours a week

https://www.forbes.com/sites/poetsandquants/2018/03/06/the-6...

gtirloni(2269) 4 days ago [-]

I didn't read that as saying you have to spend hours outside of work to learn. Learning on the job is pretty much a given these days, no? At least in my experience. Even decades ago I was always given time to research/study new things and I really like that about our industry.

lr4444lr(4160) 4 days ago [-]

I agree, and I think you hit on exactly why white-boarding persists (the least of all interviewing evils, IMHO): if this industry is so fast-moving, the fundamentals are the most reliable thing we can measure that has lasting value for (what we hope) are long-term hiring decisions.

bushido(2624) 4 days ago [-]

Why would anyone give this advice?

It seems relevant to getting a tech job. If one is looking for a job, chances are that things have changed since the last time they looked for a job, which on an average for most people these days is every 2-5 years.

This is not limited to engineering or tech, and extends to most specialized jobs in most industries.

Can we stop handing out this advice and encourage everyone to stand up for their rights instead?

In the right context absolutely. In general, its possibly a bad idea. You could always balance it out and educate the folks about the rights of an employer

If you were on my team, I would not expect you to sacrifice any of your rights. But if you kept falling behind your peers, to the determent of the team's performance, at some point you would be put on a performance plan. The unfortunate thing about performance plans is that by the time its enforced things are close to unrecoverable.

And if you did find yourself failing, I hope someone tells you that:

The technology industry is an extremely fast-moving one. Many technologies used today didn't even exist/were popular a decade ago; in 2009, mobile app development and blockchain were pretty much unheard of. Engineers constantly need to upgrade their skills to stay relevant to the demands of the job market.

Because, I will not.

nickthemagicman(4197) 4 days ago [-]

There are no employee rights in modern capitalism. In the 19th century, they worked 80 hours a week in mines, had child labor, no vacations, women were ignored, indentured servitude existed. All this and more until unions and labor laws.

Karl Marx's entire philosophy was based on the abuses of workers he observed.

Nowadays people mock progressive causes as 'socialism', mock unions and worker protections, demonize progressive politicians, and idolize oligarchs. Even though it's all against their own best interest.

Your idealistic notion of 'rights' doesn't exist and will never happen. You have no rights as long as someone controls your purse strings, which with booming inequality is more and more people nowadays as well.

I can't wait until most jobs are finally automated and we're done with this whole capitalist system entirely and have to figure out what to do next.

arvinsim(10000) 4 days ago [-]

Curious as to why the geographical locations provided are only for US and Singapore.

x0to1(10000) 4 days ago [-]

The author is from Singapore and writing from the perspective of a graduate with fang jobs to his juniors. Also, looks like it showcases some software(Docusaurus) he made at facebook.

rvz(4065) 4 days ago [-]

While this is very good to study before a technical interview, over time however I can see that this alone is going to make it 40x harder to differentiate say 100 candidates that are all perfect at interviews in general, that we are going to start asking ridiculous Oxbridge-style interview questions and expect perfect scores to advance 'good' candidates.

Perhaps companies will start asking candidates to construct mathematical proofs of data structures, algorithms, formulas and common equations from university-level entrance examinations just to do a mobile app or a web dev job.

As soon as that happens, the 'ideal candidate' companies will be expecting to interview would be a very prodigious candidate, former math Olympiad champion and decorated with titles and research papers in their name.

You guessed it: Ye olde legend of ye 10x developer.

seanmcdirmid(2200) 4 days ago [-]

Yes, basically Goodhart's law generalizes to standardized tech interviews.

chrisseaton(3058) 4 days ago [-]

> Perhaps companies will start asking candidates to construct mathematical proofs of data structures, algorithms

Perhaps we'd produce better software if people prioritised correctness like this in practice!





Historical Discussions: A Walk in Hong Kong (August 16, 2019: 740 points)

(742) A Walk in Hong Kong

742 points 6 days ago by haasted in 1287th position

idlewords.com | Estimated reading time – 24 minutes | comments | anchor

08.16.2019

A Walk In Hong Kong

My mom fainted the first time she set foot in an American supermarket. It was 1981 and we were freshly arrived in America, and some combination of the culture shock and smell and sensory overload of a Safeway was too much for her, not normally a fainting woman.

I didn't faint, but I stood transfixed in the cereal aisle. After six long years on Earth, here was a place that understood me. These were cartoon characters, made of pure sugar, that you could eat as a meal. Every box had a toy inside. How could I possibly choose just one?

Houston in that era was not an earthly paradise. But try telling that to visitors like us, dazzled by the sheer abundance of American capitalism. There were superhighways in the center city! Everyone had a giant car! You could drive that car up to a restaurant with a yellow "M" on it, and a teenager would hand you a bucket of pink ice cream through the window! According to the sign on the restaurant, this had happened billions of times. It was overwhelming.

All that prelude is to say, coming in to the Hong Kong protests from a less developed country like the United States is disorienting. If you have never visited one of the Zeroth World cities of Asia, like Taipei or Singapore, it can be hard to convey their mix of high density, mazelike design, utterly reliable public services, and high social cohesion, any more than it was possible for me or my parents to imagine a real American city, no matter how many movies we saw. And then to have to write about protests on top of it!

It's hard to write articulately about the Five Demands when one keeps getting brought up short by basic things, like the existence of clean public bathrooms.

The time and location of protests are set via social media alchemy; once you get notified about one, you descend through a spotless mall onto a bright and clean train platform, get whisked away by a train that arrives almost immediately, step out into another mall, then finally walk outside into overwhelming heat and a gathering group of demonstrators.

When it's over, whether the demonstrators have dispersed of their own will, or are running from rubber bullets and tear gas, you duck into another mall, and another train, and within minutes are back in a land of infinite hypercommerce, tiny alleys and posh hotels with their lobby on the 40th floor of a skyscraper.

Not everyone lives in a luxury hotel, man! I get it. But my eyes are like saucers. I ask forgiveness of Hong Kongers if at times I am still that six year old kid, dazzled by what to you is ordinary. You live in a kind of city we Americans can only aspire to, and it's no wonder you love your home so much you will take any risk to save it.

Saturday, August 10

Word spreads that there will be a protest in Tai Po, a residential district in the New Territories, close to the border. The protesters have been cycling events through different neighborhoods, as a way of building sympathy with the people, who can look out from their windows and watch the police reaction firsthand. At this point, the police reaction almost inevitably means tear gas, and people have been gassed in their own apartments, on high floors. This is a particular hardship for the sick and elderly, already struggling in the heat.

One of the deeper causes of the present crisis of legitimacy is the housing crisis in Hong Kong, another way in which the government has failed the people it is meant to represent, and you can see it in the extraordinary density of apartment buildings, each unit pock-marked with an air conditioner, tiny living spaces with some of the highest rents in the world.

Tai Po is easily reachable by MTR, the city's commuter train system. Everywhere except the islands is easily reachable by MTR. The MTR is the one technology the Hong Kong protests could not do without, an autonomous fiefdom that the police mostly stay out of. It is neutral territory. The train ride is uneventful until we get off at Tai Po station, where there are an unusual number of people in black, the color of the protests (lucky is the Hong Konger who started the summer as a goth or metal fan, and has some wardrobe options!)

The arriving protesters stream out of the station in groups, consolidating as they approach the rallying point in a local park. Volunteers along the way pass out flyers and hold directional signs. There is a mix of ages, though they trend young. The typical Hong Kong protester, researchers tell us, is a college educated professional in their twenties. There is a roughly equal gender balance.

The streets around the park are almost free of traffic. After a brief false start in the wrong direction, the crowd turns itself around with hand signals and begins to march. The atmosphere is festive, with lots of chanting of "ga yao" (add oil), the idiomatic phrase that is now finding its way into English.

Once every few minutes a stray vehicle wanders into our path, and demonstrators take pains to let it through, with self-selected volunteers parting the crowd. If the taxi honks in support, a cheer goes up from the marchers.

The road takes us past lush parkland, where skinny white birds observe us from the treetops, and then through a business district with no police, few pedestrians, and almost no traffic. Many stores along the route have been shuttered in anticipation of the march, but the second-floor restaurant windows are full of curious faces. People are photographing us, and we photograph them back. When someone waves or shouts support, whether from the street or from a passing vehicle, the same cheer goes up.

In the windows, we can see couples on dates, curious children with their hands pressed to the glass, and store keepers looking out with an appraising eye. Up in the high rises, old men in undershirts lean down to watch.

A Thai restaurant has stayed open. The hand-lettered sign in its window invites demonstrators in for free refreshments. There is nothing you can pay for in a protest - volunteers hand out gas masks, water, tea, and endless flyers, designed and printed with astonishing alacrity in response to each day's events. Anything you may need, people have donated.

Across the street, a man is sticking post-its to the shuttered storefront of a 7-11, creating another one of the post-it walls that are all over Hong Kong. The airport protest on Friday even had a guy styling himself Memo Man, dressed in a black body suit you could paste notes to, until he looked like a paper hedgehog.

The level of heat and humidity is almost comical. My slavic body is shutting down, in a process so unsettling to fellow marchers that concerned people start offering me water or tiny, single use tissues. These turn into wet confetti the second they touch my face. You might as well try to stop Niagara with a hand towel.

"It's okay," I tell them. "This is normal. I'm not dying—I'm Polish." They edge away.

Our group at this point numbers in the thousands. We pass through another neighborhood of residential high-rises and find ourselves at a large, empty intersection. In the distance on the left there is a pedestrian overpass full of journalists (we can tell because they've set up antenna tripods). Beyond the overpass stands a line of cops, their plastic shields glinting in the sun.

With the police visible, there is a little bit of a hush in the crowd. The march continues in a direction parallel to the police line, to another major intersection. There, volunteers are yelling to offer marchers a choice. We can turn right and walk to a nearby train station. Or we can go left and confront the police. Most people choose to go left, and are handed surgical masks (to hide their faces) if they don't already have them.

The police station is just ahead, a five-story building with large Eiffel tower-style antennas on the roof. Another line of riot police has blocked the road leading to it, and journalists rush to get pictures. The demonstrators stop and confer.

I should say a few words here about the curious way the protests are organized. The protesters learned in 2014 that having leaders was a weakness. Once the leadership was arrested, the heart went out of the occupy movement, and it lost momentum. So in 2019, there is no leadership at all. The protests are intentionally decentralized, using a jury-rigged combination of a popular message board, the group chat app Telegram, and in-person huddles at the protests.

This sounds like it shouldn't possibly work, but the protesters are too young to know that it can't work, so it works.

The protesters divide themselves into groups based on how much they can risk being arrested. The issue is not jail time, but the prospect of losing a job or being kicked out of school, now that China has shown it will crack down ferociously on companies that employ demonstrators.

The frontmost group are the people who actually come into contact with the police, and put themselves at greatest risk of arrest. Behind them there is a supply group, who passes water and other essentials forward, and helps those in front if they are hurt. And behind that is the great body of demonstrators. Those who will be up front are given leeway to make decisions, including telling the crowd to move back and make room, or deciding to disperse and reassemble elsewhere.

Now these frontline protesters have formed a huddle to discuss the tactical situation. Protesters around them shield the group from surveillance with umbrellas. The police watch, and the rest of us wait.

Journalists are all over these protests. Members of the local press wear yellow vests marked PRESS, and put on headgear if things start to get spicy. Some of the foreign correspondents elect to dress more elaborately, looking like they're about to explore the lunar surface rather than interview a tiny retiree. You can spot key events or celebrities in the protests by looking for the circle of yellow vests, and pyramid of selfie sticks raised above whatever is happening.

The protesters, like beavers, have a strong building instinct. They want to make barricades. The preliminary huddles have expressed concern that we are not on favorable terrain. The police is dead ahead, the street is bordered by a culvert and stream, and it would be too easy to cut the protesters off from behind. So a new set of huddles forms, screened by umbrellas, and after a few minutes we see that they have reassembled steel barricades into triangle shapes, held together by plastic ties, which they will assemble and tie together to block the side roads.

A great hurrah goes up as a group of masked protesters runs one of these triangles down the road, the crowd zippering open in front of them. As they run, the umbrella holders run alongside to keep them screened. Another triangle platoon runs its barricade down to the intersection.. And then - disaster! A third triangle is rushed down the street, but the squad misses their turn, and races off into the distance. The crowd roars for them to stop. Far down the road, they skid to a halt, re-form, and rush back. More cheers.

The rest of us stand and watch this business unfold. It is four o'clock and the heat is cooking me from within. A breeze is blowing towards the police line, and there is speculation that this is why the police have not fired tear gas. The foreign journalists in their body armor are all up at the front of the protest, so there is a disorienting dynamic where people further back can follow a Bloomberg live stream what is happening two hundred meters to their front. Flummoxed by the wind, the police have instead brought out pump tanks of pepper spray, like giant hot sauce dispensers, and are brandishing them at the front of the crowd.

More time passes. Finally, the decision surfaces that the position is too unfavorable. This has been another development in the protesters' tactics since the start of the summer. Rather than standing their ground, they have found it more effective to melt away and reassemble somewhere else. The tactic is a classic one, but I am impressed with the ability of a decentralized group to adopt it so effectively.

And so, everyone makes their way back to the train station, where again things transition to normal, as if a street protest with thousands of participants hasn't just happened. The trains absorb the extra passengers easily (the New York subway would be in flames), and we return to the heart of the city.

Sunday, August 11

The next day there is an authorized protest in Victoria Park, where a dour Queen Victoria in the late years of her reign sits sweltering on a metal throne.

These authorized protests are becoming rare, even though Hong Kongers are supposed to enjoy freedom of assembly. In June and July, the police would routinely grant the permits that the leaderless protests always found a way to apply for. More recently, the police have switched tactics and are now refusing permits, then tear-gassing the inevitable illegal assembly.

But today's rally is official. Demonstrators line the edges of the big open basketball courts and soccer fields, wherever there is a scrap of shade. A first aid station has been set up by the medical volunteers, and free water and electrolyte drinks are available to everyone. The air is still, making the heat worse, and for hours nothing really happens, just a crowd standing and half-listening to speakers rallying the crowd from the stage. A small group of Buddhists has set up incense and performs a ceremony in the frame of a soccer goal.

Anticipating where the real protest will happen feels just like hoping you're one of the people cool enough to be told about the afterparty. You keep one eye on the cool kids, in this case the fully masked protesters. As long as they are still around, we know we haven't missed out on anything.

Finally, around six o'clock, two lines of people form up and start to move. We are on the march.

The crowd shifts and reconfigures to start making its way out of the park. We file past groups of Indonesian women in hijab, sitting on flattened cardboard and having their picnic in the park. No amount of politics is going to deprive them of their precious day off. Queen Victoria looks down at us with disapproval, and then, just like that, we a re an illegal assembly. Speakers at the edge of the park blare out an awful, saccharine anthem as we reach the main road. I put on my hard hat.

It is hard to believe this is the tenth week of protests. The energy and numbers are just astonishing. In spite of the relentlessness of the police, in spite of the beatings from thugs who the authorities have allowed to rough up people with impunity, every weekend Hong Kongers come out to march.

We move in the direction of City Hall, down a thoroughfare lined with Times Square-like animated advertising. The crowd boos and shines lasers on a large animated sign belonging to a mainland Chinese newspaper. Then there is a sudden stop and a sea of hand gestures, thumb and pinky out, hands twisting rapidly. Someone up in the vanguard needs an Allen wrench! This is procured and passed, and then the gesture turns to a "C" shape. They need a monkey wrench!

Finally, whatever barricade full of hex nuts that is impeding our progress gets dismantled, and the crowd flows on through.

The march is entering the administrative heart of the city. It parts for a while to let through some trapped buses and taxis, and then re-forms again, moving toward Central. We are getting close to police headquarters, and soon a decision will be necessary. As we get closer, all motion stops, more consultative huddles form, and twenty or so aimless minutes go by.

Then, a hand gesture. Retreat. The police are too strong here. The crowd heads back in the general direction of the park.

A woman standing on a concrete planter by the MTR station is yelling "go to Tsim Sha Tsui!". People are looking at their phones, checking the latest news. Some of the Telegram groups around the protests have tens of thousands of participants, and information gets amplified quickly. We decide to follow the woman's advice and take the train to Kowloon.

Down in the MTR, it's a different world. Though we have just left a huge protest march, here we are in the midst of families, couples on dates, elderly people, women in saris, white expats in business clothes. The helmets come off and we again join the ranks of the lawfully assembled, the train system effortlessly absorbing a surging crowd that would turn BART into a mass tomb.

At Tsim Sha Tsui station we meet an arriving train full of demonstrators, some already wearing pink-capped respirators. We follow them up and out onto the street, emerging somewhat unexpectedly outside a mosque, where a South Asian woman on the steps is yelling "no photos!". She doesn't mean not to photograph the mosque. Rather, she's yelling at people who are taking photos of the demonstrators from its steps.

We are now in the heart of the tourist district, with hotels and fancy restaurants all around us. If I came here during the day, I would be stopped and asked a dozen times if I wanted my clothes tailored. Tonight, the tailors have stayed home. The street is a sea of black-clad people in masks and hard hats. Word goes around that tear gas has already been fired two blocks away, and I fumble to get my equipment on.

The Persian poets say the nose is the outpost of the face. I am normally proud of mine. Its great bulk has preceded me into every difficult situation in my life, sniffing out both danger and opportunity.

But the mask I bought here is designed for more delicate faces than mine. When I put it on, it somehow channels the air I exhale directly into my goggles, which fog up instantly. I can only see for a few seconds at a time before the world turns into a white mist. But, I reason, not being able to see from condensation will be much better than not being able to see from tear gas. I hang goggles and mask around my neck and await developments.

The crowd has begun to move in the direction of the police. Up ahead, someone is waving a big black flag with the Hong Kong emblem on it. They are silhouetted against a pale cloud of smoke, and as I watch I see streams of more smoke falling in arcs around the flag waver. That is the tear gas.

The people around me are filling bottles with water. Tear gas is kind of a misnomer - it's a solid dispersion that gets deliverd through a burning smoke bomb. You can extinguish it by dousing it in water The kids have a well-practiced assembly line going, delivering water to the front line, and I try to stay out of everybody's way.

Supply volunteers are handing out wet wipes (to get the chemical residue off your skin) and single-use plastic sticks of saline solution for an eye rinse. I am just moving up to get a better view of the flag waver when a tear gas canister lands at my feet, and I suddenly remember an important engagement elsewhere.

When you are in a foreign land with strangers, it is good to have shared interests. They bring people together across cultural and linguistic barriers. Right now, the crowd and I are all deeply interested in moving back a bit. As we go, medical volunteers approach people who have been affected by the tear gas and help them rinse their eyes. "Don't rub." Everybody is tended to. Hong Kong protests have better medical care than any place in America, I think, in another of those little culture shock moments that keep lifting me out of the situation.

For a while, the situation is static. I notice the flag waver has never stopped, defiantly waving right in front of the police line. Word comes that the police have now raised an orange warning flag, which means they will use rubber bullets.

The front line asks for people to move further back, and we comply. It is dangerous to look directly back into the crowd because of the dozens and dozens of green lasers being beamed in the direction of the police (who have taped foil to their visors as a defense).

At one point, I hear a tremendous cheer go up. A group of protesters rushes past with their prize, a tall section of scaffolding they have detached from a nearby work site. The lasers dance over it in celebration. It is rushed forward for barricade duty on the front line. I see other people forming a bucket brigade to pass up water bottles.

Then, for no reason I can see, there is panic. People are running flat out, and I worry about being knocked down in the surge. I catch the briefest glimpse of a woman with no protective gear, dressed normally, who is weaving her way upstream through the escaping demonstrators with a placid smile on her face.

When the running subsides, I duck behind a tree in the median. I don't know what's going on, only that people ran and now have stopped running. Someone says that the police line has pushed forward. Then there is a new sound from up front, the pop-pop-pop of rubber bullets being fired, and people start to run again. "When in Rome," I think to myself, and our little group legs it into the first side street we can find. And there we come across an open snack stall.

"Snacks!" I cry, taking off my helmet.

With tear gas, adrenaline, and the advancing riot police behind us, an appetizing line of skewers is displayed in neat rows ahead of us, in front of vats of steaming broth. The choice is obvious.

I select skewers of indeterminate seafood and wait for them to be heated by the completely unfazed proprietor. We are in little side street full of boutiques, stores and restaurants, many of them still open, the owners standing outside to assess the situation. Other demonstrators are filing past in their headgear deciding which way to go to avoid the police.

I put my mask and goggles in the backpack, and return to the role of ordinary tourist. In this respect, I am far more fortunate than Hong Kongers, who will have to worry about being stopped at random by police on their way home, and being caught with masks and helmets on their person.

I can't get over the oddness of the situation. In one direction is bedlam, in the other complete normalcy, separated by a few hundred meters. Some actual tourists are weaving about, worried, and demonstrators turn them around if they accidentally head towards the tear gas. "

Skewers eaten, we elect to call it a night, and find our way to the nearby MTR train station. The escalator at the entrance is not working, the first time I've seen anything fail to function in this indomitable train system. Underground, the station is packed with demonstrators, and illuminated signs above our heads apologize for the slight disruption in service. Trains are now running every seven minutes, instead of every two, and MTR regrets the inconvenience.

Sympathetic subway workers have opened the gates to the platform, so people can get to the platform without a potentially trackable swipe of their Octopus card (the magic rechargeable card with which you can buy almost anything in t he city).

And then, just as we are descending the escalator to the train tracks, a group of demonstrators rushes in behind us and calling for everyone to run—the police are coming.

This is the scariest moment of the night, as a quick stampede starts in the very confined space of the escalator. I am terrified that someone will trip and get trampled. On the train, people are holding the doors open, and yellow-shirted MTR personnel appear to restore order. Protests are protests, but the MTR must run!

Later that night, we will learn that in another subway station, police fired tear gas canisters directly at people, a terrible thing to do in a confined space. There is video of police beating unarmed protesters savagely on an MTR escalator. And the whole city sees the pictures of a young medic who lost the vision in her eye after police shot her in the face with a bean bag round. The outrage from those acts of violence will provoke the massive protests that shut down the airport on Monday and Tuesday.

But that news is still a few hours away, and again the MTR performs its alchemy of taking us into a world apart, the skyscrapers and hotels around Central, where the view is of tankers and brightly lit junks drifting through the harbor, and there is no sign at all of the turmoil we have just witnessed.

That is the Hong Kong I saw last weekend. I don't know - I suspect no one knows - what will happen tomorrow. The only sure thing is that people will march. There are many Hong Kongers and other smart people writing online who can explain the political context of the protests, the likelihood of intervention, and what this means in a deeper way. I am just a visitor who perhaps thinks too much about fish cakes when being tear gassed.

But I hope everyone stays safe, that the inevitable protests this weekend are allowed to proceed peacefully, and that those in a position to escalate the crisis instead recognize the fundamental reasonableness of what Hong Kongers are asking for—to be ruled by the agreed laws, applied fairly to all. Above all, I hope no one is hurt.

To the many, many Hong Kongers who have shown kindness and gone out of their way to welcome visitors to their city even in these difficult times, I want to offer my deepest gratitude. What to me is something to observe and write about, to you has become a question of survival.

Thank you for letting me join you in your city, and see what you are experiencing at first hand. I know that in America, all our hearts will remain with you, and with Hong Kong, whatever comes next.




All Comments: [-] | anchor

prawn(297) 6 days ago [-]

One of Maciej's best IMO. Informative and very entertaining.

I was struck in particular by: 'I can't get over the oddness of the situation. In one direction is bedlam, in the other complete normalcy, separated by a few hundred meters.'

In 2003, I was walking in Madrid one evening with my girlfriend when war protesters became engaged in some sort of battle with riot police. We had wandered amongst protesters up one street, just taking it all in (up to exciting but short of dangerous!), when suddenly masked protesters came running towards us, gas clouds and popping sounds behind them. We ducked first into the alcove doorway of a restaurant and then sheltered inside at the bar for the evening eating tapas. Within the restaurant, fairly normal dining. Across the road, police with shields dealing with rioters and damaged property.

woutr_be(10000) 6 days ago [-]

I live in HK, and frankly, it's quite strange. I remember having a drink with a friend, while watching the violence unfold live on TV, it felt quite surreal.

I live close to Victoria Park, where most of the large demonstrations have started from, I've seen them pass right down my street as well. All of this happening while I just go on with my life.

sammorrowdrums(4060) 6 days ago [-]

I'm from Belfast. This is exactly the case. There is almost never total constant omnipresent conflict. Nor does it spread randomly. Almost all conflict is highly localised at any given moment.

I imagine even during the war in Afghanistan there were plenty of farmers and families going about their busines while Taliban, local and international forces were live firing mere miles away.

Even soldiers cannot sustain a continuous battle. Wars are a collection of separate battles with a shared end goal.

Aeolun(10000) 6 days ago [-]

This has probably given me more insight in the protests than any news article I've read so far.

Why is it so hard to find a decent source for what is happening?

Joakal(4048) 6 days ago [-]

What do you mean by decent source for what is happening?

There's plenty of sources, including video.

Do you mean live news? There's Reddit, Guardian and more.

Some links:

https://en.wikipedia.org/wiki/2019_Hong_Kong_anti-extraditio...

https://www.reddit.com/live/133sixros7tu5

China Uncensored or Nathan Rich on Youtube if you want to see polarising opinions on Hong Kong (or China). But at least with either, you can check related videos.

Also, beware of Wumao.

kzzzznot(3990) 6 days ago [-]

https://www.hongkongfp.com/

Is a decent (non-CCP controlled/influenced/whatever source)

avip(4065) 6 days ago [-]

Because people like Idlewords are rare. And even once they exist, they don't get the exposure they deserve.

hrktb(4198) 6 days ago [-]

Fundamentally it is not what western media think as "news".

It is long, ultra detailed, first person, and doesn't cover the central point covered by other news outlets.

If a NYT editor was reading this it would cut it down to a fifth of what it is, have it reexplain the 5 demands, give a clearer political positioning etc.

We had that with french protests, standard news covering was cut so it was very digestible and non thought provoking (ton of focus on politics, the fights, injuries and non participant's reactions to the events), it felt like most places removed bits that could be too interesting.

We need both obviously, and I also share your view that more often than not when big event occurs the news coverage doesn't help much to give us a grip about what's actually happening.

albertzeyer(523) 6 days ago [-]

This is the first time I hear about the term 'zeroth world'. A quick Google search did bring up some results, but not much. Wikipedia does not seem to cover that. Of course it covers 1st/2nd/3rd world (https://en.wikipedia.org/wiki/Three-world_model, https://en.wikipedia.org/wiki/First_World, https://en.wikipedia.org/wiki/Second_World, https://en.wikipedia.org/wiki/Third_World). The terms 'developed', 'developing', and 'underdeveloped' are somewhat analogue to that (e.g. https://en.wikipedia.org/wiki/Developing_country). There is also the term fourth world (https://en.wikipedia.org/wiki/Fourth_World). Putting countries (or cities) into these categories is hard of course (and maybe does not make sense).

It seems that the term zeroth world naturally is used to say that such countries/cities are more developed than the first world. More specifically, more developed than USA. Countries/cities like Taipei, Singapore (in this article) or Norway (http://www.chaosnode.net/blog/2018/06/17/life-in-the-zeroth-...) are such examples (you probably can add some more to that list, e.g. Switzerland).

Other articles (https://www.ineteconomics.org/perspectives/blog/america-is-r...) argue that we maybe can just downgrade USA to a developing country (i.e. like 2nd world).

aboodman(3669) 6 days ago [-]

I took the term to be a snarky punch at the '1st world', which asian megacities have long since left behind.

OkGoDoIt(4137) 6 days ago [-]

I think he uses the term creatively, it's not a standard term I've seen elsewhere.

Having spent a decent amount of time in both mainland China and Hong Kong, and living in San Francisco and previously elsewhere in the USA, I can definitely relate. So many things about China feel much more modern than the USA. I get better cell phone service on top of remote mountains in China than I do at my home in downtown San Francisco. Public transit is light years better there as well. Certainly there are things that are worse in China, the main one being air quality, but often coming back to the USA after an extended trip in China or Hong Kong is quite disappointing as I reacclimate to our crappy infrastructure.

vinay427(4209) 6 days ago [-]

American living in Switzerland, and while I'd definitely agree that it's more developed than the US (on the continuum of development in my mind from also spending a lot of time in places most would consider 'developing'), I'm just not sure there's a clear difference between some of those countries named and the US. Many European countries seem around where the US is; some are more developed, while others are probably less developed. I have no idea where I would draw any distinction.

ejefz(10000) 6 days ago [-]

idlewords... this won't have any liberal bias at all.

idlewords(1358) 6 days ago [-]

What would be the more conservative take here?

TASMebWdhWc9NeA(10000) 6 days ago [-]

I am in Verona (Italy) at this time, and I'm in a hotel where some tourists from Shanghai happen to be as well. I'm alone, so I hang out in the lobby a lot. I just engaged in some small talk with some of those tourists, and the discussion somehow shifted to the Hong Kong protests.

Basically what they sad is that western culture in Hong Kong clashes with the chinese culture. They don't seem to say that the chinese way is the right way, or that the western culture is the one to accept. They just say that they are different, and that of course a shift in culture is difficult for Hong Kongers. They think it will all pan out somehow: Hong Kong has to accept that they are Chinese now, and accept all the consequences that come with that.

I understand that the viewpoint of hn, a very USA oriented site, is different, and that most people here think that the libery of HK people should not be taken away from them because democracy is the only way, but I think the chinese point of view should be heard as well, and should be taken into consideration to get a better understanding of everything that is happening.

My personal opinion on this is that China should just let them keep their autonomy, and let them be Hong Kong: a state by it's own with it's own rules and laws.

tombh(2669) 6 days ago [-]

If anyone else is open to the other point of view, I could use some help. I'm struggling a lot at the moment with how little relevance seems to be given to how significant HK was in defining China's modern identity.

How many people actually know how HK came into being? That the supposedly democratic state of Britain who had already violently colonised India, used Indian land and serfs to grow 1000s of tons of Opium to keep Chinese people addicted and thus in sustainable trade. Queen Victoria ignored a letter from China exhorting her to stop. When Britain didn't stop, China took it into their own hands, destroying all the imported Opium they could find. Britain took this as destroying their 'property' and thus went to war with them, easily winning and requiring the handing over of the port of Hong Kong so the trade would not be impeded again.

I know that was over 150 years ago now, but surely that has to be taken into account? If you don't think that's relevant to today's innocent HK'ers, then at least we have to realise that the handing over of HK was a defining factor in the ultimate end of the Qing dynasty, the closing of over 2000 years of China's political tradition and precipitating the radical changes that thrust China onto the world stage as we see it today.

I support the rights of all people to self determination. But HK is not Taiwan, it didn't naturally come to its anti-CCP ideology through an organic, internal and independent process. In fact, somewhat ironically, it came to it precisely because of an unaccountable, authoritarian regime, with no other agenda but self interest.

brendanw(4182) 6 days ago [-]

^^ CCP Propaganda Alert. This account was created 7 days ago and this is its only comment.

spectramax(4070) 6 days ago [-]

Let's replace China with X so there is no xenophobic rebuttals.

X is an authoritarian regime that has no rule of law, president has self declared perpetual status in the office, piracy is rampant, no respect for privacy of others, there is an app called ourchat that is effectively owned by the government and is increasing becoming a necessity, no media let alone any kind of investigative journalism especially against the government, your social score goes down if you buy a particular book, you cannot sue the government or even think about it, punishment can include selling your organs for arbitrary reasons, the list goes on and on.

If X were an impoverished country like Somalia, the tune would change and most people would condemn such a society. I want to do so fearlessly but sometimes people see it as an attack against the Chinese people. I've been to China and have spent many months there, made lifelong relations, etc. I have no room for any concession or bargain for the argument that authoritarian rule has benefits - yes it does but at aforementioned costs. China has risen above due to government's iron grip over every aspect of the country. It is doing so at a cost. Fundamentals don't change even if one sees the strategy panning out. An eagle in the world of doves can kill a lot of doves and have short term evolutionary imbalance. But soon, the marginal cost of turning into an eagle is so small so there are new eagles popping up in the population all of a sudden. This balance oscillates in the short term, but evolutionary pressure returns it back to an equilibrium. Fundamentals of eagle and dove dynamics don't change even though the state of this system shows "success".

I'm in the position to criticize any authoritarian regimes in the strongest way possible - be it China or any other country, it doesn't matter. I don't want to die seeing this world turn into a power grab for a few with a consequence of a dystopian society. I wish the next superpower would be a country such as Norway or Sweden, it would set such a utopian example for the world to move into the right direction.

raxxorrax(10000) 6 days ago [-]

I think the western society is objectively further advanced than the Chinese one. Not surprising since China is a country in development and most other countries have gone through similar phases. So it should not be taken as a accusation. This conflict aside, I wouldn't see them on the same level.

roca(10000) 6 days ago [-]

The idea that democracy and freedom are somehow foreign to Chinese people and can't work in their culture is a myth peddled by the CCP to serve their own interests. Democracy and freedom are flourishing in Taiwan. They are flourishing in South Korea (who aren't Chinese but whose traditions are no more democratic than China's). They're doing OK in Japan.

'The Chinese point of view' is an ambiguous term that the CCP uses to its advantage. In fact it is the Chinese Communist Party's point of view, which they have managed to indoctrinate into most Chinese citizens via their control of all Chinese media. Where Chinese people have escaped CCP influence, they tend to have quite a different point of view.

Asooka(10000) 6 days ago [-]

Basically, Hong Kong has lost its roots after decades of British occupation and forceful western indoctrination. Hong Kong is China, there are no two opinions on that. The domestic terrorists currently raising hell are misguided and work against their cultural interest. I do not doubt there will be a massacre and many will die, but this will be good for China and good for global economy.

nailer(428) 6 days ago [-]

> Hong Kong has to accept that they are Chinese now, and accept all the consequences that come with that.

HKs independence was agreed to by the PRC as part of the handover.

The PRC must accept there are two systems.

gwathk(10000) 6 days ago [-]

Go Hong Kong visit sometime. You will find British has managed to build Hong Kong into a prosperous and peaceful society with all the elements of so-called Chinese culture and Western culture. There is a church next to a mosque next to a temple in Tsim Sha Tsui district. People enjoy yum cha in their lunch and America rib-eye steak in their dinner.

I wonder whether the Chinese viewpoint stands. It is purely bad governance of HKSAR and CCP.

seppin(10000) 6 days ago [-]

> Basically what they sad is that western culture in Hong Kong clashes with the chinese culture.

Funny that authoritarianism, corruption, etc. is being excused as 'Chinese cultural differences'. This stuff ain't new, folks.

xkgt(4025) 6 days ago [-]

//Basically what they sad is that western culture in Hong Kong clashes with the Chinese culture.

This is exactly a regurgitation of the propaganda by CCP. They blamed western influence, education and cultural whitewashing of HK Chinese as the reason for the current protests and their citizens believe it sincerely. They conveniently overlook the crux of the protest, which is to ask for universal suffrage.

Even for a well educated, widely traveled main-lander, it is difficult to come out of this conditioning. I have some friends from mainland, currently staying in HK, who sincerely believe the general public is too naive to be allowed to make any decision. With the right amount of conditioning, people can be led to believe in anything even if it is contrarian to their well-being.

nesadi(10000) 5 days ago [-]

The Chinese point of view is to take HK culture and remove it, because the Chinese way is the only way. You said it yourself. I'm not sure how that's okay or what greater understanding is supposed to yield. China wants to subsume HK at any cost. HKers want to keep their culture, rights and freedom. They're going to fight for it, and China is going to crush them and force their will on them. I don't see the good here, or how a 'fair' evaluation of the situation will change these facts.

oinkl(10000) 6 days ago [-]

America's Fundamental Misunderstanding of China

https://www.youtube.com/watch?v=Ojr-tqaQQOQ

abcdefghijklmn3(10000) 6 days ago [-]

Fu Guohao was severely beaten by black shirt protesters at the airport for allegedly being a mainland spy and a police plant. Why did the protesters do that? Why did they take away his freedoms? Why do these current narratives gloss over that fact?

You can accuse me of whataboutism, and I can accuse you of indulging in propaganda.

I can't imagine getting away with shining a laser in a police officer's eyes, let alone slinging bricks or molotov cocktails. The HK police are arguably kind.

smcl(3906) 6 days ago [-]

This comment is by a green-name account, isn't responding to anyone in particular and has that online-commenter-army feel to it.

Genuine question for HN: what's the right term for this sort of comment, or commenter?

Joakal(4048) 6 days ago [-]

China has a reputation of dealing with things privately, so there's a lot of paranoia and worry from protestors.

Second point, the escalation by protestors, the government won't fulfill the protestor's demands. I can only say this by John F. Kennedy: 'Those who make peaceful revolution impossible will make violent revolution inevitable.'

spyckie2(10000) 6 days ago [-]

This piece is quite accurate - it really makes you feel like you're in Hong Kong during the protests.

References aside, as a US citizen in Hong Kong there are some things that I respect immensely and some things that really get on my nerves about HK.

I greatly respect the people and the cause. It reminds me of that one poem -

> Do not go gentle into that good night.

> Rage, rage against the dying of the light.

The people here have a heart to fight for their own identity and for their well-being. It's actually really amazing especially with the immense challenges facing them and the great risks they have as a people.

At the same time, when two parties disagree in Asian culture, it rarely turns towards resolution. Most of the time it turns towards silence (separating, parting ways, or just pretending it never happened), violence (intense arguments, passive aggressive pay back, hatred and villainfication of both sides leading to all out war), or just a lot of stiffness / unwillingness to compromise, understand the other side or reach a deal.

This reflection is not just about the protests, although the escalation is due to the culture being this way. It happens everywhere, the biggest pet peeve of mine is that it's a normal, accepted practice. For instance, if you don't like your boss, you don't say anything, you just hold it in and then you send in your resignation. I wish the culture would be more willing to engage in conflict resolution type conversation and learn how to do it. It takes a lot of practice on both sides to do it.

davedx(2636) 6 days ago [-]

What exactly is Asian culture?

rusk(3831) 6 days ago [-]

> when two parties disagree in Asian culture, it rarely turns towards resolution

This sounds a lot like politics in the UK and USA to me ...

spyckie2(10000) 6 days ago [-]

I don't want to change my original post, so I'll add to it here:

When you're on the streets with the protests, it feels extremely primeval. The micro-scale interactions of the protest movement is like looking into a microscope at microorganisms, blobs of life that just kind of move around and sometimes eat others but most of the time just float there.

It's a decentralized, rather polite mob. Angry, yes. Scary, sometimes. But otherwise, just a lot of yelling, a lot of 'tactics' - like retreat, advance, wall off here, go here, go there, run. It feels like there's no real goal or direction of the protest except to exist, and I think the author captures that feeling very, very well.

The police also have that same feeling on a micro-level, not much thought or control, just an instinctual reaction or set of rules to follow.

One party existing to express concerns and vent anger. The other party existing to restrain and disperse.

Of course, on the real level, the two parties have deeply rooted goals and feelings. The police want the protestors to stop protesting. The political party wants the people to be absorbed into China. And the protestors want to preserve the unique individuality of HK and to allow it to grow and thrive.

spacehunt(10000) 5 days ago [-]

> I wish the culture would be more willing to engage in conflict resolution type conversation and learn how to do it.

It's rather difficult to engage in conflict resolution when the other side refused to listen for at least the past 22 years...

I do agree with you, but it takes two to tango.

roboys(10000) 6 days ago [-]

Am I the only person that believes international crime will see a significant drop if China controls HK?

I'm actually glad this glorified money laundering/financial crime haven is being shut down. The sooner China gets extradition powers, the sooner we can close the criminal enterprises operating in/through HK for the past 1-2 centuries.

Running drugs/drug money through HK is something that has been going on for a very long time (thanks to the UK/US/etc), time to put an end to it.

Lot of big money manipulating kids that can't see through the haze neo-colonialism.

lawrenceyan(1295) 6 days ago [-]

Growing up in the suburbs of California, I never really understood just how central the concept of face was in Asian society until visiting China and seeing it for myself.

Over time I've come to realize that many of these social concepts exists in some form or another across all cultures, likely implicit in some part to the human condition, but it still surprises me to this day just how entrenched it still is in certain areas.

I've always been personally curious as to what the causes of these divergences in culture between different societies are. There must be some inciting reason that such a stark difference in communication exists.

smallnamespace(10000) 6 days ago [-]

This is a silly analysis because the conflict is fundamental. China wants to integrate Hong Kong. The protestors want independence. Neither side really wants an indefinite half-hearted autonomy, although the protestors are pushing to preserve it for as long as possible.

Your very reaction shows where your sympathies lie, which is very laudable, but are you willing to engage in 'conflict resolution' over your own core values, e.g. to give up some amount of democracy or independence? If not, then talk is simply window dressing, or an opportunity to spread your message.

Both sides here realize there is no true basis for negotiating, hence why they don't.

In fact, negotiation at all by Beijing would be a concession: it would legitimize the protests and give protestors a formal seat at the table. You may disagree with that starting point, but let's admit that negotiations themselves are not a neutral act devoid of consequence, hence why actual diplomats spend enormous amounts of time discussing the agenda to be discussed in formal settings: what gets talked about and by whom matters.

unityByFreedom(4199) 6 days ago [-]

> when two parties disagree in Asian culture, it rarely turns towards resolution.

This is not a fair or accurate representation of any group of Asians, especially not democratically governed areas. There have been plenty of win-win resolutions. China today is much more connected with the world than it was 50 years ago, and it's better off for it.

edit please explain your downvotes, thanks.

xster(4197) 5 days ago [-]

This is a great point. Also thanks for being neutral on this topic, especially given how hard it is to do so on Hacker News these days on this topic.

I'm typically highly critical of America but I think Asian cultures have a lot to learn from the US resolution of the civil war and general conflict resolution between parties of unequal power. One can easily reference the uneasy outcomes of Asian civil wars that didn't end in re-homogenization (such as Japan).

My running theory is still that the millennia-steeped culture is rooted in geographic dispositions and historical modes of production. In derivatives of mediterranean trading cultures like minoans, pheonicians, greeks etc, local production is not self-sufficient and equality, contracts, conflict resolution is your 'means of production' and, to put in a controversial idea, what social darwinism selects for. In continental/big plains/big rivers agrarian societies like sinocentric societies or even Egypt, unity, mass labor, hierarchy is their means of production for flood control, irrigation and other agrarian projects. Questioning traditional wisdom/methodology, parents and trying to get creative with how you plant your crops is an easy way to get yourself starved and what social darwinism selects against.

brendanw(4182) 6 days ago [-]

A few years ago a Chinese man on the BART tried asking me for directions. His english was poor. With the aid of google translate we were able to do some basic communication. Over the course of 10 minutes I learned he was trying to visit the Wikipedia headquarters. I mentioned I was surprised he could access Wikipedia in the mainland. He told me that many know how to access Wikipedia.

Let's be clear. Authoritarianism is not culture; it is an evil political ideology that the people of China are subjected to.

woutr_be(10000) 5 days ago [-]

There's honestly quite a lot of misinformation about how people live their lives in China. Many of my friends are perfectly capable of accessing blocked websites, and they're by no means computer experts. They all have been able to travel because of their increase in wealth, and most of them are in contact with foreigners from all over the world. To them, they live in a society with restrictions, but they've all benefited immensely from it in the past decades.

ausbah(3971) 5 days ago [-]

The tragedy is things like sanctions against authoritarian regimes do nothing but to hurt the average citizen. The we'll connected in such regimes have the means to evade sanction, the lower classes don't. Look at Iran, North Korea, and other countries the US has sanctioned to kingdom come - there isn't nearly as much pressure as there would be in a western state.

aboodman(3669) 6 days ago [-]

Maciej, this is amazing. thank you for posting it.

idlewords(1358) 6 days ago [-]

Thank you!

dade_(10000) 6 days ago [-]

The Hong Kong government/China have been completely inept at managing the housing situation and it is nearly at a breaking point. It is refreshing to read an article on the current situation that raises the issue.

'One of the deeper causes of the present crisis of legitimacy is the housing crisis in Hong Kong, another way in which the government has failed the people it is meant to represent, and you can see it in the extraordinary density of apartment buildings, each unit pock-marked with an air conditioner, tiny living spaces with some of the highest rents in the world.'

mback00(10000) 6 days ago [-]

But this is how the Hong Kong government has traditionally made its money. Hong Kong's tax structure is real-estate sales. This is why Hong Kong is all about huge sky rise developments of ultra-small apartments... and because the island is all hills, it is also why so many developments are built literally on a cliff.

Hong Kong is a geographically difficult place... building new plots is expensive - and the government makes it more expensive.

avip(4065) 6 days ago [-]

Interesting to contrast this with popular demonstrations in US or France. No windows broken? Burnt city infrastructure? No beaten up police? All so quiet.

namelosw(10000) 6 days ago [-]

I'm from China, I'm actually shocked to see people in the US or France do that all the time. Usually, it seems private properties are destroyed during the process.

But why? If it is the government people protest, but why do people break shop windows and burn someone's car all the time...

It seems to be a lesser kind of terrorism for me: they don't like a country, they hurt properties instead of people.

nathanyukai(4203) 6 days ago [-]

In the Chinese media the violent part of the protest is what they mostly talks about, although I do believe most protestors are peaceful. Funny enough most western media don't talk about that at all, despite that they broke into the legislative assembley and beaten people from mainland China, and HK police of course. Isn't media great these days

woutr_be(10000) 5 days ago [-]

I come from Europe, and every time someone brings up the protests to me, all I can say is, this is nothing compared to what happens in Europe. Not to downplay what's happening, but I think in general HK people are not used to violence, it's incredibly rare to see any physical altercations. I've been in verbal arguments here before, and once people see you won't back down, or you're willing to escalate, they'll quickly back off.

That's why now you see the slog 'HK police is attempting to murder citizens', even though nobody has been close to dying. It's just that they perceive violence on a different scale, and people being beaten up (not even severely) is enough to claim they're trying to murder people.

Joakal(4048) 6 days ago [-]

Across the sea from Hong Kong, the Japanese take demonstrations a step further: https://www.youtube.com/watch?v=eXjd7GkHKfU

jnaddef(10000) 5 days ago [-]

Well if you look at French media, you will believe French protesters are just there to break everything. If you look at Chinese media, you will believe Hongkongers are just there to break everything.

If you actually go to a protest in France you will realize it is pretty much the same as what is described in that text, just people trying to protest but met with unbelievable violence from police.

mytailorisrich(10000) 6 days ago [-]

They seem to have spared private property, but windows were broken, city infrastructure was damaged (e.g. LegCo, police stations), and police were beaten up.

It's like in all countries. Some protests/protesters are peaceful, some are not.

nbevans(4157) 6 days ago [-]

'All that prelude is to say, coming in to the Hong Kong protests from a less developed country like the United States is disorienting.'

This sentence got me hooked. It must be so alien to the average American that there are more developed countries/cities out there.

mensetmanusman(10000) 6 days ago [-]

More developed mostly means 'newer.'

These places will also have to figure out what it means to keep infrastructure refreshed in 50 years. It is always a struggle..

macspoofing(10000) 6 days ago [-]

I wouldn't put Honk Kong in that list though as much as I like the people and the city. I've been a few times and the infrastructure looks like it's falling apart. Electricity wires are hanging haphazardly, alleyways are dirty, and most facades of buildings are straight out of the third-world. The contrast between high-end boutiques interspaced with tiny mom-and-pop shops selling second hand stuff is a little surreal.

hrktb(4198) 6 days ago [-]

> The MTR is the one technology the Hong Kong protests could not do without, an autonomous fiefdom that the police mostly stay out of. It is neutral territory

This is a godsend. During Paris protest earlier this year the gov would just shut down public transports for about 5~10 km around and block main accesses so people couldn't massively join the protests.

At least the mainland gov. didn't get to touch that I guess.

stringyham(10000) 6 days ago [-]

> During Paris protest earlier this year the gov would just shut down public transports for about 5~10 km around

10km is nearly Paris' width so I'd say that's a bit much. If I remember correctly, they closed the main stations around the protests but not much more.

woutr_be(10000) 6 days ago [-]

Some stations have been shutdown several times in the past few weeks, I'm not sure if it was on the request of the police, or the MTR just decided to skip stations. However, this wasn't due to people going to protests, this was due to increased violence, and even attacks inside the stations.

Last week the HK police decided to fire tear gas within a station as well.

baud147258(2832) 6 days ago [-]

If you close public transport 5~10km around the protests, there wouldn't be much public transport operating in Paris. Rather there were a handful of closed subway stations, around where the protests were (like Etoile was closed, but not the closest stations). I understand the idea, a stampede in subway corridors would have caused casualties.

abcd_f(4140) 6 days ago [-]

Exactly what I was thinking. China can simply shut down the public transport system next time and that will do it. No violence, no tear gas.

unityByFreedom(4199) 6 days ago [-]

HK has come quite far, but if they can't stage peaceful protests without fear of reprisal from Beijing, then the brightest will still flee to the US.

This blog post's point seems to be, he had culture shock when he moved to the US as a child, and experienced it again when visiting HK because he feels it is more advanced than the US.

> Not everyone lives in a luxury hotel, man! I get it. But my eyes are like saucers. I ask forgiveness of Hong Kongers if at times I am still that six year old kid, dazzled by what to you is ordinary. You live in a kind of city we Americans can only aspire to, and it's no wonder you love your home so much you will take any risk to save it.

mytailorisrich(10000) 6 days ago [-]

I believe quite clear that they can stage peaceful protests. Peaceful protests have never been prevented in Hongkong.

ETHisso2017(3398) 6 days ago [-]

Hmm. I wonder how the protesters know who to trust in their Telegram groups and on LIHKG. There has to be a core of organizers, however anonymous they may appear to be.

Is there a way to track metadata exhaust from telegram for building a social graph or xss lihkg? Could be a way to ID them.

WilTimSon(10000) 6 days ago [-]

I can't find the proper link right now but there was a case of a 'journalist' with no press pass going around the HK airport when it was occupied by the protesters. He took people's photos without permission, tried to quickly get info out of others, etc. It's referenced in passing here, it seems: https://www.theguardian.com/world/2019/aug/14/hong-kong-flig...

I'd wager it's the same in their group chats. Someone's talking too much and seems too eager to find out details or incite violence? That's very suspicious. I don't think the mainland forces are trained for infiltration of this kind so they're messing up a fair bit.

loyukfai(10000) 6 days ago [-]

LIHKG's resilience in the current crisis is truly remarkable.

That being said, it's not politically-oriented at all beforehand... it's something like the Reddit or 4chan of HK whereas most people just 'shit-post' most of the time.

_petronius(3683) 6 days ago [-]

This is some serious "too interested in seeing if you could, and not stopping to wonder if you should" content. What possible positives are there in exploring how to de-anonymize people whose life and freedom requires anonymity, and who are fighting the good fight?

PikachuEXE(10000) 6 days ago [-]

Judge by speech AND actions Also keep a clear mine and use independent thinking Put faith in actions not a person/group

kerng(2600) 6 days ago [-]

When nearly half the population participates, I dont think you need much ID'ing.

odiroot(10000) 6 days ago [-]

> "It's okay," I tell them. "This is normal. I'm not dying—I'm Polish." They edge away.

Funnily enough, myself being Polish, I found HK climate much more preferable than the Central European. I dread every end of summer; every winter kills my energy and work performance a bit more.

throwaway1997(10000) 6 days ago [-]

Even us locals suffer during the summer. It easily reached 35C with up to 90% humidity.





Historical Discussions: I've reproduced 130 research papers about "predicting the stock market" (August 16, 2019: 700 points)

(700) I've reproduced 130 research papers about "predicting the stock market"

700 points 5 days ago by starpilot in 1935th position

www.reddit.com | | comments | anchor

Considering the last couple of years there's been tons of marketers pushing their affiliate links for ProQuant and Trading212, and the MT4 bridge was working until recently, there is a deafening silence from all sides on people's actual results with ProQuant. Probably ProQuant could be pushed under EU law to disclose broker-like statistics '85% of customers lost money', but it would be nice to hear from someone, anyone? A couple of the marketers I found were running on demo, which is obviously a total WTF LOL, but even demo results seem to be missing.




All Comments: [-] | anchor

lidHanteyk(10000) 5 days ago [-]

This is unsurprising. P likely is not equivalent to NP [0], and predicting the market is NP-hard [1]. It's nice to see empirical work in the field, though, and especially nice to see reproductions of published papers.

[0] https://www.scottaaronson.com/papers/pnp.pdf

[1] https://arxiv.org/abs/1002.2284

Edit for downvoters and repliers: If enough market participants are irrational, then it can still be possible for people to predict other people, instead of the market, and make money that way.

NP-hardness indeed doesn't rule out heuristic approaches, but experience with 3-SAT and other NP-complete problems suggest that there will be arbitrarily bad times, and that in those times, the amount of loss can be exponential in the length of time that the heuristic poorly predicts the market.

dvt(1699) 5 days ago [-]

Philip Maymin seems like a serious guy... but that EMH ↔ P=NP paper is absolutely not even remotely a proof. Was genuinely very curious and it's at best an intuition. Some claims, e.g. Knapsack and 3SAT are (almost?) isomorphic to the efficient market hypothesis, are pretty bold. And the justification is hand-wavy at best.

throwawaywego(10000) 5 days ago [-]

You are talking about writing an algorithm that has a 100% accuracy on an NP-hard problem, and taking the impossibility of this to discard an approach that may yield 58% accuracy.

lucasmullens(10000) 5 days ago [-]

What do you mean? People are successfully making millions or even billions on predicting the market. Seems like a stretch to relate this to P=NP

DennisP(3463) 5 days ago [-]

Your second link doesn't say predicting the market is NP-hard. It says the opposite: that the market is only efficient if P=NP.

According to the paper, if you don't believe that P=NP then you believe that the market is inefficient, which means there's profit to be made. The paper even suggests how.

IngvarLynn(10000) 5 days ago [-]

NP-hardness is irrelevant here because algotrading is not 100% formalizable.

zazagura(10000) 5 days ago [-]

Warren Buffet must have a secret proof of P=NP then, that would explain how he has made billions trading.

starpilot(1935) 5 days ago [-]

Spoilers:

> Literally every single paper was either p-hacked, overfit, or a subsample of favourable data was selected (I guess ultimately they're all the same thing but still) OR a few may have had a smidge of Alpha but as soon as you add transaction costs it all disappears.

> I should caveat that I was a profitable trader at multiple Tier-1 US banks so I can say with confidence that I made a decent attempt of building whatever the author was trying to get at.

albertshin(10000) 4 days ago [-]

Discredits a whole field -- doesn't post any papers that he attempted to replicate.

DennisP(3463) 5 days ago [-]

But also:

> Almost every instrument is mean-reverting on short timelines and trending on longer timelines.

i.e. he confirms the momentum factor, which isn't surprising since there's more solid evidence for it than anything else, going back hundreds of years.

He doesn't say what fundamental factors he looked at, so it's possible that value, size, and profitability/quality would hold up as well. All those have been studied pretty extensively in academia, in papers going back decades. The author took only a fairly random sampling of recent papers.

paulpauper(254) 5 days ago [-]

Literally every single paper was either p-hacked, overfit, or a subsample of favourable data was selected (I guess ultimately they're all the same thing but still) OR a few may have had a smidge of Alpha but as soon as you add transaction costs it all disappears.

I could have told you that without testing. If anyone had a lucrative strategy would they disclose it in a paper to the general public? I think not.

mruts(10000) 5 days ago [-]

CAPM, Fama French Three Factor, or five factor? Are you serious? They don't work anymore but they did at one point. The foundation of modern finance is built on published Chicago school papers.

lucasmullens(10000) 5 days ago [-]

Proof can be more useful than a hunch.

lettergram(1465) 5 days ago [-]

I had the same result from most papers. My personal conclusion is that when people get a winning strategy they don't publish. I personally put my money where my mouth was for a few years:

https://austingwalters.com/backtesting-our-100-yoy-profit-ge...

That being said, I try to be honest too. This can disappear any time and the model I use may only be good in this environment. I do not know. I think that's the challenge with papers, you don't honestly know when or if the strategy works. It clearly won't forever regardless.

That's why I don't share my exact method. And after doing all the research myself AND trying to sell my algorithm. I honestly don't think the industry knows what it's doing either. People are worried about sharpe ratios and all this BS stuff. The reality is for these models you mitigate risk via temporary and ever changing methods. Can't really publish on that.

trilila(10000) 5 days ago [-]

Winning strategies are not published generally speaking. You wont see many business people blogging in detail about how they went for 0 to 1. Which is shy most blogs about business are a load of BS. Reading your statement and this article confirms my long suspicion that relying on internet strategies is a bad startegy.

p1esk(2489) 5 days ago [-]

If your algorithm was successful, why did you try to sell it?

mtgx(152) 5 days ago [-]

Would this work on cryptocurrencies, too?

arnioxux(10000) 5 days ago [-]

Is your 'winning' strategy so different from other social data / sentiment analysis approaches? There is some novelty with how you weigh the sentiments (based on how much of an insider or expert they are) but I am sure existing trading strategies weren't just taking a dumb average of a twitter firehose either. Shouldn't it be easy for some large firm to replicate your approach and make the alpha disappear?

dvfjsdhgfv(2539) 5 days ago [-]

> I had the same result from most papers. My personal conclusion is that when people get a winning strategy they don't publish.

This is kind of obvious to me. It is also the reason the OP posted their results. I'm sure if they found one strategy that worked, after putting that much time into their research it would be really stupid to announce to the whole world that it works.

On the other hand, in the trading world where everyone is a competitor, you might want to deliberately introduce some confusion - but it looks like plenty of actors are doing this anyway.

vkou(10000) 5 days ago [-]

I hope that your personal implementation of your strategy takes into account 2008 ;). Because, hooo, boy, your 20% YoY returns sound great, but may not be taking enough risk of a similar collapse into account.

hobbescotch(4050) 5 days ago [-]

It seems that you consider the sharpe ratio to be not worthwhile. Would you be able to elaborate on that point? As someone who is getting into algotrading, I'm currently using the sharpe ratio to quantify risk, but would like to hear another take on it.

riku_iki(10000) 5 days ago [-]

TLDR, no results, no code, no details, just admission of failure from anonymous redditor with SEO link on his unrelated crypto-trading project.

mxcrossb(4006) 5 days ago [-]

It's baffling to me that anyone would believe this post. Do people here not have an ounce of skepticism?

dvt(1699) 5 days ago [-]

Why is this getting downvoted? The original post is clearly marketing clickbait. It's trivially true that most 'predict the stock market' papers are going to be bunk.

1e-9(10000) 5 days ago [-]

In my view, the predominant mistake made by those who seek to create profitable strategies is that they approach trading as if the market is a zero-sum game. In particular, doing things that harm the markets, like naively adding to existing momentum, is just promoting price overshoot and instability by reinforcing positive feedback loops. Such approaches hurt others and while they might make money for long periods of time, they will almost surely end up losing all that profit and more during a small number of extreme market events.

If you want to be reliably profitable, you need to first understand how the markets are not a zero-sum game and then you need to construct methods to improve the markets with your trading. There are countless ways that markets deviate from truly efficient behavior. Find some and develop strategies in areas that can benefit from your cognitive, experiential, and educational strengths. General examples of how one might improve the markets include things like providing liquidity when it's needed, limiting price overshoots when it's warranted, and incorporating new information about instrument values. The market will pay you in return if you do such things in a sound way. As long as you also do a good job of estimating and limiting your risk, you can be consistently profitable.

There will be no significant public disclosures of detailed ways to trade profitably. The markets rely on the robustness provided by many different points of view addressing market needs in a variety of ways. Any parties that overweight one of those points of view will ultimately lose money in the process of adding market instability. There's too much of that already. Don't trade until you figure out how to make the markets better.

shostack(4198) 5 days ago [-]

What are good resources for reading up in very granular executional detail how others have made markets better historically with strategies that presumably are no longer delivering alpha because they've run their course for whatever reason?

elmersglue(10000) 5 days ago [-]

This is fantastic advice.

anonu(2607) 5 days ago [-]

> you need to construct methods to improve the markets with your trading

What percentage of traders do you think will approach the market with such altruism?

I think the markets are way too complex to rationalize 'improving a market'. Markets are by definition good markets when you have long term and short term traders mixed in with technical and fundamental traders all with different alpha time horizons. This is a healthy market.

If you're a speculator, so be it. As you point out, the market may teach you a lesson at some point. Those guys go away but new ones will join.

It's a beautiful virtuous cycle.

Not to say there aren't bad actors. Most speculators are not IMHO. But they cross the line when they undertake certain market activities, like spoofing for example. This is why good markets also have good regulatory oversight (finra, sec... )

rolltiide(10000) 5 days ago [-]

yeah but you can make a lot of money preaching technical analysis to your congregation a few times per week

protip: a fibonacci retracement from a randomly selected extreme will always tell you something

protip: it takes 5 months for your congregation's account to get eaten up from transaction costs when their stop limits keep getting hit

in the mean time you can just play TA roulette and they'll always be impressed by your "uncanny" perceptive abilities

throwawaywego(10000) 5 days ago [-]

If you know that 5.6% of the users (automated, or parroting the preacher) of an exchange will use fibonacci retracements, and that 15% of the amateur market will follow the market price change caused by either buying or selling activity, then you can play roulette with a decent edge. Of course, not as much as the preacher, who is allowed to bet before (s)he will speak to his or her congregation.

When you gather enough of these commonly used technical analysis, it's like having to predict in which startup Ron Conway will invest, but you can calculate Conway in a Python one-liner, and keep up-to-date by going to weekly sermon.

omarhaneef(10000) 5 days ago [-]

How did you do 130 papers in 7 months? That's just over a paper every 2 days.

What was the setup, how did you set up a pipeline? Was it R or Python? What was the data source?

I am more surprised by your productivity than anything else.

bwilsonkey(10000) 5 days ago [-]

I don't think he actually did it. He linked to a crypto scam medium post lol...

bcherny(3706) 5 days ago [-]

It was answered in a comment in that thread. Python+Pandas+Keras.

I'd also love source code + data for this; without it, it's a claim with nothing to back it, yet.

H8crilA(10000) 5 days ago [-]

There was a fair amount of overlap in the papers. Makes the testing much easier.

> So with the papers, I found as many as I could, then I read through them and put them in categories and then tested each category at a time because a lot of papers were kinda saying the same things.

scarmig(2804) 5 days ago [-]

Wouldn't any given approach rapidly lose efficacy as soon as its published?

I would even guess that a paper being published means that, at the point the paper started to be written, its alpha had already decreased to zero. Otherwise the writers of the paper would still be using that approach. That's how it can appear to provide no value even if you extrapolate it back in time.

SubiculumCode(3536) 5 days ago [-]

The author mentions this, and said he tested for 'alpha decay' by applying method to datasets that preceded the data on which the model was tested/trained.

WheelsAtLarge(2997) 5 days ago [-]

I could bet a ton that most people will make excuses as to why the papers failed. There's something within us that wants to hit the stock market lottery.

I truly believe that there are streeks to profits in the stock market in the same way you will find streeks in any set of random numbers but they are impossible to find in a consistent manner. The road to wealth for most in the stock market is time and investing in a basket of good stocks.

Whoever thinks that they have found a system to profits in the stock market. Test and retest your method a few times. It's unlikely you have a winning system.

tomp(2363) 5 days ago [-]

No, there are trading strategies that do work, over multiple years, and can then be adjusted and refined to work even longer.

But obviously nobody is going to publish such strategies. You're looking at a negative selection of papers.

bjourne(3197) 5 days ago [-]

But there is a foolproof way of profiting from the stock market. Insider trading! I find it fascinating that people do not believe it occurs at a grand scale given the low risks and huge rewards. Exactly like how people believe athletes don't use steroids so they get all upset when every once in a while one is caught. :)

bitL(10000) 5 days ago [-]

Hasn't there recently been shown that only strategies using simple momentum-derived technical indicators were able to consistently bring returns in the stock market?

DennisP(3463) 5 days ago [-]

Not just recently, it's been known for a while. The author actually confirmed it:

> Almost every instrument is mean-reverting on short timelines and trending on longer timelines. This has held true across most of the data that I tested.

(But momentum isn't the only thing that appears to work.)

edwardy20(3643) 5 days ago [-]

Professional quant here. I have to say I strongly disagree with the conclusions of the OP.

> They were all found by using phrases like 'predict stock market' or 'predict forex' or 'predict bitcoin' and terms related to those.

Yeah, searching for any finance papers with 'predict' or 'machine learning' is literally the lowest quality tier you can get. These papers are often written by grad students who can pump an easy paper out by 'applying' some already known ML algorithm to financial markets. Of course it's not gonna work. It also kills me when I see ML models who need stationarity assumptions applied to non-stationary time series data. Yeah, good luck with that.

THAT being said, there is lots of high quality research which has been replicated over and over, showing that alpha does exist in the market (and which funds have made billions off of). I would like to see the OP try to replicate some of these instead. To give some simple examples:

1. Try searching for papers with the keywords 'and the cross section of expected returns'. For example, the momentum factor which can be tested and replicated with only linear regression. > There is substantial evidence that indicates that stocks that perform the best (worst) over a three- to 12-month period tend to continue to perform well (poorly) over the subsequent three to 12 months. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=299107

2. Statistical arbitrage strategies which were known to work well until the mid 2000s. Also been replicated many times, furthermore, you can see the gradual decline in profitability pointing to the theory that 'alpha decay' in this case is real. https://www.math.nyu.edu/faculty/avellane/AvellanedaLeeStatA...

3. High frequency strategies. No way OP or any retail trader can replicate this, but firms make billions of dollars per year consistently doing this.

In conclusion, to make a claim that there is no alpha in the market seems highly suspect, and perhaps just needs a more nuanced view of how trading firms make their profits.

gillesjacobs(10000) 5 days ago [-]

I also find it highly unlikely anyone is able to implement 130+ papers in 7 months.

This would require insane productivity, implausible access to pricing and news data resources (which are often not freely available) and expertise in machine learning, natural language processing, finance, and data science. OP had to implement financial, time-series and linguistic feature engineering pipelines, as well infer the architecture and hyper-parameters used AND train all these models.

He also claims he 'web scraped' all the data which is highly unlikely as pricing datasets are often sold for a pretty penny and not publicly available in the detail described in several of these papers.

OP must be a genius to pull this off, all the while being a trader at 'a Tier 1 US bank' (in itself that description is ridiculous).

All OP has to show for all this work is a hastily written Reddit post with dubious claims. There is no proof of the work done whatsoever, no code samples, not even result tables or graphs. And at the end OP chills his cryptotrading bot.

What's worse HN seems to gobble it up naively. Seemingly because OP is critical of something that is popular to criticize.

janny_kul(10000) 5 days ago [-]

Hey, at no point did I made a claim that there's no alpha in the market. I generated around 25% annually myself on a fairly large balance sheet and made reference to rentech's stellar performance several times. AND I'm working on a new commercial project which relies on finding alpha. but yes the rest of your comments are valid and fair.

Also, for the other comments I edited the post to add links because I was asked several times what I was up to now.

anonu(2607) 5 days ago [-]

Most academic white papers on trading strategies are horrible. There are written by academics or grad students (as you point out) that have no firm understanding of the actual market mechanics. You can often tell this right away just by the language and terminology used... Even before you get to the actual math or strategy.

Nobody will make any money in the markets relying on others work. It's just how arbitrage works. I've you want to succeed you need to be creative beyond what's already been done.

Once you're there you should follow the first rule of trading:

Don't talk about your strategy.

The second rule of trading is the same as the first rule.

I cannot emphasize enough how much you need to keep things secret in trading.

retiredcoder(10000) 5 days ago [-]

I have seen the same stationary overlook on other fields and always blows my mind that people involve would get away with that.

1e-9(10000) 5 days ago [-]

Actually, I think the author is in agreement with you that there is alpha. He apparently wrote this article to attract attention to his scheme for selling access to his cryptocurrency trading 'bots', which he claims are guaranteed to be highly profitable once he finishes developing them. Certainly, his survey is not credible and does not appear any more genuine than his business offering.

hedora(3772) 5 days ago [-]

This is a really informative comment, but the OP explicitly excluded the three strategies you mentioned. (There's a blurb about ignoring "alpha" strategies, and he only went back 8 years).

I read this article more as answering the question "Did anything useful come out of the last few batches of finance PhDs?", than "Are investment strategies are totally futile?"

paultopia(4212) 5 days ago [-]

Interesting but it would be nice if the author would, you know, write up his/her own detailed analysis with replication steps and post on arxiv or something.

defertoreptar(10000) 5 days ago [-]

I got the feeling this was more of a case of 'I did all this work for myself. Nothing useful came up, so here's what I found.' The author may be ok with spending an hour sharing the findings, but doesn't want to spend more time than that.

whoisninja(10000) 5 days ago [-]

more important than this study if done correctly is the fact that he built a framework that can ingest all this data and that he had access to all these datasets

typical hedge fund spends millions of dollars in order to build such frameworks and buying datasets, sure most academic papers fail if you replicate but the framework and datasets are very valuable because you can eventually find something on your own or an improvement on existing ideas if you keep trying hard enough + there are other sources of ideas like quant research from brokers, ideas from platforms like quantopian etc. but yea in general if you have an outstanding idea that works - you would have very less or no incentive to publish it. why would jim simons have his researchers publish anything when they can make money for him all day long everyday ... just my 2 cents.

whoisninja(10000) 5 days ago [-]

i asked the author where did he source his data, his reply was 'i scraped data'

how can you scrape pricing data? not every data in this is on public domain, otherwise there would be no Bloombergs, CapitalIQs selling data for millions(sure they're overrated and overpriced but still!). Or in other words, if he is right - he can sell data and make millions. no need of looking for an investment strategy. just my skeptical side saying :)

you need clean data to accurately test ideas. for instance getting tick data is quite expensive. most universities have free access to Bloomberg, CapitalIQ etc. datasets the reason professors can test and also the reason some smart guys in the industry work for university on the side

rossdavidh(4106) 5 days ago [-]

This is one of those cases where I would have guessed this would be the case, but it's nice that somebody else spent their time to verify, since I'm unwilling to spend my time to do so. Also nice that they shared their experience with the rest of us.

If it worked, it wouldn't be published, or at least not until it stopped working.

lucb1e(2123) 5 days ago [-]

> Also nice that they shared their experience with the rest of us.

Except they didn't share the results with the rest of us. I know you said 'experience' not 'results', but when disproving papers, the least you can do is write down three sentences about each paper as you go along reproducing them, noting what you are seeing, perhaps with a snapshot (just a zip file or so) of the code. This is calling a whole field nonsense (that everyone expects to be full of nonsense) without giving enough evidence for anyone else to dispute your claims.

linux_devil(1573) 5 days ago [-]

'The most frustrating paper:

I have true hate for the authors of this paper: 'A deep learning framework for financial time series using stacked autoencoders and long-short term memory'. Probably the most complex AND vague in terms of methodology and after weeks trying to reproduce their results (and failing) I figured out that they were leaking future data into their training set (this also happens more than you'd think).'

- Not sure how author tried to implement it , but is this not how you train LSTM networks by feeding t+1 data back into the cell again to predict t+2 data. It will be easier if author made it open source as well

jensgrud(10000) 5 days ago [-]

I wrote my thesis last year comparing different RNNs against each other using this exact paper as baseline and basically concluded that you would be better off predicting the price yesterday than using their results. Authors did not respond when prompted for implementation details or comments.

Overall, concluded that amongst RNNs the GRU architecture proved most favorable but still would not outperform simple stochastic models of the financial industry toolbox.

You can check it out here https://github.com/jensgrud/financial-forecasting-lstm/tree/...

laichzeit0(10000) 5 days ago [-]

Leaking future data in would be using t+1 for t, e.g. something like a bi-directional LSTM. I assume he means the actual training dataset had some kind of signal in the data that was also in the test data.

ricklamers(4126) 5 days ago [-]

I think this is not rigorous enough to draw any real conclusions.

If he had done a proper job of reproducing he would have created a write-up of his work explaining his reproduction methodology. The next step would be to get his work peer reviewed.

I think only then you have come close to the amount of analysis and rigour necessary to discredit so many authors of (possibly peer reviewed scientific articles) academic research.

The fact that he mentions that he doesn't know what a meta analysis is in the comments suggests that possibly _his_ results might not be what he purports them to be.

gillesjacobs(10000) 5 days ago [-]

Exactly, I also find it extremely dubious a 'Tier 1 professional quant trader' implemented 130+ papers in 7 months.

This means obtaining the same data, reproducing feature engineering and hyperparameters. Implementing learning algos. Maybe the guy is genius and god-like in NLP, finance, data science and machine learning but even then 7 months is too little time.

I was amazed at how few people call out this obvious lie here.

gingabriska(4198) 5 days ago [-]

Let's say an event occurs and you know a particular stock will go up 20% but you pump enough money into the stock to make the stock go 30% and then you let others chase the stock and publish news about this market move through media houses / content spamming / fake accounts. Then you short the stock once you get the desired movement and finally you remove the money from this stock so it goes into free fall and all people start selling. Finally, you close the short position when you notice there is no room for stock going further down. Now let's say stock ended up at 10% so you buy more stock so it goes 15% just below what you initially predicted.

Assuming you've billions to move the stock.

Why such strategy will not work?

anastalaz(2950) 5 days ago [-]

Such a strategy does work and it's also illegal. It is called market manipulation specifically 'pump and dump' https://en.wikipedia.org/wiki/Market_manipulation

zazagura(10000) 5 days ago [-]

Some of the papers I've seen are ridiculously obviously over-fitted.

For example published in 2018, but 'tested' on 3 months of 2010 prices of GBP/USD, USD/SEK and USD/THB. Quality forex data is so easy to get freely, that picking 3 months from 8 years ago on one major pair and two other random minor ones just stinks.

jobigoud(10000) 5 days ago [-]

They don't test the predictive power of their models?

quirmian(10000) 5 days ago [-]

Where can I get good quality forex data from?

SheinhardtWigCo(3890) 5 days ago [-]

I wish it wasn't this easy to get crypto-scam SEO articles to the front page of Reddit and HN.

comatosesperrow(10000) 5 days ago [-]

THANK YOU. I feel like I'm taking crazy pills watching everyone eat up the article where HE ASKS FOR YOUR MONEY.

gillesjacobs(10000) 5 days ago [-]

HN's culture of critique often involves blindly praising anything that is critical in tone. This makes it easy to guerilla market anything as long as you criticize anything established and well-used/liked.

HN is of course not the only place that has problem, but it is very obvious here.

numakerg(10000) 5 days ago [-]

Wait you DON'T want to invest your money into a black box crypto trading bot system with guaranteed returns?

For a community that prides itself on its critical thinking, HN is quick to laud a post with barely any substance when it confirms their preconceptions.

ddxxdd(4135) 5 days ago [-]

>Literally every single paper was either p-hacked, overfit, or a subsample of favourable data was selected

including methods that use:

>News Text Mining. - This is where they'd use NLP on headlines or the body of news as a signal.

I have to call this out.

Is this author suggesting that you couldn't have made money by shorting Enron stocks milliseconds after the scandal was made public? Is it impossible to make money by buying a stock in a small company, seconds after an acquisition is announced? If a CEO gets sent to prison, will that company's stocks not be affected?

And then there are other methods that use:

>Fundamental data. So ratios from the income statement/balance sheet

So buying stocks in companies with good financial health is not profitable?

Something's being left out here.

starpilot(1935) 5 days ago [-]

> Is this author suggesting that you couldn't have made money by shorting Enron stocks milliseconds after the scandal was made public?

Enron's collapse was 18 years ago. I suspect if this happened today, with today's trading environment, the answer to your question would be 'yes.' The algos today will parse an article, enter & exit a trade faster, than a human can read the headline.

> So buying stocks in companies with good financial health is not profitable?

That alone, probably not. You need to have an edge. If everyone else knows it's financial health clearly, then the price is already 'bought up.'

jdm2212(10000) 5 days ago [-]

The claim is that you can't do it consistently. Your sentiment detector has to more accurately capture the state of a randomly selected set of companies (not one selected with the benefit of hindsight, like Enron) based on news sentiment than the information already incorporated into the stock's price.

mediaman(4146) 5 days ago [-]

The important concept to understand about profitable investing is that you have to have a strategy that others are not also using.

Sure, investing in companies in good financial health is profitable. Unless everyone else does it too, and they drive up the price of the profitable companies, until all upside is gone (i.e., price is baked in). You're not better at finding profitable companies than anyone else.

Shorting stock on headlines? Sure, if you can beat everyone else. (You can't.)

The other is merely stating that, according to his analysis, apparently all these strategies did not bring an edge to the market.

dragonwriter(4199) 5 days ago [-]

> Is this author suggesting that you couldn't have made money by shorting Enron stocks milliseconds after the scandal was made public?

No, he's claiming that any of the actual published systems, if they would correctly have made money on that one special event, would not do so on enough other events to make up for those they would lose money on plus transaction costs on all the trades they would make to actually beat a broad market index.

It's pretty easy to (with hindsight) design a system that would make money on Enron or any other isolated event. It's harder to build a system that will consistently beat the market on future events that it's not designed against.

rco8786(4137) 5 days ago [-]

Right? Just like how you could have made money shorting Facebook right after their recent FCC fines came down. Oh wait.

Market is irrational in the short term. Hindsight is 20:20.

huac(2736) 5 days ago [-]

they're probably not accounting for HFT. that's not unreasonable to expect.

> So buying stocks in companies with good financial health is not profitable?

everybody has the same common sense, prices reflect all available information (at least, if you believe the efficient market hypothesis, which I do to some extent). so you shouldn't expect the method to be profitable in excess of the overall market profit -- what we call alpha.

leet_thow(10000) 5 days ago [-]

The stock market is a complex adaptive system where the agents are constantly changing their strategies so that even if you were to find inefficiencies or patterns, they are only ephemeral.

deepnotderp(2366) 5 days ago [-]

This is why the truly successful quant groups like Renaissance continuously adjust their strategies and come up with new ones. Renaissance in particular has invested heavily into their data processing pipeline which enables them to have a significant advantage over the rest of the field.

alanbernstein(4180) 5 days ago [-]

From the post:

> The easiest way to test whether it was truly Alpha decay or just overfitting by the authors is just to reproduce the paper then go further back in time instead of further forwards. For the papers that I could reproduce, all of them failed regardless of whether you go back or forwards. :)

jmpman(10000) 5 days ago [-]

Has anyone tried a simple approach - trying to predict which of the S&P 500 will have the lowest 10% returns, and build an (S&P 500 - 10%) index? It seems obvious that the S&P is stacked with some great companies and some old dogs. Does that method not work?

smnrchrds(4133) 5 days ago [-]

Since it's obvious, everyone knows it. And since everyone knows it, it's already priced in. You cannot find an edge by acting on widely-known public information.

dwohnitmok(3851) 5 days ago [-]

Intuitively the problem you run into is that occasionally those obviously bad companies have amazing comebacks and you completely miss out on those, so you can still end up underperforming the S&P 500.

yodon(3809) 5 days ago [-]

There are no quick easy hacks that give you reliable above market returns. If there were, enough people would use them that the pricing would correct for it because of demand and the above market return opportunity would disappear.

bArray(10000) 5 days ago [-]

Currently I remain skeptical, 130 pages in 7 months plus meaningful experiments is quite some going. A list of the papers (so at least the authors can defend themselves), the source code and data used (because some of these methods require social media inputs) would definitely help.

After doing so much work though, why wouldn't you go the extra small few steps to publish? That way the work can be peer reviewed and the Scientific community has a chance to learn from it.

gillesjacobs(10000) 5 days ago [-]

Yeah the paper does not pass a basic smell test.

OP also claims he does not know what a meta analysis is while doing all the work for a what could be a great meta-analysis.

He also answers evasively when people ask for results or proof.

bem94(2995) 5 days ago [-]

> That way the work can be peer reviewed and the Scientific community has a chance to learn from it.

After trawling through so much peer reviewed work which in their view is utterly broken, I can understand why they'd be despondent/mistrustful of submitting it to a peer review process.

lucb1e(2123) 5 days ago [-]

Yeah when I got near the end of the post, I was quite surprised. You spent a boatload of time on this and all you do with it is make a PSA on reddit going 'hey lol I looked at all these papers and they're all overfit'. Someone in the comments asked about sharing the work and was brushed off with 'nah, the code is a mess'.

This reeks of unscientific work, exactly what s/he describes the papers to be. While the results are in line with everyone's expectations (hence it getting upvotes despite offering zero proof, people already believed it and like it being confirmed), and so I don't really doubt they did try to reproduce papers and failed, it also isn't a reliable source in the slightest. If they were serious about disproving the papers, they could just have written down the steps, even if it's just a few lines of notes with snapshots of the code for each paper.

Just a list of papers isn't going to help the authors defend themselves, the methodology (if it can be called that) is so vague that the authors would basically be starting a 'he said she said' discussion.





Historical Discussions: Kerbal Space Program 2 (August 19, 2019: 657 points)

(659) Kerbal Space Program 2

659 points 3 days ago by xucheng in 4043rd position

www.kerbalspaceprogram.com | | comments | anchor

With the original Kerbal Space Program having become one of the most beloved games of all time and now bigger than ever, Kerbal Space Program 2 has been fully redesigned from the ground up to meet the demands of modern and next-generation space exploration, all while maintaining the monumental foundations of the first game. Build a space program, construct powerful spacecraft, design resource-gathering colonies, and much more to uncover the secrets of the galaxy. A plethora of exciting new features will captivate veteran and returning players, as well as usher in a whole new wave of Kerbonauts to the ingenious and comedic world that has entertained millions of players.




All Comments: [-] | anchor

eropple(2901) 2 days ago [-]

I was pretty bummed to hear about Squad doing their people dirty; the company seemed to consistently underpay and burn out their developers. Have they improved? (With cites?)

philipov(10000) 2 days ago [-]

Squad is not developing this game.

daniel_rh(4177) 2 days ago [-]

I can't find any information on linux support with KSP2. Does anyone have a link with more linux information? One of the many reasons I loved KSP was robust linux support.

TeMPOraL(2629) 2 days ago [-]

KSP is the only case I've seen where people actually learned to use Linux just so that they could play the game - KSP actually worked better on Linux, because for a long time Windows didn't have a 64bit build, and 32bit build run out of RAM very quickly once you started modding the game.

(And mods for KSP are a thing to behold; they turn a somewhat curious game into a masterpiece and deepest treatment of space exploration in videogames to date.)

loudmax(10000) 2 days ago [-]

The Original Kerbal Space Program is playable on Mac and Linux. If its sequel is a worthy successor, I hope it will follow suit.

albertzeyer(523) 2 days ago [-]

Last time I read about KSP (https://news.ycombinator.com/item?id=16581913), there was lots of criticism about the developers, or their working conditions. It sounded so bad that this was one of the reasons for me to not buy it. Has this improved?

enragedcacti(10000) 2 days ago [-]

It was purchased by Take-Two so the developers are probably only treated as badly as normal game devs now.

https://forum.kerbalspaceprogram.com/index.php?/topic/161355...

philipov(10000) 2 days ago [-]

Squad is not developing this game.

roddux(10000) 2 days ago [-]

I wonder if version 2.0 is written in Unity? I hope not. I recall a lot of the performance issues in KSP1 being blamed on the choice of engine...

joveian(3761) 2 days ago [-]

They say on the forum it is in Unity.

crysin(10000) 2 days ago [-]

Unity is not inherently bad. It's garnered that reputation because its super easy to learn and make a somewhat workable game out of, even if you have absolutely 0 understanding of the workings of your code. I'm not familiar with Kerbal's code or any blogs / videos on it but I would imagine they're doing Newtonion physics simulations which are going to be rather intense for a game and probably add an unavoidable burden to the overall performance of the game. They also could have just been doing things that are not efficient in Unity and not realized at the time and maybe will not repeat the same mistakes in Kerbal 2 if it is written in Unity. This meme that Unity is inferior to either Unreal or a custom built engine are annoying. Sure, if the Kerbal team built their own engine from the ground up and hired on their own engine team I'm sure they'd be able to make micro-optimizations to make the game run better but at a much higher development cost.

mjevans(10000) 2 days ago [-]

At the time Unity was a reasonable choice; a mix of decent licencing terms at different scales and cross platform support (even if it used to be wonky, it might still be TBH).

I've heard that Unreal's cross platform support matured in that time and I recall that Unity made some headlines for less progressive changes to their licencing terms... within the last two years I'd like to say?

sarcasmatwork(10000) 2 days ago [-]

Looks fun, anyone play previous versions?

The trailer was hilarious without sound. Watching guy fall out landing module, or they wreck their entire space ship.

celeritascelery(10000) 2 days ago [-]

> The trailer was hilarious without sound.

There was no foley, so it was just as funny with sound.

BitwiseFool(10000) 2 days ago [-]

Highly recommend picking up a copy. It's the most rewarding game I have ever played.

Hates_(2737) 2 days ago [-]

Yes. I found it extremely challenging while managing to remain fun and educational.

gnulinux(3467) 2 days ago [-]

It's a pretty good game with a very realistic physics simulator. They simulate everything from orbital mechanics to aerodynamics. There is a planet in the 'solar' system with an atmosphere close to Earth's (Kerbin) so you can go there with a gigantic rocket, bring your airplane and fly in a different planets atmosphere. Pretty amazing.

pinewurst(1979) 2 days ago [-]

It's a great game - played the original KSP since early beta.

6gvONxR4sf7o(10000) 2 days ago [-]

I've been playing it semi-regularly since 2012. I can't recommend it enough.

joveian(3761) 2 days ago [-]

To counter the glowing reviews others have left, it really depends what you want out of it and if you install mods or not (and if you do if all the mods you want work together). I don't use mods and want a more relaxing gamelike experience than a lot of folks here (but with lots of building and little micromanagment or random limitations just for difficulty) and it is meh in my opinion, although there aren't a lot of alternatives that are available DRM free and in some ways it is better than the alternatives. Look at actual gameplay, these trailers are useless for evaulating it as a game. Kerbals falling off the rocket is mildly amusing but they are mostly there to fly the rockets and don't really do much or have much to interact with. If they add ways to build stuff to interact with when you go somewhere (it sounds like they are) that would be nice.

There is a quite a bit of fun in building and launching a rocket. The learning curve is very steep but can be its own challenge if you like that kind of thing (I do). It takes quite a long time to do anything. Lots of people have posted interesting stuff that they built if you get frustrated and want inspiration. The wiki [0] has extensive documentation (except for the recent Breaking Ground extension so far) although the game much less so. Look at threads of the month in the Announcements forum for some amazing stuff people do with KSP [1] (seems like it would take an unreasonable amount of time to do any of that).

[0] https://wiki.kerbalspaceprogram.com/wiki/Main_Page

[1] https://forum.kerbalspaceprogram.com/index.php?/forum/11-ann...

Each version has lots of bugs [2] and some fundamental stuff was only implemented recently or not at all. DeltaV estimation and accurate maneuver planning are two things that only appeared about a year ago and are huge improvements. There is still no quick base game way to check the center of gravity of a plane with empty fuel tanks, although it won't fly well after a while if that is much different from the full center of gravity and you might not find out for a half hour or more since it takes forever to get anywhere and it is hard to balance a plane well enough to be able to use time warp. There are a few auto direction holding options if you have the right tech but nothing to keep your plane wings level with the ground or fly holding altitude. You can recover the parts of rockets that you land but you can't recover anything you separate before getting your main craft in orbit so you can switch focus to the debris (the huge fuel tanks that some people complain about in other comments are for those of us who want to recover everything and not leave space junk).

[2] https://bugs.kerbalspaceprogram.com/issues

I'd recommend both of the extensions if you are going to try it (although maybe try the base game first to make sure you minimally like it), particularly Breaking Ground that adds some minimal automation ability and the ability to get science by crashing stuff in to a planet or moon (this has bugs, although it should be fixed shortly). In career mode it takes forever to unlock all the science and almost none of it is at all related to building rockets, it is just doing the same few trivial things in a variety of locations.

Once you do get somewhere, on your home planet or elsewhere, there isn't really anything there, even with Breaking Ground that does improve it a bit. It used to be all terrain scatters (tree, rock, etc. looking stuff that you can go right through) and hills, now there seem to be two things you can interact with (also rock and tree looking things) that are the same all over a planet. So much of the fun is in the building and making up your own stories as you test a design. Or you can make stations and visit your stations.

I've played quite a few games due to poor health. Better in many ways IMO are Infinifactory (other than the horrible story), RollerCoaster Tycoon 3 (and 1 and 2 for that matter, with OpenRCT2 a nice requires-the-original-game upgrade mostly to be able to use higher resolution), and SimCity 3000 (unfortunately the GOG version crashes randomly after a few hours). Tropico (any version) and Surviving Mars also but they do the mandatory micromanagement and random limitations thing that I don't like (Surviving Mars more so but it has some good ideas too, like a number of camera options for taking in game photos). Theme Hospital (difficulty focused) and Banished (fairly building focused once you figure out the mechanics, although not that many layout options) are much simpler games but have some advantages from the game aspect and at least some building (Project Hospital looks nice and has more building but still has game breaking bugs nine months after release). Spore is also worth a mention; the building is mostly cosmetic and the gameplay has a bunch of issues, but it is still one of the few others that have a build and fly a space ship aspect (only with no atmosphere at all :/). OpenTTD (fully open source) and Cities in Motion focus on transportation building and are limited but decent for what they are. Widelands and Unknown Horizons are also fully open source builder clone games that are IMO improvments over the games that inspired them, Settlers II and the Anno series, although still not all that great. I hear Planet Coaster is good but it isn't avilable DRM free. There are a few more DRM free options but I get the impression they are difficulty focused rather than building focused.

gnulinux(3467) 2 days ago [-]

Ah, what a coincidence I just bought KSP about a month ago. Very good game but lacks a lot of automation. kOS and MechJeb solves some of these issues, but if you want to play 'non-cheaty' it's still a very manual game; there is no way to make use of computers whereas in real space programs everything is computed and organized so precisely.

Aperocky(10000) 2 days ago [-]

IMO kOS is worth 75% of the game itself, by which I mean the game has 25%.

That's how it should work, I also know a mod that provides python binding, but I still preferred kOS as it sorta became its own ecosystem. And it almost perfectly replicates the automation that are there in 1970s.

TeMPOraL(2629) 2 days ago [-]

RemoteTech gets you there. It adds a flight computer that can be set to execute maneuver nodes for you - which is crucial for remote operations e.g. when you have to execute a burn at a point where the planet of interest is between your probe and your comm network (before they added comm networks to stock, RemoteTech was what you used to introduce comm range, relays and speed of light delay).

nkrisc(4193) 2 days ago [-]

I always thought MechJeb and such should have already been part of the base game. Learning how to do manoeuvers manually is an important and rewarding part of the game. But after flying your 1000th craft into orbit, I just want to automate and manually fly the stuff I haven't tried yet.

shostack(4198) 2 days ago [-]

Is KSP1 any fun for casual gamers with only an hour or two and a joystick? Will I have to read up on lots of math and orbital mechanics to get anywhere in it?

TeMPOraL(2629) 2 days ago [-]

Hour or two is 1-5 'missions', depending on complexity of your craft and planned flight. However, you'll quickly discover you want to do 'just one more mission', and suddenly it's 5:00 AM.

As for math, you won't have to read up much - KSP is the ultimate textbook. Watch some introduction videos by Scott Manley or whatever the community currently recommends, and fire up KSP. Make your first orbit, then first transfer to Mun or Minmus, then back, learn to change planes, intercept objects and dock... you won't even notice when you've gained intuitive understanding of basic orbital mechanics.

hughes(4170) 2 days ago [-]

I have hundreds of hours in KSP and have never wanted a joystick. It's extremely forgiving and encourages learning by experimentation. This includes developing an intuition for orbital mechanics.

joveian(3761) 2 days ago [-]

You do not need math but do need quite a bit of time. I suggest some DRM-free builder alternatives at the end of my long comment above, but even most of those need quite a bit of time and all much less than KSP. I'm not sure builder games in general are the best option on limited time, but maybe you are much faster than me :).

BitwiseFool(10000) 2 days ago [-]

No math needed at all. It's hard to explain, but you end up understanding orbital mechanics visually and by feel. It's really just a game about speeding up and slowing down.

godtoldmetodoit(10000) 2 days ago [-]

Can definitely just pick it up and go, no need for a joystick. Just slap some boosters together and have fun blowing stuff up.

ziddoap(10000) 2 days ago [-]

It's one of those games that has a fair amount of breadth for those who just want to jump in and have some fun, but a ton of depth for those who get hooked.

mhh__(4160) 2 days ago [-]

The thing that KSP got right was making orbital mechanics 'intuitive' e.g. you just drag maneuver nodes and go by eye.

You can do the mathematics if you want (or look at a summary in the game) but it's not required.

headcanon(4139) 2 days ago [-]

You won't need to read a textbook or anything, but you'll want to invest more than an hour or two before you make orbit for the first time.

The great thing about this game though is that it teaches you orbital mechanics in an intuitive and fun way, much better than reading a textbook full of vector calculus ever could.

standardUser(10000) 2 days ago [-]

I consider myself a connoisseur of ridiculously time-consuming single-player games (Civ series, Crusader Kings, city builders and the like) and I have found no game as ruthlessly time consuming as KSP.

If you happen to have heroic self-control, you could find a lot of enjoyment simply doing a few moon landing missions over the course of 5-10 hours.

frittig(10000) 2 days ago [-]

What I liked about the current KSP is that everything is real. NASA actually sent a satellite to all of our Soler system's planets. They actually developed a rover that can drive on the moon, and they preformed rendezvous with two spaceships. They also had to deal with a finite amount of fuel and parts that could be destroyed if there was too much pressure or heat on them.

Now with colonies and mining it is less realistic and more sci-fi. I can see the appeal of that, but it is a different appeal. I would have preferred that they spent energy developing things to make it more realistic. For example, Lagrange points, different forms of propellents, n body physics, automation, ets. Recently with Bereshit I learned that before (certain?) rockets can fire their main engines they have to fire small ones to provide acceleration in order to move the fuel in the big tank closer to the main engine. It would have been nice if KSP also took that into account.

On an unrelated topic, I wonder how KSP 2 will make multiplayer work with timewarp

topmonk(4183) 2 days ago [-]

There are some mods for ksp that make things more realistic, including a real solar system and more realistic rockets.

TeMPOraL(2629) 2 days ago [-]

It doesn't seem to me that the new things stray much from the 'real but not necessarily actually done in the humanity's past' path. Colonies and mining is what you did (or at least many people did) after nailing their first couple landings and dockings. It's what real people in real space exploration business want to do, it's what they're preparing to do.

The video seemed very realistic to me, in the hard sci-fi, 'this is how it might look like under constraints of known physics and technology' way. The most outlandish things they've shown on KSP2 video are Orion and Daedalus rockets, rings for centrifugal gravity, and domed greenhouses, all of which are things taken out of real plans done decades ago, and have no reason for not working in reality except that we run out of funding for Space Race 40 years ago. There's also precedent with original KSP, which had its version of NERVA nuclear-thermal rocket engine, which never flown in space either.

exDM69(3992) 2 days ago [-]

> On an unrelated topic, I wonder how KSP 2 will make multiplayer work with timewarp

This is what I'm wondering too.

I have had a 'multiplayer space simulation with time warp' game design brewing in my head for quite some time, but it's a turn based concept. Essentially you submit a turn saying 'this is everything I intend to do from today until Mars launch window 2033', and the server consolidates the moves of all players.

Space is a pretty big place, so direct interaction between two players' spacecraft is a relatively rare occasion. In real-life space mission close encounters there is usually only one 'active' spacecraft (e.g. Dragon or Soyuz) and the other party is 'passive' (like the ISS). Even when both craft are capable of maneuvering (like Apollo Lunar Orbit Rendez-Vous or Gemini/Agena missions), only one of the spacecraft were assigned an active role.

So I have an idea how you could make a pseudo-realtime simulation with a turn-based multiplayer.

This wouldn't allow going on EVA or doing some rendez-vous shenanigans in real time with your friends, though. And that's the kind of thing I imagine appeals to the 'kerbal crowd'.

> Recently with Bereshit I learned that before (certain?) rockets can fire their main engines they have to fire small ones to provide acceleration in order to move the fuel in the big tank closer to the main engine. It would have been nice if KSP also took that into account.

Afaik there's a mod for this which adds 'realistic' rocket ignition, throttling and the need for ullage motors for liquid fuel rockets.

What comes to space colonies and other sci-fi elements, I think they might be a welcome addition to KSP. Now the career mode contains too much grinding for 'science points' and when you've advanced far enough down the tech tree, you get to grind for minerals and biomes. The gameplay could certainly use some late-game goals.

SketchySeaBeast(10000) 2 days ago [-]

I've sunk a ton of hours in the original Kerbal Space Program, its pre-launch state is one of my favourite games ever.

The only thing there that excites me here is interstellar travel. I'm worried about this being a soul-less cashing in, with a 'less, but pretty' situation - much like every versions of the Sims.

TylerE(3985) 2 days ago [-]

I just hope it's actually a new game, and not further layers of goop on the janky Unity core.

CydeWeys(4038) 2 days ago [-]

Pretty, in this kind of game, goes quite far. You could fix some of the issues with modding in KSP, but only to an extent, and you were sacrificing stability to do so, plus it wasn't an out-of-the-box experience (i.e. must players didn't have it).

AcerbicZero(10000) 2 days ago [-]

Same here. My hope is that they don't try to add to much to KSP1, and just focus on fixing the obvious issues. Most of those problems were due to the engine anyway so perhaps this will work out.

TeMPOraL(2629) 2 days ago [-]

Little context for the trailer: it seems to be a homage to this fan video: https://www.youtube.com/watch?v=RkDOOsGg-9I, which is one of the most well-known videos in the KSP community, and something people show to people to get them into playing this game. I must say the new, 'official', video does capture the atmosphere very well.

If the game will look a third as well as on the trailer visualization, I'll buy it without thinking (who am I lying to, I'll buy it anyway). One thing though: I hope the modding capability and freedom won't be diminished.

If KSP is a great game by itself, mods make it a literal order of magnitude better - a huge amount of solid fan work was done to expand every imaginable aspect of KSP. And I mean it: among many things, mods give you many more realistic and unrealistic building blocks, life support management, communication networks including speed of light delay, trajectory planning, UI/UX improvements, graphics improvements, orbital assembly, working space telescopes, realistic aerodynamics model (including supersonic physics), n-body simulation and non-symmetric gravity fields,... I don't think there's another game where so much of such highly skilled effort was spent for free to make it deeper.

I really hope they won't do anything that would make this depth of modding impossible.

toomanybeersies(4200) 2 days ago [-]

I'd say that Gary's Mod has had a similar amount of work put in to it by modders. But it was also designed specifically to be modded, the default game gets boring really quickly.

Diederich(10000) 2 days ago [-]

> ...n-body simulation and non-symmetric gravity fields...

I want to agree with everything said here, and add that the KSP mod ecosystem is no joke. The few things mentioned here barely scratch the surface.

As a point of comparison: I'm aware of and long involved with the Minecraft community, and it is huge and excellent. While the total amount of mod material is likely greater for Minecraft compared to KSP, I'm comfortable saying that the mods in KSP are deeper and wider in scope.

Aperocky(10000) 2 days ago [-]

I really REALLY need kOS to have fun playing this game... Hopefully mods are not hard to port over (Or a default scripting language, I can get behind that too)

peeters(4206) 2 days ago [-]

> One thing though: I hope the modding capability and freedom won't be diminished.

There was a video posted alongside with a lot of development interviews and one of them was very specific about mentioning they were creating a platform and they wanted to expose the framework they use to the community. Hopefully not just lip service but as far as signalling goes it was important enough for them to mention.

Abishek_Muthian(3966) 2 days ago [-]

Website mentions modding, but I wonder whether it would be limited only to the PC version. But, I'm really glad that the consoles are getting KSP2.

sixothree(10000) 2 days ago [-]

Thanks for the context.

gowld(10000) 2 days ago [-]

> it seems to be a homage to this fan video:

The end of the KSP2 trailers says 'Special thanks to Shaun Esau #buildflydream', which is for that video.

exadeci(10000) 2 days ago [-]

From the website:

>Multiplayer/Modding >The technological developments made to the foundations of Kerbal Space Program 2 will build on the beloved modding capabilities of the original game, as well as deliver on the long-requested addition of multiplayer. Soon players will be able to share the challenges of deep space exploration. More details on these features will be revealed at a later time.

erigeron(10000) 2 days ago [-]

This also looks inspired by the short film Wanderers, about humanity's expansion into the solar system. https://vimeo.com/108650530

haberman(3386) 2 days ago [-]

Love KSP and I'll be looking forward to this!

I really don't get cinematic trailers though. If it's not actual gameplay, then what does it tell me about the game? What does it tell me that I don't already know from KSP1?

Pfhreak(10000) 2 days ago [-]

They are communicating their aspirations. That they want to do colonies. That they want bigger stations. Interstellar travel. Multiplayer. Unusual drive concepts.

And they tell you the tone. Note all the stuff crashing, but also the hopeful message of expanding. It's an emotional appeal to you. It's a way to communicate to your audience that, 'Hey, we get you. We know what you want from this series, and it's in good hands.'

astrodust(4063) 2 days ago [-]

'Multiplayer/Modding'

I'm in.

TeMPOraL(2629) 2 days ago [-]

Oh yes. A proper multiplayer is the final thing I miss in KSP. I'm not really sure how it could work reliably, given the need for regular timewarping, but I would so love to build and operate a space station with other players simultaneously.

martindale(3114) 2 days ago [-]

As long as I can self-host and keep it running on the LAN throughout our 6-month playthrough as a family!

gzu(10000) 2 days ago [-]

Will be excited to see if it uses a re-vamped game engine. KSP 1 is based on Unity and bottlenecks at single core cpu.

legohead(4203) 2 days ago [-]

from the dev interview [1] it looks like they are still using Unity for KSP2?

[1] https://www.youtube.com/watch?v=2wmJlnTqjSg

siekmanj(10000) 2 days ago [-]

Unfortunately, looks like KSP 2 will also be using Unity: https://forum.kerbalspaceprogram.com/index.php?/topic/187315...

inopinatus(3723) 2 days ago [-]

From the extant details it's hard to tell if it's even a new game.

The cinematic launch video is pretty funky but is clearly marked as 'Not actual gameplay'. The press materials talk about new parts and new places to visit, all of which could simply indicate a very large expansion pack on the original. There's nothing in the accompanying 'developer journey' video to suggests it's a new base game either, let alone a new engine; quite the opposite: everyone is wearing Unity t-shirts.

I'm hoping it's both, because the current gameplay vastly exceeds the game's performance envelope. In particular, the physics engine: I get 5 frames per second when docking large space stations, and KSP is running its physics engine on two CPU cores out of 10, and not even warming up a GPU.

squeaky-clean(10000) 2 days ago [-]

KSP2 is still based on Unity, but Unity has changed a lot throughout the years. (It's like how Apex Legends and Half Life 2 use the 'same' engine). The new entity-component-system (supposedly) makes it much easier to write games with a job oriented approach that scales to multiple cores very well.

TeMPOraL(2629) 2 days ago [-]

AFAIK they switched to a multithreaded physics engine some time ago, during one of the Unity version bumps. Can't test it now, but my vague memory of looking at htop when playing recently seems to agree with that.

AcerbicZero(10000) 2 days ago [-]

I was a huge fan of KSP, bought into it back in like ~.8/.9 days before there was anything to actually do, and I loved it. Talked tons of people into playing it and I got a lot of joy out of it. I didn't like the way Squad did things, and the bigger they got the worse things seemed to get, but they still ended up producing a solid, relatively stable game, which made space 'fun' in a way not many games could.

Squad threw away the majority of the good will they originally generated (mostly by treating their community, and especially their modders, like crap) but the core concept of KSP is solid enough that even with their questionable behavior it still managed to thrive. I wont be backing this early on, but perhaps when it gets closer to release it'll be easier to judge what kind of product they're going to end up with.

philipov(10000) 2 days ago [-]

It looks like Squad threw away more than just good will, because they're not developing this. Some studio named Star Theory Games is listed as the developer for this on Steam. Does anyone have any more color on them?

Aperocky(10000) 2 days ago [-]

If there's something I'll get behind more than a KSP 2.0, it would be a community driven open sourced KSP.

I can even withstand more bug than 0.13 and learn C# just to create mod.

t0astbread(10000) 2 days ago [-]

KSP always looked very cool but I've heard a couple of things that threw me off:

- KSP devs are or were treated poorly by their employer (low pay, no compensation for the success of the game)

- apparently the program collected a lot of data (I can't remember what exactly) about its users and the machines it was running on, to the point where even non-privacy minded people were calling it spyware

Does anyone know anything about this? Afaik discussion about these issues just kinda stopped after a while but we never got a conclusive answer to these points.

t0astbread(10000) 2 days ago [-]

Aside from those controversies, does anyone else find it kinda weird that KSP is not FOSS? I mean, I don't wanna sound like the guy that points an index finger at devs making proprietary software but I would've guessed they have the right kind of audience for that. (In the sense that their audience would still pay for the program and they might actually benefit from open sourcing it.)

hadlock(10000) 2 days ago [-]

Squad was/is a marketing company based out of Mexico. The owner of the company agreed to let his lead technical guy work on KSP as a side project as a bribe to keep him from leaving to do other things. KSP took off, and the team grew to more than five people. After the 1.0 release most (all) of the original team left, especially after they outsourced the console port to an outside team. The original KSP team are all now working on independent projects.

KSP2 is sort of a Battlefield 5 type game, where somebody still owns the IP but the original team that built the engine/brought the magic is gone. It might be good, who knows.

In my opinion I'm a little skeptical about KSP2 as the base KSP1 game was so moddable that I can't imagine what they would add to KSP 2 that isn't in the base game through mods already. But maybe it will be even better than the first.

fsloth(4187) 2 days ago [-]

If you don't want to buy products made by employees treated badly don't buy games. Unless they are the few, cool, indie ones like Papers Please and so on.

Employee treatment is a wide spread problem in the games industry. No unionization, and the pool of hires is continuously refreshed by new young naive people who've 'always dreamt of working in games'.

I'm not saying that this is not an issue, but it's kind of obvious that this is a shit market for employees.

Faark(10000) 2 days ago [-]

As for the spyware accusation... from what I read, Take Two slapped their usual privacy policy on the game after they bought it. The community didn't take that nicely.

I don't remember ever seeing any evidence for spying beyond policy discussion and would be surprised if there is anything beyond common telemetry.

jammygit(10000) 2 days ago [-]

The spyware was called red shell, it would apparently track your web browsing to figure out how marketing campaigns were paying off.

https://www.reddit.com/r/KerbalSpaceProgram/comments/8stnm3/...





Historical Discussions: Start Your Own ISP (January 16, 2018: 938 points)
Start Your Own ISP (August 17, 2019: 632 points)

(632) Start Your Own ISP

632 points 4 days ago by mahathu in 3818th position

startyourownisp.com | Estimated reading time – 4 minutes | comments | anchor

Start Your Own ISP

This site is dedicated to helping you start your own Internet Service Provider. Specifically this guide is about building a Wireless ISP (WISP).

This guide is focused on the very earliest stages of starting a WISP - determining feasibility up through connecting the first few customers. There are many challenges that will come up at 100, 1,000 or 10,000 customers that are not (yet) covered in this guide.

If you'd like personalized assistance with a project feel free to book some time with me.

Join the discussion! Chat with me (the author) and others interested in this kind of thing here: #startyourownisp:matrix.org.

Follow Along on Twitter @syoisp

Getting Started

What is a WISP? And why might you want to build one? Also defines some terminology.

Costs What does it cost to build a wireless Internet Service Provider? (Link to a Google Sheet that you can copy and customize.)

About Me Who am I? Why am I doing this?

Step by Step Guide

Step 1: Evaluate an Area: Make sure your area is a good candidate for a Wireless Internet network.

Step 2: Find a Fiber Provider: Find a building where you can purchase a fiber connection and use the rooftop to start your wireless network.

Step 3: Find Relay Sites: Extend your network wirelessly toward your customers.

Step 4: Pick a Hardware Platform: Evaluate available options for wireless hardware.

Step 5: Billing and Customer Management: Make sure you're able to get paid and support your customers.

Step 6: Network Topology: Design your network topology to make your network reliable and scalable. Routers, switches, IP addresses, VLANs, etc.

Step 7: Build your Infrastructure: Install hardware for your fiber connection and your relay sites.

Step 8: Install a Customer: Get your first customer online!

Step 9: Marketing: Let people know about your service so they can experience a better Internet connection!

Step 10: Maintenance: Keep your network running smoothly, even in bad weather.

Miscellaneous

Form 477: How to prepare and file with the FCC Form 477 is used by the FCC to determine which providers are servicing which areas. ISPs must file this form twice a year.

Tools you'll want to have A list of the tools you'll need to install relays sites and customers.

Aim a Backhaul A guide describing the proper techniques for aiming backhauls. Designed to be printed out and taken to the site for reference.

Backhaul List If you just need to get a solid wireless connection from Point A to Point B then use this list to pick the right equipment and get it set up.

RF Basics and Channel Planning Avoid self interference by carefully choosing channels for your access points and backhauls.

MDUs (Multiple Dwelling Units) Best practices for providing service to apartment buildings, condos, attached townhomes, etc.

Guide to Google Earth Some tips and tricks for using Google Earth to plan and build your network.

Roof and Ladder Safety Stay safe out there!

Sponsored by BroadbandNow. Looking to donate? Patreon or BTC 1JKa1Kdrp3r4xPSXBRJ6nPC6YYdLcqQ4Bp. Many thanks! © 2019 startyourownisp.com – Documentation built with Hugo using the Material theme.



All Comments: [-] | anchor

xvector(2897) 4 days ago [-]

Would be cool to see community-run non-profit ISPs.

war1025(4204) 4 days ago [-]

Both my hometown and where I currently live have municipal broadband.

https://osage.net/internet-services/

http://centraliowabroadband.org/

q3k(4094) 4 days ago [-]

https://bgp.wtf/ (Warsaw, Poland)

dbmueller(10000) 4 days ago [-]

I think there are quite a few in France:

https://ffdn.org/fr/membres

zajio1am(10000) 4 days ago [-]

It is interesting how phrase 'community-run ISP' is interpreted in different ways. I would understand it to mean ISP run by independent non-profit organization or consumer co-op, but definitely not ISP run by government/municipal organization. But many posts in this thread (and also in other discussions) use it for municipal ISP.

TomMarius(10000) 4 days ago [-]

That's all over the place in the Czech Republic. Really the best way.

colechristensen(10000) 4 days ago [-]

Just last week I was wishing exactly this existed. There are so many businesses where the main hurdle to get over is the basic institutional knowledge.

I wonder how cheaply it could be done for 1 customer (my parents house has terrible wireless internet and I'd like to move home and try doing a startup)

madamelic(3906) 4 days ago [-]

>I wonder how cheaply it could be done for 1 customer

Depends on your definition of cheap. Your biggest expenses would be monthly bill to your fiber provider, then any necessary rent for backhaul sites, then the up-front cost.

I can't say for sure the monthly cost but it would definitely be in the low four figures.

faitswulff(3383) 4 days ago [-]

About how much money should you be prepared to spend if you were to follow this guide exactly?

jedimastert(4085) 4 days ago [-]

There's a spreadsheet linked [1] that says (for 10 customers, approximately) $24,000 upfront and $2,800 monthly.

https://docs.google.com/spreadsheets/d/1jjUYOQMuZ4cRyTv1M5X8...

techntoke(4203) 4 days ago [-]

I always wanted to have an open access RADIUS server so that I can resell my unused Internet. I personally think this should be allowed.

liara_k(10000) 4 days ago [-]

The problem with this is that consumer Internet access is priced based on the assumption that it will be 'bursty'. Heavy users are subsidized by light users, both on a moment-to-moment basis and overall.

Reselling consumer-level Internet access to means you're taking away the users that would otherwise be subsidizing your service (because they're buying from you instead) while still expecting the local ISP to shoulder a large share of the costs for bringing your packets to their final destinations.

That issue goes away if you pay for a dedicated line (IE, explicitly contracted reserved bandwidth) to an IXP or place somewhere will sell you bulk transit, and you're paying for that too. That's fine. The problem is with expecting to make a profit by being the ISP to your whole block, based on a $100/mo GBPS fiber drop.

Not to say that the ISP mono/duopolists aren't loathesome--excessive market power is the root of... not all, but lots of socio-economic evil. The economics of telecommunications is complicated.

dboreham(3222) 4 days ago [-]

You mean resell the thing you agreed not to resell?

xwdv(10000) 4 days ago [-]

I see this post come around every once in a while, and while the idea seems alluring, the fact is if you're not in the right location and market, this kind of business will be unsustainable.

Maybe you can start your own ISP, but do not assume it will be in the same place you live. Analyze the country and find the pockets where you can thrive and grow a successful business, otherwise don't even bother. Also bring a decent amount of cash and a good credit line.

Ayesh(3837) 4 days ago [-]

I think there will be more demand once people realize there are other options. I saw a post on HN a few days about someone starting a small ISP in NYC. I suppose NYC market is saturated as it can get.

One interesting I ran across is how mountainous villages in Nepal has their Internet connections. I was in Annapurna trail last month, and in Manang village (which has about 20 guest houses), there is one big satellite connection and everyone shares from it. The owner charges the consumers a flat fee. From what I hear, and it could be exaggerated, they pay about $500 per year. I think a remote location, one uplink satellite, and a few wired subscribers can be a viable business!

madamelic(3906) 4 days ago [-]

Also since these would be WISPs (Wireless ISPs), it depends on topology as well (signals can't go through mountains, backhauls need to have line of sight to multiple customers, etc). It's a tricky balance of multiple variables.

est31(3893) 4 days ago [-]

Yeah, the story is usually like this:

* People found an ISP $NEWCOMER in an area where $ESTABLISHED has bad service and high fees.

* The newcomer invests loads of money into cabling the neighbourhood with fiber (or wireless internet) and offers deals with lower fees and better connectivity than $ESTABLISHED

* In a static world everyone would switch to $NEWCOMER and they'd make a lot of money, being able to expand to other places. But executives at $ESTABLISHED aren't stupid. They invest as well and lower their prices, too, just enough to be lower than $NEWCOMER.

* $ESTABLISHED waits until $NEWCOMER runs out of money. They rise prices again. It doesn't even matter if its price reductions mean $ESTABLISHED loses money in that region. The regions where there is no newcomer bring in enough cash to pay for the few regions with (temporary) losses.

Of course even if $NEWCOMER runs out of money, someone will buy their infrastructure and offer similar services, but there are more components in the competitive moats of established ISPs like (local) government granted monopolies in exchange for providing public buildings like school with free internet, or the bundling of streaming services with internet deals. Those moats usually are wide enough that there is no avail for small ISPs to break through. That being said, if you break through the moat, it's an insanely profitable business. DTAG had to invest tons of money until they got there and now they are in the place where the big ISPs can't harm them any more and the american daughter company is now the favourite of the DTAG shareholders.

unnouinceput(10000) 4 days ago [-]

Start your own ISP in US - there, I fixed the title for you.

I would love to start my own ISP in my country, except that dealing with my corrupt government and corrupt bureaucracy existing here would mean I would have to have triple the cash of official investment, because here without 'greasing the wheels' nothing works, and all of that is done by discretely leaving an envelope with cash on the corner of the discussion table that nobody in your presence will touch it. Also the amount of cash itself is a mind game on its own. You must discreetly research the person to see how much money they take as bribe. Too much and news about you being a newbie will spread like wild fire and all your future discussions will drain you of even more cash in discreet envelopes. Too little money and your request will get denied making you face the same person handing over another discreet envelope, and you better have at least the correct sum in there. So waste time asking for this mind game, waste cash that you'll never be able to officially report as deductible and then start implementing the points in article. Welcome to Romania ̄\_(ツ)_/ ̄

allana(10000) 4 days ago [-]

Sounds like the perfect place for a bribe transparency website to name and shame officals that take bribes.

timurlenk(4154) 4 days ago [-]

Romania has 14.3M (1) Internet users for an active population of 9.2M (2)

As the same time, about 98% of the population has mobile phone coverage (which implies some form of data access) with terminals that can be had for 0 upfront cost and ~10€ per month subscription with very large traffic allowance. Telekom (previous Romtelecom) can offer DSL services in most villages where a phone exchange is present.

Corruption or not, its probably a bad idea to start a wireless Internet provider in Romania because it already has excellent commodity connectivity virtually everywhere.

1. https://www.internetworldstats.com/europa.htm 2. https://www.romania-insider.com/romania-active-population-do...

FDSGSG(4166) 4 days ago [-]

Weird, I'd imagine the ubiquitous and insanely cheap pole fiber would be a much bigger problem for WISPs in Romania than the corruption.

Seems like it would be very hard to make a profit unless you can find hundreds of customers close to each other who are willing to pay unusually high prices for internet connectivity.

Hard_Space(3734) 4 days ago [-]

Ironic, because this kind of neighborhood ISP scheme was how Romania got a good foothold on internet connectivity in the 1990s, according to my Romanian friends who remember the period.

Rerarom(10000) 4 days ago [-]

If one has their own ISP, can they pirate stuff without being caught?

elcomet(4186) 4 days ago [-]

You'll receive emails from copyright owners of they detect torrenting. But that doesn't mean you have to act on it.

emptysongglass(4083) 4 days ago [-]

I'm starting a WISP in the Oslo area if hackers in that area want to join forces. I could use a native Norsk speaker!

morphle(4169) 4 days ago [-]

I would like to join forces and bring a Norsk speaker with me. Please contact me. MerikAtFiberhoodDotNl.

jkilpatr(4186) 4 days ago [-]

My company Althea (https://althea.net) is making open source router firmware that makes it easy to people to set up incentivized mesh networks in their communities. It allows routers to pay each other for bandwidth which means that everyone hosting a node earns money for the packets they forward.

This ends the incumbency problem by making bandwidth a commodity. Want a byte, buy a byte, switching to the best provider automatically. No software knowledge required, fully plug and play. (equipment still required, can't break the laws of RF. it's just regular WISP gear though)

Here's a talk I gave on exactly how the system is designed and implemented.

https://www.youtube.com/watch?v=G4EKbgShyLw

https://docs.google.com/presentation/d/1izRgUr-Tm-ixnqpd7NGW...

Operyl(3996) 4 days ago [-]

Huh, how do you deal with the problem of law enforcement assuming that an IP is a single subscriber? Even if law enforcement believed it up front, it still is a hassle to the users now dealing with law enforcement requests.

eximius(4190) 4 days ago [-]

My problem is that I want to do this with fiber, not Wireless, which is considerably more expensive.

If only regulatory capture wasn't a thing...

iptrans(10000) 3 days ago [-]

If you want help with getting the costs down for fiber builds, hit me up. Email in profile.

fiatjaf(1930) 4 days ago [-]

Here are some talks from the guy behind that site: https://www.youtube.com/watch?v=cq4mWTQZmnI&list=PL1MwlVJloJ...

These are very dense talks with tons of information you'll probably not need unless you're reeeally going to do it, but it's good to watch for an overview of all steps involved anyway even if you don't.

The talks are from Altheapalooza from https://althea.net/, but they can be applied to non-Althea ISPs also (although you should consider Althea as it's great despite the shitcoinery).

emptysongglass(4083) 4 days ago [-]

Ethereum is not a shitcoin, not really sure why you felt that dig was necessary.

topkai22(10000) 4 days ago [-]

Thank you, this is fantastic. I am working with a 100+ employee company doing highly technical engineering work in a very rural location, currently trying to run their business off of 10 MBPS. They are getting quotes exceeding $500k from most people they talk to just get any additional bandwidth to them (presumable fixed line fiber), which has resulted in this critical upgrade being put off for quite some time. This guide makes me think they could quite possibly build out their own backhaul system to a neighboring town for substantially less money. Should make my next conversation very interesting.

mciancia(4111) 4 days ago [-]

500k? This is crazy

techslave(10000) 4 days ago [-]

it's probably not wise to build your own backhaul if you're not in that business. don't know the particulars of course but most likely DIY is not the way to go.

startyourownisp is not really about that.

now if you can do a microwave or other backhaul that's great but it still sounds like something you should be able to get someone to provide for you.

iptrans(10000) 3 days ago [-]

I've helped many rural locations get online. If you want to email me your address, I can see if there are more reasonable options available to you than $500k.

Email in profile.

grahamburger(4198) 4 days ago [-]

Oh hello everyone! This is my website. Not sure why it hit HN today but cool too see it here! Thanks @mahathu!

I'm not at a computer tonight I'll run through later today or tomorrow and answer questions. My contact info is on the website as well.

MichaelApproved(2518) 4 days ago [-]

Do you mind sharing how you found out it was posted here today?





Historical Discussions: Every productivity thought I've ever had, as concisely as possible (August 19, 2019: 601 points)

(608) Every productivity thought I've ever had, as concisely as possible

608 points 3 days ago by less_penguiny in 3045th position

guzey.com | Estimated reading time – 25 minutes | comments | anchor

Every productivity thought I've ever had, as concisely as possible

created: 2018-08-07; modified: 2019-01-16 Table of Contents

I combed through several years of my private notes and through everything I published on productivity before and tried to summarize all of it in this post.

If you're unproductive right now

Here's what you should do if you've been procrastinating for an entire day:

  1. Accept that you won't do anything today and try not to get angry at yourself
  2. Set the alarm for the time you will be preparing to go to bed today
  3. No, really. Do it. It will take 20 seconds
  4. Procrastinate for the rest of the day
  5. When the alarm rings, put your laptop and everything you need for work in your backpack
  6. When you wake up, try to not check social media, email or anything else. Do not take anything out of your backpack
  7. Get dressed, take your stuff, and go to a library, caffe, whatever else where you either
    • never been to
    • have been to but never procrastinated within the last 6 months
  8. While getting to that place, figure out what you want to be doing today
  9. Do it
  10. Return home in the evening. Don't take anything (especially your laptop) out of your backpack. Repeat steps 6-10

Every productivity system stops working eventually and there's nothing you can do about it

You've most likely tried the pomodoro technique. You set the timer for 25 minutes, then take a 5 minute break, then set the timer for 25 minutes again, then at some point you take a longer break and so on. I predict that pomodoro technique eventually broke down for the following reasons:

  1. you stopped adhering strictly to 5 minute breaks and they started turning into 6-7-10-15-20-or-more minute breaks
  2. you've gained an aversion towards 25 minute timers, even while remembering that you should set them, and started finding excuses like "oh this task is too short", "oh i don't need a pomo right now", "i will wait till round time (:00 or :30) and start the pomo then" and these excuses started happening more and more frequently
  3. you started to outright forget about pomodoros, instead just doing your stuff the old way and once in a while realizing that you should've been running a pomodoro

It seems that every productivity trick / system stops working in exactly the same way I described above. Most productivity tricks develop aversion around them. All of them lose salience.

The only way to avoid encountering problems with productivity is to make the stuff you want to be doing in the long-term to be the most exciting stuff you can do at any moment in time, which is perhaps possible if you, e.g. work at a startup, but is untenable in almost every situation.

Context intentionality as the key difference between home and every other place on planet earth

You never wake up at work having forgotten to fill out to dos for the day and feeling slightly depressed.

However awesome you feel you are, this does occasionally happen at home.

Home is the default place. Home lacks intentionality, which means that sometimes you will feel that "I need to do something" rather than "I will do something something specific right now". As Kaj Sotala puts it: "I'm starting to suspect that I may have MASSIVELY underestimated the negative motivational impact of not having a clear sense of one's next action in a project."

Having no clear idea what to do next increases the probability that you won't feel like following all the rules you came up with massively. The only solution I know is to avoid working from home as much as you can.

If you aren't working from home, your workplace should be at least a couple of minutes away (better: an hour away), so that you would not fall into the same trap with it and always had the time to think on what specifically you're going to do once there. This is also why designating a special room at home as an "office" is probably a bad idea: you will frequently enter it on autopilot, without intentionality.

Thus, even if you can work at home, you probably shouldn't. I personally try leave home as early as possible, go to the university, and return home as late as possible.

A trick to aid leaving home and going to work someplace else is to try explicitly forbidding doing anything productive at home. This way, you can no longer tell yourself you'll start working "soon" and then proceed to waste the entire day procrastinating.

Interlude: "eliminate the distractions" is the worst productivity advice I've ever seen

With my present system, YouTube, reddit, agar.io, etc. are always just two clicks away, but it doesn't seem to matter at all. And yet, when I had StayFocusd installed with Nuclear Option turned on (forbidding to visit any sites that aren't on the white list), I picked up Windows Minesweeper and Solitaire, would often literally bang on the keyboard staring at the monitor wanting to scream and eventually found out a way to uninstall StayFocusd even when I purposely made it — as I thought — impossible to do so.

The very fact that your to do list feels ughy means you're doing something wrong. The very fact that you need to fight the urges to procrastinate means you're doing it wrong. The utility function itself is warped in fucked up contexts.

How I work and rest, how my system is different from all the others, and why I like it so much

For the last several years, all Did I hear somebody say "preference for routine"? my working time has been structured as follows:

  • work for 25 minutes from :05 to :30
  • take a 5 minute break from :30 to :35
  • work for 25 minutes from :35 to :00
  • take a 5 minute break from :00 to :05
  • every three hours (at 12-3-6-9) the :05-:30 work cycle is substituted for a break, which lasts 35 minutes.

For example

  • work: 9:35-10:00
  • break: 10:00-10:05
  • work: 10:05-10:30
  • break: 10:30-10:35
  • work: 10:35-11:00
  • break: 11:00-11:05
  • work: 11:05-11:30
  • break: 11:30-11:35
  • work: 11:35-12:00
  • break: 12:00-12:35

It's important that your clock shows seconds, so that you could know the start/end of pomos/breaks precisely, instead of constantly trying to guess them. On Windows I use T-Clock for that; for Android you can read my instructions detailing how to make status bar's clock show seconds.

But didn't I just write that every productivity system breaks down eventually? Yep, this one breaks down as well. However, I found that this arrangement:

  1. prolongs the period during which the system works.
  2. provides me with 625 minutes of work, interspersed with 275 minutes of breaks (provided my workday is 15 hours). This ratio of work / breaks means that I
    • have a relatively easy time convincing myself to put off impulsive things (because the maximum waiting time is less than 2.5 hours — until the next long break)
    • don't burn out, since such a large portion of my day is specifically dedicated to doing pleasurable, rewarding in the short-term stuff
  3. is impossible to forget. Frequently, we simply forget about productivity tricks. Once whole life is built around one, it becomes pretty difficult to forget about it.

You may say, "but isn't this basically Pomodoro Technique TM?" Kind of. There are several important differences.

  1. I don't care about doing only full pomodoros: suppose, I got home at 20:15. Do I dick around and wait till 20:35 to start a pomo? No. 20:15 is time during which I'm working, so I get to work, and then round this pomo up or down , depending on the circumstances.
  2. I don't care about pomos being "distraction-free": suppose, I got distracted in the middle of a pomo. Do I start over? No, I just get back to work, and round this pomo up or down, depending on the circumstances.

The fact that I never have to think "do I work right now or do I take a break right now?" removes most of the friction of the pomodoro technique and means I no longer have to actively think about starting pomodoros. Complice's LessWrong Study Hall served as an inspiration for me. Given these modifications, it seems that I was able to make my system somewhat of a natural equilibrium.

Some additional rules / heuristics that I follow:

  1. if I'm in the flow and don't notice that it's a break right now, then I skip it. This happens regularly to short breaks; occasionally, e.g. if I'm writing a really exciting post, to long breaks
  2. I usually divide the stuff I work on in 3 hour chunks. For example, my today's to do list is:
    • -9 work on the most exciting thing about site
    • 9-12 productivity post
    • 12-13:30 retrospective on college post
    • 13:30-15 clean up OneNote
    • 15-18 data post
    • 18-21 clean up incoming information
    • 21- figure out site's todos
  3. if I'm obsessed with something, e.g. very exciting research, I usually fill the entire day with it
  4. if I finish my task in the middle of a pomo, then I move on to the next task immediately
  5. if I decide that I don't want to finish the pomo on the task I planned, e.g. after realizing that the textbook I picked is bad, I try to finish the pomo doing as similar to the original task as possible, e.g. starting to read another textbook on the topic
  6. if I have a thought pop up in my head during a pomo, I write it down to an "Incoming.md" file always open in Notepad. I clean this file up during the breaks and on Sunday
  7. during the short breaks I'm only allowed non-distractive stuff, i.e. social media, email, most of reddit, slack are prohibited but
    1. if I have an ongoing conversation with someone, then during the next short break I'm allowed to check if the person wrote anything and reply to them
      1. if I think the conversation does not wait, then set the phone timer for 5 minutes, and open the conversation then
    2. if I need to open a specific email conversation, I go to https://mail.google.com/mail/u/0/#search/NAMELASTNAMEORTOPIC, which, along with hidden number of unread messages, means I don't get distracted
    3. if I need to open my email inbox, I think if I can wait till the long break
      1. if I can't, set the phone timer for 5 minutes, and open email then
      2. if I can, open the "Incoming.md" file and write the thing I want to check there, so I don't forget about it
    4. if I need to open Twitter / FB for some reason
      1. open it incognito, so that I don't see my timeline, notifications, and DMs
      2. if I need to use my account, e.g. to tweet something or to write someone
        1. write the tweet down and wait till the long break
        2. set the phone timer for 5 minutes, and open Twitter then
  8. Chrome
    1. if I need to open a webpage for something, I'm only allowed to get one page away from what is immediately needed for the task, e.g. I'm writing this post and I want to see what Wikipedia writes about GTD, so I'm allowed to go to GTD, then from GTD to David Allen (and to any other page linked at GTD) but I'm not allowed to open any pages from David Allen. This is the exploration / exploitation tradeoff I enjoy
    2. if I'm researching something, I'm only allowed to open one page at a time, e.g. when I'm doing a literature review, I open the page, read it, close it, and then open the next one. This prevents me from accumulating a unread tabs too fast
  9. if I have an ongoing real-time conversation with someone during the long break, I can continue it
  10. if I'm chatting with someone IRL and I need to open social media because of this, e.g. to send the person I'm chatting with a tweet, I'm allowed to open social media anytime, but only for that specific thing
  11. if I want to listen to a music, 5 minute timer (because music is somewhat distracting and starting listening to it impulsively is dangerous)
  12. If I just published a post, I can check social media whenever for the next 6 hours.

I have similar rules for all the other stuff, key being that I must either ensure that I don't get distracted (incognito) or to make sure I'm not doing anything impulsive (5 minute timer, other person as a trigger). If I don't follow this, it may be impossible to distinguish between opening, e.g. Twitter because I really needed it or because I really wanted to check my notifications. And if I can't distinguish between these two, then the simplest explanation is the impulsive one, which means that I'm losing control. Which is really, really bad.

My model of the brain is the following: if it believes that you will do as you planned to do, it will occasionally probe you with impulses, but nothing major, and you will be able to do what your "system 2" Where system 1 is "fast" and system 2 is "slow". wants to do. But if you start to give in to these impulses, the brain increases their power and their frequency, and naturally, you start to give in more and more, right until the point when you do nothing but what your impulsive brain wants you to do in the moment.

Note that being aware of doing something on impulse doesn't magically remove the adverse consequences of the action on rules you're breaking, i.e. being aware of breaking a rule does not make it ok to break it.

Rules are about exceptions

Why the hell does my "some additional rules" list look like the table of contents for War and Peace? Why don't I just say "distractive stuff allowed only during long breaks"? Because rules are about exceptions. Exceptions are inevitable. Sometimes I will have to open a social network or check email during pomos. If I didn't have exceptions explicitly written down, then I would break my rules, thereby decreasing their future strength. I wrote up a formal model of this process in my Bachelor's thesis.

Still, it's impossible to write down all exceptions, which means that sometimes I still break my rules and sometimes the rules break down completely. Thus, I need a way to reinstitute them.

Interlude: guilt

When a rule / productivity system breaks and you start wasting your time, don't take it too personally and try not to blame yourself. The only thing you can do now is write your observations down, think about how to improve them for the future, and then try to implement them.

Rules stopped working. What next?

First of all, an important qualification: rules don't just stop working. They stop working in specific contexts, e.g. if you go to a novel location, you will find that the old patterns of behavior — whether productive or destructive — are much weakened there. This suggests a natural solution to the problem of reinstating broken rules: go to an unexplored coffee shop or library and work there until the new location breaks down. Often, people work in libraries and coffee shops precisely for this reason — because these locations allow to create a novel self-pattern specifically for them. The problem with this is that if there are too many possible locations, there's no incentive to maintain the rules structure in them, so they break down too fast.

I found that I have 3-4 places in town I especially like and rotate my work between those in which rules work at the moment (the rule following ability seems to reset naturally within several days-weeks-months for each location). I have one specific place in which the rules are extra strict, meaning, upon entering this place I turn off mobile internet on the phone and can only turn it on after a 5 minute timer.

Pro tip: you don't usually go from 0 procrastination to 100 in an instant. If you learn to recognize when you're at 20 and switch the location preemptively you will save yourself a lot of time.

Physical place is not the only context that can be broken. For example, if you have trouble playing games on your phone everywhere, buying a new one and committing not to play any games on it from the very start can work very well (at least for some time).

Bullshit test for the previous section

The previous section is exceptional in that you can test whether what I wrote there is bullshit rather easily (this is basically the same thing as in the beginning of this post):

  1. Think of a task you're currently putting off, even though you shouldn't.
  2. Open Google Maps and find a particular coffee shop/library/etc. you've never been to.
  3. Go to that place strongly committing to only doing the particular thing you decided to do there and not get distracted either physically or mentally on other things. This means as little chatting with people as possible; no checking email (unless it's absolutely needed); no checking social media; not doing anything at all that is not directly related to the task at hand. Basically, maintaining the rule structure similar to the one I described in the beginning of the post.
  4. Notice how easy or difficult the task was to work on and how easy or difficult it was to not let yourself be distracted, compared to your usual environment.

My prediction: however off-putting and ughy the task is, there's going to be close to none difficulty in concentrating on it and there will be no other issues that usually prevent you from accomplishing it.

Break rules sometimes

Adhere to anything religiously enough and you start to forget why you decided on doing it in the first place. With rules turning into Chesterton's fences. Stoics suggest negative visualization and occasional intentional deprivation of things we take for granted, e.g. living off bread and milk for a few days; I suggest to occasionally forget all the rules you have for a while and see what happens. This will happen naturally at some rate, due to rules being gradually broken, but sometimes it might be worthwhile to explicitly let yourself not be bound by any rules designed to enhance productivity and see how it pans out. It seems that this kind of experience may serve as a sort of a springboard for the future.

For example, a couple of times per year I clear out several days, during which I play Civilization 5 for 16-18 hours a day. At some point, I become so nauseated and frustrated by the game that I naturally just can't play it anymore. The realization of just how easily I just sent like 50 hours of my life down the drain helps to gain the perspective on a lot of stuff, including why I avoid videogames so scrupulously at all other times. Spending a day procrastinating / playing videogames is equivalent to reading a book for an hour every day for two weeks.

A couple more tips

How to not forget about productivity tricks?

You can literally just add a reminder a week / a month in the future which asks if the new productivity trick is used and, if not, why.

Podcasts and audiobooks

If you think you can't listen to podcasts and audiobooks, you're probably wrong. Just speed them up and you'll be able to concentrate on them just fine. You probably have at least 30-60 minutes of downtime every day available for audio content. That's basically a free hour a day to read a book. The other possibility is that you've just tried to listen to a wrong book and gave up too early! I found that about half the books I can't listen and need to actually read.

The way to highlight books while listening to them is to catch an identifying phrase near the segment you want to return to, write it down to a special file, and after having finished the book, open the written version of the book and find all the highlights there, using the identifying phrases.

See my audio content apps recommendations and headphones recommendations in my Tools / Gear post.

Browser tab management

Tabs have a tendency to blow up. However, there's a natural upper limit for how much the can blow up, since at some point they overflow and you no longer have access to the rightmost tabs. This is so frustrating that we can naturally turn this to our advantage. So, create two soft rules: à la FIFO.

  1. only close tabs from the left-hand side
  2. only open tabs in the end (but can close just opened tabs).

These rules are pretty easy to maintain (since you can still do anything you need while following them) but they're really frustrating to abide by, which creates a really good incentive to clean tabs up. The equilibrium is to have about 50-80 tabs (depending on the size of your monitor), just about to overflow. Note that you might need a lot of RAM (like 16gb) for this to work.

Figuring out the core issues behind procrastination

I will quote Marco Vega of Sapien here:

99% of the time procrastination is not about you being lazy or lacking work-ethic. Your body/brain is sending your some important information about the tasks at hand, and it's important that you listen to those signals empathetically.

For example,

A - The task requirements and goals might not be clear enough. If you are trying to get yourself to "plan for a project" or "write a book" then it's hard to identify the next actionable items. Put some time aside to figure out what physical things you can do to move the project forward. Try break down the larger tasks into the smallest pieces possible. The goal of the project might need identifying, or the requirements fleshed out from a supervisor.

B - The task might exceed your current competency. Sometimes we know what we have to do, but don't know how to do it, and then we become avoidant rather than admitting this. In this case, it's worth figuring out what you do know how to do and what you don't know how to do, and be honest with that. Then slowly ask for help or read up on the things you don't know.

C - The tasks might really not be worth it. Sometimes you are assigned tasks that don't actually help you achieve your long-term goals, and so your brain demotivate you from doing them. Maybe the payoff is low, maybe you don't learn anything new from them, or maybe a colleague you don't like will gain credit for the tasks, or maybe you just wont be rewarded or appreciated for getting the tasks done.

As a general rule of thumb. If you notice yourself procrastinating, don't beat yourself up about it. Just notice the behaviour and put some time aside to have an honest conversation with yourself for why you might be unconsciously avoiding these tasks. There is no shame here. It's very difficult to move forward without self-empathy and self-understanding. 'Pushing yourself' is OK in small doses, but if you make it a habit, you are increasing your chances of burnout!

Task order

If you just have a bunch of things to do for the day, try using random.org to decide on the order of your tasks. This both

  1. doesn't allow you to just do the easiest tasks first
  2. makes the very act of choosing the next task pretty exciting

Help me make this post better!

If you've read to this point, I would guess you enjoyed the post. If you decide to try any of the tricks I describe — do let me know, so that I check back with you in 1, 3, and 12 months and see if this post is actually helpful in the long-term.

Acknowledgements

Thanks to Anastasia Kuptsova for many helpful comments.

Also see

Productivity by Sam Altman

Bibliography

Many people influenced my thinking on productivity. In particular, I should note:

Scott Adams' How to Fail at Almost Everything and Still Win Big (probably the most important book I've ever read)

Scott Alexander's The Lottery of Fascinations

David Allen's Getting Things Done

Tiago Forte's The Holy Grail of Self-Improvement

Malcolm Ocean's A ritual to upgrade my Face

Roko's Ugh fields

Adam Strandberg's Time Depletion

Between ages fourteen and sixteen I read a lot of Scott H Young and Tynan and some Steve Pavlina, so although I don't remember almost anything they wrote, I'm pretty sure there are some traces of their ideas in me.

Stuff with similar thoughts I discovered while writing this post

Peter Hurford's How I Am Productive

Scott Alexander's Applied Picoeconomics

Eliezer Yudkowsky's Execute by Default




All Comments: [-] | anchor

ssorallen(4168) 3 days ago [-]

RE: 11.3 Browser tab management

No manual, rule-based tab management ever worked for me, which is why I took over development for Tab Wrangler a while back and have been continuing to update it. Garbage collection for old tabs without me having to intervene.

This has been the best system for me, and it has worked well for years. Old tabs are automatically cleaned up when I haven't looked at them in a while.

It's all open source: https://github.com/tabwrangler/tabwrangler

mackrevinack(10000) 3 days ago [-]

dayum. that's sounds like something I really need. every time i sort out my tabs it takes only a week before it's back to normal.

when the tabs are closed and stored are they put into one folder or can they be put into month folders?

bobowzki(3765) 3 days ago [-]

Whenever I read one of these posts I'm always very curious about the age of the writer. I find my thoughts on this topic has changed a lot from mid twenties to mid thirties.

guzey(10000) 3 days ago [-]

Hi! I'm the author of this post. I'm 22. I wrote most of this post when I was 20. I still agree with basically all of it. Also, I spent like 6 months writing it because most of the productivity advice is bs and wears off quickly and I wanted to make sure I only include things that stick.

arjan_sch(4196) 3 days ago [-]

Esp. when reading the part about 15 working hours per day.

Draiken(10000) 2 days ago [-]

I often get surprised how deep we got hooked into the system. We tell ourselves unless we're doing something (no matter how useless that is) we're worthless. Everything becomes a different kind of procrastination.

We feel productive creating yet another CRUD app, idle game or advertisement optimization tool that ultimately does nothing for anyone but the capitalists on the top of the chain. Only to feel bad about doing anything that doesn't generate profit to someone.

It saddens me that I don't see a way out of this hole. The system has won and everyone either follows it willingly or is forced to by society.

If only I could be like the OP who seems to live happily in this productivity cycle without gazing into the abyss that is the meaninglessness of it all.

zettatron(10000) 2 days ago [-]

I'm very glad you have raised this point. This concept of productivity is quite new. Unfortunately, it has not remained flexible enough to offer the same kind of meaning today, that it did for workers at it's inception.

Fortunately, many engineers, among others in technology-centric roles, have co-opted this definition to mean their intellectual output.

Depending on whether or not an individual's intellectual currency is manifested via productive output, this hyper-productivity needn't be unhealthy in itself.

It is when this desire overtakes those who don't derive value from the 'hobby' of productivity that it becomes dangerous.

scottishfiction(10000) 2 days ago [-]

There are places where you can find well-paid work that isn't for 'capitalists at the top of the chain'. Well, at least not directly. Do you think you'd find the public sector equally meaningless?

CPLX(3303) 3 days ago [-]

This is actually a pretty good list. Obviously it's personal and specific to him but it gives food for thought.

With that said I've read a million posts like this and the only thing that's ever made me feel like I had genuinely changed my perspective and understood what was happening more clearly was reading the Getting Things Done book.

The key insight for me is that so much of procrastination results from a lack of clarity about what exactly should be done next, and keeping a mental load of trying to keep track of everything. Separating the three basic concepts of planning, making decisions, and actually doing the work, into discrete sessions, has been a life changer.

I still fuck off constantly and hate myself for procrastination from time to time, obviously, but using the core GTD framework and returning to it when I stray has really helped.

reubenswartz(4005) 2 days ago [-]

Yes. And if you're not sure what to do next, the next thing to do is figure out that next step. Sounds circular and silly, but helps tremendously to realize, hey, I got stuck, let me take a step back and figure out what I'm really trying to do and what tasks I need to complete to get it done.

toxik(10000) 3 days ago [-]

I would definitely recommend against having an hour's commute to work. It is a really strong predictor for life dissatisfaction. Fifteen minutes is the number I heard to be a good balance.

tomlong(10000) 3 days ago [-]

I currently have an hours commute, which I find works quite well for me. It is a little bit unusual though - it is a train journey with no changes, I always get a seat, and I live about a 2 minute walk from the train station, and work about a 2 minute walk from the train station at the other side. The train journey is about 50 minutes.

It's a good period of time to slowly engage and disengage - I do the things that might affect my productivity in the day - read HN, catch up on personal (and sometimes work) email. I sometimes read a book, or message friends.

Everyone's different but this works nicely for me - when I get in from work I tend not to feel the draw of the screen, so my leisure time is less at risk from being sucked into for example reddit, and when I get to work, the period of the day that in the past I have previously lost a lot of productivity to - a 'few minutes' on catching up with the world, 'relevent' tech news etc I've already scratched that itch.

I think it just sort of works with my fairly ill-disciplined personality traits in a way that helps me be productive.

I think much longer though and I would feel I was losing too much of my day. (I'm typically out of the house 07:00-18:00).

galfarragem(564) 3 days ago [-]

> Fifteen minutes is the number I heard to be a good balance

I would rephrase it as: a 15 minutes walk is the sweet spot of commutes.

Tade0(10000) 3 days ago [-]

My coworker has a 1.5-2h commute, which I couldn't wrap my head around until I found that he has two children.

This is his 'me time'.

eitland(3409) 3 days ago [-]

I typically have about an hour of commute each way adding up to two hours a day.

It kind of works. Lately it has been 1.5 each way which starts eating seriously into other things.

The things that makes it tolerable are typically:

- if I can work while commuting, because that means shorter time in the office. Ironically - and depending on project - I can get more work done on the train than in a similar time slot in the office.

- if I can work on a side project

- if not, if I can read or study

- if I could sleep (I find it hard, but sometimes it works)

But it is tiring to be away from home 10-11 hours a day just to fill 40 hours.

traviscj(10000) 3 days ago [-]

Piling on, I'd also recommend against a 15 hour workday.

dahart(3778) 3 days ago [-]

Yep. I'll add though, personally I was happier doing a 90 minute train commute than a 45 minute drive by car. Seemed doubly productive because I spent the commute time working on a fun side project and not battling with my negative inner monologue about other drivers.

tonyedgecombe(3983) 3 days ago [-]

Fifteen minutes does seem a good time, too far to pop in on a whim but not so far that the daily journey is onerous. Bonus points if it's a fifteen minute walk.

rootusrootus(4203) 3 days ago [-]

That seems like good advice. I currently commute 20 minutes and it's fine. Not too long, but long enough that work is far away. But if it were half again as long I'd probably get stabby.

Reedx(765) 3 days ago [-]

Strongly agree, though if you find yourself in that situation there are ways you can turn it into productive time. Especially if you otherwise don't have blocks of alone downtime.

- Use the time to think. Driving the same commute every day becomes automatic and you can solve problems, practice a talk, iterate or come up with ideas and so on. Take notes via voice.

- Podcasts. There's an endless amount of learning available. Some recommendations: 99% Invisible, Making Sense, Hardcore History, Freakonomics, Business Wars, Planet Money.

- Audiobooks.

Make it a habit and you can actually get to the point where your commute is almost painless and even something to look forward to (particularly if it's the only place you listen to podcasts/audiobooks). Sometimes I actually end up waiting in the car extra time for a Podcast to finish or to clean up notes.

timerol(10000) 3 days ago [-]

> If you aren't working from home, your workplace should be at least a couple of minutes away (better: an hour away)

The notion of a commute being beneficial is weird to me. If it takes more than 5 minutes to get to work, I end up spending the time engaged in something else, which leads to me being significantly distracted when I arrive. Thinking for an hour about what to do when you arrive seems like the opposite of productivity

Scarblac(10000) 3 days ago [-]

I travel about an hour, but I can read or work in the train. It's not that bad.

I need the ~10 minutes it takes me to talk home from the train station to switch between work mode and be-nice-to-excited-kids mode.

wry_discontent(4193) 3 days ago [-]

By happenstance I just ended up in an apartment that's a 5 minute walk from my office. It's the best thing that's ever happened to me. I go home for lunch. I spend 0 time commuting, I'm basically home as soon as I'm ready to leave.

Compared to my other jobs, which tended to have more like a ~30 minute commute, I feel like I have so much more time in the day. I'm never taking a job where I have to commute again. It sucks the life from me.

mdszy(10000) 3 days ago [-]

My workplace is 10 minutes away, I would hate life if I had to commute an hour every single morning for work.

freediver(3930) 3 days ago [-]

The simplest productivity hack is finding passion. I never seen a windsurfer procrastinate. If you are truly enjoying your work and can't wait to do it, there is nothing going to stop you.

lm28469(4203) 3 days ago [-]

For most people as soon as your passion becomes your money earning occupation it ceases to be a passion.

'Follow your passion' requires a lot of willpower and very often more work than a salaried position, unless you lower your standards drastically.

stared(624) 3 days ago [-]

The worst productivity advice I am hearing over and over.

I ensure you that there are many passionate writers, graphic designers and... programmers, who procrastinate a lot.

One of a big finding for me was that it is NOT nearly as much if I like something or not[1]. Is about knowing the next step and maintaining focus (especially with keeping the time 'now' not 'not now'). (However, I speak from ADHD perspective; maybe for non-ADHD people passion is sufficient.)

[1] For a stark comparison: I procrastinate on open-ended side-projects I love (sometimes for years), but when there is some accounting issue I solve it ASAP.

wry_discontent(4193) 3 days ago [-]

Unfortunately, that's not an easy thing to do. It's not practical for 90% of people. Being software developers, we're lucky, but even though I love what I do, I often hate doing it for other people.

generatorguy(10000) 2 days ago [-]

Windsurfers always sitting on the beach not windsurfing!!

kranner(3815) 3 days ago [-]

> If you are truly enjoying your work and can't wait to do it, there is nothing going to stop you.

Clearly you don't have an anxiety disorder.

billfruit(4152) 3 days ago [-]

Sometimes one can be in a situation of the Buridan's Ass, there are multiple things interesting and exciting, but one can't fully commit to any of them, and end up doing none of them.

CPLX(3303) 3 days ago [-]

Find someone who has to windsurf for a living and you'll find one who procrastinates.

captainbland(10000) 2 days ago [-]

That's more than likely because the moment you do start procrastinating as a professional wind surfer, like in any sport, you immediately get ousted by every hopeful who's willing to put in the hours to get access to an extremely limited pool of things that pay in that field.

thomas(2290) 3 days ago [-]

Serious question: Is anything less productive than reading other people's productivity thoughts? It's a combination of procrastination and finding out what works for someone who is presumably more productive than you (ie: guilt).

egypturnash(10000) 2 days ago [-]

You never know when you're going to find the next trick that will actually work to fool your brain into letting you get shit done for a while. These days I mostly rely on my own version of the Pomodoro method but I have definitely found value in other ideas now and then.

Ensorceled(10000) 3 days ago [-]

In the article you just read, the author writes an entire section: '2. Every productivity system stops working eventually and there's nothing you can do about it'

If you're like me and have ADHD, you read looking for your next system that will work for a while.

rgoulter(10000) 3 days ago [-]

Not everyone who comes across this post will be spending more time reading about productivity than being productive.

The term I've heard for that is 'productivity porn'. (as in, getting the satisfaction from observing productivity, without actually putting in the effort yourself).

vadansky(10000) 3 days ago [-]

I was procrastinating and read about the Pomedoro technique and it worked pretty well for me. Sometimes you need a trick I guess.

marmada(10000) 3 days ago [-]

I think it can actually be very productive if you engage with the article and take notes.

Example: I read the following article over the course of 2 days. While I was reading it, I had VSCode open and was taking notes in Markdown. While this article only mildly changed how I acted, it definitely planted a few seeds in my head, which I am trying to cultivate. And now I have a source of notes on productivity I can look at any time!

https://www.lesswrong.com/posts/P3zrurj5hHKFKDL3M/productivi...

adamsea(4204) 3 days ago [-]

Commenting on them ;)

pleasecalllater(4198) 3 days ago [-]

One less productive thing I can think about is reading a comment about how unproductive reading thoughts about productivity is :)

dfboyd(10000) 3 days ago [-]

I looked at the guy's pie charts of working time, and unless I'm reading them wrong, he _never sleeps_. He has all 24 hours of the day marked as mostly work.

lanekelly(4198) 2 days ago [-]

He's mapping his working time onto the pie chart. For example, if he works 9am-5pm, during that time he follows the pie chart's structure for that block of time.

naushniki(10000) 3 days ago [-]

I wonder when achieving personal productivity has become a widespread problem and why did this happen.

zettatron(10000) 2 days ago [-]

The democratization of capitalism oversold the benefit of productive output

Scarblac(10000) 3 days ago [-]

I need just one thing: a way to get rid of a bad habit that doesn't involve changing the circumstances in which I have that habit.

I'm a Web developer with a mindless tic-like habit of opening sites like Reddit and HN all the time. Even a second after closing it. I can't very well get rid of Web browsers.

Edit: ohh, but the tip of going to a place where I've never procrastinated before and sitting down to think what I actually want to do next, that doesn't involve a Web browser. Nice.

trefn(2832) 2 days ago [-]

I have a similar tic. I just edit my hosts file to redirect those requests to localhost. Then when I mindlessly navigate to one of those sites it doesn't work and I realize I did it unconsciously. I do the same for gmail because I have the same tic for checking my mail.

If I actually want to go to one of those sites I just edit the hosts file and comment out the redirect line.

On my mac, the /etc/hosts file looks like:

  # #
  # Host Database
  #
  # localhost is used to configure the loopback interface
  # when the system is booting.  Do not change this entry.
  ##
  127.0.0.1 localhost
  255.255.255.255 broadcasthost
  ::1             localhost
  fe80::1%lo0 localhost
  127.0.0.1 news.ycombinator.com twitter.com www.facebook.com reddit.com
  127.0.0.1 mail.google.com
vadansky(10000) 3 days ago [-]

Edit: ohh, but the tip of going to a place where I've never procrastinated before and sitting down to think what I actually want to do next, that doesn't involve a Web browser. Nice.

That is a great idea, but what do you call it when I do not want to do it because I start thinking about all the awesome things I could have done in my life if I started doing that earlier?

noah_boy(10000) 2 days ago [-]

This works great for me: http://www.stayfocusd.com/

You can limit your time on specified sites

misterman101(10000) 3 days ago [-]

What's worked for me every time I get in that routine is to force myself to do one to five push ups every time I think about switching tabs.

During that time I think about what it is that I really want to be doing. After a while I automatically skip to this step without having the desire to refresh HN.

jedberg(2244) 3 days ago [-]

> If you aren't working from home, your workplace should be at least a couple of minutes away (better: an hour away)

One great hack for this is if you work from home, get up and get dressed, then go out for a walk around the block as your 'commute' to work. At the end of your work day, take a walk again in the opposite direction as your commute home.

Even though the ploy is totally obvious, it will put your mind in a work vs play mode depending on the direction of the walk.

ryanstorm(10000) 3 days ago [-]

Great idea. This also sounds a lot like it falls in with the 'ritual' concept from Deep Work by Cal Newport.

guzey(10000) 3 days ago [-]

This is a great idea and I used to do it for a while when I tried working from home. The problem for me is that I usually give on this ritual after a week or two. How long have you been able to sustain it for?

birdman3131(10000) 3 days ago [-]

An hour away seems like a waste of 2 hours of your day to a commute.

frlnc_throwaway(10000) 3 days ago [-]

I highly recommend Jean Moroney's blog. She digs deep into the psychology behind productivity and goal-setting. I found many unique insights in her articles. Thanks to her writing I've come to believe that procrastination often stems from deeper emotional issues or an unacknowledged clash of priorities (for example, when you try to force yourself to do something that you don't, in fact, want to do, the deeper issue is that you haven't resolved the clash between your short-time desires and your long-term goals).

E.g:

https://www.thinkingdirections.com/dont-motivate-yourself-le...

https://www.thinkingdirections.com/three-steps-to-following-...

General list here:

https://www.thinkingdirections.com/category/time-management/

mrieck(10000) 3 days ago [-]

More and more research is showing procrastination to be an emotion management problem. [1]

That's why it's been really helpful for me to start keeping a journal. It helps increase self-awareness and helps when I get off-track on my side projects. I can can realign my goals and ease back into being productive again.

[1] https://www.fastcompany.com/90357248/procrastination-is-an-e...

dominotw(2041) 2 days ago [-]

> So often, people take on tasks because they think you have to do them, or the task meets some criteria that someone else has set. You feel you have to do the project because it's required for a class, or your boss told you to do it, or it's the only way you see to get the credential you need to take the next step in your career.

Remided me of passage from J.K's book.

'There are two kinds of action. One brings you reward, and the doing of it strengthens the ego, the 'me'. The other kind of action, the action which you love to do, has no reward or punishment and is not concerned with what the neighbour says, or with gods or with the priests or with belief. You do it because it is the only thing to do. You rejoice in the very doing of it, not for heaven or the avoidance of hell. You just do it and in the very doing of it is the delight. This action is of freedom from society and has nothing whatsoever to do with morality. This action is from nothingness. When there is this, you can look at the world from that silence of nothingness.'

Seems like you need all kinds of productivity tools/hacks for the former and none for the latter.

> clash between your short-time desires and your long-term goals

Or maybe its a clash between the artist in you with cog in the machine that you need to be live the life you want

viach(3551) 3 days ago [-]

> for example, when you try to force yourself to do something that you don't, in fact, want to do

This is called a day job, even for programmers.

> you haven't resolved the clash between your short-time desires and your long-term goals

Is it even possible to resolve? One would like to read books (play guitar, gardening) for the rest of his life instead of trying to get his docker-compose file working properly. How a productivity technique could resolve such a discrepancy between goals and desires?

crb002(4110) 3 days ago [-]

Needs to add in exercise.

guzey(10000) 3 days ago [-]

I experimented a lot with exercise. I found that it doesn't influence my mood, productivity, etc. at all, so I didn't include it. Admittedly, I'm an outlier here.

nurettin(3954) 2 days ago [-]

My productivity tip is to get old. If you are too young for that, watch an old person complete an overwhelming task. Model after their resilience and patience.

nurettin(3954) 2 days ago [-]

And I don't mean old as in obese diabetic vitamin d and iron deficient cranky old. I mean reliable old.

vinceguidry(10000) 3 days ago [-]

Personal productivity deals in the hoary space between defining who you are as a person and figuring out the best way to move reality towards the vision.

If you're not treating subconscious input with as much seriousness as humanly possible, (i.e. you're procrastinating for a good reason, one that probably relates to why you think this thing you're procrastinating doing is the best way to move the needle on that vision) then you will forever be living in a hell of your own making.

Making yourself is an artistic pursuit and should be treated with every bit as much care and nurturing as making a painting.

maxerickson(1026) 3 days ago [-]

How else can you reach your final form!

gandalfgeek(2245) 3 days ago [-]
tonyedgecombe(3983) 3 days ago [-]

That's more interesting than the title implies.





Historical Discussions: Rails 6.0 (August 16, 2019: 588 points)

(589) Rails 6.0

589 points 6 days ago by vipulam in 3589th position

weblog.rubyonrails.org | Estimated reading time – 4 minutes | comments | anchor

Dealing with incoming email, composing rich-text content, connecting to multiple databases, parallelizing test runs, integrating JavaScript with love, and rewriting the code loader. These are fundamental improvements to the fundamentals of working with the web and building fast and fresh applications. This is the kind of work we've been doing for the past fifteen years, and we're still at it. THIS IS RAILS SIX!

And that's just the headline improvements! Since Rails 5.2 was released a little over a year ago, we've continued our high pace of improvement all over the framework. In 2019 alone, we've had 341 code contributors submit improvements and fixes. We've tried to summarize just some of the goodies in the release notes, but there are many more than that.

While we took a little while longer with the final version than expected, the time was spent vetting that Rails 6 is solid. In fact, GitHub, Shopify, and Basecamp, as well as plenty of other companies and applications, have been running the pre-release version of Rails 6 for months and months in production. We might not have caught everything, but if it's good enough for GitHub, Shopify, and Basecamp, it's probably good enough for you too!

So what are you going to get with Rails 6? Check it out:

Action Mailbox routes incoming emails to controller-like mailboxes for processing in Rails. It ships with ingresses for Amazon SES, Mailgun, Mandrill, Postmark, and SendGrid. You can also handle inbound mails directly via the built-in Exim, Postfix, and Qmail ingresses. The foundational work on Action Mailbox was done by George Claghorn and yours truly.

Action Text brings rich text content and editing to Rails. It includes the Trix editor that handles everything from formatting to links to quotes to lists to embedded images and galleries. The rich text content generated by the Trix editor is saved in its own RichText model that's associated with any existing Active Record model in the application. Any embedded images (or other attachments) are automatically stored using Active Storage and associated with the included RichText model. The foundational work on Action Text was done by Sam Stephenson, Javan Makhmali, and yours truly.

The new multiple database support makes it easy for a single application to connect to, well, multiple databases at the same time! You can either do this because you want to segment certain records into their own databases for scaling or isolation, or because you're doing read/write splitting with replica databases for performance. Either way, there's a new, simple API for making that happen without reaching inside the bowels of Active Record. The foundational work for multiple-database support was done by Eileen Uchitelle and Aaron Patterson.

With parallel testing support, you can finally take advantage of all those cores in your machine to run big test suites faster. Each testing worker gets its own database and runs in its own thread, so you're not pegging one CPU to 100% while the other 9 sit idle by (y'all do have a 10-core iMac Pro, right 😂). Hurray! The foundational work for parallel-testing support was done by Eileen Uchitelle and Aaron Patterson.

Webpacker is now the default JavaScript bundler for Rails through the new app/javascript directory. We're still using the asset pipeline with Sprockets for CSS and static assets, though. The two integrate very nicely and offer the best trade-off of advanced JavaScript features with an it-just-works approach to other assets.

Xavier Noria's new Zeitwerk code loader for Ruby. No more const_missing, no more code loading gotchas, hello Module#autoload!

Those are just some of the marque additions, but Rails 6.0 is also packed with minor changes, fixes, and upgrades. Just some I'd call out: Proper Action Cable testing, Action Cable JavaScript rewritten in ES6, protection against DNS rebinding attacks, and per-environment credentials. Also, Rails 6 will require Ruby 2.5.0+ now. You can check out everything in the individual framework CHANGELOG files for the nitty-gritty rundown.

This release was shepherded by release manager Rafael França with support by Kasper Timm Hansen.

Thanks again to everyone who keeps working on making Rails better! Thanks to everyone who uses Rails! I'm incredibly proud to see this open-source framework continue to thrive outside the pressures of market terms and reciprocal guilt. This is a gift we give each other and expect nothing in return.




All Comments: [-] | anchor

jacobkg(4128) 5 days ago [-]

The multiple database support is really exciting. Anything that makes it easier to push database reads to replicas is huge!

gotts(10000) 5 days ago [-]

The cool thing about multi-database feature that's coming to Rails is setting like

>config.active_record.database_selector = 2.seconds

Rails provides a way to make sure that at-least for first two seconds after the write operation, read requests are handled by the write instance. This assumes that your read replica will get updated within 2 seconds with the live data.

How cool is that

hirundo(3695) 5 days ago [-]

We've been doing this for years on a Rails 3 app. We yield from a `slave` method, with the connection pointing to the replica DB until yield returns. It's just a short method. It's great if Rails now supports this directly, but a shame if people have been waiting for it.

phodo(915) 5 days ago [-]

What is the lightest weight, best practice / pattern for using rails, not django/flask, to call python-based programs (e.g. python machine learning inference such as tensorflow/pytorch, or other scientific computing like scikit or numpy) on a lean server (keeping costs down)?

ewalk153(10000) 5 days ago [-]

I would probably write a flask wrapper and expose an api to the Rails app. I've seen this done successfully at scale in a large Rails app.

cutler(4198) 5 days ago [-]

I think Node.js is the better choice for this than Rails. You want your Python number-crunching API accessed async leaving Node to take care of auth, logging and database reads. With your Python API returning JSON I would imagine Rails serialisation would slow down performance relative to Node.

bbmario(10000) 5 days ago [-]

What is the fastest web server for Rails today? I've always wanted to run something serious with Rails, but the performance always puts me off.

jacobsenscott(10000) 5 days ago [-]

Any server is fast enough. If your web app is literally doing anything other than rendering empty pages the time a request spends in the web server will be an insignificant fraction of your total response time.

WJW(4144) 5 days ago [-]

You should probably ask Github, they run their main site on Rails.

bbmario(10000) 5 days ago [-]

Wow, got down voted for asking a question on which web application server I should use with Rails today. Really?

prab97(10000) 5 days ago [-]

Do you plan to run something more serious than github, basecamp or shopify? Usually it is not rails, which is the bottleneck, but the architecture of your application.

wasd(1611) 5 days ago [-]

Puma is definitely the fastest or one of the fastest. It's also the default in Rails now. https://github.com/puma/puma

dwheeler(10000) 5 days ago [-]

I don't know if it's the fastest, but I've had good success using Puma with Ruby on Rails in a real-world production site. That said, if performance is your goal, then the web server is only a small part of it. In my experience, the way to get a high-performing Ruby on Rails application is primarily through fragment caching and using a CDN. You can get excellent performance with very little effort that way.

chachra(4202) 5 days ago [-]

Support for multiple databases? Django added that 9 years back!? - https://django.readthedocs.io/en/1.4.X/releases/1.2.html#sup...

Are they both talking about the same thing, or I am missing something?

peteforde(1246) 5 days ago [-]

It's a lot more than talking to pg and mysql at the same time, thankfully!

https://edgeguides.rubyonrails.org/active_record_multiple_da...

In short, to do this feature family properly and in an opinionated, predictable way means that you have to cover a number of non-trivial cases. Multiple primaries and their replicas is just the beginning:

- automatic connection switching on a per-model basis

- Automatic swapping between the primary and replica depending on the HTTP verb and recent writes

- Rails tasks for creating, dropping, migrating, and interacting with the multiple databases

Sharding, cross-cluster joins and load balancing replicas are coming soon.

I think what's important here is that every site over a certain scale previously needed to implement their own take on this, and there were a lot of thorny edge cases, race conditions and heisenbugs to do so. It's kind of like the old chestnut that if you find yourself writing original crypto code, you're probably doing it very wrong.

The fact that any project can uncomment a few lines and get all of this for free is properly awesome.

reneherse(1746) 5 days ago [-]

Great news, and a hearty congrats and thank you to all the contributors!

As a frontend designer, I'd been hoping for some mention of faster SASS/SCSS compiling, but it looks like it's the same asset pipeline for that. Can anyone recommend a current resource for decreasing compile times?

I love designing in a Rails app, but my non-Rails projects all seem to inject CSS changes into the browser almost instantaneously, whereas with Rails there's often a wait of up to 10 seconds. The lag is pretty disruptive to the design workflow. I'm pretty new to Rails so any tips on what I might be doing wrong would be appreciated :)

tomphoolery(4199) 5 days ago [-]

sassc-rails is helpful since it binds to libsass.

peteforde(1246) 5 days ago [-]

Definitely switch over to sassc-rails, which should give you a massive speed boost.

I also run an instance of bin/webpack-dev-server which handles all of my dev-time Sass pushing needs.

One thing I'm a little sad about is the decision to continue using sprockets (aka Asset Pipeline) for CSS and images. My understanding is that a lot of gems would break if they changed but I can't help feel an opportunity was lost to bite the bullet and do the big switch all at once. I guarantee you that it's coming.

Luckily, I can say that switching to using Webpack for CSS is very doable. The best instructions are here: https://gist.github.com/andyyou/834e82f5723fec9d2dc021fb7b81...

sansnomme(3899) 5 days ago [-]

Use Webpack for SCSS (and if necessary enable PostCSS, though I am pretty sure Rails' Webpack config supports SCSS by default) and enable livereload, the problem will fix itself.

heeebeeejeebees(10000) 5 days ago [-]

Finally made it! Congratulations!

Zawinski's Law "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."

--- Action Mailbox routes incoming emails to controller-like mailboxes for processing in Rails. It ships with ingresses for Amazon SES, Mailgun, Mandrill, Postmark, and SendGrid. You can also handle inbound mails directly via the built-in Exim, Postfix, and Qmail ingresses. The foundational work on Action Mailbox was done by George Claghorn and yours truly.

rxhernandez(10000) 5 days ago [-]

I sincerely mean no harm by this but:

Web developers are to the rest of developers what internet people in the us are to the rest of the world. That is, I think y'all forget that there are many other types of development that look nothing like web development.

I think I might come very close to losing my mind if I worked somewhere where someone arbitrarily sticks code to read mail in the firmware I was working on.

craz8(10000) 5 days ago [-]

Hah! And with not a lot of code and an ActionText integration, you can write some nice looking emails too!

I can build my own Gmail in an hour!

jjeaff(4090) 5 days ago [-]

Funny, I first saw this law mentioned yesterday in a discussion about a new PHP Laravel package for handling incoming mail.

thrownaway954(4200) 5 days ago [-]

Congrats on shipping!!!

Been doing Rails for the past year coming from a C# and ColdFusion background. God I love what you can accomplish in a short amount of time with Rails.

ColdFusion became a nightmare since there is NOTHING for it any more.

C#, though awesome, is just too verbose.

Rails and Ruby are a God send.

bdcravens(1015) 5 days ago [-]

I'm a long term ColdFusion developer (since 1999) - about 5 years ago we (the company I work for; I have autonomy to choose the stack and tools) switched to Rails (from CF) for the ecosystem.

imafish(4157) 5 days ago [-]

I'm considering learning rails for a new project. Also coming from a .NET background. Any good resources you can recommend?

buf(3346) 5 days ago [-]

It's christmas eve already?

gls2ro(3959) 5 days ago [-]

I think you are confusing Ruby (the language) with Ruby on Rails (the framework).

Ruby - the language - is the one released usually on Christmas.

masonhensley(3916) 5 days ago [-]

Cool! After a hiatus from Rails, I threw together an internal tool this week with 6.0.0.rc2 - breath of fresh air in a world of microservices and complex architectures.

Good job rails team!

===

Edit: P.S. - Thanks rails community too! Was super simple to build thanks to the vibrant community...

- Corporate SSO via devise + omniauth

- CSV uploads (activerecord-import gem)

- pagination (will_paginate gem)

- external API integration

- DB seeding

- audit logging on edited/created data (audited gem)

- wizards (wicked gem)

- etc, etc.

altmind(10000) 5 days ago [-]

Conventions in rails are so strong, that I quite possibly know how to seed and migrate your database, can integrate additional omniauth providers and know how to pull audit data even without looking at your code.

greenie_beans(4131) 5 days ago [-]

on a whim...do you know of any good guides for integrating an external API? i'm new to rails and working on a rails project that needs to communicate with a third party api -- there are a lot of messy guides, particularly for oauth

mhartl(880) 5 days ago [-]

Congrats to the Rails team! With this new major release, I'm planning to update the Ruby on Rails Tutorial accordingly. See here for details:

https://news.learnenough.com/ruby-on-rails-6

You can join the Learn Enough/Rails Tutorial email list to receive a notification when it's ready:

https://news.learnenough.com/email-list/

mud_dauber(2961) 5 days ago [-]

Really, really happy to hear this. The Hartl tutorial is my gold standard. Props.

heylook(10000) 5 days ago [-]

Add me to the long, long, long, long list of people living a better life because of your tutorials. Thanks.

metanoia(10000) 5 days ago [-]

I also am a senior engineer now because of this tutorial :) I'd taken a mess of CS courses in college but had never done professional web development.

Along with many, many other things, this tutorial taught me the basics of TDD, which for me was the big unlock, keeping me out of the debugger for every little thing.

Many thanks, Michael.

PopeDotNinja(4165) 5 days ago [-]

That is indeed a fabulous tutorial.

briandear(1352) 5 days ago [-]

I work as a senior engineer at a FAANG and this whole road started with The Rails Tutorial. It isn't hyperbole to say that Mike Hartl changes lives. Him and the old RailsCasts dude deserve to be in the Rails Hall of Fame.

HalcyonicStorm(4181) 5 days ago [-]

Thank you so much for your work! I owe a lot of my initial success in getting into the industry and setting up my career to your book!

jmuguy(10000) 5 days ago [-]

Thank you Michael! I will forever be grateful for your work on Rails Tutorial, it started me on my path to becoming a developer and the Learn Enough series has been super awesome.

felipeccastro(4193) 5 days ago [-]

I've been working with Rails for several years. First, it was fullstack rails, then we got a giant mess of jquery, tried to solve it by moving to an SPA with Vue... and since then our productivity really dropped.

While Vue works, I miss the productivity of having a single server app to maintain, instead of two. I'm considering moving back to Rails only, with the aid of some library like Unpoly for the nice ajax unobtrusive javascript helpers.

Congrats on the release, and thanks for helping to keep alive the model of server-rendered apps, while everyone else appears to be moving to SPAs.

acangiano(197) 5 days ago [-]

You might find Rails + Turbolinks + Stimulus to be relatively headache-free.

PopeDotNinja(4165) 5 days ago [-]

There's nothing wrong with some good ol server side template rendering. Non trivial pages will take longer to render though.

hartator(3657) 5 days ago [-]

Love the parallel testing thing. I wonder if it's already working with Rspec.

jihadjihad(4054) 5 days ago [-]

They mentioned in the PR that it seems like an adapter for rspec/cucumber needs to be added for that to work out--I, too, am hoping that such a thing comes to fruition.

mintplant(1649) 5 days ago [-]

Would someone more familiar with Ruby mind explaining what Zeitwerk is/does? I think I have a vague idea, but both the announcement post linked to from this one and the project's README assume a lot of prior knowledge about the use case.

Dangeranger(1014) 5 days ago [-]

Sure, I'll give it a shot.

Ruby classes usually need to explicitly require the other classes and modules which they use. Rails implemented a different way to achieve requiring dependencies using a method called 'const_missing' which requires the missing dependent when it is not found in the known constant list.

The problem with this is that if your classes and modules do not adhere to a strict arrangement within your source code directories then Rails can sometimes not find the file you are looking for, or worse find the wrong file.

Since 'require' has an effect on the loaded constant namespace, the order of requiring dependencies can make a difference in the presence of bugs related to which code has been required.

There's more information here: https://github.com/fxn/zeitwerk#motivation

cookrn(3691) 5 days ago [-]

Sure, I can try! One part of the Rails 'magic' that the framework tries to abstract away from an app developer is code loading. So, instead of each file in your app needing to explicitly specify using 'require' statements which other files it needs to run, Rails uses Ruby to automatically 'require' files for you. This system is dependent on constant names and namespaces matching file names and directory structures. So, a constant (Rails model) like 'Sports::Player' used in a controller might be found via lookup automatically in 'app/models/sports/player.rb'.

A system which does this has been a part of Rails for a _very_ long time, and Zietwerk is an upgrade to it by a longtime Rails contributor that fixes some of the previous downsides and gotchas.

nickjj(2731) 5 days ago [-]

I think one of Rails' biggest strengths is:

> In fact, GitHub, Shopify, and Basecamp, as well as plenty of other companies and applications, have been running the pre-release version of Rails 6 for months and months in production.

Test suites are nice and all but nothing beats having tons of real world production workloads thrown at something to really test it and iron out a good developer API. There's just a massive amount of confidence you have that you won't be the one having to pioneer some weird edge case bug.

Can someone list at least 1 other popular web framework that has multiple individual sites pushing billions of page views through it before it hits a stable release?

krob(4209) 5 days ago [-]

Symfony. Last i heard, pornhub ir youporn was using it and their traffic dwarfs most other sites.

ksec(1707) 3 days ago [-]

I would not be surprise if there was a PHP framework that rivals that in page view. But I don't think there will be any other framework that gives as much confidence to developers as Github and Shopify.

I think Cookpad wants to test Rails Pre-Release in the future as well, that is a site possibly bigger than Github. I wonder if Apple Music is still on Rails.

ludwigvan(3740) 4 days ago [-]

Not multiple sites perhaps, but Facebook uses React's unstable versions on production.

faitswulff(3383) 5 days ago [-]

Probably WordPress.

maz1b(4211) 5 days ago [-]

Very exciting. While in medical school, I taught myself a decent amount of Ruby and learned the Rails framework due to an old mentor of mine.

The same app that I learned with is now a full fledged startup, with thousands of future doctors using it, and one of our DB tables has nearly 1m objects.

Kudos to the rails team, really. I know a lot of titles and headlines talk about 'is Rails dead?' and this should put it to rest. It might not be the most cutting edge technology, but it's proven.

bigzyg33k(10000) about 21 hours ago [-]

Is this passmed, by any chance?

hirundo(3695) 5 days ago [-]

Apparently reports of Rails' death are greatly exaggerated.

onlyrealcuzzo(4111) 5 days ago [-]

I hated on Rails forever. It has its flaws for sure. And granted, I now have 5 years of experience with it, but there is nothing that compares to it for whipping something together quickly.

Don't get me wrong, Python is a great language. And Django is a great framework. There's lots of good languages and Frameworks.

But Rails has the right magic of incredibly stable tooling, a huge community, and tons of documentation & guides written for beginners. It definitely has a unique (and valuable) place in the world / businesses.

ethagnawl(4066) 5 days ago [-]

Say what you will about performance, but Rails continues to be the most productive framework for building a database backed web application or (non-trivial) REST API. It's secure by default, includes optional peripherals (ActiveJob, ActiveStorage, ActiveMailbox, etc.) to handle most common tasks and has excellent documentation.

Congrats to the Rails team on this release and for continuing to plug away after all these years.

beefhash(813) 5 days ago [-]

Rails just became boring technology. It works. The hype is gone, but it continues chugging along happily and is generally an understood problem.

peteforde(1246) 5 days ago [-]

Rails: still dead after 15 years.

Will it ever hit critical mass?

dragonwriter(4199) 5 days ago [-]

Has it ever been accused of being dead as a project? Falling out of favor for new projects, sure, but a new major release doesn't counter that.

nullwasamistake(10000) 5 days ago [-]

Eh, no. The whole MVC pattern is obsolete and every web project I've worked on less than 5 years old is a SPA. It's gone the way of PHP. It will always be around in legacy stuff but hardly anyone is building new project with it.





Historical Discussions: IBM Open-Sources Power Chip Instruction Set (August 20, 2019: 562 points)

(568) IBM Open-Sources Power Chip Instruction Set

568 points 1 day ago by Katydid in 675th position

www.nextplatform.com | Estimated reading time – 9 minutes | comments | anchor

It has been a long time coming, and it might have been better if this had been done a decade ago. But with a big injection of open source spirit from its acquisition of Red Hat, IBM is finally taking the next step and open sourcing the instruction set architecture of its Power family of processors.

Big Blue is also moving the OpenPower Foundation, which it formed with Google, Mellanox Technologies, Nvidia, and Tyan to help create an ecosystem around the Power architecture six years ago this month, under the administrative control of the Linux Foundation. (Considering how many open source projects are under the control of the Linux Foundation, perhaps it is time for that organization to consider a name change. The Open Foundation is probably appropriate, and probably already owned by someone else.)

In any event, if you have ever wanted to create your own Power processor and was lamenting how expensive it might be to license the technology from IBM, now is your chance.

IBM's long journey to opening up the Power architecture began a long time ago, starting with the creation of the PowerPC Alliance between Apple, IBM, and Motorola back in 1991, just as Big Blue was starting to get serious about the Power architecture for RS/6000 Unix systems – Unix was all the rage then, and Sun Microsystems and Hewlett Packard were circling IBM's proprietary mainframes and minicomputers like starving wolves, with a very lean and hungry Oracle snarling nearby. Behind the scenes, IBM was preparing to move its proprietary AS/400 enterprise systems to a common hardware platform with the RS/6000, a credible Windows Servers was years away (and would very briefly run on Power iron), and a young Linus Torvalds had just created the first Linux kernel (which would eventually be the key to keeping Power iron alive in HPC centers in particular and in some enterprise datacenters).

The history is long and complex, but suffice it to say that Motorola and IBM both had their challenges bringing server-class processors to market and the move to 64-bits was particularly difficult. Interestingly, it was IBM's AS/400 processor team in Rochester, Minnesota which saved the day by creating a very good 64-bit PowerPC chip that also had a double-pumped vector processor embedded in it, and it is this processor, not the ones designed by the AIX people down in Austin, that is the very kernel of all Power chips and systems that have followed since. Eventually Sun Microsystems went up on the rocks with its UltraSparc-III systems, and Hewlett Packard and Intel created Itanium, which had its own litany of woes, and this left the door wide open for IBM to be a spoiler in the early 2000s. And it was just then, back in 2001, when IBM got its first dual-core chip and its first processor to clock above 1 GHz out the door – that would be the Power4 "GigaProcessor" – and IBM brought the hammer down in Unix, delivering twice the bang for the buck as Sun and HP did in Unix, eating market share like crazy.

At the same time all of this was going on, the Motorola 68000 series of chips, which were at the heart of Apple PCs as well as myriad kinds and untold millions of embedded controllers. Arm may rule in controllers today, but back then its was Motorola 68Ks, and the kind of unified processor architecture spanning from embedded devices to datacenter gear was first done – and actually realized – with the PowerPC architecture.

Of course, since then, the Unix market has been largely replaced by X86 systems running Linux and Windows Server, and Sparc from Sun and PA-RISC from HP, and Itanium from Intel are all dead. Motorola has ceded the embedded controller market to Arm, and IBM has been trying to breathe some life into Power, first through the Power.org in 2004 and the OpenPower Foundation in 2013. With each move, IBM has opened up its technology a little more and broadened its appeal. It is a question as to whether this will be enough, with an ascending AMD providing an alternative to Intel processors and the Arm collective fielding many good processors, all using Arm licenses and many adding their own special tweaks to the Arm designs while not violating the Arm architecture.

No one is saying that the OpenPower Foundation will have an easy time growing its ecosystem, despite the many architectural advantages that Power holds over other ISAs, but it now has an easier time than a more closed architecture has. It doesn't hurt that the Power ISA is being given away royalty free, either.

"We started OpenPower six years ago because the industry was seeing the decline of Moore's Law, and we were seeing the need for more powerful systems to support HPC, artificial intelligence, and data analytics," Ken King, general manager of OpenPower at IBM, tells The Next Platform. "We needed to find other ways to drive system performance, and with limitations on the processor, the ability to integrate and innovate up and down the stack was becoming more critical. This led to things like NVLink with Nvidia, a close relationship with Mellanox on interconnects, and OpenCAPI for other devices, and we have seen some progress here. But we are also seeing a shift in the industry, with companies moving to more open hardware. IBM opening up Power to the point where we would license the CPU RTL to others so they could design their own processors was limited in its effect because there were not that many people who wanted to spend many hundreds of millions of dollars – not for license fees, but for full development – to create their own high-end CPU. We did make some progress in opening up our reference designs, and there are over 20 vendors who are now making Power-based systems. We are seeing interesting developments with the nascent RISC-V architecture, and hyperscalers are hiring their own chip designers and building their own CPUs and interconnects. They are getting into the hardware space, even if they are not going to be hardware vendors, to drive that performance."

In that environment, now is as good time as any to get the Power ISA opened up and see what kind of uptake it might have against RISC-V and Arm and a closed X86 architecture from AMD and Intel.

To be precise about what IBM is doing, it is opening up the Power ISA and giving it to the OpenPower Foundation royalty free with patent rights, and that means companies can implement a chip using the Power ISA without having to pay IBM or OpenPower a dime, and they have patent rights to what they develop. Companies have to maintain compatibility with the instruction set, King explains, and there are a whole set of compatibility requirements, which we presume are precisely as stringent as Arm and are needed to maintain runtime compatibility should many Power chips be developed, as IBM hopes will happen.

The OpenPower Foundation, working under the Linux Foundation umbrella, will have an open governance model, with IBM having precisely the same one vote as other OpenPower Foundation members as to what changes can be made to the Power ISA in the future. IBM will have retain the right to make whatever changes that it wants to the architecture to suit its own needs, but all other changes will require a majority vote of the members to ensure compatibility. "Everything has to stay in the compliance guidelines because we do not want a bunch of fragmentation," says King. This presumably also applies to Big Blue. If companies want to make a non-compliant change, it takes a unanimous vote of the members to do so. For instance, this may be for some specific set of instructions for a very precise workload set. Anyone can do a custom chip, too, but they will fall out of compliance with the ecosystem.

In addition to all of this, IBM is providing a softcore model of the Power ISA that has been implemented on FPGAs – presumably from Xilinx, not Intel's Altera devices – that people can play around with.

In addition to this, IBM is also taking its OpenCAPI accelerator interface and its OpenCAPI Memory Interface variant, which is a key feature of the Power9' (that's a prime symbol, not a typo) processor that is coming out sometime this year, and it is actually contributing the RTL for these reference designs to the OpenCAPI Consortium, which is independent from OpenPower.

Some work needs to be done to reduce the number of communication methods and protocols that are being employed to link CPUs to each other, to accelerators, and to storage. Gen-Z, CCIX, CXL, OpenCAPI, NVLink, Infinity Fabric – the list keeps getting longer and the differences between them all are disruptive in a bad way. We need one or two standards, maybe. Perhaps this last bit is a step in getting us there. IBM just wants for companies to make OMI memory, which we have talked about before and which we will be detailing shortly based on a presentation that IBM made this week at Hot Chips. OpenCAPI memory may be the best way to get most of the bandwidth benefits of HBM memory without having to resort to stacking up and packing it up, but keeping it in DIMM form factors.

And so, right here right now, King is extending the olive branch out to Intel, much as it did back in the late 1990s when InfiniBand was created largely by Intel and IBM as a replacement for the PCI-Express bus.

"OpenCAPI and OMI are architecture agnostic, and the goal is to enable others to create their own coherent accelerator and memory interfaces in an open standards environment," says King. "We are hoping over time – and we have been having these discussions – that with OpenCAPI we are able to work with Intel and converge OpenCAPI and CXL to converge them into one, common standard. And there is a lot of interest in that, without getting into the specifics."

We suspect that Google, Facebook, and maybe a few others have some good ideas about how this might get done, and that this time around, they will have a big say on how these standards will converge. One thing is for sure: They won't tolerate six standards when one or two will do.




All Comments: [-] | anchor

cipherboy(10000) 1 day ago [-]

I'd like to recommend the friendly people at Oregon State University Open Source Labs [0] who host POWER resources for open source projects. If you're looking to see what the ISA can do on P8 or P9 system, I'd definitely contact them and see if you can get a VM.

There's also a cool vector library [1] that bridges the gap between different versions of the ISA and different compiler versions.

[0]: https://osuosl.org/services/powerdev/ [1]: https://github.com/open-power-sdk/pveclib

tpearson-raptor(10000) about 22 hours ago [-]

Shameless plug, but you can so grab a POWER9 micro VPS (and large ones too) without any human intervention at integricloud.com . Those are commercial / paid though, not free.

ecnahc515(10000) 1 day ago [-]

As someone who previously worked as a student at the OSUOSL, thanks for promoting it!

bryanlarsen(3634) 1 day ago [-]

Will this do any better than open source SPARC, which was open sourced in 1999?

https://www.eetimes.com/document.asp?doc_id=1140292

blihp(10000) 1 day ago [-]

I can't see why it would. This would have mattered 20 years ago when there weren't more compelling ISA's out there. But that's not today's world: ARM is fairly ubiquitous and dirt cheap while RISC-V is a promising and open source up-and-comer. This seems like a relatively non-event (or worse: confirming that it's effectively a dying/dead platform) unless one has a significant investment in Power.

sandfly(10000) 1 day ago [-]

How is does this different from publishing the instruction set, which was presumably already available (if it weren't, how would one write an application for it)?

Koshkin(4145) 1 day ago [-]

I guess, publishing something does not necessarily mean making it public domain.

rgblambda(10000) 1 day ago [-]

Naive question: What would this mean for Xbox 360 emulators?

floatboth(10000) 1 day ago [-]

They were fine anyway, in fact Xenia has been in development for a long time already.

ISA royalties are for chip vendors. QEMU implements like most ISAs out there and I don't think anyone ever had to explicitly acquire any rights for that...

monocasa(2845) 1 day ago [-]

As someone writing a 360 emulator, probably not much. The Cell docs are already available (Xenon is a modified Cell PPE), and the ways that it's different are a one off for Xenon that probably won't be documented in this dump.

EamonnMR(3879) 1 day ago [-]

Or Mac, for that matter.

mepian(3130) 1 day ago [-]

Nothing at all.

hajile(4188) 1 day ago [-]

I wouldn't think it would matter very much for emulators, but there has been some amount of work with old game consoles running on an FPGA to give the actual experience (and you can add quite a few old console chips on a single modern FPGA). If someone is up to re-design a POWER chip, running games would be a lot easier.

crb002(4110) 1 day ago [-]

IBM is dead until it gets Z mainframes and Power servers in big a three data center (AWS, GCP, Azure).

Guessing Microsoft will start with putting mainframes in the West Des Moines data center since so many insurance companies are still dependent on DB2 batch crunching.

wolfgke(1414) about 20 hours ago [-]

> IBM is dead until it gets Z mainframes and Power servers in big a three data center (AWS, GCP, Azure).

Google is using POWER9 servers in their data centers:

> https://www.forbes.com/sites/patrickmoorhead/2018/03/19/head...

> https://www.nextplatform.com/2018/03/26/google-and-its-hyper...

Of course, most of their servers are still x86; as far as I am aware, the reason why Google also uses POWER9 servers is that they don't want to be too dependent on the two manufacturers of x86 CPUs.

haolez(10000) 1 day ago [-]

What does it mean to open source an instruction set? What can we do now that we couldn't earlier? (genuine doubt)

albandread(10000) 1 day ago [-]

Probably means they wont sue you or bill you if you use it in your own processor designs. The article mentioned releasing a softcore for FPGA; that at least means you can run the instruction set on something. There are even enthusiasts that might be interested in using an FPGA to build a whole computer; for example the Amiga community have a PowerPC based operating system. Instruction sets must also be an acquired taste; Power seems complex to me; but a lot of people clearly like it.

andyjpb(3396) 1 day ago [-]

An open, high end CPU design is really going to change the cloud market. An ISA like this is a first step in that direction.

Facebook and Google already have their own compute projects and, like Amazon, have access to custom versions of silicon from a variety of vendors.

With a properly open CPU design we'll start to see the first tightly integrated, vertical 'cloud' products that maybe still have a 'commodity' API on the top (or maybe not?) but are custom all the way down from there.

With the end of Dennard Scaling, if not Moore's Law, Open ISAs and Open CPU designs will radically change both the hardware and compute markets and ecosystems over the next 5 to 15 years, similar to what we saw with Open Source in the 1990s.

Of course, it's not clear that POWER will be the one to do that, and RISC-V isn't going to be making a grab for Intel's crown any time soon, but this looks like IBMs bid to lead in that area.

When the cloud vendors start building systems like this they'll not look too much different from mainframes and IBM wants to continue to own that market.

verall(10000) 1 day ago [-]

I'm not sure about this - there are many open processor designs in academia if a fb/google wanted to pick them up - the difficulty is integration and software. They could easier just work on ARM, the reference designs are available if you are fb or google.

I guess what I'm saying is, even if a reletively modern, 2-issue, OoO, with SMT and 256b vector proc, came out open source, would anybody really bother to integrate it and fab it?

From what I see fb and Google work with silicon vendors because they don't want be silicon vendors.

mlyle(10000) 1 day ago [-]

It's a far, far cry from an open ISA to having multiple competing vendors, let alone open CPU designs.

It was much earlier, but OpenSPARC's impact was limited-- and that was full RTL.

If POWER is open, does anyone really want to make competing high-performance designs-- let alone open them? Better to take something like RISC-V and come up with the first high performance design.

This is especially true when you consider IBM's vertical integration: IBM is the only real POWER OEM and the only real POWER semiconductor vendor.

(If we really assume a reduction of innovation in processors, and a 15 year time horizon... expiration of IP becomes a significant factor, too. Why not just make generic ARM?)

danrl(10000) about 24 hours ago [-]

> An open, high end CPU design is really going to change the cloud market.

I agree. It's only that POWER does not appear to be very high end to me. At best it is performing acceptable for the energy it consumes. Lowering energy consumption is what drives the margins. As a Cloud vendor I would stay as far away from POWER as possible.

jart(4169) about 19 hours ago [-]

The patents have expired on i486. Does that mean x86 qualifies as a free/open ISA? Patents will expire on 64-bit soon.

pjc50(1383) about 20 hours ago [-]

'Tightly vertically integrated' and 'open' are somewhat at odds with each other.

I think far too many people seem to think that the instruction set is something you can just drop in to a chip and start stamping it out, without any appreciation for the amount of device-specific engineering that has to happen. The reason things like a 'true open source' Raspberry Pi haven't happened is the $5m - $10m of work required. And for high end devices that would be required to be competitive in the cloud, that number goes up a lot.

I've not heard of Facebook, Google or Amazon doing significant custom silicon projects themselves, as opposed to just working with vendors for some customisation. The only FAANGM in that space are Apple.

IBM are the like the pastoralists living in the ruins of Rome in ~1000AD. They're a consulting firm with a grand name and history.

mrtweetyhack(10000) 1 day ago [-]

Do you really think you'll see any benfit from Facebook, Amazon, or Google building their own chips? I mean they would benefit but their only goal is to get more money out of you for themselves.

std_throwaway(10000) 1 day ago [-]

Is there a second independent manufacturer for Power processors?

blihp(10000) 1 day ago [-]

NXP/Freescale

alttab(3963) 1 day ago [-]

Use xlc once. I'm done thanks.

arghwhat(10000) 1 day ago [-]

Why in the world would you use xlC, unless you're masochistic enough to target IBM-AIX? PowerPC is a very well supported architecture, supported by common compilers and operating systems.

lsllc(3700) 1 day ago [-]

IBM are switching away from xlC in their compiler development to Clang so they can focus on the LLVM backend for POWER and leverage the Clang frontend to support the newer C/C++ standards (for both AIX and POWER Linux).

classichasclass(3208) 1 day ago [-]

So use gcc or clang.

pjmlp(299) 1 day ago [-]

Last time I checked it was one of the compilers mostly used in HPC.

For example,

https://hpc.mines.edu/using-xl-compilers/

bem94(2995) 1 day ago [-]

I have so many questions:

- Where can I get the ISA specification?[1]

- Where can I get a compiler?

- Is there a link to the 'softcore model'?

With RISC-V you can start very simple and small (micro-controller) and work your way up in understanding and implementation to a very large core (application class). POWER is a monster of an architecture, designed more for 'big iron'. I guess that might limit the 'hobbyist' factor RISC-V has.

1. This I think, all 1200 pages of it: https://openpowerfoundation.org/?resource_lib=power-isa-vers...

blattimwind(10000) 1 day ago [-]

PowerPC and POWER are relatively mainstream. It's supported by IBM XL, GCC, Clang and most major JITs (including luajit).

kop316(10000) 1 day ago [-]

I own a Talos II (https://www.raptorcs.com/TALOSII/) computer. It actually runs an official port of Debian (https://wiki.debian.org/PPC64) on it, which includes a compiler.

shawnz(10000) 1 day ago [-]

> POWER is a monster of an architecture, designed more for 'big iron'.

It's the same architecture as PowerPC, designed for desktops, isn't it? Have things really changed so much since then?

circuit(10000) 1 day ago [-]

- Where can I get a compiler?

PGI has a free POWER compiler https://www.pgroup.com/products/community.htm

floatboth(10000) 1 day ago [-]

> all 1200 pages of it

Sounds weak, one of the versions of ARMv8 has a spec that's exactly 6666 (!) pages. I would expect IBM to be more detailed lol

ajross(10000) 1 day ago [-]

> POWER is a monster of an architecture

FWIW: the original RS/6000 devices were 20-40 MHz in-order CPUs with architectures objectively simpler than a RISC-V microcontroller like the E310.

benchaney(10000) 1 day ago [-]

In response to your second question, I believe that gcc and llvm both support power.

ajdlinux(1949) 1 day ago [-]

The toy soft-core VHDL model that is referred to there will be available at https://github.com/antonblanchard/microwatt at some point in the next couple of days.

ksec(1707) 1 day ago [-]

And more questions, Correct me if I am wrong,

So this is opening up of POWER ISA, since there is quite a few different version or Revision of that, I assume that is the one beings used in POWER9 and in the future POWER10?

And it is more like RISC-V ISA open source rather than MIPS open source ?( I believe POWER was previously opened but with Cooperate Protection speak all over it ).

And this does not include Implementations, like POWER9?

I mean, if all of these were true, without implementation , or at leats licensing it for cheap, it still doesn't change the market one bit.

gigatexal(4094) 1 day ago [-]

Phoronix ran benchmarks and current Talos II systems run hotter and slower than equivalent x86 systems but maybe they will improve over time.

std_throwaway(10000) 1 day ago [-]

AFAIK those benchmarks weren't specifically optimized for that processor architecture. Most programs are optimized with x86 systems in mind. The results for your specific application might come out differently.

llampx(4195) 1 day ago [-]

If only this had happened as Opteron was taking a beating in the server market.

Now with EPYC Rome, I wonder just how many takers IBM will have.

brianwawok(10000) 1 day ago [-]

There seem to be general paths for OS.

1) OS from the start. Develop in the open. Maybe lock some features behind a paywall.

2) OS when something is not hot anymore. Take your formerly private stuff you charged a lot of money for, and because so much better stuff has come out.. meh, let's OS it.

This is clearly a case of #2....

mshook(10000) 1 day ago [-]

Yeah, it's the question I guess: too little, too late?

Because it's hard to beat the x86 mammoth for so many reasons (on top of my head):

- huge market share in servers/workstations

- Intel has more resources than pretty much anyone else

- AMD is now back in the game and started a core/performance/price war with intel

- x86 is 'cheap'

- market shares for 'cheaper' stuff will probably be taken by ARM and RISC-V

- so much time was invested in optimizing compiler, code and so on for x86 because that's what everyone has

- the Torvalds argument which is to say developper 'will happily pay a bit more for x86 cloud hosting, simply because it matches what you can test on your own local setup, and the errors you get will translate better,'. So as long as you don't have cheap Power workstations, it'll be a moot point. I remember working on AlphaPC and pretty much nothing was 64 bits clean back then, it was a huge mess. Now that part is solved but not everything else...

I definitely get the appeal for the Googles of the world to challenge Intel and for niche (internal) products, and for myself because honestly I don't really need an intel compatible CPU but in the long run, I am not sure it'll go anywhere...

ch_123(3442) 1 day ago [-]

I think the best bet for POWER would be for Chinese manufacturers, given the trade war and general distrust of American-made tech in their systems.

There are already manufacturers who have licensed the EPYC IP from AMD, but a for-free design could be compelling.

marcthe12(10000) 1 day ago [-]

The funny thing is that IBM caused x86 to be successful due to the IBM PC

vasilia(10000) 1 day ago [-]

Power clusters are used for ML because they have integrated NVLink and as I remember OpenCApi for FPGA. It's not general purpose platform.

olivierduval(10000) 1 day ago [-]

Naive questions:

- Will Huawei be able to use this processor design (now that it is open sourced) to build it's own chips, bypassing ARM restriction & US IP ?

- Are these processor designs usable in mobile device, or only in workstations and servers (using to much power for example) ?

somepig(10000) 1 day ago [-]

POWER is about as far from a good fit for most ARM applications as you can possibly get.

It's all about shoving a ton of hot power hungry multithread cores as close together as you can and running them at full bore.

Symmetry(2083) 1 day ago [-]

What's being released here is the instruction set architecture, not the microarchitecture for any particular processor design. As RISC ISAs go Power is relatively pragmatic, though not to the extent 32 bit ARM is, so it has relatively good code density compared to SPARC and MIPS. Plus it doesn't have annoying misfeatures like branch delay slots or register windows.

For mobile processors it seems about as good as 64 bit ARM but with a bit less software support in the mobile world, though a good history of software support in general.

blattimwind(10000) 1 day ago [-]

IBM targets the scale-up market (few big, fast machines) with POWER instead of scale-out (many small, slower machines). Consequentally they are high performance but not particularly tuned for high efficiency, because performance is the more important design goal of the system.

xvilka(2603) 1 day ago [-]

RISC-V is a better fit for mobile devices. But Huawei does produce network hardware that might benefit from a good POWER-based platform.

f00zz(10000) 1 day ago [-]

About the first question, UltraSPARC has been open source for a while. You can even download the Verilog code. We haven't seen any UltraSPARC-based processors, so I don't see why they would use this.

pkaye(10000) 1 day ago [-]

> Will Huawei be able to use this processor design (now that it is open sourced) to build it's own chips, bypassing ARM restriction & US IP ?

Which operating system would the use? Is that supported with the power instruction set?





Historical Discussions: Decrement carbon: Stripe's negative emissions commitment (August 15, 2019: 568 points)

(568) Decrement carbon: Stripe's negative emissions commitment

568 points 6 days ago by mhoad in 3054th position

stripe.com | Estimated reading time – 7 minutes | comments | anchor

As part of Stripe's environmental program, we fully offset our greenhouse gas emissions by purchasing verified carbon offsets. Starting this year, we're going a step further. In addition to our offset program, we are committing to pay, at any available price, for the direct removal of carbon dioxide from the atmosphere and its sequestration in secure, long-term storage. We're announcing this commitment to solicit technology partners and to urge other companies to follow suit.

The need to remove CO2

Urgent global action is needed to halt greenhouse gas emissions, and it looks increasingly likely that in addition to emissions reduction, humanity will need to remove large amounts of carbon dioxide from the atmosphere. In its most recent summary report, the IPCC notes that most scenarios that stay below 2°C of temperature increase involve "substantial net negative emissions by 2100, on average around 2 gigatons of CO2 per year."

Early adopters and technology learning curves

Carbon capture and sequestration technologies are nascent today. While a growing set of startups (including Carbon Engineering, Climeworks, and Global Thermostat) are actively working in the space, there's no clear market price for permanent CO2 removal. If there was scalable, verifiable negative emissions technology available in the vicinity of $100 per tonne of CO2 (tCO2) captured, it could be a trillion dollar industry by the end of the century and complement emissions reduction in halting anthropogenic climate change.

Researchers and entrepreneurs are working today to create this new industry, and continued progress will require funding for experimental prototypes with initially high cost per tCO2. This provides an opportunity for Stripe and like-minded early adopters to shift the trajectory of the industry. Experience with manufacturing learning and experience curves has shown repeatedly that deployment and scale beget improvement. This phenomenon has been seen across TVs, phones, semiconductors, DNA sequencing, and clean energy technology. The cost of solar panels was about $30 per watt in 1980, $10 per watt in 2000, and less than $1 per watt in 2019.

Negative emissions technology is likely a long-term part of the world's climate solution, and early adopters can help by buying in early. If a broad coalition of buyers commits substantial investment, we're optimistic that the price curve will start to move.

Current efforts

In surveying the technology landscape (good surveys are here and here), we see many promising routes towards scalable negative emissions. Technological effort is needed because the earth's natural CO2 sinks, reduced by human land use change, are overwhelmed by the ~35 billion tCO2 emitted by humans each year as shown in this video of the carbon cycle over time. With this in mind, Stripe has identified at least three categories of ongoing projects that we would anticipate funding:

First, improving natural carbon sinks is a goal of current land management projects, such as forestation initiatives, soil management reform, and agricultural techniques. These initiatives are already beneficial today, but there remain open problems on scalably verifying how much CO2 is stored each year and how long it will remain stored. There is an opening for scientists or entrepreneurs to increase the duration of the storage by, e.g., hacking plant roots to store more CO2 for extended periods of time.

A second set of exciting projects is enhanced weathering (aka CO2 mineralization). CO2 in a gas or liquid reacts to form carbonate minerals when it encounters silicate minerals (e.g. olivine) and rocks rich in Ca and Mg. This carbon is then sequestered for centuries+ in the mineral. Such weathering processes operate extremely slowly as natural carbon sinks, and many research initiatives are investigating how to accelerate these processes. Interesting ideas include crushing rock with natural forces and forcing CO2-rich fluids through natural rock formations.

A third family of exciting projects is direct-air capture, such as the prototypes deployed by Carbon Engineering, Climeworks, Global Thermostat, and others. A direct air capture plant is an industrial installation that uses energy (ideally from clean sources) to force air into contact with a CO2-sorbent such as hydroxide solutions or amine-functionalized solids. The CO2 is then separated back out from the sorbent and transported to long-term storage sites. While more research on safe storage is needed, current estimates are that the earth can accommodate tens of trillions of tonnes of CO2 in secure long-term storage, which is many times the amount emitted by human activities to date.

The approaches mentioned here likely just scratch the surface of what's possible. Humanity will need these and more techniques to reach mega-project scale to meet our collective need for negative emissions in the coming decades.

The commitment

This leads to Stripe's Negative Emissions Commitment. We will seek to purchase negative CO2 emissions at any price per tCO2, starting immediately. We expect that the best price will initially be very high: almost certainly more than $100 per tCO2, as compared to the $8 per tCO2 we pay for offsets. We don't expect to be able to sequester all of our carbon emissions, both because the relevant technologies are not yet operating at sufficient scale, and because it would be financially infeasible for Stripe. And so we commit to spending at least twice as much on sequestration as we do on offsets, with a floor of at least $1M per year.

We will work with experts to select winning carbon capture solutions based on cost-effectiveness, and we may choose to buy from more expensive providers whose approach we think is particularly promising. Eligible solutions are ones that sequester greenhouse gases from the atmosphere in secure, long-term (century+) storage. We're reaching out to entrepreneurs and researchers who believe that they can create negative emissions. We know that the market is early-stage and are open to funding projects that have a strong plan for delivering negative emissions in the coming decade, even if they are not yet ready.

Our call to action

  1. For teams currently commercializing negative emissions technology, please reach out to Stripe so that we can work together. We will choose our initial solution to purchase in 2019Q3.
  2. For other companies: Please reach out to Stripe to join our commitment and participate in joint-buying together.

We can be reached at [email protected]. Thank you!




All Comments: [-] | anchor

papreclip(10000) 6 days ago [-]

They might make up for their electricity bill 1:1 but they'll never go neutral on the amount of carbon wealth they consume in payment for their services.

Imagine I have a business where I busk for money with my guitar outside of a coal plant and a single coal minor is my patron. I may be 'carbon neutral' but in fact I owe my whole existence to coal being mined and burnt. Coal is my life blood and the sole source of wealth in my life. I may as well mine the coal myself and play guitar for myself, and if you consider me and the miner to be a unit it's obvious the two of us are living on coal and the fact that he does all the work while I play guitar hardly means anything.

Companies like stripe are sufficiently separated from coal and oil that they can believe they don't suck at the teat of fossil fuels, and that the only coal wealth they consume is whatever powers their light bulbs and servers. But ultimately you can track back the wealth that comes their way - all of it - to operations that generate quite a bit more emissions.

We can run subsidies and credits in little isolated corners of our economy, but it's like running a refrigerator in a closed room. Ultimately you're just burning wealth which creates a greater load on the wealth-generating machine that is the fossil fuel industry.

Real advances that make it possible to generate wealth without fossil fuel money in subsidies should be celebrated. These tech guys taking $1m of their $100m in profits (the coal miner pays the busker using stripe) to pay their $1m light bill is a shell game

Nyandalized(10000) 6 days ago [-]

I'm not quite sure what you're trying to say. Are you discounting their efforts? Their motives?

You're not responsible for everyone else, or if you were, they'd not need to be for themselves.

iamleppert(4201) 6 days ago [-]

What kind of margins do they have that enables them to just throw money around like this? Is the payments industry that big of a cash machine to allow this kind of spending that has nothing to do with their core business?

Nyandalized(10000) 6 days ago [-]

You would find that banking and it's related fields are indeed very profitable.

gitgud(3354) 6 days ago [-]

So they're basically going to pay for the CO2 emissions at around 100$ per tonne (10 cents per kg), right?

Well, we can calculate the C02 exhaled by a typical human, as [1] 36.84 mg = 0.03684 kg of CO2 per breath. (0.03684 kg @ 10c per kg) Therefore each breath you exhale is worth 0.003684 cents....

So how do you want to pay me @stripe?

[1] https://www.quora.com/What-is-the-mass-of-the-carbon-dioxide...

rwz(10000) 6 days ago [-]

Are you saying you're going to stop breathing after receiving the money?

itcrowd(3986) 6 days ago [-]

This is an amazing pledge and very powerful statement.

However, I am not 100% convinced morally of CCS solutions. The main counter argument being that CCS technology enables the status quo of fossil fuel burning to continue.

On the other hand, there are some heavy industries (steel production?) that do not have any alternatives to coal burning in order to operate and CCS can offset the carbon footprint in these industries. I would however not call this 'negative emissions' but rather 'emission-free' or similar.

What is your opinion on CCS? Can people convince me in either direction?

bobbygoodlatte(10000) 6 days ago [-]

I think it depends what the lower limit of this technology is in terms of cost-per-ton. If the technology can get 100X cheaper, it would be immoral not to pursue it. If it only gets 2-3X as cheap as it is today, it might create the moral excuse you cited and be a net-negative.

My instinct is it could be an extremely important technology for fighting climate change. And while it develops it's important to continue to insist that it doesn't excuse the emissions it might offset.

ajross(10000) 6 days ago [-]

> However, I am not 100% convinced morally of CCS solutions.

So... let's put off saving the planet because someone has moral qualms? No. This is a legitimate emergency. If we have something that has a net positive impact, we need to do it and stop arguing about it on the internet. Nothing is without tradeoffs, and some are simply going to have to be made. There are no easy solutions. We have seven billion people to feed while we manage to undo all the damage too, remember.

I'm not saying capture solutions are the best choice, mind you. I'm saying that decision needs to be made with numbers and not 'morals'.

Ma8ee(10000) 6 days ago [-]

SSAB and LKAB are experimenting with burning hydrogen instead of coal for steelproduction.

moultano(3067) 6 days ago [-]

Limiting climate change to 2C is such a hail mary at this point that we should look favorably on anything, no matter how imperfect, that moves us in that direction. There are no silver bullets, we need to do everything at once.

godelski(4168) 6 days ago [-]

It's a fair question, but I'll ask you 'what's the alternative?'

We can't plant enough trees. Nor can we always plant them in the most needed places (North, where carbon accumulates). We can't build even public transport (which is still positive emissions). We can't get everyone to go vegan (which still is positive emissions. Even lab meat won't make negative emissions). We can't stop air travel and shipping without significantly affecting people's lives (including their ability to live, as drugs, equipment, and experts aren't all created locally). We can't get everyone to leave their homes and concentrate into cities.

Even if we could do all that, we'd still be positive. Worse, how do we get every country in the world to do the same? It seems infeasible to me. And it doesn't address the problems like that carbon builds up in certain areas that we just can't even plant trees.

But here's the thing. The argument isn't 'planting trees vs CC', it is 'trees + CC'. Actually it's more! It's about using every available method we have at our disposal. There is nothing wrong with burning carbon actually. The issue is burning so much that we are destroying the plant. Destroying the planet is the issue. If we have negative emissions total (and aren't creating carbon hot zones) then who cares how much we emit? As long as it's captured and doesn't damage the planet.

Leaving CC off the table is like telling a bunch of starving children that they can't have hamburgers because some of them might eat too many and get fat. The concern is completely missing the point of the problem.

tlb(1183) 6 days ago [-]

There isn't some intrinsic moral sin to burning carbon. There are just practical problems with having too much in the atmosphere. If you can remove carbon, especially if you can also convert it back into fuel, that solves the problem.

(Carbon isn't the only emission from burning coal -- it contains sulfur and thorium too. But these can also be mitigated with some expense. And if you're converting CO2 directly to fuel, there's no increase in pollution at all.)

doctorpangloss(4061) 6 days ago [-]

You could just buy oil and not burn it. Far simpler than CCS.

kingbirdy(4198) 6 days ago [-]

The status quo of burning fossil fuels will continue whether we employ CCS or not, so we may as well use every tool available.

mcot2(10000) 6 days ago [-]

Why not invest in new solar power generation instead of removing co2 at this point.

JimiofEden(10000) 6 days ago [-]

Why not both?

At least in terms of priority, the damage is done and is already in the atmosphere. imo, we need to first focus on cleaning that up and getting some form of homeostasis before it gets much worse, WHILE working on solar/nuclear/renewable tech to make sure we don't just relapse and make the problem worse again.

I'm optimistically hoping that's the route we'd take anyways. Money tends to say otherwise tho. =(

hugoromano(10000) 6 days ago [-]

Our normal state should be doing good for the planet, and not bragging for doing the right thing. For starters, Stripe staff should fly far less, invest in remote working, or research how to make payments with the lowest carbon footprint. Cash is the baseline. I hope that there isn't an hidden agenda of carbon credits as a business.

TeMPOraL(2629) 6 days ago [-]

Should. Doesn't mean it's possible.

When the market doesn't favor doing the right thing, companies trying to do it die quickly, outcompeted by those who don't care. That's one reason why this has to be a gradual process, involving carbon credits and companies bragging about doing a bit of the right thing, and hopefully having that bragging be positively received.

esotericn(4203) 6 days ago [-]

Proud.

I'm in the process of setting up a new consulting business, and one of my major commitments is to donate a significant percentage (at least 10%; likely more) of profits towards CO2 reduction / capture efforts.

I'm watching this space eagerly.

chrdlu(10000) 6 days ago [-]

Trail of Bits (a security consulting firm) recently subscribed to an offset program that offsets all of the carbon from their employees!

https://twitter.com/trailofbits/status/1155840018797256711

titojankowski(4036) 6 days ago [-]

For people really into carbon removal --- Check out AirMiners, the largest community of entrepreneurs, researchers, and funders exploring opportunities in negative emissions and carbon removal. Folks from every major carbon removal startups like Climeworks, Global Thermostat, etc.

Join 320+ of us on Slack!

Apply here to join: https://docs.google.com/forms/d/12L8drO9a6OZf3-I9578PieBI0fw...

There's also a public meetup group in the Bay Area: https://meetup.com/CarbonRemoval/

titojankowski(4036) 6 days ago [-]

woohoo, 3 new applications so far!

enraged_camel(2196) 6 days ago [-]

Let's say I'm a regular web developer and know nothing about climate science, carbon removal, or any related science or technologies, but might be interested in donating my time and and skills.

Do you have any recommendations?

jrv(4025) 6 days ago [-]

I'm really interested in the topic and would love to just lurk and read what people say on there for a while, but the text in the signup form makes it sound like a person like me wouldn't be accepted, since I don't have super concrete plans to make that my next profession (although I'm generally interested in exploring this field if I really started thinking that my skills would make sense there).

sremani(3501) 6 days ago [-]

Data centers are factories without smoke stacks, they are voracious consumers of energy and since 'cloud' everything computing is the name of the game at least for medium term, tech companies would do a favor if they focus on tackling the datacenter related issue.

Also chip fabrication is a dirty dirty dirty business, as the consumers of the computer hardware, we as an industry has responsibility to pollution reduction of this process.

Planting trees is good, giving aid money is good, but cleaning your own shit is better!

Nyandalized(10000) 6 days ago [-]

I'd argue that 'the cloud' makes traditional computing more efficient. Since their motto is 'pay for what you use', it automatically aligns with motives for the most efficient use.

Not to mention that the usage of computers has reduced the need for much more power hungry methods. We can now efficiently calculate much stronger, lighter, easier to manufacture parts and building patterns.

There is no need to drive/ride/walk to a place to hear the message, radio broadcasting is much more effective, even though it itself consumes power.

brunoTbear(10000) 6 days ago [-]

I'm surprised by the size of the commit. If $1M is the floor for what a Stripe-sized org is willing to spend, how expensive would a ca