Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

December 03, 2022 13:38



Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: Show HN: Trading cards made with e-ink displays (November 30, 2022: 1144 points)

(1144) Show HN: Trading cards made with e-ink displays

1144 points 4 days ago by jonahss in 10000th position

www.wyldcard.io | | comments | anchor

This has been my most ambitious project yet. I learned how to design electronics and circuit boards. I learned a new programming language called Rust and wrote my first driver, I upped my CAD skills, 3D printed, and did my first resin casting. I generated the images pictured on the cards above using an AI image synthesizer which I'm hosting on my own server.

I haven't designed a game for them yet! I've got some ideas, and I'm curious to explore the new interactions and game mechanics these unlock. If you know anyone out there who'd want to play around with these, try out ideas, give suggestions, or nerd out on game design, send me an email: [email protected]

Special thanks to Ish, who inspired me to actually build something, and to Brady who taught me everything about electronics.




All Comments: [-] | anchor

upwardbound(10000) 3 days ago [-]

Amazing!! Are you taking orders? Have you considered setting up an Etsy where you can sell a few of them for $100 per tile when you feel like making a couple now and then?

jonahss(10000) 3 days ago [-]

oh dang! I'd totally sell the cards, for less than that even. Sign up for the mailing list on wyldcard.io and if you shoot me an email I'll totally sell you cards, and even a base station (though those are pricier)

[email protected]

HardwareLust(10000) 4 days ago [-]

That is a really cool idea! I really want to see what kind of new games that can be created from this.

jonahss(10000) 4 days ago [-]

Thanks! Yeah me too! I've got an idea for a game, but I'm not an experienced game designer. If others want to try to design a game for it, I'd be open to sharing with them :D

Mostly wanted to inspire something new.

k_sze(10000) 3 days ago [-]

Somehow this makes me think of the "Deck of Dragons" in "Malazan" because the faces can change.

jonahss(10000) 3 days ago [-]

Adding to my list of fictional card games, thanks!

lightedman(10000) 4 days ago [-]

Digital Tarot readings would be a possible use for this. You'd need to program the thing to randomly turn cards 180 degrees.

jonahss(10000) 4 days ago [-]

Yeah! I was going for a Tarot/Occult vibe.

Turning 90 degrees... maybe I could do some magic with magnets.....

ShredKazoo(10000) 3 days ago [-]

I guess the advantage of this setup, relative to playing a board game on an iPad, is that keeping info hidden from your opponent is more natural with cards. I suggest to exploit this, you either make the cards thin enough to hold in a hidden 'hand' the way someone might do in magic the gathering, or (much easier) have your plinth include a hidden hand prop section ala Scrabble.

In terms of product development, I think the best path would probably be to create or find an opensource game engine for cardgames and have a nice development environment plus API where a dev can write a game for the engine that either runs on a PC or runs on the cards, with no code changes required. (PC support makes debugging and iterating on game design much faster.)

jonahss(10000) 3 days ago [-]

I hadn't thought of a Scrabble style card holder, good idea.

Ah yeah, I think the game design community mostly uses tabletop simulator and one other such program for testing out ideas. Being able to deploy to both is a good idea.

an_aparallel(10000) 4 days ago [-]

ha! these are rad. But tbh they look more like electronic dominos. The docking station makes them less card like too (but love the way you've arranged the contacts)

jonahss(10000) 4 days ago [-]

Ah thanks, I worked hard on the contacts.

There's a thin and flexible e-ink display I could use instead, but they're $20 each instead of $4. I agree they're a bit chunky in this form :)

I agree the docking station detracts a bit from the naturalness of just playing on a table.

I decided to build what middle-school-me would have wanted to play with. I took a lot of inspiration from Yu-Gi-Oh which has the same docking station concept (just executed much better, where you wear it on your wrist).

999900000999(10000) 4 days ago [-]

Anyway to add NFC. I imagine a special gameboard which sends data to the cards based on game state.

Very impressive!

jonahss(10000) 4 days ago [-]

Heh heh, that's like phase 4 ;)

I want kids to be able to play at school or at camp, so didn't add any connectivity. Plus I'm trying to stay cheap. But I can imagine adding bluetooth to the base or to your phone or something. Then I can go all 'BILLS PC' and let you store your spirits on the cloud, let you load them in and out of cards, etc etc.

Except I don't want to turn it into an online game. Computer games are awesome, I think it's lame to emulate physical cards with a computer, a'la' Hearthstone.

retendo(10000) 4 days ago [-]

Finally, a way to play physical Sabacc[1] in real life.

[1]: https://starwars.fandom.com/wiki/Sabacc

jonahss(10000) 4 days ago [-]

I'll add that to my list of fictional card games!

I took inspiration from Yu-Gi-Oh, Card Captors, Angelic Layer...

mbreese(10000) 3 days ago [-]

I got a Sabacc deck someplace a few years back. I think it might have been from a storefront in Disney(land|World). But, it looks like you can also buy a set online from a few places.

My son was really into it for a while, but they included betting "credits" with the game that weren't nearly as fun as normal chips would have been.

kevinmhickey(10000) 4 days ago [-]

Scrolled to find this comment and was not disappointed.

zumu(10000) 3 days ago [-]

Can I load up whatever images I want? The dream would be proxying Magic cards for playtesting. If you could load net-decks and shuffle via an app over NFC somehow, that'd be amazing as well!

Lots of possibilities here!

jonahss(10000) 3 days ago [-]

Pretty much any image you want, but it's only 4-color greyscale and pretty low resolution. I'd have to make a nicer UX than the current 'scp the images onto the raspberry pi' setup I have now.

jojo259(10000) 3 days ago [-]

I REALLY love this idea. The way that this could combine the tactile nature of physical games with the benefits of digitization is awesome. The example you gave somewhere of the vampire card is so cool.

jonahss(10000) 3 days ago [-]

Thanks! You get it!

hifikuno(10000) 3 days ago [-]

I haven't read all the comments here yet, but I reckon this would be cool to use as a dungeon exploration game. Each card is a room and you can lay them out as you explore new rooms. When certain events are triggered you put the card back on the machine to be shuffled then placed back where it was. Get in, get as much loot as you can and get out without dying.

AndrewDucker(10000) 3 days ago [-]

If you haven't encountered it you should check out Legends Untold https://www.kickstarter.com/projects/legendsuntold/legends-u...

Obviously not eink, but exactly that kind of card game exploration.

jonahss(10000) 3 days ago [-]

Oh cool, good idea!

gigel82(10000) 3 days ago [-]

There are a lot of 'electronic price tags' which are basically the same form factor as your trading cards, except they mass produce them (have nice plastic cases) and they usually include a 3-color eink (black-white-red or black-white-yellow) plus a wireless transmitter (usually proprietary protocol, but sometimes plain Bluetooth and/or NFC) for OTA updates and a 10-year battery (sometimes replaceable CR2032). Also, if you can grab them at $6 / piece, I imagine they're being produced for a lot less than that (random AliExpress link: https://www.aliexpress.us/item/3256803094207083.html).

If you're thinking of mass-production might be worth reaching out to one of those manufacturers; you can buy in bulk if nothing else (but I'm sure they'd be open to customizing it a bit - maybe some branding on the plastic molding and whatnot).

jiveturkey(10000) 2 days ago [-]

Seen these at Ross and/or other discount stores. It boggled my mind. How could these possibly be cheaper than the staff swapping bits of paper. 10 year battery but I assume they break (and you have to walk around to find the broken ones) in 1-2 years, or get retail abuse, or other calamities.

The 4' ones (close to paper tag size) are $20.

jonahss(10000) 3 days ago [-]

Heh, the screens I use are exactly the ones contained in those products.

Huh, the battery lasts for ten years? I wonder how many refreshes you get for that. Maybe I should just be using those.....

fxtentacle(10000) 3 days ago [-]

The Waveshare ones are even powered purely by NFC. So they don't need a battery, yet you can rewrite them with ease.

pjerem(10000) 3 days ago [-]

Looks like you need a gateway (not that expensive but too expensive to just hack with those tags at home).

I'd love to find cheap e-ink tags that can just receive a picture over some cheap wireless protocol and display it.

polishdude20(10000) 3 days ago [-]

Woah that's really impressive! What sort of microcontroller are they running? Why Rust?

jonahss(10000) 3 days ago [-]

See my other comment. I wanted to learn rust, but by the end, once I knew it, I wouldn't have chosen rust for a prototype if I started again.

These prototypes are running on a raspberry pi, though I wouldn't want to put them into production like that.

yumraj(10000) 3 days ago [-]

A very simple game could be Rock-Paper-Scissors if you can add a gyroscope or something to detect shaking and then show Rock, Paper or Scissor at random.

Sorry for adding more work. :)

selcuka(10000) 3 days ago [-]

I believe they don't have any batteries in them but only powered when connected to the dock. E-ink only requires power when changing the image, not for displaying it.

cableshaft(10000) 3 days ago [-]

This is very cool. I think you'll have a hard time finding a traditional board game publisher willing to put money into this (there might be one or two out there, but most will see this as prohibitively expensive for them), but you might be able to pull off a successful Kickstarter for them on your own.

Kind of like the Blinks game system, these little hexes with colored lights in them that each have a separate game in them and can 'teach' the other hexes they connect to.

One of the Blinks Kickstarters: https://www.kickstarter.com/projects/move38/blinks-smart-boa...

piyh(10000) 3 days ago [-]

Card games are generally played with 10-20 cards once you hit your mid/endgames. I can't think of a system where you'd have mechanical benefit of all those being epaper before you should go full digital. Having a few choice mechanics tied to them is the way to go.

What does epaper get you that paper doesn't? A microprocessor and persistent state. RFID cards on a board game might be a good middle state for most cards. Scan them as played. Build gamestate or dynamics to be populated to epaper.

Epaper would be good for storyline branches, timeline progressions, evolving characters. Make them traveling characters being owned by different players. Differentiate them from becoming a static scoreboard that could be represented on a single tablet.

safety1st(10000) 3 days ago [-]

Firstly super novel idea with tons of space for both technical and gameplay innovation. Really love it!

If we're talking business models, the one which immediately comes to mind as appropriate is that of a gaming console. Sell the hardware (e-cards, maybe a mat?) at a loss, then sell games for it at no marginal cost. Let other people build games for it because hit games are what sell platforms.

Since this essentially a portable gaming device, consumers may compare it to e.g. a Nintendo Switch - if you undercut the Switch and you have a blockbuster title or two, you could have incredible product on your hands.

Presumably the lucrative economics of trading card games could be applied here as well...!

jonahss(10000) 3 days ago [-]

Thanks!

Yeah, it's a weird in-between. I don't think a game publisher would take a risk on it, but neither would silicon valley VCs.

Kickstarter would be my bet too. I was going to put this project down for a little while and start a 'real' startup though.....

yurikoval(10000) 3 days ago [-]

the big question is can it display NFTs?

jonahss(10000) 3 days ago [-]

sure

CamperBob2(10000) 3 days ago [-]

You should probably start talking to a patent attorney right about now. 99% chance this goes nowhere, 1% chance you're the next Gary Gygax.

jonahss(10000) 3 days ago [-]

I'd settle for Richard Garfield ;)

You think I should patent? Isn't that going to cost me like 20 grand? Plus if Wizards of the Coast really wants to, they can work around my patents somehow.

If I could patent for only a couple thousand dollars maybe I would..... But I WANT other game developers to use this idea. Just, if they want to skip ahead, they can work with me, I'd be happy to share.

lifeisstillgood(10000) 3 days ago [-]

This is amazing - but don't just limit yourself to table top games. While the killer app has not popped into my head I am sure there are business uses as well - from name tags to ... err - but just the very existence of these things means people will find new uses. fantastic

jonahss(10000) 3 days ago [-]

The trick then, would be making an easy way to write images to them. Maybe a phone attachment...

crote(10000) 3 days ago [-]

Ooooh, I love this! Are the design file available somewhere?

jonahss(10000) 3 days ago [-]

Ah, not at this time, but maybe I should open source em?

bee_rider(10000) 3 days ago [-]

Echoing everybody else here, really neat!

Questions:

Do the cards contain any battery? I'm wondering if they have always-running CPUs in there, or if all the compute happens in the base (and then it just writes to the screen).

I guess color eInk would probably be cost-prohibitive, huh?

jonahss(10000) 3 days ago [-]

Thanks!

Yup, you called it. The cards are totally passive. Just the screen and an eeprom memory chip, no battery.

All the compute happens in the base.

Color e-ink way too expensive for now, but if Wizards of the Coast wants to invest 100MM we can make em in color, thinner, and flexible!

zozbot234(10000) 3 days ago [-]

If this was a fully hackable product, it could be used for so much more than just playing games. Think the stuff that e-ink price tags are already used for, e.g. in product retail, plus so many uses in the office; no need to awkwardly write things down on post-its or simple slips of paper, when they can instead be shown on a trading card with information being reliably drawn from a single source of truth.

jonahss(10000) 3 days ago [-]

I'm down to make it fully hackable.

The trick would be an easy interface for writing to them. They don't have batteries, so you need to send all the power to them along with information.

dankwizard(10000) 3 days ago [-]

My first thought is card battler

You could actually have a characters 'hitpoints' accurately reflected instead of having to keep track of, ala Hearthstone.

jonahss(10000) 3 days ago [-]

yeah!

cbreynoldson(10000) 3 days ago [-]

This is huge. Are you going to sell these as kits?

jonahss(10000) 3 days ago [-]

Oh, that's not a bad idea. I could do a kickstarter, sell kits or fully-assembled ones that way. It would be about $300-$400 for 3 cards and a base in this iteration and at low volumes.

Think I should do it?

bashmelek(10000) 4 days ago [-]

This is amazing! I've fantasized about doing something similar—-what resources did you use to pick up all of those skills?

jonahss(10000) 4 days ago [-]

I come from a web dev background, and spent the last 3 years running the cloud and embedded firmware teams for a 3D printer startup. I learned that hardware isn't so hard, once you get past all the mystery. My coworkers gave me tips on which tools to use, and helped me when I got stuck.

The best advice I got was 'Read the datasheet'

The biggest step for me was deciding to spend money on experimenting. With a budget, suddenly it was fun to order stuff, buy tools, try things out.

Rust Book was the best for learning Rust.

bckr(10000) 4 days ago [-]

I really like this and look forward to future iterations.

For anyone curious, the symbol made by the contacts is the Tree of Life from the kabbalah

jonahss(10000) 4 days ago [-]

Yeah, I'm going for an occult/mysticism/witchy vibe.

Magic the Gathering has a monopoly on fantasy, and the cyberpunk genre is played out. Plus I want the inner-workings to be somewhat mysterious.

I had a lot of fun thematically laying out the contacts. For example: Keter is VCC and Malchut is GND

qup(10000) 4 days ago [-]

This is a really impressive project. That's an assload of new stuff to learn. Congrats on shipping.

jonahss(10000) 4 days ago [-]

Thanks! Well, not 'shipped' per se. Nobody's got one but me so far :P

JetAlone(10000) 4 days ago [-]

Why are the ports for the card bays Sephirot shaped? Does this game's lore have some Talmudic angelology inspiration?

jonahss(10000) 4 days ago [-]

Yes! Going for a mysterious witchy/occult vibe.

cwkoss(10000) 3 days ago [-]

Super cool. Taking a crack at rules for a game using this.

---

Sample game idea: CAKE DECORATION game - players have to try to decorate as many cakes as possible. Requires 10 cards.

One card is the 'design recipe' card and never leaves the base station. It shows an amount of ingredients, ex. '1 cup of buttercream, 3 shakes of sprinkles, 4 squirts of whipped cream'.

Players start with either 3 blank cards or 3 random low-amount ingredient cards.

On a players turn, they select from three cards: each card is an ingredient, an amount, and a modifier. So cards could be '1 shake of sprinkles (2x)' '2 squirts of whipped cream (+1)' '1/4 cup of buttercream (+1/4)' (could make variations or add other kinds of things that might go on a cake)

The player draws a card, base station detects which card is missing, and the other two ingredient cards increase in amount based on the modifier value displayed on the card. ex. '1 shake of sprinkles with a 2x' will fully complete the recipe if not drawn for two turns (because 1 x2 x2 = 4 shakes, which is enough for the recipe)

With 4 cards in their hand, if the player can pay for the whole recipe, they win the round and get a point. On win, new recipe appears and all cards in the winning players hand and on the base station get rerolled to new low-amount ingredient cards. Score could be displayed on the margins of the recipe card.

If they cannot pay for the recipe, the player places a card back on the base station. And their turn is over. (Should ingredients be re-randomized when replaced on base station? Should that be a player decision whether to reroll it?)

First player to decorate N cakes wins the game.

---

I think something like this could make for a viable game with a low number of cards, but could be more fun with a greater number (17 ideally?) which would allow for 3 active recipe targets (+2), using the 4th slot for another ingredient and displaying recipe as a disconnected piece(s) (+1), and larger hand sizes (5 -> +4) to allow for more complex recipes

An interesting game design question is how random you want the cards to be? Fair random would probably be viable, but since the 'deck' can know and make decisions based on the state of cards not connected to the base station, you could deal unfairly if less randomness would make the game more interesting or fun.

jonahss(10000) 3 days ago [-]

oh my gosh, I love it! Thanks!

I'm glad you kept the total card count low, that's a limiting factor because of their price and size.

I'll try it out!

cwkoss(10000) 4 days ago [-]

Very cool!

In terms of making a game with them, I think a design where the pins are on the face of the card may be useful. I want to be able to 'draw' a card without knowing what is on it.

Certainly can be useful with pins on the back (and I totally get how this orientation is probably more size-efficient), but I think front pins would be more 'playable'. Maybe a design could be achieved with holes that pass through all the way so it can be written with either orientation?

Being able to 'power up' an existing card in the upwards orientation could be really cool for situations where you kind of want 'counters' applied to a portable card. Could have a base-station that allows you to 'add' the qualities of one card to another target card, or 'evolve' a pokemon, etc.

eru(10000) 3 days ago [-]

> In terms of making a game with them, I think a design where the pins are on the face of the card may be useful. I want to be able to 'draw' a card without knowing what is on it.

That would work. But you can also make that work with current hardware (I think): you just need a delay between pressing the button and the screen changing? So that the player can press the button, pick up the card, and five seconds later, it's revealed.

jonahss(10000) 4 days ago [-]

Ah yeah, originally I wanted the contacts to wrap around the edges of the card, so then you can stack them and address whichever one in the stack you want.

Maybe I should raise the priority on that.

Definitely I'm into the idea of evolving, or breeding cards like pokemon. Powerup or 'combine' would be cool too.

Folcon(10000) 3 days ago [-]

This is really interesting jonahss, well done for completing it! Have you looked at the physical games and convention spaces?

Things like DnD roleplaying games or Megagames could both use something like this.

I'd personally be interested in both those applications, what kind of wireless communication can they do?

jonahss(10000) 3 days ago [-]

Oh for sure, I'm hoping to inspire the game developer community. I know a lot of things using electronics have been tried in the past and it usually comes out as a gimmick, but I tried to design these to hide their electronic nature, hoping to enter the same spot that dice and counters inhabit.

I was going to post about it on some game dev forums next, any suggestions of where I should post?

They don't do wireless as of now, they have to sit on the plinth to get an update. There are options for how to add some form of wireless connectivity.

I'd be interested in continuing the conversation, I'll send you an email, or email me: [email protected]

metaloha(10000) 3 days ago [-]

Trading or collectible card games that have cards that modify themselves with play would be great here. SolForge is the example I'm thinking of, but I'm sure other games do it too.

SamBam(10000) 3 days ago [-]

I was thinking trading cards too, but thinking of specifically trading them.

I wasn't really watching the images in the demo video, so at first I though he was transferring one image to another. This made me think it would be really cool if there were exactly one way to get a new image on your card: put it on a device that swaps two cards. You'd get a new image, but they'd lose theirs. So you'd actually have to commit to trading.

I guess that sounds a little like NFTs, but hopefully without the douchey scamminess.

Semaphor(10000) 3 days ago [-]

RIP Solforge. That was my favorite digital card game before it crashed and burned, then resurrected and finally fizzled out.

8note(10000) 3 days ago [-]

It would let you make a real version of Inscryption under any of the different game varieties

jonahss(10000) 3 days ago [-]

Yeah! I'm thinking a mixture of Magic the Gathering and Tamagotchi. Where the cards can level and grow like an RPG.

DoingIsLearning(10000) 3 days ago [-]

Out of curiosity what connectors did you use between the card and the plinth?

Are they some sort of pogo pins?

jonahss(10000) 3 days ago [-]

Pogo pins were too expensive, I'm (totally mis)using spring fingers.

Benjamin_Dobell(10000) 3 days ago [-]

This is absolutely awesome! I've had (terrible quality) 3-color e-ink displays and some associated electronics sitting in my office cupboard for a few years because I wanted to prototype this project, but I never did. I'm so glad you did this!

Since I just want this to come to fruition, I'll explain what my intended launch strategy was. I'm a developer (just a contractor, not owner) of Tabletop Simulator and do some stuff in that community where there's an overlap between physical tabletop and digital. My plan was to launch a card game (largely designed) and the e-ink cards simultaneously via Kickstarter. However, before that, a digital implementation of the game on TTS. Basically as much as I love this idea, I couldn't see it being monetisable on its own unless you can bring the cost down significantly. It also didn't seem like a defensible business on its own. But if the product is a game that uses said functionality, well, that'd be just swell.

Anyway, congrats, this is awesome!

jonahss(10000) 3 days ago [-]

Thanks! And thanks for the tip! I may reach out to continue the conversation and shoot ideas back and forth if that's okay :)

Garlef(10000) 3 days ago [-]

Maybe make it an NFT based CCG and hardcode a transaction fee of 5% to make money on the secondary market.

Perez418(10000) 3 days ago [-]

Great work, this is awesome. I'd love to connect and hear about your journey. I've always wanted to make a board game.

jonahss(10000) 3 days ago [-]

Love to! Email me? [email protected]

YetAnotherNick(10000) 3 days ago [-]

Off topic, but where are you getting e-ink display from. I could think of lot of usage for them if they don't require any refresh or power and is available for $10.

jonahss(10000) 3 days ago [-]

Oh, the screen alone is less than $10.

Bought off Alibaba, I think they're so cheap because they're designed for grocery store shelf price tags.

Good Display GDEW029T5D

junon(10000) 3 days ago [-]

Damn, impressive OP. This looks really nice. The design of everything is awesome.

jonahss(10000) 3 days ago [-]

Thanks!

blindseer(10000) 3 days ago [-]

If you could put even a single button on the 'card' itself, then it'll open up a whole layer of game design.

This isn't an actual game, but I'm thinking it would be great to pick a card face up that's 'red' and be able to make a decision to turn it into 'cyan'. Others could pick 'green' and 'blue' face up, but can click a button to turn it to something else, and then the game resolves itself when everyone reveals it. Like if everyone can discuss and if everyone reveals the primary colors everyone wins.

I'm sure there's interesting game mechanics you can come up with when you give a decision to the player.

Additionally, I think this could make an interesting programmable suite of games. You don't have to sell just one game.

jonahss(10000) 3 days ago [-]

Agree on the suite of games, just like a deck of cards :)

Ah, the cards have buttons next to them on the base. I figured they could display icons on the card next to each button (each spot on the base has 3 right now). I call it the 'ATM' style UX.

jtolmar(10000) 3 days ago [-]

This is very neat!

I don't think traditional board games would be interested unless you got the price down very very low ($2/unit or less).

I think you could make something happen if you marketed and packaged it as a 'board game console' though. Like a fixed number of cards and larger boards, plus some sort of base station or phone app that'll flash everything to the starting state of a game of your choice. Then people buy new games digitally from you, same as a modern video game console.

(I'd be interested in the game dev facing API and writing some games for such a thing, so let me know if you're pursuing it and we can swap contact info.)

jonahss(10000) 3 days ago [-]

Yes please! Email me? [email protected]

jonwest(10000) 3 days ago [-]

That's a great idea! Rather than selling it as a single game, sell it as a platform that people can build on top of. There's a pile of potential in that.

chasebank(10000) 4 days ago [-]

Sweet product! I always thought pet name tags would be a fantastic application for this!

nixpulvis(10000) 3 days ago [-]

Can I make a pet tag for my NTF charizard?

TylerE(10000) 4 days ago [-]

I'm confused. Do you rename your dog every week?

Tworthers(10000) 3 days ago [-]

Seems like a bit of a solution waiting for a problem.

Given the current limitations in the system I would go with a monster battler game with a starter set of just 2 cards which would be yours and your opponents monster and then add a selection of power-ups/abilities these could be much cheaper components that could be played on the base station for an effect. These could be swipe cards or even just resistors basically anything that can be measured by the base station. With this hybrid system you could get the cool evolving over time monsters as well as lots of pieces to play with. Lots of scope for cool and hidden interactions. (Ideas taken from digimon tamers)

jonahss(10000) 3 days ago [-]

Ooh yes!!

redorb(10000) 3 days ago [-]

Honestly 2 of these per person and 5 for the turn,flop,river - and we have a texas holdem game without cards and shuffling - just chips...

so a 6 person game, 12 of these + 5 ~ 17 of these ~ $450 ~ about the cost of real nice chips and cards...

eru(10000) 3 days ago [-]

Well, you could also 'just' put an iPad in the middle of the table for public game state (like turn, flop river in Poker), and everyone has their smartphone for private game state (like your hand in Poker) and you can play most boardgames quite nicely.

Instead of the iPad in the middle you could use a TV screen, too, but then you can't directly interact with that via touch screen.

jonahss(10000) 3 days ago [-]

hey, yeah! The commenter about the iPad is also right, but, there's something satisfying about the physicality.

That said, you can play poker with a deck of cards and a pile of beans.

But yeah, I'm looking for something somewhere between Poker and an iPad.

See my other comment, a lot of interesting mechanics are available.

I'd sell you a poker set if you want to impress your friends :D Email me: [email protected]

dalbasal(10000) 3 days ago [-]

One card pp might be adequate for the hold cards. It's easy enough to represent two cards on one little screen. Feels tidier to me.

For holdem, you ideally need a plinth that can 'deal' cards upside down. Maybe it's more general to have a plinth that can update cards in either orientation.

Congrants jonahss on your art. Thanks for sharing.

_osorin_(10000) 3 days ago [-]

Love it! I suppose it's possible to display animations right? If so it'd be very cool to play Yu-Gi-Oh! and Magic with animated cards. Endless possibilities.

schwartzworld(10000) 3 days ago [-]

Update speeds are going to be too slow for meaningful animations. Also, the lifecycle of the display is measured in screen refreshes, so having to flash through the animation frames would shorten the life for sure.

jonahss(10000) 3 days ago [-]

Animations seem to be a tricky prospect for e-ink displays. Definitely possible, but may require more expensive hardware? I've seen pebble watches and remarkable tablets which can do it amazingly well, maybe I just need better firmware...

culanuchachamim(10000) 3 days ago [-]

Very nice! Congratulations!

From a long time ago I'm trying to build a e-ink magnet for the fridge with relevant updated relevant information about some school programs.

But I haven't really found a cheap e-ink display.

Edit: I see that you answered it:

>Bought off Alibaba, I think they're so cheap because they're designed for grocery store shelf price tags. Good Display GDEW029T5D Where do you get yours for $20?

realslimjd(10000) 3 days ago [-]

You can get displays from Adafruit for $22 [0] or from Waveshare [1] even cheaper if you're willing to do a little more work. I've used both and they're both pretty nice.

[0] https://www.adafruit.com/product/4777

[1] https://www.waveshare.com/product/displays/e-paper.htm

jonahss(10000) 3 days ago [-]

Oh the display is only like $4-6 on Alibaba. After you add the resin and my custom PCB it's like $20 at the low scale I'm purchasing.

jonahss(10000) 3 days ago [-]

Actually, the cards already have magnets in em, so these would work on your fridge already.

But you need a base station to update them, and a way to connect to your network to upload images/text.....

If you email me your address, I'd be happy to send you one for your fridge :D

xrd(10000) 3 days ago [-]

This is really cool.

When the pandemic started, it seemed like there was an opportunity to build games where you still are participating in physical space, but certain aspects of the game are shared across a network and then transform back into something that occupies physical space. I thought that could be something like AR but your cards open up other doors.

It would be fascinating to have a deck that shares cards in multiple distant places, with people playing the game from far away but connected through your cards.

Adding an element of time delay to the synchronization of the deck would be fun too.

Please provide a way to follow along on your progress, like a mailing list! Such a great project.

8note(10000) 3 days ago [-]

My friends and I did that with gloomhaven.

You have your local deck and player, and then the actual board is on TTS

jonahss(10000) 3 days ago [-]

Oh! I've got a mailing list signup on wyldcard.io !

Should have added that call-to-action to my post; *facepalm

Your idea is kind of like the hologram chess that Ed plays in Cowboy Bebop

unsupp0rted(10000) 3 days ago [-]

I applaud that you made a thing, and it's cool that you learned Rust, wrote a driver, did CAD, 3D printed... the works! It's far more than I can do.

But let's say all trading cards in the world were like the e-ink ones you invented, and then somebody came along and said, 'I printed 52 trading cards on glossy card paper, in vibrant color. They are 1 millimeter thick, weigh practically nothing, and cost about a penny each to produce. The drawback is you have to shuffle them in your hands'.

The paper version would be a massive upgrade in almost every sense.

voiper1(10000) 3 days ago [-]

The e-ink screen changes much more than that:

>The cards also contain a memory chip, so they can store stats, moves, and keep changes and status effects from one game to the next. ... >I think these cards have the potential to unlock a new paradigm of tabletop gaming. They are rooted in the physical world, but can implement complex game mechanics run by a computer.

e.g. Magic The Gathering has a TON of modifiers - +1/1 counters, etc - and this could live-update and keep track of those in a physical/offline setting.

knlje(10000) 3 days ago [-]

Really interesting idea and I can see many game mechanics that benefit. What do you think is the minimum thickness using any technology available?

jonahss(10000) 3 days ago [-]

Oh, if you were willing to spend $50 a card, I think you can get them down to the thickness of a credit card.

If you are willing to spend even more, you could bundle the card circuitry with the silicon embedded in the e-ink display already, and there are FPC e-ink displays. So it'd be about half the thickness of a credit card and flexible!

(at that point, we'd need to change the plinth interface, probably go for NFC like people are mentioning in these comments)

Euphorbium(10000) 4 days ago [-]

This could actually replace mtg, which has run its course, jumped the shark and is on its deathbed.

JetAlone(10000) 3 days ago [-]

I haven't played MTG in ages, and decided to check out their website to see what the new cards are like. Aoparently there's a new mechanic 'More than meets the eye' letting you cast Optimus Prime for one less mana...? Apparently it's just, part of the Brothers' War set now. It's like those fortnite ads I see for Naruto or Goku being in the game now. Games just aren't allowed to have their own aesthetic anymore because crossover deals are apparently too profitable. Totally felt off.

jonahss(10000) 4 days ago [-]

I so agree! MTG is desperately scraping the bottom of barrel for new mechanics.

That's why I wanted to get this concept out there, to inspire the next thing.

Scalene2(10000) 3 days ago [-]

I think marketing this more like a console and less like a game might be the way to go.

jonahss(10000) 3 days ago [-]

You may be right, a few other commenters said something similar.

jrpt(10000) 3 days ago [-]

What would a board game made with these cost, $100?

jonahss(10000) 3 days ago [-]

I'm trying to target $80 for a 'starter set' of the base and 3 cards. I think this is doable at real scale.

squokko(10000) 3 days ago [-]

I know you probably have nothing besides the display in the card to keep it small, but that probably increases the number of Pogo pins you need if you need to run the entire display protocol over Pogo. If you had a demultiplexer in the card you could use only 3 Pogo pins.

jonahss(10000) 3 days ago [-]

I wish it were pogo pins, but they're so expensive. I'm using spring fingers and they're surprisingly reliably and durable (so far).

I'll look into using a demultiplexer, because maybe then I could afford the pogo pins.

CrypticShift(10000) 4 days ago [-]

I love it ! you should make a color version [1] !

[1] https://news.ycombinator.com/item?id=31191850

jonahss(10000) 4 days ago [-]

Ha! I'm really trying to keep the price down, these are the cheapest screens I could find and they're still ~$4.

If they took off and were produced at scale, we could definitely do color, and make them thinner and flexible as well!

vegasje(10000) 4 days ago [-]

This is incredible! I'm so excited to see what you do with this in the future.

May I ask what e-ink displays you're using?

jonahss(10000) 4 days ago [-]

Bought off Alibaba, I think they're so cheap because they're designed for grocery store shelf price tags.

Good Display GDEW029T5D

netman21(10000) 4 days ago [-]

Looks awesome. I just want to use it to play three card monte.

jonahss(10000) 4 days ago [-]

Oh hey, I've been looking for small demo games to implement on it, before jumping into a massive RPG or something. I can try implementing that.

rychco(10000) 3 days ago [-]

This is incredibly cool & inspiring. How familiar were you with Rust before starting? Any resources you can share for programming the circuit boards you use? I've had some embedded Rust ideas myself but am not sure where to start!

jonahss(10000) 3 days ago [-]

Thanks!

I knew no Rust at all before starting. Yikes was it frustrating to learn a completely new language. I had to go through the whole rust book, not skipping any basics, in order to get there. I also assigned myself a homework assignment to get used to the concepts. See the other blog post on wyldcard.io/blog

Honestly, by the time I got to the finished prototype, I could have just done the whole thing with Javascript. Although intending to use an SDM32F7 for the finished project, it was really convenient to use a raspberry pi for development. Using VS Code, I could program remotely on the pi from my Mac and iterate really quickly.

the awesome-embedded-rust[1] repo was very helpful.

TBH, embedded rust really feels like it's not ready for hobbysists who aren't embedded experts already.

[1] https://github.com/rust-embedded/awesome-embedded-rust

riatin(10000) 3 days ago [-]

Amazing project - I love the art on the 'cards'! Who was your artist or where did you source it from?

jonahss(10000) 3 days ago [-]

Ahh... I used Stable Diffusion!

riskable(10000) 3 days ago [-]

I read a lot of these comments and people are thinking about traditional card games and electronic games of the past that are kinda sorta similar (but not really). They're missing the ONE HUGE ADVANTAGE of epaper displays: They don't need power.

It seems the big assumption everyone's making in the comments is that the displays will just stay there on the electronics box. The reality is that you can refresh the display and then take it back into your hand. You could even trade it with other players while disconnected.

Furthermore, these displays can be refreshed hundreds of thousands of times (some can do millions) and therefore can last a lot longer than regular cards. They're also stupidly cheap in bulk! Example of a cheap NFC-powered one: https://www.aliexpress.us/item/3256803070283516.html (not the best example since it's kinda thick but you can get much thinner ones around the same price).

You just need to come up with some example games that take advantage of the trading dynamic. Also, if you're up to the engineering it would be neat if you could use one epaper 'card' to modify another one. Much like how in many card games you can add an 'enchantment' or 'modifier' style card on top of another to give it additional capabilities.

I also recommend adding a cover or shutter to the cards so that they can be refreshed without the other players seeing what it is. Once the card is back in your hand you can move the cover out of the way and see what you got. Alternatively, you could make it so that the cards refresh their displays upside down. That way you can refresh it without anyone seeing what's there.

jonahss(10000) 3 days ago [-]

Yeah!

A few comments have mentioned being able to refresh the cards while upside down, or covered by another. Maybe my next prototype can put the contacts on the edges and have them wrap around or pass through, so they work on both sides.

That will be a bit more complexity for fabrication, but sounds like a fun project.





Historical Discussions: Intentionally making close friends (November 28, 2022: 1063 points)
Intentionally Making Close Friends (July 16, 2021: 3 points)
Intentionally Making Close Friends (June 29, 2021: 3 points)

(1064) Intentionally making close friends

1064 points 5 days ago by yamrzou in 10000th position

www.neelnanda.io | Estimated reading time – 31 minutes | comments | anchor

Introduction

One of the greatest sources of joy in my life are my close friends. People who bring excitement and novelty into my life. Who expose me to new experiences, and ways of seeing the world. Who help me learn, point out my blind spots, and correct me when I am wrong. Who I can lean on when I need support, and who lean on me in turn. Friends who help me grow more into the kind of person I want to be.

I am especially grateful for this, because up until about 4 years ago, I didn't have any close friends in my life. I had friends, but struggled to form real emotional connections. Moreover, it didn't even occur to me that I could try to do this. It wasn't that I knew how to form close friends but was too anxious to try, rather, 'try to form close friendships' was a non-standard action, something that never even crossed my mind. And one of my most life-changing experiments was realising that this was something I wanted, and actually trying to intentionally form close friends.

It's easy to slip into a passive mindset here, to think of emotional connections as 'something that take time' or 'need to happen naturally'. That to be intentional about things is 'inauthentic'. I think this mindset is absolutely crazy. My close friendships are one of the most important components of my life happiness. Leaving it up to chance feels like passing up an incredible opportunity. As with all important things in life, this can be optimised - further, if done right, this adds a massive amount to the lives of me and of my future close friends.

The first half of this post is the story of how I approached intentionally forming close friends, and the second half is an attempt to distill the lessons I learned from this. As such, this post is more autobiographical than most. Feel free to skip to the advice section if you don't want that. Further, what you value in close friendships is highly personal - this post will focus on what I want in friendships and how I try to get it, but you should adapt this to your own situation, values, and what feels missing in your life!

Exercise: Think about your closest friends, and how these friendships happened. What needs are you fulfilling in each other's lives? Are you happy with this state of affairs, or is something missing? What could be better?

My story

The Problem

Back when I was in school, I never had close friends. I had friends, people I liked, people I spent time with, whose company I genuinely enjoyed. But I was pretty terrible at being vulnerable and forming emotional connections. These friendships rarely went beyond the surface level. In hindsight, I expect these could have been far richer (and I've formed much stronger friendships with some of these friends since!), but I never really tried.

I find it hard to introspect on exactly what the internal experience of past Neel was like, but I think the core was that trying wasn't available as a possible action. That I spent much of my life doing what felt socially conventional, normal and expected, for the role I saw myself in. And 'go out of your way to form emotional connections' wasn't part of that. It wasn't an action I considered, weighed up the costs and benefits, and decided against - it never even occurred to me to try. It didn't feel like a void missing from my life - things just felt normal. It was like playing a video game, and having a list of actions to choose from, like 'ask about their day', 'complain about a shared experience' or 'discuss something cool I learned recently'; but this list contained nothing about 'intentionally form an emotional connection'. It wasn't in my reference class of things I could do.

One of the core parts of my life philosophy now is the skill of agency, of actually doing things. The skill of going out of your way to make opportunities. To identify what's missing in my life, and in the world. Finding the actions that I don't need to take, that no one else will make me take, or do for me, and deciding to take them anyway. Fixing that which is broken. Finding that which is not broken, and deciding to make it better anyway. Exploring and trying new things. Challenging my self-image and growing. Fundamentally escaping the mindset of needing permission, and breaking past the illusion of doing nothing. I think this is one of the most valuable skills anyone can learn, and one I cherish, though I am far from perfect at it. And this experience is a large part of why I value it. Not realising I could make close friends was a failure of agency, an unknown-unknown that cut out a massive amount of potential happiness, without even realising it.

The solution

Despite all this talk of agency, I stumbled my way out of this problem pretty much by accident. When I was 18, in my final year in school, I ended up in a long-term romantic relationship (with a girl who, thankfully, was far better at taking initiative than me!). And this was one of my first times really feeling a deep, emotional connection with someone. And, surprisingly, found that this was great, and added a ton to my life! And further, got a bunch of surface area on what emotional connections actually felt like, and how they formed.

That relationship ended in my first year of university. And as part of trying to move on and recover, I did a lot of introspection on how the relationship had changed me, and what now felt missing from my life. And one of the biggest things missing was having a deep emotional connection with someone. So I decided to fix this.

The obvious next question was, what to actually do? In full 19-year-old-Neel fashion, I took a pretty reductionist approach to this. I made a list of all the people I considered close or close-ish friends, and tried to figure out how we became close friends. And in each case, I identified the main shifts in our relationship after intense, 1-1 conversations, where we were both being emotionally vulnerable and authentic, and talking about personal things. So, to make more close friends, all I needed to do was engineer more of these 1-1 conversations!

This was also inspired by a time when I was 17, and at a rationality camp for high-schoolers. We were doing a workshop on Comfort Zone Expansion (CoZE), where the intention was to identify something we were uncomfortable with but wanted to explore and try in a safe environment. I and another participant noticed we were both uncomfortable with being vulnerable and authentic, and tended to use humour to deflect from anything personal. So, we decided to find a private place and spend two hours having a fully authentic conversation, with no deflection allowed. This was kinda terrifying, but also a really great comfort zone expansion experience, and I felt much closer to him afterwards.

One decent way of engineering an authentic 1-1 conversation is to go through a bunch of personal and vulnerability-inducing questions together, a la 36 Questions that Lead in Love (after cutting the 2⁄3 of questions that I found dull). So I made a list of questions I considered interesting, which I expected to lead to authentic and vulnerable conversations. And then went up to the 10-20 people I felt most friendly with, explained the experiment, and asked if they'd be interested in blocking out a few hours, and going through the list together.

Somehow, this worked! About 80% of the people I asked said yes, and I felt much closer with about 50% of them afterwards. Some people were weirded out, but most of my friends were down to try the experiment. With some people the questions felt awkward, but with some people I really vibed. And some people were extremely enthusiastic about the idea from the start - I explained the idea to a guy I vaguely knew, he loved the idea and suggested doing it together, we hit it off immediately, and he's now probably my closest friend.

If you're interested in the questions, you can see the full list here. Some of my favourites:

  • What's the best way to get to know you as a person?

  • What's your life story?

  • What traits do you envy/value in those around you?

  • What do you feel insecure about?

    • This one is higher variance - I don't recommend leading with it!

  • What do you value in friendships? What are the best ways they add to your life?

  • How, historically, have you become close to people?

  • If you could design a personal set of social norms for how your friends interact with you, what would they be?

  • How would other people describe you? How does this compare to how you want to be perceived?

  • What in life do you get truly excited about?

Retrospective

I am incredibly happy I ran this experiment. It has made my life massively better. And, in hindsight, I am still really surprised that the success rate was so high! If this idea sounds compelling, I would highly recommend people try it - it was an excellent growth experience.

That said, I still somewhat cringe looking back on that. I think having a literal list of questions made the interactions much more artificial. Since then, my conversational style has evolved to be a lot more natural, while trying to preserve the spirit. I really like asking questions, and will often weave these questions into a conversation if appropriate. And strongly try to create an atmosphere where people are comfortable being honest and vulnerable, and where I show vulnerability in turn.

One of the main reasons I'd recommend others try this is that it broke me out of a bad equilibria. I was trapped in a 'normal' mode of conversation - making small talk, being inoffensive, feeling aversion to being weird, respecting where I thought other people's boundaries were. But when I tried something totally different and super weird, it often went great! Sometimes other people hate small talk too, and also seek a genuine connection. But by default, this would never have happened - it took one of us taking initiative to break past the trap of social norms. And only by having a bunch of unusual and scary conversations yet having them go great did I develop the courage to be weird. Respecting other people's actual boundaries is important, but the conventional approach assumes an un-negotiable, one-size-fits-all to boundaries. And empirically, this picture is often totally off - some people think I want them to act normally, but are open to a much wider range of conversational styles if I initiate it.

Another key lesson is that closeness isn't just about spending a lot of time being friends - intentional, authentic, 1-1 time together makes a big difference. Sometimes I feel closer to someone after a single amazing conversation, than to people I've considered friends for years. Emotional connections aren't something that just happen to me - they're something I need to actually try to form. And effort, intelligently applied, can really pay off.

Advice

So, that was my story. Now, I want to try distilling some key lessons that I think might apply to other people's quests to form close friends.

A key caveat to everything that follows: I argue for being much more intentional about social things than normal. When doing this, it's easy to come across as cold and calculating. I think it's super important to try to remain authentic, and to signal authenticity. I find it helpful to generally be friendly, make jokes, be honest and transparent, and be willing to be vulnerable. Eg, being open about the strategies I'm running and why, if it ever comes up.

Seek excitement

A key mindset I use when forming connections via conversation is: "If we aren't both excited about this conversation, do something differently". This applies especially when talking to someone I don't know well, and want to figure out whether we might become good friends.

Most social norms optimise for conversations that feel safe, not ones that feel exciting, so I need to do something differently! This means asking the other person questions. This means taking a genuine interest in what we're talking about - and if I can't take a genuine interest, then I am doing something wrong.

A tactic I find helpful here is what I call recursive curiosity. I lead by asking an open-ended question that invites a detailed answer. Then, I introspect and try to notice excitement, find the part of their answer I find most interesting, and ask a follow-up open-ended question about it. Then, I repeat this process on their new answer. After about 3-4 iterations, we've normally gotten somewhere that feels alive and novel, where we're both learning, rather than the same stale conversations they have all the time. The follow-up questions don't need to be thoughtful or elaborate, often just '[specific detail] sounds interesting, tell me more' or '[specific detail] didn't really make sense to me, can you clarify? Did you mean [naive interpretation]?' are more than enough. Introspecting on confusion or curiosity also works well. Often, rather than having a clear purpose to my questions I try to maximise surface area - just asking questions that point at my confusions and try to maximise the new information I gain, and the amount that I learn. This tends to feel fairly reactive - just responding to whatever was most interesting in the last thing said.

Sometimes I feel trapped in a boring conversation direction because the structure of small talk feels hard to break out of. When this happens, I like to go meta, eg observing 'man, I feel like I keep having the same kinds of conversations at these places' or 'let's get the boring questions out of the way - [standard small talk done rapidly]'. If they seem to empathise, this is a good opener for a more fun question, eg: 'what kind of things do you get excited about?', 'what's something cool you learned recently?', or 'have you had any particularly memorable conversations in [this context]?'

I've found that practicing this skill has made me much better at forming instant connections when meeting new people, and is much more fun than small talk! Even beyond meeting new people or following concrete algorithms, the spirit of 'seek the most exciting thread of the conversation' makes talking to friends way more fun!

Warning: these tactics sometimes get the other person to monologue - I am fine with this, and most people enjoy talking to an engaged audience, but some people feel bad at one-sided conversations. If you apply these techniques, I recommend being willing to monologue in turn if the other person seems interested - otherwise it can feel like you're being insincere.

Being vulnerable

For me, vulnerability, and especially shared vulnerability, are really core to forming emotional connections. But this is difficult to manage because vulnerability, by definition, is hard. Different people have very different boundaries and comfort levels, and respecting boundaries is super important here. But, conversely, successfully creating shared vulnerability is really valuable and worth striving for.

My main approach is to create a space in which it feels safe to be vulnerable, but try to avoid creating obligations. I try to be honest and vulnerable myself, and freely share things that feel authentic throughout the conversation. I prefer to express lots of small vulnerabilities throughout the conversation rather than sharing something major and making it feel like a big deal - the latter tends to create an obligation/expectation of reciprocation, while the former better establishes a 'I consider this fine and normal' norm. I also find that both are effective for breaking people's social scripts/default ways of acting by being weird and unexpected - I find this is often a good first step to actually having a meaningful conversation. I find that sharing anxieties and insecurities can work particularly well here - almost everyone has them, it feels stigmatised to discuss them but people tend to respect you when you do, and they're often much more common and relatable than people think. I've had a bunch of these conversations, and still find it exciting (and sad) when I meet someone with really similar problems to me!

The 'without obligations' point is particularly important here, and hard to thread - sometimes people would enjoy sharing something vulnerable, but fear that it would make me uncomfortable. I like to ask questions that invite a vulnerable response, but to give the person an 'out', some kind of reasonable excuse they could use to deflect without losing face. And by gauging their reactions, and seeing how much further to probe.

Overall, this is pretty hard to gauge and balance. It definitely takes a lot of practice, and I'm far from perfect at it. But I find it very worthwhile to practice.

Personally, I tend to be very anxious about whether I'm making other people uncomfortable, so I find this technique pretty aversive at times. My main approach to motivation here is internalising that I want to be a person who actually does things - that being vulnerable and welcoming vulnerable are skills I find uncomfortable but value, and I am growing as a person if I cultivate them. And, empirically, I've found this really useful to practice. I find this often leads to really awesome interactions, often in my first interaction with someone.

Alternate title: Making friends like an r-strategist

Another key insight about friendship is that it's all about upside risk. I will meet many, many more people in my life than I could ever sustain friendships with, let alone close friendships. Thus, if I am meeting new people and want to find potential close friends, I want to filter fast for compatible people. Further, compatibility is heavy-tailed - I won't really vibe with most people, but some people are awesome. I want to explore and optimise for information. This pushed towards high-variance strategies. If I meet 100 people, and want to pursue a friendship with just a handful, this is great!

This is a very, very different mindset from standard social norms, which push me towards being bland and inoffensive, and minimising the probability of bad interactions. A bad interaction (so long as it doesn't damage my reputation) is just as useless as a mediocre interaction for finding potential friends. Instead I want to maximise the probability that, if someone is compatible with me, we have an awesome interaction. This is a key part of why I push for excitement and vulnerability - many people won't vibe with that, but it makes it much more likely that I hit it off with the right kind of person.

Further, I am not constrained by the number of people I could meet - there are a lot of interesting people in the world. This means it's OK (but sad) if some people I could be compatible with don't vibe with my approach. Some people are pretty closed at first, and take a while to warm up to new people, but are awesome once this happens - my strategies around eg minimising small talk work much less well on this kind of person, which is sad. But the ability to filter fast is crucial. (Note: The trade-off between efficiency and precision depends on your situation, and I expect I'm further towards efficiency than most readers)

An important part of this is that a good filter is something that identifies people I'm compatible with and convinces them that they're compatible with me - if it feels like I'm coldly analysing or interviewing them, this is unlikely to go well. This is another part of why I am excited about approaches centred on excitement + vulnerability, those tend to go well if reciprocated.

Warning: This logic does not apply with people who I will need to interact with regularly anyway, eg co-workers/classmates. Social norms around minimising weirdness/potential for bad outcomes make much more sense in those situations, since downside risk is much higher. These mindsets work best when eg meeting people at a party or meetup or friends of friends, where I won't necessarily interact with them again.

Another key part of hits-based befriending is meeting lots of people, and exposing myself to lots of possible hits. Some of my favourite approaches:

  • Going to meetups

  • Going to events that will attract people with similar interests

  • Talking to people around me in talks/lectures

  • Asking my friends for intros to their friends - both generically ('do you know anyone I might get on with?') and specifically ('can you introduce me to [specific person]?')

  • Proactively reaching out to people who seem interesting

    • I find this pretty anxiety-inducing, but it has a surprisingly high success rate. Most people are flattered!

  • Having public forms on my website for people who want to have a chat or go on a date

Exercise: What traits do you value in your friends? What kind of person would you love to be friends with? How could you identify these traits in someone in a first meeting?

Take Social Initiative

See longer form thoughts on this in Taking Social Initiative

A key second step to the hits-based befriending mindset is to follow-up once I identify someone cool! I try to make sure I get their contact info, and reach out shortly after meeting them trying to arrange a call/meetup. I find that many people are too socially anxious to do this, but this is a really useful skill to practice. The vast majority of my current friendships would not exist if I was bad at reaching out. Most people find this great and flattering, don't overthink it.

If you feel convinced of this logic, but still feel anxious about it, my main advice is to practice. Find some safe-ish ways to try it at first, eg with people you really hit it off with, or who seem incredibly friendly, or who you feel really comfortable around. If you're overthinking it, talk it through with a trusted friend and let them talk you into it. If you're concerned you won't know what to talk about, do some research on the person and make an agenda: a list of possible topics or questions to ask them. Initially, it takes a lot of willpower and effort, and may feel super anxiety inducing - this is normal. But after I did it a few times and it went well, my mind started to update, and it now feels like a habit. I find that a similar strategy works for most forms of comfort zone expansion.

Another tactic for overcoming anxiety is to other-ise. Imagine you met someone at a party, they thought you were cool, and messaged you afterwards asking to meet up again. I don't know about you, but I'd find that pretty flattering. Or, imagine a specific friend coming to you with an analogous situation, asking whether they should follow-up. What advice would you give to the friend? And, if it differs from your internal thoughts about your situation, why? Personally, I have yet to find a situation where the advice I give to the friend is worse than the advice I give myself.

A related and important skill is keeping in touch. Most people are really bad at keeping in touch, especially without a structure like university that keeps you in frequent contact. This means that many friendships fizzle without this structure. And this is really sad! I find a common mindset is 'if friend X really valued this friendship, I wouldn't be the one to always reach out'. But, empirically, I am confident many of my friends value our friendship, but also suck at reaching out. Being conscientious and organised is just hard, and varies a lot between people. Some people are very organised and keep in touch with everyone with ease, others easily lose track of close friends. 'Does this friend reach out to me?' is an incredibly noisy signal for how much they like you, and I consider it to convey approximately no information. I want to be good at keeping in touch, because I want to be able to remain friends with less conscientious people. And if I want to know if a friend actually likes me, there are much more direct ways of asking them.

But, fundamentally, keeping in touch should not be that hard - you just need to regularly reach out to arrange a call/meetup. This can be solved by being highly conscientious and having a good memory, but if you're lazy like me, the correct way to solve this is with systems. I have a pretty barebones spreadsheet (see template) that lets me set an interval of N days to reach out to each friend, and reminds me to reach out N days after our last call, which has completely solved this problem. Other systems, such as Calendly, are great for streamlining both following-up and keeping in touch, by making it trivial to schedule things.

For me, much of the anxiety around following-up and keeping in touch centre on being a burden, and bothering other people. A mindset I find helpful is reframing it all as providing a public good. Taking social initiative is hard and most people aren't very good at it. But most people do value fun social interactions. And, for some reason, people often enjoy interacting with me. This means that by taking social initiative, I am creating more opportunities for both of our lives to be better, which is something I find deeply motivating. 'I want to be a person who creates win-win situations' is fairly core to my identity. Whether it's following-up, keeping in touch, organising parties, suggesting group activities, etc, I want there to be more people in the world who do this. So I want to cultivate this skill myself.

Deepening Friendships

See my post on Friendships for a deeper dive into what my ideal friendship looks like

My guess is that a surprising amount of the variance in friendship quality comes from finding the right people, and that things can often flow easily from there. But I think that it is also clearly valuable to practice the skill of deepening existing friendships. I have less time actively optimising this skill, and feedback loops are harder, but here are some thoughts:

  • All my thoughts so far on vulnerability, excitement and authenticity also work for deepening friendships - I find that having regular authentic and meaningful conversations really help me feel closer to people

  • Spending time together

    • In particular, searching for unusually fun/fulfilling ways to spend time together. Pay attention to their interests, experiment, and take social initiative!

    • Relatedly, actually make time to spend together. Protect your Slack, and spend it on the people you care about. It's easy to make the mistake of considering social stuff the low priority thing to cut when I'm busy.

  • Don't feel constrained by fear of seeming weird, or social norms

    • Some social norms are good, some are bad. But if I really care about someone, I want our relationship to be optimised for their preferences. And this means figuring out what they want, and setting explicit boundaries and norms with each other

      • Eg, do they value honesty? Politeness? Bluntness? Proactivity? Compliments? Affection?

  • Relatedly, have good and clear communication about what we both want out of the friendship, and how the other person adds to our life

    • People often find this hard - it's hard to eg communicate about ways the other person annoys you without harming them. For me, a key is creating clear and explicit common knowledge that we both value this friendship and are invested in it. This frames all clear communication as being on the same team - we're trying to work together and share information, so we can both forge a better friendship

  • Seek positive externalities - be a pleasant person to be around, and find ways to add joy to the lives of those around you

  • Try to form a coherent model of how they add value to my life and vice versa. Once identified, try to actively optimise for the ways I add value.

    • Obvious caveats: Make sure to remain authentic, account for uncertainty in the model, check for consent, etc

  • Practice relevant skills:

    • As above, good communication

    • Love languages - different people have very different ways of expressing affection, and it's easy to typical mind fallacy. Some people really value appreciation, others value your time, others value gifts, others value physical affection, etc. I find it hard to empathise with people with different love languages, so it's important to explicitly notice and account for this, and understand what my friends want

    • Emotional support/debugging - one of the most valuable parts friends play in my life is providing emotional support, and help solving my problems. And I want to be able to provide this in turn. But this is really hard, and definitely a practicable skill!

      • My main tip is to avoid jumping to conclusions about the problem and what is needed, and instead to explore the problem more than feels necessary. I often explicitly ask 'what kind of help are you looking for?' - sometimes they want a solution, sometimes they just want to vent

      • See this post for more advice

But remember, these are just my takes, according to my friends and my values. You should experiment, try things, and figure out what works best for you!

Exercise: Make a list of your good friends. Which of them do you feel closest to, and what has led to this? What blocks are there to being closer to some, and what are you going to do about it?

Conclusion

Close friends are very important to my life. And solving the problem of intentionally making close friends has added a ton of value, helped me make many friendships I cherish. Further, I've managed to add a lot of value to their lives. Friendship is one of the best mutually beneficial trades I've ever made.

The ideas in this post are all specialised to my tastes, and my experiences. I am a massive fan of 1-1 conversations, and of shared vulnerability. But your mileage may vary! You should experiment, try things, and forge your own vision of what intentionally forming close friendships looks like for you.

A point I've made throughout is that this is a skill. These are all things you can practice, experiment, iterate and get better at. It's easy to think of friendships as just something that happens to you. But I assert that, for most people, the cap on how awesome their friendships can be is far higher than where they are right now - it would be weird if that wasn't the case! Relationships are complex.

Finally, my key lesson from all this is that I need to take agency. It's easy to go through life never solving this problem, never even noticing the lack. Nothing will go visibly wrong. Nobody will stop you, or solve this for you. If you need permission from someone to do something differently, let this post be it - if you want your social life to be better, the only one who will fix this is you.

If the ideas in this post have resonated, I encourage you to take a moment to stop and reflect on your social life: Are you happy with how things are going? Do you feel happy with your ability to emotionally connect? Could it be better, and if so, how? What have you tried to do about this so far? And looking forwards, what are you going to do about this?

Exercise: Set a 5 minute timer, and list as many concrete ideas as you can for experiments to run, and things to try doing differently.

Bonus exercise: Actually do something about it.




All Comments: [-] | anchor

thenerdhead(10000) 5 days ago [-]

Serious question.

For those who have had close friends in their late teens to mid 20s, do you still have close friends after a significant life event like getting a job, getting married, moving to a new city, having a baby, etc?

I relate with the author of this article in my 20s, but cannot relate at all in my 30s because my life has significantly changed (for the better).

inkcapmushroom(10000) 5 days ago [-]

For me the only close friends I have kept through my 20's and many life-changing events are ones I made in high school, and mostly because of Discord. Being able to virtually hang out has made it possible for me to maintain relationships after a move which I wouldn't have been able to keep up before.

zerkten(10000) 5 days ago [-]

No, but a better answer is probably that the relationships are different. As with most HN advice it's completely contextual.

Close friends from before the transitionary period are still people I can connect with to a large extent. Not all are people I'd now want to spend a lot of time with because time is so limited.

Making new close friends is something I've somewhat given up on based on the fact that I've moved so far away, now have had to prioritize work and family, etc. I suspect there will be fleeting opportunities as the kids create introductions and I perhaps get more time for hobbies/volunteering.

I think things are very dependent on circumstances. People commenting and giving advice only have knowledge of X circumstances, but there are really Y or X number of them out there. This means that I get left feeling like I'm doing something wrong, missing out, etc. based on how others describe life. Anxiety and other mental challenges affect how I feel too. Things aren't that bad when I actually take a big step back and think about life though.

jrochkind1(10000) 5 days ago [-]

Wait, you're saying you can't relate anymore because you don't have close friends anymore and don't want any? Or because you have close friends naturally so don't need to analyze how to do it?

I am older than my 20s, and have moved jobs and cities a couple times, but I don't have a spouse or kids. I still have some close friends (including one or two i've had since my 20s and one or two i've met subsequently), but would personally like more and closer. I'm not sure if that answers your questions!

My guess is a lot of people in our particular society in the USA stop having close friends when they have a spouse and kids... but I don't this this is true in all places and times, and I personally suspect it's not very healthy and people would be doing so much better in many ways, including unexpected ones in ways people don't realize would be affected by this, if they had more non-family intimacy. But whatever works!

nonameiguess(10000) 5 days ago [-]

Nope. I had plenty of close friends through my mid-20s, but now at 42, after moving away to another state, I still believe we have enough accumulated trust and love that I'd give those people just about about anything, and feel comfortable sharing anything, and I'm reasonably sure they all feel the same way, but we haven't kept in regular contact. I'm not a phone conversation person and never have been, and quit all social media years ago to preserve my sanity. Regular visits aren't feasible because of the distance and the visits I can manage are ones in which I prioritize seeing my family over anyone else.

I don't have any serious complaints. I'm not lonely and people aren't super-important to me. I've got my wife and my cats and that's enough. I have no idea what I'd do today to make new friends that would ever be anywhere near what my old friends were. Those were people I saw day-in and day-out for years since we were in school together, we shared major life events and formative experiences that can't ever be repeated, and spent a level of time together that simply isn't feasible for so many reasons, from having a full-time job to having to prioritize time with my wife to no longer having the capacity and energy to go out and stay up late on any kind of regular basis.

Again, I have no complaints, but I don't see how it's realistic or possible in most cases to ever make friends as an adult that are as close as the friends you make and maintain throughout childhood and teenage years, other than a life partner or partners you actually live with.

SoftTalker(10000) 5 days ago [-]

Nope. I had my closest friendships in high school and college. I haven't talked to any of those people in 30+ years (nor have any of them reached out to me). Careers, relocations, families happened. Life has forks like this where paths separate and never rejoin.

And no, I didn't make new friends in the same way. Being young with a lot of time and no responsibilities doesn't ever come back.

sabellito(10000) 5 days ago [-]

I've kept 3 very close friends (they don't know each other), and now we live in 3 different continents. One I met in my teens, another in my early twenties, and another when I was almost 30. We're all almost 40 now and keep daily contact via text and have a call once every 2 months or so. They're the people I ask advice to and share good and bad news.

There's no secret to it, we're just on eachother's minds and ping one another frequently.

viburnum(10000) 5 days ago [-]

This is way too complicated. All you have do is show up to the same place regularly to places where other people also show up regularly. School, work, dog park, gym, etc.

dragonsky67(10000) 5 days ago [-]

I think so. Seems so many people want a 'how to' on every aspect of their lives. Friendship should not be something you try to design or have a strategy for, I can think of nothing that would drive me away faster.

Friendship happens, if through shared experience, interest or just chance. True friendship doesn't happen because you woke up one morning and 'Make a Friend' was at the top of your todo list for the day.

wpwpwpw(10000) 5 days ago [-]

I agree with this. From my experience, people who feel lonely (and feel saddened by that) tend to become overly worried about what others think of them and obsessed on making an impression. That usually leads to them becoming an echo chamber of themselves and starting to act 'artificially', which is noticed by others as not being authentic. I'd say that sharing, which arises from mutual interest, is the cornerstone. So being in places where people authentically share mutual interests and cooperate is a great place to make friends. Sports associations (team sports, climbing, hiking, with the added bonus of physical activity being awesome for mental health) community centers with activities (chess, musical groups, dancing)...

kelnos(10000) 5 days ago [-]

Showing up isn't enough. You have to engage with people, and push past mundane surface-level topics of conversation. And after that, you have to follow up and schedule time outside of whatever place/activity where you met.

I know plenty of people who go to work or a gym or whatever every day, but don't come away from it with close friendships.

pessimizer(10000) 4 days ago [-]

That's quite obviously not true, because almost everyone shows up to places regularly, and some people fail to make friends. Unless you assume that all the people who are having trouble making friends are jobless independently-wealthy agoraphobic shut-ins.

batiudrami(10000) 5 days ago [-]

Showing up definitely helps but in my experience the transition from acquaintance to friend comes through a bonding moment where there was an emotional connection - either struggling through or achieving something together or vulnerability from one or both of you.

There are a lot of bad things about drugs and alcohol but one of the reasons I like them is they tend to enable/accelerate these moments.

standardUser(10000) 5 days ago [-]

I'm in my 40's and I have a lot of close friends, both in my city (mostly newer friends) and around the country/world (mostly older friendships).

My trick is, I cheat. I'm happily single, live alone and have no children. That gives me the luxury of doing the heavy lifting in my friendships. I travel a lot and most of those trips are to stay with friends in their homes or to meet them at a third destination. That's been the most effective way to stay very close to my friends, as well as to become close to their spouses and kids. I understand most people can't do this because of their own family obligations. But maybe it's something people with families could do just once or twice a year to visit people important to them?

Beyond that, I'm usually the one to start group chats or send memes on social media or schedule video chats. Sometimes it's a chore, but usually it's effortless for me. And the close friends I've maintained to this point are with people who are usually very eager and able to stay in touch on their end. The others have fell by the wayside over the years. And some I have consciously excluded from my life when I realized they were bad friends.

I'd also add that, as a man, I find that women make better friends then men, especially past our 20's, and I'd encourage all men to consider finding and building more non-romantic relationships with women.

shepherdjerred(10000) 5 days ago [-]

> building more non-romantic relationships with women.

I completely agree; I've always preferred being friends with women, but in my opinion it's not easy to do.

Things sometimes get complicated quickly if there's attraction on either side which often happens to at least one side if you're close friends. Even without romantic feelings, things get weird/awkward if either party is in a relationship or married.

mikesabat(10000) 4 days ago [-]

I agree with a lot of this comment, but honestly having kids is just another connection point with other people - another reason to form a friendship.

A few months ago I organized a Dad's Drinks at a local bar. I walked there with a neighbor and realized that on every block there was at least one dad chilling in his house that wanted to come out. Having kids builds a habit to stay home, but isn't always a necessity - especially once the kids are 3 years old or older.

andreyk(10000) 5 days ago [-]

Very brief summary: a personal anecdote about forming good friends followed up by advice for how to have better conversions (vulnerability, curiosity) and how to meet more people who will become friends (meet people but filter out most of them, take initiative with following up), and lastly how to deepen friendships.

As someone who has a fair number of close friends, I think this is a good post with lots of solid advice. And it's written well! Personally I've been lucky to just naturally meet and form my best friends over time, but I definitely think taking initiative and following up are important to maintaining a close friendship.

A general point in this post I also like is that making and keeping close friends may require work and energy, rather than being something that life just throws your way. I intentionally try to periodically message friends, come out to places they live, just generally keep in touch; as one gets older people move about and it's harder to maintain your closest friendships, but it is possible!

Last comment: this is a very pragmatic and analytical post with a lot of discussions of things you can do, and not much discussion of how you should feel. I'd add this - just CARE. Appreciate the people in your life and let that appreciation guide you.

ngold(10000) 4 days ago [-]

pro tip. Don't be a flake like me.

ngold(10000) 4 days ago [-]

don't be a bummer

aeturnum(10000) 5 days ago [-]

IMO this is the #1 takeaway:

> I was trapped in a 'normal' mode of conversation - making small talk, being inoffensive, feeling aversion to being weird, respecting where I thought other people's boundaries were.

If you want something else, you gotta let others know somehow! Everyone's life is different and has differently shaped spaces for relationships, interaction, etc. It's very common for people to be open to growing closer to some of their social circle, but be unsure about who to focus on. It can be you if you simply indicate you are also interested!

On the other hand, I think this is more wrong than right:

> A point I've made throughout is that this is a skill.

Calling 'Growing closer' a skill suggests it is a craft that 'you should' practice - but it's not. Growing closer requires work and attention and you will get better at it over time - but to think of it as a skill 'you are doing' instead of a mutual process that advances only at the shared pace you are getting into manipulation. 'Growing closer' is measured by the combined 'distance' by which you and the other person have adjusted your lives to be closer to one another.

Maybe OP means that having deep and vulnerable conversations is a skill - and THAT I would agree with, but it has nothing to do with growing closer! You can have deep conversations with strangers (in fact it's often easier to have them with strangers).

People who discover the power of intimacy in our atomized world risk accidentally leading others to believe they want a deeper relationship than they have the capacity to maintain.

kelnos(10000) 5 days ago [-]

> Calling 'Growing closer' a skill

My take on it that the skill isn't the act of growing closer itself, but the actions you can take to elevate conversation above small talk and see if there could be greater compatibility there for a closer relationship. And that latter part is maybe a big part of the skill: I agree that just having a deep/vulnerable conversation isn't going to give you a close friendship, but I think that's a necessary step on the path. Determining whether or not -- from those deeper conversations -- whether or not a deeper friendship is going to develop is also a skill worth mastering.

(Maybe the author did actually mean what you thought, in which case I agree with you... and so I like my explanation better.)

andreyk(10000) 5 days ago [-]

I think the 'this is a skill' just refers to the various things needed to form and maintain friendships (being good at conversations, taking social initiative, following up, etc.) More of a set of skills. And to be fair, my interpretation is that it is a set of skills both people in the process of friendship possess and need to practice to make things work (it is possible after all to be a bad friend or a bad partner due to your own lack of effort or skill with eg communication or vulnerability).

ordu(10000) 5 days ago [-]

> to think of it as a skill 'you are doing' instead of a mutual process that advances only at the shared pace you are getting into manipulation.

A mutual process needs some coordinated actions from both sides. If I'm unable to play my part, then there will be no mutual process. I believe I understand what you are trying to say, but it is the next stage of education on 'how to make friends'. When you have skill to make friends, then you can think of how to apply it mutually. If you have no such skill, then all your argument about manipulation just doesn't apply.

> If you want something else, you gotta let others know somehow!

There is a question of how to do it. I do not know, for example. Have no idea. Several times different people tried to make friends with me, I tried to be open for it, and it didn't work out.

balaji1(10000) 4 days ago [-]

> work and attention

Attention is such a good choice of word. Attention with memory is key. I feel memory/remembering is the work, because we are constantly flooded with new things all day. Otherwise, once I am out of a conversation, I am into another app or meeting.

> accidentally leading others to believe they want a deeper relationship than they have the capacity to maintain.

Damn that's true. Though I think our instinct is to try to connect on a deeper level, and with just good intentions. We can be good friends with a lot of people. And be close to very few. That's ok, and better to be clear.

lordnacho(10000) 5 days ago [-]

Great article, though it does give a sort of programmy-algorithmic answer to one of life's great questions. Not saying that's bad, just unexpected.

I was actually thinking about it today. Here's an awkward truth about friendships that everyone needs to get comfortable with: often one of you is more interested in the relationship than the other. You know what I mean. For instance, I was invited to a wedding a few weeks ago. It never occurred to me to write to this buddy in many years, I didn't invite him to my wedding, but he invited me to his. I had a great time, we caught up and had a good talk. The same has happened the other way round, I'm sure. People I like a lot and contact, but they don't contact me. Yet when we hang out, everyone has a good time.

If you act weird about these relationships, you lose them. You don't want to do that, because marginal relationships are maybe the most rewarding to maintain, IME. People doing different things to you in different places bring a lot more into your mix than the ones you see every day. There also tend to be many of these relationship-seeds, so your close friends will grow from some of them.

CretinDesAlpes(10000) 5 days ago [-]

It's a good point, but I would say it's not awkward, just expected unless exceptional circumstances (for example two single persons sharing a flat and having similar interests at that time). Like any relationships, friendships evolve with time, you might get close at some point but things change and you are not in touch much for the next few years, and then the situation changes again and you are back closer. Don't you think?

I see this happening especially when people get in and out of intimate relationships. I think it's just life...

balaji1(10000) 4 days ago [-]

'Traditions are solutions for which we have forgotten the problems' - I think this quote fits to so many more situations as I get older. Relationships (w/ friends and family) are easily strengthened by traditions.

Traditions are very algorithmic. Traditions sometimes force us to connect/meet people and resolve conflicts if you follow the 'rules'. Traditions can be hard to follow (as in overcoming some inertia or discomfort).

PS: Don't have to think of just cultural traditions. A gang of college friends can have their own silly but decades long traditions and inside-jokes.

SubuSS(10000) 5 days ago [-]

I think that's any close relationship: the internets like to say if you don't get 50/50 - hit the lawyer, but reality is it is never 50/50. You won't be in any long term relationship if you don't accept that.

Michelle Obama posted this recently (that I agree with very much)

'''

michelleobama

Verified

As an adult, I've lived in a number of places, but as far as I'm concerned, I've only ever had one real home. My home is my family. My home is Barack.

But here's the thing—our marriage has never been perfectly 50-50. One of us is always needing more or giving more. We have to be willing to listen to each other, honestly and without defensiveness. Only then, can we evolve together.

Over the years, a lot of young people have asked me about marriage. And my response usually goes something like this: You have to prepare yourself for long stretches of discord and discomfort. You have to learn how to make real compromises in the way you've lived as an individual. Glamorizing a relationship while you're dating will lead you straight to difficulty once you're married. You can't paper over problems when you're living with someone day in and day out.

So you've got to ask yourself: What are you trying to get out of this relationship? Have you truly thought it through? Do you want a wedding or do you want a lifelong partnership? Those are two very different things. Together, you are answering the question: Who are we and who do we want to be?

'''

zug_zug(10000) 5 days ago [-]

I appreciate you calling this out.

I want to take a step back and generalize a little here, because I think there are a huge number of hypothetical 'ideals' we tell ourselves friendships (and families and relationships) are 'supposed' to be, but in reality basically never are - and when we hold ourselves to these ridiculously high standards literally we just end up isolating.

For example I think a lot of people act like lying is a huge betrayal of trust, but white lies are more-or-less a social lubricant (even comfort-lies that might prevent the person from growing, like 'Yeah it was him/her, not you.').

Another example is romantic idea that we're 100% into 1 person romantically and 0% into anybody else until the moment we break up.

great_reversal(10000) 5 days ago [-]

I'm 23 and have lived most of my life without having close friends. I'd say I'm very extroverted and can strike up engaging conversations with most people I meet. I'm easy going and don't really have a fear of talking to new people.

The problem is that I end up with dozens of friends that I regularly have good banter with, but it just never feels like one of those 'best friend' or 'close friend' situationships. Most of my own problems or vulnerabilities are dealt with by myself, internally, perhaps with some advice from others if I discuss it. It feels like you need to lean on people emotionally in order to actually become closer.

What OP has mentioned does actually work and I have used some similar approaches, without knowing it, to improve my friendships. But you also need to attach a high value to regular conversation and communications. If you don't keep in contact on a regular basis, the friendship will diminish, regardless of the closeness.

zug_zug(10000) 5 days ago [-]

> It feels like you need to lean on people emotionally in order to actually become closer.

I do feel like a lot of friendship comes down to interdependence, which I see as a good thing. But it's not always emotional, it could be that he helps you fix your motorcycle and you let him rant about his ex-wife. Or maybe you rely on this person's great taste in video games.

But yeah in general I think being open enough to gently 'ask for help' in one form or another from people helps them grow close to you.

photochemsyn(10000) 5 days ago [-]

Perhaps I'm an outlier, but having to sit around making small talk and sharing personal life experiences with a like-minded cohort is literally a nightmare scenario for me. On the other hand, I love working with people on collaborative effort-intensive projects, working toward shared goals and so on (but I don't have much interest in getting involved in their personal or family lives afterwards). Also, I may be unusual in that I don't suffer from loneliness, there's always something interesting to get involved in or work on. Is this a very uncommon type, I wonder?

Additionally, financial restrictions on activities are a reality for many people these days, ruling out a large number of extracurricular activities that would be fun and interesting to share with others. When one's economic future is on the line, things like friendships become luxuries that might have to fall by the wayside, as people have to move in search of employment, and budgets get tighter. People in a college environment are often not under such immediate pressures.

standardUser(10000) 5 days ago [-]

'making small talk and sharing personal life experiences'

It sounds odd to me to combine these concepts in the same sentence. Sharing personal life experiences is arguable a necessary part of building deep friendships, and it's often a vulnerable and intimate act. Small talk is none of that, if not the opposite.

l33tbro(10000) 5 days ago [-]

This is kind of sad. It just feels like OP has built this enormous apparatus over some emotional void they've picked up in their development. Making friends just isn't that hard for the vast majority of people.

If you're having to use reverse engineering and problem solving to form emotional connections, then you might want to spend the time in therapy and working through what caused you to be emotionally stunted.

kelnos(10000) 5 days ago [-]

> Making friends just isn't that hard for the vast majority of people.

Is that actually true? Things I've read seem to suggest that, at least for men, forming close friendships is hard, especially as we get older.

I'm feeling it, too. I'm 41, and with the exception of my partner, I don't think I've made a close friend in nearly a decade.

(Also, note that this is talking about close friendships, not just people to hang out with.)

But really, this 'apparatus' seems to work for this person, and he seems to be happy with the results, so who are we to criticize? If others who also have trouble making friends can find this useful, all the better. If not, that's fine too.

alocasia-1(10000) 5 days ago [-]

Therapy can literally just be talking through the same kinds of mechanisms as described here (with the plan to apply them to real life). Source: it was for me. How are they going to become second nature if you don't practice?

ptato(10000) 5 days ago [-]

You're making a lot of assumptions. Sometimes a blog post just isn't meant for you.

archagon(10000) 4 days ago [-]

Why are you so weirdly judgemental/insecure(?) about this very earnest post?

> Making friends just isn't that hard for the vast majority of people.

Good for you and the vast majority of people, I guess...?

xxEightyxx(10000) 4 days ago [-]

Making friends is easy, keeping friends and growing the relationships are difficult. I am mostly introverted but have periods of extreme extroversion. I'm 34 but all of my best friends have passed away from various accidents over the past decade.

I feel like a shell of my former self where all I do or have a desire to do is work. The only person I consider my friend is my former boss who, after losing his wife to ALS, I did everything I could to be there and support him because he took a chance on me by offering me a job that ultimately has led to a great career in the tech field.

I've noticed my personality has slipped away, I no longer have hobbies or anything of real passion in my life anymore and thus there's nothing I have to offer another person in a relationship.

It's incredibly lonely and a poor place to be yet I really don't know how to escape it. Most other dudes my age have families or many of them only care to go out to breweries which I don't enjoy.

There's still a bit of hope deep down that all isn't over yet and a ray of sunshine can still propagate itself somewhere, sometime.

g4e2t(10000) 4 days ago [-]

Make two lists. One, an activities list. Every time you hear something fun to do, put it on the list. Eventually, after a couple years, you'll have nearly everything. If it's on the cheap side it's a good sign, although expensive things can be fun too.

The second list, is a content consumption list. Basically, building a twitter, but without the timeline. Write down categories you like, and people you like, and get a YouTube subscription so you don't have to have adds. After around 2 years, you should be rarely adding people & content to the list.

My advice: have a separate categories for movies to watch & tv shows to watch & only do so with friends & loved ones. Never watch tv or movies by yourself, would be my advice.

I understand loneliness. It can be a sad journey.

hermitcrab(10000) 4 days ago [-]

Sorry to hear that. Recommend you try an activity that involves physical activity and other people. Kayaking, capoeira, rock climbing, whatever feels interesting to you.

ArekDymalski(10000) 4 days ago [-]

> there's nothing I have to offer another person in a relationship.

I think that the most valuable (and appreciated) thing you have to offer are your atttentive ears and your time. That's what many people are looking for in a true, deep friendship.

The real frienship isn't about super-intresting hobbies or flashy personality but really basic things like listening each other, supporting i small or large difficulties etc.

the_only_law(10000) 5 days ago [-]

Every few years I get this urge to "start over" somewhere new where I'm going to turn everything about my life around. Whenever I've tried to make friends or be social, it seems to fall into this almost cyclical pattern.

I'll find a group of people who hang out and join that. Tend to make pretty good inroads early, and people seem to like me.

Then over time the group slowly starts to fracture into "cliques" none of which I eventually fall into. It starts to feel at some point that I'm no more than a court jester, there to make opportune jokes to entertain the other cliques. People care less about what I have to say and I guess any eccentricities about me that intrigue people get old quick.

Sooner or later there's a big drama caused by some petty disagreement and the group fractures along those cliques. Sometimes reconciliation is tried, but by that point no one's schedules line up, or half the people are always sick, etc.

OmarShehata(10000) 5 days ago [-]

I've seen this happen but it's really interesting that you say this is cyclic/it always falls into this pattern for you.

Have you found that reaching out to individual people in the group to be worthwhile?

I often find it hard to make/maintain meaningful connections in these groups (and the 'people care less about what I have to say' etc feels relatable), but I've had great success with just asking individuals like 'wanna grab coffee/lunch sometime' either 1 on 1 or in a group of 3. It can be kind of awkward, and there's always a sense of 'Oh you guys were hanging out? Why didn't you let me/XYZ know!' but I just embrace that and it's been a much better way for me I think to make close friends.

nerdponx(10000) 5 days ago [-]

This is interesting because I have a handful of close friends, but I've never made a close friend intentionally. I suspect that even 'naturally social' people might benefit from thinking about their social lives and relationships with more intention, especially as we/they get older and opportunities for random friendship development tend to wane, especially in highly-individualized Western society where people spend most of their time focused on themselves and their nuclear families.

Enginerrrd(10000) 5 days ago [-]

It is my experience that the most 'naturally social' individuals ARE operating with a lot more intention if they let you really pick their brain, or you watch them interact with disparate groups of new people in different settings.

ianbutler(10000) 5 days ago [-]

I've only really intentionally made a close friend one time. However, my longest closest friend was not that way. Him and I bonded pretty much immediately after getting into a fist fight on a high school wrestling mat during practice and our coach had to separate us. Now he's been there through most of my achievements and I his. Good guy, doing a PhD in chemical engineering at Case Western now.

harry-wood(10000) 4 days ago [-]

yes. I see what you're saying.

heads out into the street to start a fist fight

AtNightWeCode(10000) 5 days ago [-]

That font is super-annoying. You are not a friend...

emadabdulrahim(10000) 5 days ago [-]

Maybe you could be his friend and help pick a better typeface that works for prose ^^

sys42590(10000) 5 days ago [-]

I really like reading this article and I think it's a good start to get rid of a loner lifestyle.

There's however one caveat... making close friends is quite a culture dependent topic in my opinion. Making close friends requires differerent approaches depending on the culture you're living in. Strategies that work in San Francisco may not work in Miami, Buenos Aires or Trondheim.

adamsmith143(10000) 5 days ago [-]

I think the strategy of 'Go do things with the same group of people repeatedly' is quite universal.

coldtea(10000) 5 days ago [-]

>I find it hard to introspect on exactly what the internal experience of past Neel was like, but I think the core was that trying wasn't available as a possible action. That I spent much of my life doing what felt socially conventional, normal and expected, for the role I saw myself in. And 'go out of your way to form emotional connections' wasn't part of that.

The whole intro part of the post sounds a lot like ASD 1 musings, including the approach regarding the solution...

mjfl(10000) 5 days ago [-]

They are numbered now?

jdkee(10000) 5 days ago [-]

'What's fascinating is why that disconnect happens. Since the 1950s, sociologists have identified three factors that need to be in place for friendships to take hold.

They are:

• Proximity: You're around each other regularly.

• Unplanned interactions: You see each other even without putting an appointment on a calendar.

• Privacy: You're in a space where you can exchange confidences.'

See https://www.businessinsider.com/things-that-help-people-make...

hoseja(10000) 4 days ago [-]

Each one of these has been destroyed haha. Surely nothing bad will come of this haha.

legerdemain(10000) 5 days ago [-]

The author is a child. The core of his anecdote is 'I dated a girl and then I went to a summer camp and got my camp mates to do that '36 questions' exercise the New York Times popularized several years ago.' I think this advice would be useful to socially awkward college students who need explicit instruction for talking to classmates.

yunwal(10000) 5 days ago [-]

Yeah, I was so ready to agree with this article at the beginning, but then got to the 36 questions part and felt queasy reading it. Imagining a friend coming up to me and reading off some script about 'who is your ideal dinner guest' makes me depressed.

I do think there are some decent observations in the beginning of this article that a lot of people take too long to realize.

1) There comes a time in most people's lives where making friends won't come without a lot of effort. You will have to go out of your way to create situations where you can get to know people.

2) Asking probing, slightly uncomfortable questions can lead to deeper friendships. If you're the type of person who doesn't feel comfortable digging into people's feelings, it's worth it to force it.

That being said, you need to treat your friends like they're unique people. You can't read questions off a script and expect them to feel good about it.

hahamrfunnyguy(10000) 5 days ago [-]

While the author is young, they've made some accurate observations - at least in my experience.

tinglymintyfrsh(10000) 5 days ago [-]

Disclaimer: no unsolicited advice please.

I'm old(er) mid-40's, single, no kids, and never married. I don't have much family or close friends. Hookup culture is de rigueur but it doesn't interest me. Where I'm at, there isn't a context to meet people and most random people in public around keep to themselves and treat me like I'm invisible. I do volunteering but that also doesn't go anywhere.

If you have to constantly take the social initiative, then you're carrying the relationship. Sometimes absence is better. Instead of dwelling on loneliness or self-pity, keeping productively busy seems a better alternative.

I'd like to have a family and kids, but an inability to find and make friends precludes that. The thought of getting old alone, having no one to check-in on me, no one to bury me, no one to care that I'm gone, and no one at a funeral seems depressing. I don't see how I won't end up in a 'potter's field' somewhere.

leobg(10000) 3 days ago [-]

Also, an unsolicited book recommendation: Hesse, Steppenwolf. You may recognize yourself in it. I did.

afr0ck(10000) 5 days ago [-]

It breaks my heart to see Western people living such lives. I love the West for so many reasons, but the social construction is a complete broken mess. Family and community is completely destroyed. People live and work only for themselves. The amount of loneliness and isolation is just mind blowing and unbearable. IMO, it's a selfish lifestyle. People who manage to make great success (due to a combination of lack, intelligence, hard work and sometimes just beauty) can live very good lives but the rest are doomed to live miserable, lonely lives and no gives a shit about them until they die. Very sad. I wish you happiness and fulfillment (whatever they are for you).

spyckie2(10000) 4 days ago [-]

Some people are too, too risk averse. They subconsciously avoid, not just pain, but the possibility of discomfort. It's not just one or two actions - it's their filter that they see the world through.

Pain avoidance is often disguised as disinterest, but really it's just a cost function to minimize the possibility of feeling embarrassment, awkwardness, discomfort, heated discussion, or emotional distress. Sometimes this is personality related; other times, this is a result of an aggressive, hurtful, or manipulative upbringing.

I have a friend like this and he is very hard to be friends with. He hit this point where he was stuck in an unhappy life situation (have few good friends, can't find love, etc) but vehemently opposed anything slightly uncomfortable that will move him from his unhappy cage.

What's happening subconsciously is a deep seated trauma/fear freezing him in place. For my friend, he was bullied as a kid, and his parents also were emotionally angry and abusive. He grew up with the fear that out of nowhere, a seemingly trivial sentence or action could send him into a world of confusion and hurt.

Today, meeting new people and sharing emotions is a giant minefield to him. But most people just see him as stubborn, whiny, hard to socialize with and doesn't listen to advice.

But it's hard to be his friend not because of his awkward socials but because he complains about his distress so much but doesn't do anything about it... for years. As someone who wants to solve problems and make progress, it is very frustrating to see him move so slowly.

He is very, very slow to go out from his comfort zone, even just a little bit.

valarauko(10000) 5 days ago [-]

> Hookup culture is de rigueur

I don't think that's generally true, especially if you're looking to date around your own age. Some in the age bracket may be averse to marriage, but not a relationship in general.

kelnos(10000) 5 days ago [-]

> If you have to constantly take the social initiative, then you're carrying the relationship.

I don't think that's true, and the article does talk about this a bit. Relationships are never actually 50/50. If you're expecting that, you're gonna have a bad time.

But really, think about outcomes more than trivial details: if you are the one who always has to be the one to reach out, but you end up having a good time when you talk or meet up with the person, then isn't that still a good outcome?

But ultimately everyone gets to make their own choice, of course. Would you rather have no friends or family, or would you rather have to put in more effort than the other person at some aspects of maintaining a friendship/relationship? Ideally it shouldn't be this stark dichotomy, but if it is, you gotta make your choice.

(Also, 'no unsolicited advice please' is a weird thing to say when you're posting about a personal situation on a public forum. If you don't want to take this as advice, then take this as disagreement with what you're saying, and my expression of frustration that your choice seems to be 'be alone and sad' when the alternative is 'put in a little more effort than the other person'.)

mrcrumb1(10000) 5 days ago [-]

This line of thinking seems pretty prevalent but the way I see it, there's no such thing as carrying the relationship. If you're the one with the social need, the onus is on you to get that need filled. This generally means doing more work than others but it's not on behalf of them. It would be nice if other people filled our needs for us, but that's generally just wishful thinking.

xyzelement(10000) 4 days ago [-]

Apologies if this is in the category of unsolicited advice. I noticed that you had a submission describing a similar problem set in the work setting, so I am seeing a theme. It sounds like you are in a lonely place and are hoping for something better.

There's no magic to it but I can recommend some things that work very well for me.

First, do you have a curiosity about people? The folks you're working with, or folks at your volunteering events - do you strike up a conversation? Do you know where people are from, what their families are like, what their hobbies are, what they think and worry about? I don't mean to encourage you to engage in robotic small talk, but asking you whether these things are actually of interest to you? Wanting to know someone is the first step of a relationship.

Assuming you have the interest, ask whether you've done anything to learn these things? If you realize that there are 30 people you could have known these things about but don't - why not? Are you very shy? Do you think people will think you're rude? Do you not know how to chat? Answer this for yourself and seek resources on-line for how to close those gaps. I went from a shy mush-mouth to a pretty good shmoozer in a few years, it's possible.

If you decide that you actually don't have an interest in people (like, you genuinely don't care what anyone's life is like) then there's a strong disconnect between than and the desire to have a family/kids/friendship. If you have the ability to, seek some sort of talk therapy to explore this question. Chances are there's some 'wound' that's blocking your ability to want to connect with people if this is the case.

Finally, you sound like someone who closes doors before they are even open. I suspect when you say 'hookup culture...' what you are really saying is 'I have a reason for why I am not on dating apps.' When you say 'people treat me like I am invisible' - you are writing off an entire town as not being interested in you, which gives you an excuse to not try to connect with them (yet, somehow they had all met each other - so it's probably your vibes - and that's in your control to change). If you really believe that your location is a problem that's preventing you from having relationships, then move. If you have a desire for family and kids and you think that's not possible in your location, why are you there? (and, if you are about to give yourself an excuse as to why that's impossible, pay attention to that too)

kleer001(10000) 5 days ago [-]

:Not advice, but many questions and some sharing my experience:

Feel free to answer or not as you like.

> I'm old(er) mid-40's

I was in my early 40s when I found a lovely mate, married, and reproduced.

> Hookup culture is de rigueur

What makes you say that? What geographic region is that? Is that just the kids or those in your demographic?

> Where I'm at, there isn't a context to meet people

Are you in a place without personal ads?

> I do volunteering but that also doesn't go anywhere.

Is that volunteering at the same place or different ones? What kind of volunteering?

> I'd like to have a family and kids,

Do you want to make your own kids? Are you willing to be a step parent?

> but an inability to find and make friends precludes that.

That's not my experience. There's popular advice that one should marry one's friend. But in my experience one can just look for a marriage partner and leap frog that nonsense.

> The thought of getting old alone

Ah, so essentially human, thank you for sharing. I too have that at my core.

standardUser(10000) 5 days ago [-]

'Hookup culture is de rigueur but it doesn't interest me.'

That's media bullshit that people really like to believe. There is no shortage of people looking for relationships.

leobg(10000) 3 days ago [-]

Our neighbor is in his 70s. No kids, no wife, no family. Spends his time watching crypto videos on YouTube and gardening in summer. He is polite and helpful, but we try to avoid getting into conversations with him because those end up becoming monologues of him telling us about the apocalypse.

Living alone today is easy. It's also comfortable, because you save yourself all the drama and overhead of having to deal with another irrational human being. It's also easy to feel superior for living this way, telling yourself that you are rejecting a sick society. The thing is, with nobody knowing you intimately and giving you feedback, you can end up being totally delusional and not ever realize it.

ChrisMarshallNY(10000) 5 days ago [-]

It's a great article, but is not the way that I have worked.

I'm a bit 'spectrumish.' I'd make a great hermit. I don't really find a need to have close relationships.

One thing that I've found in any relationship, is that a 'power dynamic' changes things a lot. Most romantic or working relationships have a 'power dynamic,' where one party has some kind of leverage over the other, or that one party gets more from the the relationship, than the other.

This isn't necessarily bad (after all, we get married and have families all the time), but I feel that it adds a different 'color' to the relationship.

Most of my closest (male) friends are ones that I don't have a 'power dynamic' with. We usually have drastically different vocations, and don't really need each other. We generally have common interests (I suggest that volunteer work is a great thing), but find each other's company enriching and interesting.

xyzelement(10000) 4 days ago [-]

// Most romantic or working relationships have a 'power dynamic,' where one party has some kind of leverage over the other,

This is a very weird way to think about it. Something I realized after I got married and bought a house and had kids, is that I love it and none of it would be possible without my wife. Likewise, she couldn't have any of this without me. We are partners.

If we wanted to look for power imbalances, we could find them. Like, we could obsess over who makes more money, or does more child care, or who had a greater impact on our selection of where we live etc, but that would be a very weird thing to put front and center. Much more importantly is that both of us are willing to put a lot into the relationship and consistently get even more out of it, enabling us to lead deeply meaningful lives together.

mikesabat(10000) 4 days ago [-]

This is a great post. A few immediate thoughts.

Friends become harder to make (mostly for men) as you get older and have children. It simply takes more effort and you'll have less time, but having friends that you see regularly is probably more important when you are 40+. It's worth the effort and easier to systemize.

Following up & reaching out can be a big blocker. I just decided that I'm always the one to reach out and I even built myself an Airtable tracker to do this.

Vulnerability is trending up (for lack of a better description). Writing this post was vulnerable and it's #1 on HN. Finding the right way to open up, make it interesting and not seem needy is an very valuable skill.

floppydiskette(10000) 4 days ago [-]

Why mostly for men?

erdos4d(10000) 5 days ago [-]

This guy should move to scandinavia, he'd learn that conversation is cheap. I can't tell you how many times I've spoken to a scandie for literally hours just to have it come to nothing. No new friend, not really even an acquaintance, it was all just talk and meant nothing at all.

SoftTalker(10000) 5 days ago [-]

Scandinavians are very friendly and pleasant people on the surface, but it's hard to penetrate that surface into deeper friendship. To really get into someone's inner circle of close friends you pretty much have to have grown up with them or known them for years.

timoth3y(10000) 5 days ago [-]

"If you go out looking for friends, they are scarce. If you go out to be a friend, you'll find them everywhere."

― Zig Ziglar

zelphirkalt(10000) 4 days ago [-]

The number of people I have tried to be friends with, but who not treated me like one -- I think it sounds catchy, but is not true at all, at least in my circles.

galaxyLogic(10000) 5 days ago [-]

And in the end the love you take is equal to the love you make

- Beatles

jcmontx(10000) 5 days ago [-]

This fellow had higher emotional IQ at age 17 than me at age 27. Great read!

hnuser847(10000) 5 days ago [-]

If it you makes feel better, at the age of 34 my therapist handed me a 'feelings wheel' as a way to help me name my emotions[1]. Prior to that, my emotional vocabulary essentially consisted of 'anger' and 'joy'. This in turn lead me down a very deep and painful rabbit hole to figure out why I had so much trouble accessing my own emotions, opening up to people, and trusting others.

[1] https://feelingswheel.com/

MetaMalone(10000) 5 days ago [-]

Think we have been forgetting the necessity of relationships in a post-internet society. It can be a good thing to be intentional about relationships. The benefits don't only apply to introverts, it can also be important for extroverts to cut off harmful people from their lives and deepen connections with those they need most.

standardUser(10000) 5 days ago [-]

Plus one for intentional relationships. I've become much more careful about who I form bonds with as I've grown older, and I've left a handful of long-term friendships behind when I realize I didn't want those people in my life. I'm not an extrovert, but I do make friends easily and I learned the hard way that if you just make friends with everyone who comes your way, you'll end up with some bad friends.

Barrin92(10000) 5 days ago [-]

I don't really understand this tendency to create these artificial systems for having friends. To me it's in the same category as dating apps, or weird exercise routines.

The best way to have deep friendships is to live a social life. Engage in the community you livee in, take on responsibility, be politically active, start a business, become an open source maintainer, if you're religious go to church/mosque/etc. If you're active and engaged with the people around you'll basically have no choice but to have deep friendships. We're social animals by nature, there's no need for a ten step program.

yshklarov(10000) 4 days ago [-]

We're bipedal animals by nature, but not every individual is able to walk. Likewise, although we're social animal by nature, not every individual can easily form close social ties.

There are many reasons a person might struggle to make friends. For instance, some people were poorly socialized, or have some kind of early-childhood developmental trauma that makes it difficult to feel the appropriate emotions when talking to people. Others naturally have trouble with empathy and so are unable to smoothly navigate social interactions. And some just naturally aren't interested in learning about other people's lives, or in telling others about themselves. So they have to force themselves somehow.

I agree with you: these artificial systems are pathological and unpleasant. But sometimes it can seem like the only option.

My impression is that you've never struggled with any of these very basic aspects of life. Consider that not everybody's experience is the same.

lezojeda(10000) 5 days ago [-]

You are describing ways of getting acquaintances/just friends, the article describes ways of deepening those friendships such as paying more attention to questions that lets you better know people, I found it very helpful, it may come 100% naturally to you, but not to all of us

micromacrofoot(10000) 5 days ago [-]

I don't think your experience translates well to everyone, hence the need for different methods.

I'm active in my community, politically active, have started and sold a business in the past, etc... and all I have are acquaintances. I don't think anyone outside of my immediate family even knows when my birthday is or how old I am.

antisthenes(10000) 5 days ago [-]

You're describing a way to make colleagues and acquaintances, not friends.

The more active social life you lead - the less time you have to spend on every individual you meet, hence, it is not enough to become close friends.

altdataseller(10000) 5 days ago [-]

Some of the suggestions you offer don't really increase the chances of making close friends.

Like being an open source maintainer - how is that really going to lead to close friends?

Or starting a business - sure you might network more, and meet business partners, but do they really become close friends?

I will say that going to a church regularly was the #1 thing that really led to lots of friends for me. Like instantly.

camtarn(10000) 5 days ago [-]

This really isn't true. I lived a pretty active life for a decade or so, as a drummer in various drum groups (samba, etc). I was well-liked, people knew my name and seemed to enjoy hanging out with me, and I had some decent conversations. But I didn't really have any people I'd be super comfortable chatting with 1-1, and nobody I'd call up and specifically hang out with.

It was only when the pandemic hit that I realised that, without the regular group socials I was used to, I literally had nobody to chat to. At that point I started doing more or less exactly what this article said, and deliberately trying to make closer friends. I'm happy to say that the strategy worked pretty well.

Perhaps you are naturally good at making deep friendships in these circumstances, but I think a lot of people are not.

sfusato(10000) 5 days ago [-]

Excellent article, but don't make having close friends part of your main goal. Once you are comfortable with who you are, do things on your own, work on new hobbies and interests, then you will attract the right people. Being confident and positive are a people magnet.

kelnos(10000) 5 days ago [-]

> don't make having close friends part of your main goal

Why not? If that's what you want, then, by definition, that is your goal. Sure, you might be falling prey to the XY problem, where your actual problem is that you're feeling emotionally unfulfilled, but erroneously believe that having close friendships (when you currently have few or none) will help solve that problem. But... I kinda think that's likely to be a pretty good solution.

s1artibartfast(10000) 5 days ago [-]

I strongly disagree, or at least don't think that's Universal. There's more to meeting and making friends then simply being yourself. There's taking risks and putting yourself out there. There is making time and priority for them. Maybe this works for someone who is naturally an extrovert and naturally makes friends, but that isn't everyone. Not everyone can or should try to take a wait for them to come to me approach

Xcelerate(10000) 5 days ago [-]

I fortunately have close friends, but not through any of my own doing. Over the years, there were a lot of friendly extroverts that encouraged me to hang out and join them for burger night, or they'd message me when they were in town. Everyone I dated took the initiative to ask me out, including my wife. I always feel awkward when I reach out to people, as though I'm bothering them somehow, so I tend to avoid doing so. Once someone is a close friend, I go out of my way to maintain the connection, but it's that in-between stage of "acquaintance" where I have a hard time.

I had a bit of a revelation when I left my last job. There were very few comments from coworkers when I left. I don't think I was disliked (hopefully?), but I don't think anyone really considered me their friend either. Looking back I think I came across as somewhat of a NPC to coworkers. I preferred to eat lunch by myself and I only discussed the business topic at hand during meetings unless someone else brought up a personal discussion.

I wouldn't mind to start more personal discussions, but I'm always concerned it might come across the wrong way, so the furthest I seem to get is "how was your weekend?"

replwoacause(10000) 5 days ago [-]

I am this exact same way.

-NPC Person

nier(10000) 5 days ago [-]

NPC as in non-playable character?

One of the first search results reads:

«[...] slang used by millenials to denote people who act as if they are being told by someone to act in a specific manner, although that may not be the case. Using this abbreviation, the commentator or the creator is poking fun at the other person for acting in a certain manner.»

bgroat(10000) 5 days ago [-]

I am one of these friendly extroverts.

Something that's hard for me to explain is that 'making friends is so hard it's easy'

What I mean by that is, social anxiety is so high, population-level self-absorption is so high, people are so lonely... that for the most part if you just approach someone and behave as if you're friends... they'll go for it.

There's so much DEMAND for friends, that people will gobble up your supply if you offer it (so long as you're not a total monster)

AntoniusBlock(10000) 5 days ago [-]

If people are going out of their way to approach you (especially women), then it's probably down to you being attractive or at the very least above average.

JumpCrisscross(10000) 5 days ago [-]

> had a bit of a revelation when I left my last job. There were very few comments from coworkers when I left

Can't remember who said this to me, but it stuck: nobody remembers congratulatory calls for a new job. But you never forget each person who reached out when you were fired or let go.

It's fun but ephemeral to celebrate. It hurts to share pain, but it's rewarding. When I find people my gut wants to share burdens with, I listen: they became close friends.

Workaccount2(10000) 5 days ago [-]

> I always feel awkward when I reach out to people, as though I'm bothering them somehow, so I tend to avoid doing so.

Sometime around 30 it hit me like a swinging log to my introverted face that the reason I felt that way was because that's how I felt when people reached out to me.

It really shook my perspective to realize that many people like having someone reaching out to them and they are not annoyed by it.

pessimizer(10000) 5 days ago [-]

Your Friends Have More Friends Than You

https://scribe.citizen4.eu/your-friends-have-more-friends-th...

Math makes this part of the human condition. I honestly think that the solution for the shy (like you and I) is to value the relationships that these extroverts offer. At least a couple of the extroverts will think you're wonderful, and the other people that they will accidentally bring into your life through their gregariousness will cover the range from extreme introvert to extreme extrovert and everything in between. You will be meeting those introverts through a person who they trust, which is your best opportunity for successfully connecting with other shy people.

The only caveats are that you can't focus on one extrovert; get to know more than one, they're happy to talk. Focusing your explorations on one extrovert puts too much burden on the extrovert, and they want to share their time with a lot of people. The second caveat is to not let the extrovert take advantage of you, because a lot of extroverts are con-men/sales-types.

edit: 97% of the good friends I have had in my adult life were the result of a single extrovert talking to me out of the blue in my high school cafeteria. Through her, I met my first real girlfriend, and met two of the members of the band I'd end up singing in and touring with in my 20s. Still friends 30 years later, so I guess she liked me.

Foivos(10000) 5 days ago [-]

> I preferred to eat lunch by myself and I only discussed the business topic at hand during meetings unless someone else brought up a personal discussion.

Probably, they thought that you did not like them.

nicbou(10000) 5 days ago [-]

Someone posted succeedsocially.com in another thread, and it's an excellent resource that might help you. I'm astonished by how on-point it is.

kelnos(10000) 5 days ago [-]

> the furthest I seem to get is "how was your weekend?"

Not a social expert here, but that isn't a bad start! Sure, it's a bit small-talk-y, but it can open up more interesting conversation topics. Assuming the person you're talking to give you more than just 'oh, it was fine', anyway.

You can also modify it to 'did you do anything interesting over the weekend?' And then if they say something specific, you can follow up with deeper questions about what they tell you they did, and go from there.

I get that it's hard, though. I considered myself an 'extroverted introvert' (and even an 'introverted extrovert', on my more socially-energetic days), though pandemic isolation has weakened the extrovert part. I often feel like asking deeper follow-up questions is intrusive and somehow bothersome, but I think the first step is to just make yourself not care, and ask anyway. Listen to how the person responds, and you should be able to get an idea if they do want to talk more about it, or if they'd rather not. If not, you can slow down and disengage. But if they do, then that's a good sign to continue.





Historical Discussions: FTX's collapse was a crime, not an accident (November 30, 2022: 1057 points)
FTX's Collapse Was a Crime, Not an Accident (November 30, 2022: 28 points)

(1061) FTX's collapse was a crime, not an accident

1061 points 3 days ago by mcone in 10000th position

www.coindesk.com | Estimated reading time – 51 minutes | comments | anchor

Privacy policy: https://www.adition.com/en/privacy-platform

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year 213 days 16 hours 53 minutes 20 seconds
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://adacado.com/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: http://adelement.com/privacy-policy.html

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 352 days 12 hours
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://theadex.com/privacy-platform/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 90 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.adition.com/en/privacy-platform

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year 213 days 16 hours 53 minutes 20 seconds
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.adman.gr/privacy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Select personalised ads
  • Measure ad performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Your Consent:
  • Consent expiry: 10 years
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.adobe.com/privacy/policy.html

Purposes (Consent):

  • Store and/or access information on a device
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 180 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.adtiming.com/en/privacypolicy.html

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Your Consent:
  • Consent expiry: 30 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.alliancegravity.com/politiquedeprotectiondesdonneespersonnelles

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
Features:
  • Match and combine offline data sources
  • Link different devices
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.agma-mmc.de/datenschutz

Purposes (Consent):

  • Store and/or access information on a device
  • Apply market research to generate audience insights
Special Purposes:
  • Ensure security, prevent fraud, and debug
Features: Your Consent:
  • Consent expiry: 1 year 1 day
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://aaa.artefact.com/privacy-policy.do

Purposes (Consent):

  • Store and/or access information on a device
  • Measure ad performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://privacy.audienceproject.com

Purposes (Consent):

  • Store and/or access information on a device
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://bam-interactive.de/datenschutz/

Purposes (Consent):

  • Store and/or access information on a device
  • Create a personalised ads profile
  • Measure ad performance
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 day
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://cp.tonefuse.com/privacy-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Select personalised content
Your Consent:
  • Consent expiry: 1 hour
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.bertelsmann.de/meta/datenschutz/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Match and combine offline data sources
  • Link different devices
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://bmind.es/aviso-legal/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
Your Consent:
Privacy policy: https://www.captifytechnologies.com/privacy-notice/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features: Special Features:
  • Actively scan device characteristics for identification
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://connatix.com/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
Features:
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 30 days 2 hours 13 minutes 20 seconds
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://cynapsis.de/impressum/index.html#datenschutz_d

Purposes (Consent):

  • Select basic ads
  • Select personalised ads
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Match and combine offline data sources
Your Consent:
Privacy policy: https://www.cpex.cz/pro-uzivatele/ochrana-soukromi/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 1 year 23 hours 59 minutes 49 seconds
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://emarketingsolutions.es/privacy-policy

Purposes (Consent):

  • Select basic ads
  • Measure ad performance
  • Measure content performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Your Consent:
  • Consent expiry: 30 days
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.clickonometrics.com/optout/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 30 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://e-volution.ai/privacy-policy-3/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 day
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://vdx.tv/privacy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 90 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://global.com/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
  • Actively scan device characteristics for identification
Your Consent:
  • Consent expiry: 1 year 175 days 16 minutes 40 seconds
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://kervit.com/privacy-policy/

Purposes (Consent):

  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features: Your Consent:
Privacy policy: https://havasmediagroup.com/data-protection-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
Features: Your Consent:
Privacy policy: https://www.nonstoppartner.net

Purposes (Consent):

  • Store and/or access information on a device
  • Measure ad performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Your Consent:
  • Consent expiry: 60 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.heimspiel.de/privacy-policy-iab-vendor/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Measure ad performance
  • Measure content performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 2 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.ingenioustechnologies.com/privacy-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Measure ad performance
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.koenigsteiner.digital/datenschutzerklaerung

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Develop and improve products
Special Purposes:
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.dataexchanger.pl/privacy-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Create a personalised ads profile
  • Apply market research to generate audience insights
Your Consent:
  • Consent expiry: 3 years 1 second
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.jwplayer.com/privacy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
  • Actively scan device characteristics for identification
Your Consent:
Privacy policy: https://carbonrmp.com/privacy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Develop and improve products
Special Purposes:
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 89 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.marfeel.com/docs/compass/privacy/policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Technically deliver ads or content
Features: Your Consent:
  • Consent expiry: 180 days
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.mrpfd.com/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Actively scan device characteristics for identification
Your Consent:
  • Consent expiry: 1 year 6 hours 40 minutes
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://mobpro.com/privacy.html

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Measure ad performance
Features:
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
Privacy policy: https://www.nanointeractive.com/privacy-statement/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Your Consent:
  • Consent expiry: 182 days
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://near.com/privacy-policy/#business-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Match and combine offline data sources
  • Link different devices
Special Features:
  • Use precise geolocation data
Your Consent:
Privacy policy: https://nextmillennium.io/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Measure content performance
Special Purposes:
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
Your Consent:
  • Consent expiry: 10 hours
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://s.on-device.com/privacyPolicy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 30 days
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://adalyser.com/en/terms

Purposes (Consent):

  • Store and/or access information on a device
  • Measure ad performance
Your Consent:
  • Consent expiry: 2 years
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.oan.pl/en/privacy-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.brightcom.com/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Measure ad performance
  • Measure content performance
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
Features:
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
Privacy policy: https://optoutadvertising.com/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year 1 day
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://orangeclickmedia.com/privacy-policy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
Your Consent:
  • Consent expiry: 1 hour 50 seconds
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.prequel.tv/terms-and-conditions

Purposes (Consent):

Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
Privacy policy: https://datmean.com/politica-privacidad/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 30 days
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.7pass.de/psdd/datenschutzbestimmungen-cdb.html

Purposes (Consent):

  • Store and/or access information on a device
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Develop and improve products
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 90 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.pulselive.com/privacy-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Actively scan device characteristics for identification
Your Consent:
  • Consent expiry: 30 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://qualitymedianetwork.de/datenschutz/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 90 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.reppublika.com/privacy-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
Features: Special Features:
  • Actively scan device characteristics for identification
Your Consent:
  • Consent expiry: 93 days
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://richaudience.com/privacy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.scoota.com/privacy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Select personalised ads
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Your Consent:
  • Consent expiry: 1 year
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://docs.roku.com/published/userprivacypolicy/en/us

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year 30 days 16 hours 18 minutes 18 seconds
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://schober.de/datenschutz/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Create a personalised content profile
  • Develop and improve products
Features:
  • Match and combine offline data sources
  • Link different devices
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 2 years 170 days 1 hour 40 minutes
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://score-media.de/datenschutzerklaerung/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 90 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.selectmedia.asia/terms-and-privacy/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 90 days
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.snigel.com/privacy-policy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Actively scan device characteristics for identification
Your Consent:
  • Consent expiry: 90 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://adx.space/policy.html

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Your Consent:
  • Consent expiry: 9 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://spicymobile.pl/polityka-prywatnosci

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
Special Purposes:
  • Technically deliver ads or content
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 28 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://cognitivemarketing.tid.es/tad.pdf

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year 355 days
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://cognitivemarketing.tid.es/tde.pdf

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 1 year 355 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://cognitivemarketing.tid.es/tid.pdf

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 90 days
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://cognitivemarketing.tid.es/tme.pdf

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 1 year 355 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.ozoneproject.com/website-privacy-cookie

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Apply market research to generate audience insights
  • Develop and improve products
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 90 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://tvsquared.com/privacy-policy

Purposes (Consent):

  • Measure ad performance
  • Apply market research to generate audience insights
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 2 years
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://tacs.c0nnectthed0ts.com/policy1/data_privacy.html

Purposes (Consent):

  • Store and/or access information on a device
  • Select personalised ads
  • Select personalised content
  • Develop and improve products
Features:
  • Match and combine offline data sources
Special Features:
  • Use precise geolocation data
Your Consent:
Privacy policy: https://www.united-internet-media.de/de/datenschutzhinweis/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://www.usemax.de/?l=privacy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://wemass.com/privacidad/

Purposes (Consent):

  • Store and/or access information on a device
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 3 years
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details
Privacy policy: https://www.yieldlab.de/meta-navigation/datenschutz/

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Measure ad performance
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Match and combine offline data sources
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Special Features:
  • Use precise geolocation data
Your Consent:
  • Consent expiry: 1 year
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies only
  • Show details
Privacy policy: https://corp.fanbyte.com/legal/privacy

Purposes (Consent):

  • Store and/or access information on a device
  • Select basic ads
  • Create a personalised ads profile
  • Select personalised ads
  • Create a personalised content profile
  • Select personalised content
  • Measure ad performance
  • Measure content performance
  • Apply market research to generate audience insights
  • Develop and improve products
Special Purposes:
  • Ensure security, prevent fraud, and debug
  • Technically deliver ads or content
Features:
  • Link different devices
  • Receive and use automatically-sent device characteristics for identification
Your Consent:
  • Consent expiry: 30 days
  • Cookie expiry may be refreshed during the lifetime.
  • Tracking method: Cookies and others
  • Show details



All Comments: [-] | anchor

hinata08(10000) 3 days ago [-]

'a crime'

I'm surprised by 'DeFi' nerds who promoted cryptos to escape government interventions during bull markets, but who invoke state institutions like justice when they have lost everything.

Crypto wasn't dead to me until I saw that post.

I hope governments won't do any investigation into this, as it's what anyone who gave money to FTX invested for.

baby(10000) 3 days ago [-]

FTX isn't DeFi. I think you're conflating centralized entities building on top of crypto (which should be regulated the same way banks are due to multiple historical crisis created by shady banking) and DeFi which are much more transparent in how they operate (by design)

dmix(10000) 3 days ago [-]

Not all crypto people are 'crypto-anarchists'. That is a tiny fringe within a fringe (and FTX was hardly pushing that idea either)

Even among the hardcore anti-government anarchists I doubt many of them saw it as anything but something they wanted in the distant future, not something that exists today.

electic(10000) 3 days ago [-]

FTX isn't DeFi. It is a centralized exchange, similar to a bank or a brokerage. DeFi, ironically, is the complete opposite of this.

lordnacho(10000) 3 days ago [-]

Pretty long article to say some simple things.

Well said, though. It's pretty straightforward they shouldn't have used the customer funds, or tied Alameda so closely to the exchange.

It's still a bit slapstick though. I can see some young kids with no organisation falling into 'oh let's borrow some from the customers, I bet banks do it all the time'. It's still wrong of course but with the whole nerd act you might wonder whether there was any genuine thought about whether it was allowed. Even if you don't know the rules, you have to follow them.

The whole thing beggars belief. How they got that big, while playing video games, having no oversight, with backing from real names. And then blew up seemingly in the space of a few days, but possibly 'mortality wounded' for a long while before.

ilaksh(10000) 3 days ago [-]

How? Because everything operates around the level of delinquent 12-year olds following the latest trends.

Centralized 'crypto' exchanges are antithetical to cryptocurrency. It means NOT using cryptocurrency and giving it to a bank/speculators and then day trading off-chain. The only good reason to keep them around is for the fiat exchange capability. Which could better be served in other ways.

The amazing thing is that most people still don't know what cryptocurrency actually IS, and think this FTX scam and other similar nonsense means that 'crypto isn't trustworthy'. When in reality it is all a perfect example of why the capabilities of actual cryptocurrency (such as public ledgers and smart contracts) are such important advances.

unionpivo(10000) 3 days ago [-]

Sure, that's is what it looks at first, but then if you look closer you see:

- Hiring compliance officer, who was on tape helping his former boss get away with fraud[1] (and that was not recent revelation)

- Using self deleting messages and encouraging others to use them, for important discussions [2].

- Alameda research having backdoor into FTX, just so they could do stuff without rising any red flags [3]

There are other things that point into direction of fraud, but this are the main 3.

[1] https://nypost.com/2022/11/20/ftxs-ex-chief-regulatory-offic...

[2] https://d1e00ek4ebabms.cloudfront.net/production/uploaded-fi...

[3] https://www.wsj.com/articles/alameda-ftx-executives-are-said...

gamblor956(10000) 3 days ago [-]

In today's NYT Dealbook, SBF claimed that 'unspecified third parties' were behind the 'hack' around the time they went bankrupt, that they spent 'thousands of hours' on compliance, that dealing with the CFTC is 'too much work', that they had 'too many boards' overseeing their work (internally), and that customers might be made 'whole' if FTX were recapitalized like Bitfinex was. Also, he claims he took all the money FTX/Alameda loaned him and then used contributed it back to them so he's super duper not sure where all the money is. (This last bit by itself isn't usually illegal; law firm and accounting firm partners do this all the time when they first buy their stakes, but FTX appears to have double- or triple-booked this money: first when received, possibly again when loaned out to SBF, and again when recontributed back to the company.)

He claims he has no hidden funds, he's down to his 'last credit card' and last $100k in his bank account.

I expect many of his statements today to be used against him in court. Assuming he hasn't 'died' by then.

rosnd(10000) 3 days ago [-]

> and that customers might be made 'whole' if FTX were recapitalized like Bitfinex was

The obvious difference being that Bitfinex seems to be run by adults.

adam_arthur(10000) 3 days ago [-]

He cashed out $300M in FTX's last funding round. So where did all that money go?

Waterluvian(10000) 3 days ago [-]

A lot of people using the OTPP to push narratives need to read up on it a little. It's a massively successful pension fund that's out there crushing it for public school teachers (32 year average return of 10%). The amount they lost here is not even a blip. It's just part of the process of investing $240B of assets. You can captain hindsight it all you want but they are completely correct to make some higher risk investments along with the lower risk ones too.

I'm not suggesting that the OTPP is the best pension fund around. Just that there's no story here. They invested a penny in a high risk venture and lost it badly.

lotsofpulp(10000) 3 days ago [-]

How is OTPP's performance (including expenses) relative to a low cost broad market index fund like VOO or a low cost target date retirement fund?

arkis22(10000) 3 days ago [-]

This isn't just comingling accounts and bad accounting. They borrowed between firms. They absolutely knew where liquidity was, and where liquidity wasn't - liquidity follows profits. Their solvency was based on their own manufactured tokens.

They took money from clients because they thought they could pay it back because they think they are smart and they are not smart. Their accounting was good enough to keep track of large loans to insiders though.

totalZero(10000) 2 days ago [-]

Not to mention that his kimchi spread origin story is based on an arbitrage.

How could it be possible for an arbitrageur to not know the difference between entity A and entity B?

P&L up/downs and internal funds transfers are an essential part of running that kind of business. You can't sell something in Asia and simultaneously buy it in the US without thinking carefully about how you finance your trading activity.

babypuncher(10000) 3 days ago [-]

Whatever you want to say it was, I thought it was pretty fun to watch.

baby(10000) 3 days ago [-]

At this scale it still isn't large enough to cause a 2008-type of crisis like banks have done, thankfully

TacticalCoder(10000) 3 days ago [-]

> I thought it was pretty fun to watch

As in: 'Fun to watch people losing money they sent to someone who they thought was honest because he was on the cover of many mainstream publications who were presenting him as an altruistic genius'

?

neilv(10000) 2 days ago [-]

At the same time, Coindesk.com front page includes this:

https://www.coindesk.com/layer2/2022/11/30/a-self-regulatory...

> A Self-Regulatory Organization Is the Best Way to Advance Crypto While Protecting the Public

> The financial industry, collectively, has greater domain expertise than can be expected of regulators. This reality invites transparent, fully accountable, self-regulation.

I'd think anyone could see that it's virtually all scams atop schemes atop scams, and find grounds to shut it all down.

Of course people who would like to run these scams and schemes would like to self-regulate it.

jasonhansel(10000) 2 days ago [-]

Can you imagine anyone proposing this for the ordinary financial sector? The hardcore libertarians would approve, but nobody else would take such a proposal seriously.

nroets(10000) 2 days ago [-]

The regulators didn't catch Madoff.

SBF's greatest punishment will not be jail. It will be that he can't show his face in public for fear of running into someone who lost money.

Noone will want to do business with him anymore because of reputational risk.

Ftx most likely had deposits from some really cleaver hackers with real anger management issues. Any parcel he opens could be a bomb.

Prediction: The next mega entrepreneur in the crypto space will be extremely transparent.

Animats(10000) 2 days ago [-]

Here's the NYT article.[1]

It was an remote interview with Bankman-Fried. The reporter (interviewer?) was David Yaffe-Bellany.[2] He came from Bloomberg.

When, as a reporter for a major paper, do you call a CEO a crook? That's a tough call. Especially when the CEO is in total denial at the time. Especially when such an article will crash what's left of the company. I wonder what discussions took place within the NYT editorial staff.

The interview was on Sunday, Nov. 13. The Binance buyout deal was supposedly happening as of Nov. 8, Binance was making backout noises on Nov 9th, and the deal was dead by Thursday Nov. 10. There was still talk of other deals past that point.

It wasn't clear as of Nov. 14th that there had been out and out theft. The hard info on that came later. Take a close look at who knew what when.

[1] https://www.nytimes.com/2022/11/14/technology/ftx-sam-bankma...

[2] https://yaffebellany.com/

pgwhalen(10000) 2 days ago [-]

> It wasn't clear as of Nov. 14th that there had been out and out theft.

The leaked balance sheet on Nov. 12th was as much clarity as I needed. Plainly, it demonstrated that customer assets were not segregated as promised, but were gambled with.

AmericanChopper(10000) 2 days ago [-]

Why do you think a reporter should be deciding who is and who is not a crook? If they adequately report the facts then the reader can be sufficiently informed to make up their own mind.

bhk(10000) 2 days ago [-]

Seriously?

'Asked whether he was overly dependent on that small group, Mr. Bankman-Fried said his circle of close colleagues numbered about 15.'

What a softball! Perfectly calibrated to his mistakes-not-fraud PR push.

In contrast, this Coindesk article is exactly the kind of in-depth, no-nonsense article focused on the relevant facts that people used to expect from the NY Times

anonymoushn(10000) 2 days ago [-]
SpelingBeeChamp(10000) 2 days ago [-]

> When, as a reporter for a major paper, do you call a CEO a crook? That's a tough call.

No, it's not. The answer is never — unless that person has been charged with and convicted of a crime.

To do otherwise would expose the reporter to enormous potential civil liability. Defamation is a thing.

Prosecutors charge people with crimes, not reporters.

csours(10000) 3 days ago [-]

Bad corporate culture is not always a crime in and of itself, but it does encourage crime. If a company has bad corporate culture, you can find crimes associated to that culture.

I will also say that many people in crypto did not want to look down and see that they were standing on air. This does not excuse FTX or SBF. They had lawyers that told them to implement corporate controls and they ignored that advise.

jeremyjh(10000) 3 days ago [-]

No, this isn't just bad culture. This is FTX saying 'trade on our exchange, and your funds will be held only in the coins they are recorded in. We will not make investments with your funds.' This is the promise and expectation of every brokerage account. It was a lie, and the investment of their customer's funds into their own investment fund is simply theft.

scoofy(10000) 3 days ago [-]

'It was a crime' seems to be the cope du jour that for those who invested in an asset intentionally designed to avoid regulation.

I don't want this to come off as dunking on anyone who has touched by this tragedy. I feel for the people I know involved in this, just as I see these events as part of the inherent risk structure of the space.

Am I wrong here? I really want to know if these cries of foul are substantively different than, say, the required reserves to prevent these types of collapses. Or say, the existence of FDIC insurance in cases of mismanagement or even outright illegal trading by banks?

gary_0(10000) 2 days ago [-]

Crypto is pretty much just people tearing down Chesterton's Fence and then going 'surprised Pikachu' when most of the protections they bypassed turned out to be necessary.

It can also be hard not to blame the victim after they responded to skepticism with shrill political diatribes, although that's not a new trick when it comes to fraud schemes. The guys trying to sell bogus patents for tapwater-powered cars in decades past used similar rhetoric in order to make pawns out of the gullible. 'It's not a scam! That's just what they want you to think!'

TimJRobinson(10000) 2 days ago [-]

The problem is most outside crypto don't understand the difference between centralized crypto (FTX, Celsius, Binance etc) and decentralized crypto (Uniswap, Aave, GMX)

Almost everyone in the crypto industry wants the centralized institutions regulated, because they are opaque companies that could do market manipulation or steal funds at any time.

In stark contrast is DeFi, the second group, who are completely transparent and have no way for the owners to run off with user funds. These ones should not be regulated (scammers should be prosecuted if they code secret backdoors, but they don't need normal bank reporting, kyc, etc)

The distinction is very important.

yt-sdb(10000) 2 days ago [-]

Let's say I start an exchange to trade magic beans. The beans are worthless, but lots of people like trading them. In my terms of service, I say I won't use user funds for anything else and won't rehypothecate user beans if posted as collateral. I claim to make money off fees. Then I do both, lose all the money, and people lose billions of dollars---which they realize when they ask for their money back. You may think magic bean trading is dumb. You may think the people trading them are dumb. But what I did would still be a crime, and calling it a crime doesn't necessarily mean you're delusion about beans.

Put differently, it sounds like you think people who put money into FTX are dumb. Fine. But the law still protects dumb people, as it should.

xiphias2(10000) 2 days ago [-]

I got a loan against my BTC as collateral in the regulated Blockfi company and put it into highly regulated storage at Gemini.

The part I missed is that the contract contained a hidden part where Blockfi can take out the money from the regulated storage and do whatever they want with it without notifying me.

I was paying regular loan repayments and had no idea that they put it in a casino.

When people don't go for known thiefs and mix them with other people, it just gives me more confidence in the important of decentralization.

phist_mcgee(10000) 2 days ago [-]

Cope de jour.

Classic.

lmm(10000) 2 days ago [-]

> I don't want this to come off as dunking on anyone who has touched by this tragedy. I feel for the people I know involved in this, just as I see these events as part of the inherent risk structure of the space.

> Am I wrong here?

I think so. The overwhelming majority of the lost money comes from some combination of a) criminals b) investors in crime-enablement technology c) speculators who thought they could make money off a) and b). It's not much of a tragedy, and dunking on them is warranted.

> I really want to know if these cries of foul are substantively different than, say, the required reserves to prevent these types of collapses. Or say, the existence of FDIC insurance in cases of mismanagement or even outright illegal trading by banks?

The problem with being outside the law is that you're outside the law. Banks do dumb things and lose money all the time, but the rest of us support them because they're fundamentally a part of society; they submit to society's rules, they pay the fines they get charged without complaint (even when those fines are often blatantly unfair), they generally follow the law as best as they can and go through the proper processes when they want to get the law changed (which is not to say they don't lobby for their own interests - they absolutely do - but they play by the rules).

harlequinn77(10000) 2 days ago [-]

It looks like a pretty clear cut case of embezzlement and fraud.

They're both crimes.

jasonhansel(10000) 2 days ago [-]

Just based on the usage patterns of crypto as a whole, it's likely that a fair number of FTX's users were involved in human trafficking, opioid dealing, and/or money laundering. I'm not sure how big of a percentage that was of FTX's user base—maybe it was too trivial to matter—but it does make me less sympathetic.

djbebs(10000) 2 days ago [-]

Theft is a crime regardless of what is being stolen, so is fraud.

No need for regulation to deal with this.

hinata08(10000) 2 days ago [-]

that's exactly what I commented, and the comment was downvoted, contradicted, flagged, and has vanished (really, deleting comments ? is that what this site is about ? I thought is was a site for the open minded ) (edit : it was folded, not deleted. but still)

I share the same perspective, but I'm surprised by everything that is happening around.

Users on Tweeter and some crypto websites are speculating and investigating on how ftx got a banking license (shouldn't they have done it in the first place, before they invested ?)

They're also defiant to the media (as in the intro of the featured article)

And posts on HN about the very nature of the crypto industry are hot.

I find all of that pretty amateurish yet still convinced, after seeing both the wave of tech guys who believed in cryptos for what you said, then of more or less proud 'crypto bros' who just saw a financial opportunity, with hardly any financial, tech, or political knowledge. The former didn't see the latter come in and ruin their usage of the crypto, and the latter ignored the warnings from specialized journalists

What is happening right now is very interesting to watch. I've spent two years to get a better paid job before I could have more money to invest, so I'll watch around to learn.

I've already seen basically the same, on documentaries about Wirecard.

The first lesson I'll take will be about being humble, being able to listen outside of bubbles, and to take criticism without flaming other commenters. The second one will be that collapses can arrive very fast. The poor investors lost everything in a snap. It looks pretty bad

shp0ngle(10000) 2 days ago [-]

The way mainstream is covering SBF is downright bizarre and it starts to confirm conspiracy theories to many people.

indigodaddy(10000) 3 days ago [-]

Watching the NYT SBF interview tonight, in which SBF is constantly characterizing billions of customer dollars being funneled illegally to Alameda as, "we didn't realize that Alameda's 'position' in FTX was so large..."

This is just straight up nonsense correct? Or does he have something to grab hold to here?

55555(10000) 3 days ago [-]

The basic idea is this: If he just sent customer funds from FTX to Alameda, that would definitely be fraud/theft as the FTX TOS said they wouldn't do that. But instead he did something else. You know how you can take a margin loan in your brokerage account against the value of your stocks? You can use the money from the loan as you wish, but if your stocks go down X% then they will 'margin call' you and liquidate your assets to pay back the loan. SBF's claim is that Alameda took a margin loan as an FTX customer just like any other customer might, and then lost their money gambling. But unlike other customers, Alameda was not subject to any margin requirements (??) and could not be automatically liquidated (??). These facts(?) probably ruin this defense, which otherwise might hold some water, as it's not at all a defensibly arms length transaction. IANAL.

So they took a margin loan using all their FTT as collateral, then lost all the money. Normally you wouldn't be able to lose more than you've left as collateral, but they weren't like other customers (see above). But there's also the fact that the collateral was semi-BS, so FTX can't sell it to get the money back anyway. FTX printed the FTT out of thin air and then sent it to Alameda (surely for some purported reason). Alameda then used this semi-fake collateral to loot all FTX's customer deposits. (It was fake in the sense that you can't sell all of it at the mark-to-market/last-match price). You can see how FTX is involved at every step of the way here, and is at the root of the scheme, not merely a criminally negligent player.

But honestly it just keeps going deeper and deeper, as it sounds a lot like funds were missing prior to any of this happening, and that even if they could pay all this money back, there wouldn't be enough money, and that the margin loan wasn't just used for investing but also frivolous spending. Don't try to make sense of SBF's actions, they don't make much sense (Why go on a shopping spree that worsen's your already-insolvent exchange? Why tie up withdraw-at-anytime depositor's funds into 10-year-time-horizon VC funds? IDK, why talk so much publicly against your lawyer's wishes and your own best interest?)

m3kw9(10000) 2 days ago [-]

Does claiming ignorance help him at all?

wnkrshm(10000) 2 days ago [-]

It's a common human experience: Normalization of deviance. [0]

You take a (what you think is) calculated risk, nothing bad happens immediately and it worked out - which reinforces the belief that your risk estimate was correct and not just pure luck.

Then you take the same sort of risk all the time and it becomes your new normal - you take another risk (which is now a larger step, since the previous risk-taking is just business as usual) and so on until the probability of failure is 1.

Obviously this can easily happen to people who have no advisor they respect more than their own opinion.

[0] https://en.wikipedia.org/wiki/Normalization_of_deviance

indigodaddy(10000) 2 days ago [-]

Is the outward exterior of earnestness and sincerity (I have to admit I'm almost falling for it) a facade? It seems like he actually believes everything he is saying and there is an absence of malice in how he projects in the interview. He's either a psychopath who is faking (very well) this earnest persona, or he actually is fully deluded by his own fantasy creation/dollhouse that is FTX.

andirk(10000) 3 days ago [-]

He gambled. Lost. Gambled more. Lost bad 'even when times were good'. My guess is he figured if he could borrow from Peter to pay Paul one more time... one more time that he would eventually be back in the black. I think he personally set up those 'positions' over and over, but he's not good at gambling so now it's all gone.

https://www.nytimes.com/live/2022/11/30/business/sam-bankman...

chewbaxxa(10000) 3 days ago [-]

I spoke to a few Alameda/FTX people over drinks before summer. They were delighted to brag about the lack of separation between the two entities. Really there wasn't any.

1vuio0pswjnm7(10000) 2 days ago [-]

'In interviews, he talked a stream of nonsense tailored to snowjob outsiders about an industry that's already full of jargon and complicated tech.'

He is still talking. For example,

16 Nov 2022

https://www.youtube.com/watch?v=6DezodR9hNI

(Sounds like he is in the bathtub.)

20 Nov 2022

[another to be published phone call with the YouTuber above]

30 Nov 2022

https://www.nytimes.com/live/2022/11/30/business/sam-bankman...

https://slate.com/technology/2022/11/sam-bankman-fried-new-y...

Internet discussion seems to center on criminal charges but there will be numerous civil suits first. Poetic justice would be if he has to ask his parents for money. Every time he opens his mouth, plaintiffs lawyers are listening.

In the 16 Nov interview he reveals that he told his lawyers to 'go fuck themselves'. He said 'I don't think they know what they are talking about' and 'They only know what they're talking about in the extremely narrow domain of litigation.'

Maybe he should be put on a suicide watch if he isn't already.

If I understand correctly, he will be on The Crypto Roundtable next.

https://podcasts.apple.com/us/podcast/the-crypto-roundtable-...

He is booked to appear on Good Morning America next week.

How many times can someone say 'like' in 20 minutes. Listen to an SBF interview and find out.

maxbond(10000) 2 days ago [-]

> [another to be published phone call with the YouTuber above]

https://www.youtube.com/watch?v=xP54LZB3WRw

skybrian(10000) 3 days ago [-]

The New York Times did another interview with Bankman-Fried today, very much against the advice of his lawyers. I didn't watch it, but from a tweet summary [1], it sounds rather amazingly self-destructive.

Getting the lies on the record seems useful?

[1] https://twitter.com/edmundlee/status/1597957472761057282

earthscienceman(10000) 3 days ago [-]

Why did I just get linked to a tweet of a news article?

https://www.nytimes.com/live/2022/11/30/business/sam-bankman...

oldgradstudent(10000) 3 days ago [-]

> very much against the advice of his lawyers

According to Sam Bankman-Fried. Not the most reliable source.

chrchang523(10000) 3 days ago [-]

We'll see. From my perspective, this could still go either way: he could be put in prison for a long time partly as a consequence of how much is on the record, or he may still be protected regardless of what is on the record.

ratsmack(10000) 3 days ago [-]

He doesn't sound like the genius kid that the media had made him out to be. He sounds more like a pawn in some scheme where he was chosen to take the fall.

DevKoala(10000) 2 days ago [-]

This comment section is hilarious. Some don't want to acknowledge the obvious, and instead try to tip toe around the subject with funny excuses such as, "MSM are not crypto experts", "his 40M in donations to democrats do not hold any value anymore", "his family ties to Democrat and academic leadership only makes his inner circle judgement more impartial", etc.

There are none so blind as those who will not see.

seabird(10000) 2 days ago [-]

Yeah, it's pretty wild watching people wonder why a top political donator is getting so much peel for somebody who is either a crook or dangerously stupid. There's a lot of people who like to imagine that media and journalism is a lot more impartial than it really is.

mdotk(10000) 3 days ago [-]

Finally, some reporting by media that hasn't been bribed.

readthenotes1(10000) 3 days ago [-]

I would call CoinDesk 'the media'.

They used to be a competitor and are existentially scared that people will believe that crypto is gambling in a crooked house.

bandyaboot(10000) 2 days ago [-]

Finally, media saying things that I like.

puffoflogic(10000) 3 days ago [-]

Uh, they're not called 'bribes', they're called 'campaign donations'.

bandyaboot(10000) 3 days ago [-]

I have to admit, I'm really having a hard time caring about whatever drama is happening in the pretend money world.

anonporridge(10000) 2 days ago [-]

People sent him 'real money' (all money is pretend) which he then funneled to other projects, specifically massive political contributions to influence US elections.

That should absolutely concern you even if you think all cryptocurrency and bitcoin is fake and doomed to eventual collapse.

richbell(10000) 3 days ago [-]

CoffeeZilla covered this in a recent video. The effort to paint over such blatant fraud and negligence and pretend it was all a mistake from a well-meaning kiddo is disgusting.

https://youtu.be/0rL35_WV3lE

classified(10000) 2 days ago [-]

SBF looks like a pathological liar who had gotten away with it all his life. He must be so disappointed.

Alex3917(10000) 3 days ago [-]

> The effort to paint over such blatant fraud and negligence and pretend it was all a mistake from a well-meaning kiddo is disgusting

Let's not forget that the folks using FTX were only doing so in an attempt to steal money from others. If the key innovation of crypto is that it's supposed to reduce the risk of lending by bringing realtime transparency to the assets and liabilities of borrowers, then why was FTX paying vastly higher interest rates than traditional banks? It's hard to feel sorry for those who lost their money because they thought they were in on the con or whatever.

You can hear the folks who lost money saying this themselves, e.g.: https://www.youtube.com/watch?v=9cATvKsZA0A

And to those who think I'm being uncharitable, I've been saying the same thing on HN since long before the FTX collapse, e.g.: https://news.ycombinator.com/item?id=28457273

iancmceachern(10000) 2 days ago [-]

CoffeZilla has been killing it on this. Youtube creators and similar have really taken over from traditional media.

quickthrower2(10000) 2 days ago [-]

Even if we assume the best: meaning kiddo, zero sympathy. Because it is negligent. (Unless he is a child under 16 which he isn't)

foobarbecue(10000) 2 days ago [-]

Aw, but his hair is so messy, he can't be that bad!

Honestly I wonder how intentional this Boris Johnson tactic of engendering sympathy through unkempt hair is. It seems to work. Maybe it reminds parents of their helpless teenagers or something.

deaddodo(10000) 3 days ago [-]

Yup, 100% this. My own feelings on crypto notwithstanding (the whole thing is a fraud/built on hype); the crypto community seems to have pretty much agreed on this being a case of straight fraud. It's only the tangential and traditional financial community (and some journalists) who are painting this as a "well-meaning company/crypto genius who made a mistake".

TacticalCoder(10000) 3 days ago [-]

> The effort to paint over such blatant fraud and negligence and pretend it was all a mistake from a well-meaning kiddo is disgusting.

I agree and this thread is honestly plain disgusting too. Many here are happy and find it funny to see people's crypto being stolen by fraudsters.

From TFA:

'It is now clear that what happened at the FTX crypto exchange and the hedge fund Alameda Research involved a variety of conscious and intentional fraud intended to steal money from both users and investors.'

And from the bankruptcy filings:

'On-chain analysis has found the bulk of movements from FTX to Alameda took place in late 2021, and bankruptcy filings have revealed that FTX and Alameda lost $3.7 billion in 2021. This is maybe the most befuddling part of the Bankman-Fried story: His companies lost massive amounts of money before the 2022 crypto bear market even started. They may have been stealing funds long before the blowups of Terra and Three Arrows Capital'.

People may hate on Bitcoin, Ethereum, ape monkeys yacht club etc. (or whatever these NFTs are called) as much as they want, it's not reason to rejoice when thieves do scam people (including a canadian teachers pension funds' money).

There's also another angle: the very same who are cheering on people losing all their money to a scam were the first to say: 'Pay your taxes on any crypto gain you made' (so that, in the end, they get their share of these winnings too through the state's spendings)...

If the state collects taxes on gain made on crypto, the state go after people stealing other people's crypto.

As simple as that.

P.S: and I hope people are happy that their politicians got greased by stolen funds and aren't giving these bribes (pardon, donations) back.

nostromo(10000) 3 days ago [-]

I can't figure out why the media is doing such a terrible job covering this (other than Matt Levine, his coverage has been flawless).

Is it because SBF gave a bunch of money to politicians? Is it because FTX got investment dollars from prominent VCs? Is it because journalists do not understand crypto, like, at all? Is it because he bamboozled journalists by talking so much about Effective Altruism?

Nobody, literally nobody, said Bernie Madoff was 'suffering from a solvency crisis.' They said he lied and stole money. And yet SBF's alleged crimes are quite similar in nature, but he's portrayed as suffering from market conditions beyond his control. So was Madoff. If Madoff got lucky and his portfolio doubled in a year, then he too would have been able to make customers whole. But even if that happened, he still lied and stole, just like SBF.

jiveturkey(10000) 2 days ago [-]

I call it PR done to perfection.

lajosbacs(10000) 2 days ago [-]

I don't get this obsession with mainstream 'the man' controlling the narrative. For example the FT slams him so much it's boring. They even have a (great) data driven article about how he sucked at LoL.

oldgradstudent(10000) 3 days ago [-]

Of course it was an accident.

They never intended the fraud to be found out, just like any fraudster and Ponzi scheme operator in history.

FireBeyond(10000) 3 days ago [-]

My girlfriend is working on her CPA and is fascinated with forensic accounting to the point where she thinks I should do it, because of the little things I notice.

In an interview the other week, SBF was asked what he might do differently given the chance.

'More careful accounting'.

Girlfriend: 'Huh. Just a casual phrase. But not 'more accurate/better' accounting, but 'more careful'.'

paulpauper(10000) 3 days ago [-]

It took almost 3 years to indict Enron executives after its bankruptcy. This could be a while.

LatteLazy(10000) 3 days ago [-]

Enron was a much more complex, longer, more wide ranging fraud though. This was just straight up 'CEO moves client money to his other company' theft. It's almost staggeringly simple and mundane...

yawnxyz(10000) 3 days ago [-]

For a fraudulent app, I'm surprised at how well FTX as a platform worked.

I only played around with it for a while, buying/selling $250 worth of coins to try it out. I thought it was fairly well-designed, fast, etc. and on par with other tools like Schwab or Robinhood.

To hear that the entire thing was a fraud makes me think that they put a lot of engineering/design effort into making the FTX platform work. It was a fully-functioning product, not some Theranos-like facade!

rootusrootus(10000) 2 days ago [-]

That makes sense, though, the app wasn't fraudulent, just what they were doing on the back end with customer funds. It's of course in their best interests to hire good enough software talent to build an elegant platform that will inspire confidence and attract more 'investors.'

pessimizer(10000) 3 days ago [-]

It was a 'crime', but we treat the financial industry and anybody with money with such kid gloves that it doesn't really matter that it was a crime. I have no confidence that his prosecution won't end up as difficult as Rob Blagojevich's (notoriously corrupt ex-governor of Illinois), in that Illinois ethics law has been so carefully crafted to allow bribery to and from officials that it was surprisingly difficult to convict him, and irt the details of the process probably unfair. Blagojevich's actual crime was not having being popular with people who had clout, not his open selling of a US Senate seat.

This dude, however, is attending speaking engagements with the NYT and sharing the stage with war criminals after his obvious fraud in a shady-ass industry. He's probably still got plenty of friends, and his co-speakers are an example of what impunity and getting off scot-free when everyone knows you're guilty might look like for him.

StanislavPetrov(10000) 3 days ago [-]

Reminds me very much of the John Corzine situation. If you are connected with the right set of politically influential and wealthy people, you are above the law.

jandrese(10000) 3 days ago [-]

Eh, he stole from rich people. He's going to jail. It's not like he only ripped off a bunch of retail investors.

asimpletune(10000) 2 days ago [-]

So I've been watching the NYT dealbook interview and want to add some clarity that I think is missing here. Let me add a disclaimer that I don't condone SBF at all. I just want to use his words to provide clarity. Everything below is paraphrasing.

Q: Did you co-mingle funds?

A: I did not mean to.

Q: It says in the TOS that you would not do that. How is it ok then?

A: It says elsewhere in the TOS that customers can take margin positions, so in some cases the co-mingling was to be expected.

Q: How did then such a large amount of money get co-mingled then, because it seems that a much larger amount of customer funds were affected than just the accounts trading on margin?

A: In the early days, we did not have a bank account, so when customers wanted to trade on our platform, they deposited money to alameda who then credited the money to us. I don't know for sure, but it seems that the money that was credited wasn't always put in the right account (ie safe money, per the TOS)

So as I've understood so far, it seems SBF is claiming there was just incompetence and mismanagement. A lot of regulation in finance is to prevent people from being able to do this. Anyways, to those who have said, "stealing money that I traded for magic beans is still stealing", I think it's a little more complicated than that.

pjc50(10000) 2 days ago [-]

Margin trading is still not supposed to be co-mingled (or allowed if you're running a regulated brokerage). There's supposed to be a separate bank account labelled 'client money' which has the client money in. https://www.handbook.fca.org.uk/handbook/CASS/5/5.html

You could argue that anyone sending their money to a company of a different name to the one they expected, in an offshore jurisdiction, 'deserved' what they got, but I don't agree with that. SBF is clearly trying to claim incompetence rather than criminal intent, but that should not save him from a gazillion wire fraud charges.

NhanH(10000) 2 days ago [-]

Q: Did you kill those people?

A: I did not mean to. I just loaded the gun, closed my eyes, pointed to the crowd and pulled the trigger.

If you are a trading exchange without a bank account, something is seriously wrong and you don't have an exchange. Choosing to take on customer money in any capacity is just straightforward trying to do something shady.

Yeah it is a little more complicated than normal to do illegal bullshit and fleece retail investors, that's the whole point of regulations and is not really an explanation for anything.

mrobins(10000) 2 days ago [-]

To quote the article:

> Bankman-Fried has decided to "confuse, evade, distort."

Even if you believe he's smart enough to build a massive exchange that takes in billions of dollars of customer money but is too incompetent to institute the most basic safeguards that seems fraudulent (and criminally negligent) to me.

Also, all the 'incompetence' seemed to benefit him and FTX senior leadership. Funny how things shake out like that sometimes...

nigerian1981(10000) 2 days ago [-]

How does incompetence and mismanagement explain SBF, his parents and senior executives buying properties worth nearly $300M in the Bahamas with FTX funds as has been reported by multiple news outlets?

system16(10000) 2 days ago [-]

Playing dumb and framing himself as just a kid who was out of his element appears to be his team's strategy to try and minimize his personal consequences in all of this. He is a highly educated individual that was literally born on Stanford's campus to professors. I'm not sure many will buy his Steve Urkel 'Did I do that??' defence.

quyleanh(10000) 3 days ago [-]

A bit off topic, but I hope this thread will not be flagged as mine days ago.

Negative discuss about FTX is seems sensitive... I still don't know why many people still defense for FTX and SBF. Thousands of people lost all their money.

eldenwrong(10000) 3 days ago [-]

He belongs to a very protected group of people who happen to own all the boats and medical outlets

lettergram(10000) 3 days ago [-]

The dude took our $1B in a personal loan with customer wealth. He then used it to donate to random politicians, invest in effectively crazy one off ideas, etc.

hn_throwaway_99(10000) 3 days ago [-]

> The dude took our $1B in a personal loan with customer wealth.

Let's correct that to say 'stole $1B in customer wealth'. You can't 'take out a loan' from someone who doesn't agree to loan it to you (and, indeed, FTX expressly said they wouldn't borrow customer funds).

themagician(10000) 3 days ago [-]

What amazes me about FTX is the speed and severity of the collapse. Never seen anything like it. The website and app simply disappeared overnight. The never sent statements via email or postal mail. If you had any money in the only proof you have some transactional emails and lines in the statements from the banks which you used to fund your FTX account.

It really is remarkable. I would think the hardest part about getting indictments at this point is collecting enough information to actually present to... anyone.

If you don't have physical or digital copies of statements, transactions, and account numbers and the company you are dealing with just comes up with some bogus, 'We were hacked, it's all gone, we are now bankrupt, goodbye,' and then proceeds to delete their website and disconnect their phone how do you prove that you even have a claim?

jasonhansel(10000) 2 days ago [-]

I suspect that this is surprising in part because we have so little firsthand experience with bank runs. Before the FDIC, a bank could collapse fairly quickly and with relatively little warning.

muskmusk(10000) 2 days ago [-]

I cant hold it in: Yeah it would be really convenient if there was some sort of global database that could hold all this data.

ejlangev(10000) 3 days ago [-]

This article, while probably true an accurate, seems designed to make SBF the fall guy by portraying him as a uniquely bad actor in the crypto space. In reality, the crypto space is full of fraud that is probably criminal and he is just the latest example. That doesn't excuse his behavior and he should probably go to prison for this but it would be a shame if people get taken in by thinking of him as just a bad apple.

hn_throwaway_99(10000) 3 days ago [-]

He is not the 'fall guy' considering he, along with Alameda's CEO, make the direct, conscious decision to embezzle customer funds to prop up Alameda. Just because he stole what looks to be a billion dollars doesn't mean there are no other bad actors.

I think this article is great. I've commented on this for a while, but I still can't fathom why most of the media isn't highlighting that this was plain old, straightforward theft. There is no gray area here, the lack of crypto regulations are pretty irrelevant, and I'm glad a journalist is saying clearly that this was just fraud and theft.

bhk(10000) 3 days ago [-]

I totally missed the part where they say that there have been no other criminals in crypto. Could you point me to that?

Or are you just thinking that the PR narrative of well-intentioned-but-made-some-mistakes would be better at protecting people from future scams?

dmix(10000) 3 days ago [-]

I watched the full NYT interview today and he's either an amazing well-coached liar (very possible) and or he really is a kid in way over his head, who got to take advantage of reputation circles (VCs, media, politicians, etc) to make reckless massive gambles without any adult risk-adverse old-school finance guys in the room.

Basically demonstrating zero personal responsibility for the the money he had control over.

Maybe more will come out to show he's is in fact a modern Bernie Madoff, with a grand hidden scheme at work to directly enrich himself, but that's not 100% obvious yet. I'm looking forward to what comes out in the future.

notch656a(10000) 3 days ago [-]

MIT grad, ex Jane Street trader, comes from lawyer-professor family, garnered VC capital from big names like Kevin O'Leary and lead the pitches.

Dude knew what he was doing and if he didn't it was done out of such criminal negligence that he's still to blame.

jstx1(10000) 3 days ago [-]

Not segregating customer funds isn't a gamble.

hula_lula(10000) 3 days ago [-]

Its easy to look like a sheep when you have not internalised any sense of responsibility. It was literally a game to him

fatneckbeardz(10000) 3 days ago [-]

the proof would be to find how many people he fired and blackballed because they disagreed with him or told him he was wrong.

hnews_account_1(10000) 3 days ago [-]

I'm a year older than him and in finance without an Ivy League education or any pedigree and I can swear to you that innocence act fools no one except people outside the field. This is like a programmer saying he drowned the company because he didn't realize how much he was spending on cloud costs as he left the auto scaler on. It is the stupidest excuse, except the autoscaler was running on infra the programmer created and owned so he just took all the money. Which is just basic fraud.

It is such basic fraud that it happened even in 2008 and it'll keep happening. Cherub faced little fucks will evolve to fool you out of your money until the end of all time. Don't fall for it.

p1necone(10000) 3 days ago [-]

'I was too ignorant to run a giant investment company/pseudo-bank correctly' shouldn't get you off though. He needs to face the same charges regardless of whether he was malicious or stupid.

NelsonMinar(10000) 3 days ago [-]

Note this op/ed is published by CoinDesk, the cryptocurrency journalism outlet that broke the story about FTX's financial situation that precipitated their collapse. Different authors but same publication. https://www.cnn.com/2022/11/24/media/coindesk-scoop-ftx-reli...

valar_m(10000) 3 days ago [-]

What's your point?

inasio(10000) 3 days ago [-]

Probably not the angle parent was going for, but CoinDesk is owned by Digital Currency Group, who had significant exposure to this mess (and their Genesis trading arm appears to be currently evaluating bankruptcy), so in some sense CoinDesk precipitating FTX's collapse had severe effects on their own situation.

woodpanel(10000) 2 days ago [-]

Great article. It even goes there and mentions BCCI.

And with this four-letter acronym alone the clandestine literate [1] might be able to draw his/her own conclusions about the how, the why and the kid gloves with which the media handles SBF.

Imagine a three letter state agency looking for ways to secretly pay off shady foreign allies from terrorists to cartels. What kind of infrastructure would you need to do that on a geopolitical scale? Sigh, if there only existed such a technology irony off!

No, that doesn't mean SBF is crypto's James Bond, but it does shed an intriguing light on certain stablecoins as well [2, 3]

[1] https://www.washingtonpost.com/archive/opinions/1991/08/11/t...

[2]: https://asia.nikkei.com/Spotlight/Datawatch/Tether-cryptocur...

[3]: https://www.ledgerinsights.com/tether-shadow-ftx-saga/

jessaustin(10000) 2 days ago [-]

Not so much a 'crime' as an 'op'. Like various other distractions to which we're supposed to devote attention, while the actual crimes take place elsewhere.

barathr(10000) 3 days ago [-]

This seems a bit like a 'doth protest too much' article -- while the things FTX did are no doubt shady, this article seems to be trying to heap scorn on FTX as if it is unique in the cryptocurrency space. (And there are factions within that world -- is FTX in a different faction than Coindesk? I'm not sure.)

It seems quite possible that a couple of years from now we'll find out that most major cryptocurrency companies were doing things similar to FTX. With that possibility, this article could be an attempt to avoid FTX bringing down other companies in the space.

valar_m(10000) 3 days ago [-]

Well, FTX is unique in the crypto space in that there has never been a loss of funds anywhere near the magnitude of this debacle.

Odd to me that anyone would look at a company that managed to lose billions of customer and investor dollars under extremely suspicious circumstances, and then react dismissively to credible allegations of fraud by framing it as an attempt to 'heap scorn'.

crtified(10000) 3 days ago [-]

No doubt it was both, to some degree. The latter doesn't excuse the former.

And the whole matter is only made worse by (or, perhaps, 'highlights') the surrounding incestuous cesspool of other crypto exchanges and funds, and the 'crypto media' arms which they own and run. Several of whom are almost certainly engaging in, at the least, grey-area practices, in order to accrue vast and lightly regulated profits.

The fact that many of crypto's best-known names are basically digital bankers-for-profit, illustrates how the scene went off the rails some time ago. From a tech perspective.

xmodem(10000) 3 days ago [-]

If you're building a system that's certain to fail once it comes under any sort of stress, when the inevitable happens, that's hardly an accident.

djbebs(10000) 3 days ago [-]

If FTX Terms of Service state that customer assets belong to the customer--not to FTX and those assets are used by FTX for any purpose that is a crime. Full stop.

gjvc(10000) 3 days ago [-]

Why are you defending fraud?

jmull(10000) 3 days ago [-]

I think the crypto people keep driving the NYT-is-in-bed-with-SBF narrative to distract from the real issue: the great bulk of crypto efforts are essentially scams that want to lure real money in with the promise of riches.

They are all feeling the crunch now and want to turn the narrative away from ones that will cause people to stop putting real money in.

loeg(10000) 3 days ago [-]

I assure you, non-crypto people are baffled as to why big media organizations have been lobbing softballs to SBF, too.

RustLove(10000) 2 days ago [-]

None of Theranos enablers seemed to suffer at all after Theranos went down. Why do any of their board members still have careers? Why is Stanford's Channing Robertson still advising companies?

I presume the dozens of people, news organizations, celebrity endorsers, politicians, and investment gurus who hyped FTX won't suffer a bit either. And I think that's wrong.

college_physics(10000) 2 days ago [-]

your problem is that you are a minority (in a society where those with the right information and ability to act on it actually think its a-ok). It is also called moral bankruptcy and is closely associated with financial bankruptcy.





Historical Discussions: Tales of the M1 GPU (November 29, 2022: 1010 points)

(1012) Tales of the M1 GPU

1012 points 4 days ago by rawrenstein in 10000th position

asahilinux.org | Estimated reading time – 22 minutes | comments | anchor

Hello everyone, Asahi Lina here!✨

marcan asked me to write an article about the M1 GPU, so here we are~! It's been a long road over the past few months and there's a lot to cover, so I hope you enjoy it!

What's a GPU?

You probably know what a GPU is, but do you know how they work under the hood? Let's take a look! Almost all modern GPUs have the same main components:

  • A bunch of shader cores, which process triangles (vertex data) and pixels (fragment data) by running user-defined programs. These use different custom instruction sets for every GPU!
  • Rasterization units, texture samplers, render output units, and other bits which work together with the shaders to turn the triangles from the app into pixels on the screen. Exactly how this works varies from GPU to GPU!
  • A command processor that takes drawing commands from the app and sets up the shader cores to process them. This includes data like what list of triangles to draw, what global attributes to use, what textures will be used, what shader programs to use, and where the final image goes in memory. It then sends this data over to the shader cores and other units, to program the GPU to actually do the rendering.
  • A memory management unit (MMU), which is in charge of limiting access to memory areas belonging to a specific app using the GPU, so different apps can't crash or interfere with each other.

(This is all very simplified and in reality there are a lot more parts that vary from GPU to GPU, but those are the most important bits!)

In order to handle all these moving parts in a reasonably safe way, modern GPU drivers are split into two parts: a user space driver and a kernel driver. The user space part is in charge of compiling shader programs and translating API calls (like OpenGL or Vulkan) into the specific command lists that the command processor will use to render the scene. Meanwhile, the kernel part is in charge of managing the MMU and handling memory allocation/deallocation from different apps, as well as deciding how and when to send their commands to the command processor. All modern GPU drivers work this way, on all major OSes!

Between the user space driver and the kernel driver, there is some kind of custom API that is customized for each GPU family. These APIs are usually different for every driver! In Linux we call that the UAPI, but every OS has something similar. This UAPI is what lets the user space part ask the kernel to allocate/deallocate memory and submit command lists to the GPU.

That means that in order to make the M1 GPU work with Asahi Linux, we need two bits: a kernel driver and a user space driver! 🚀

Alyssa joins the project

All the way back in 2021 when Asahi Linux started, Alyssa Rosenzweig joined the project to start working on reverse engineering the M1 GPU. Together with Dougall Johnson (who focused on documenting the GPU shader architecture), she started reverse engineering all the user space bits, including the shaders and all the command list structures needed to set up rendering. That's a ton of work, but less than one month in she was already drawing her first triangle! She's amazing! If you haven't checked out her series on dissecting the M1 GPU you should visit her website and take a look! ✨✨

But wait, how can she work on the user space driver without a kernel driver to go with it? Easy, she did it on macOS! Alyssa reverse engineered the macOS GPU driver UAPI enough to allocate memory and submit her own commands to the GPU, and this way she could work on the user space part without having to worry about the kernel bit. That's super cool! She started writing an M1 GPU OpenGL driver for Mesa, the Linux userspace graphics stack, and just a few months later she was already passing 75% of the OpenGL ES 2 conformance tests, all on macOS!

Earlier this year, her work was so far ahead that she was running games on a fully open source Mesa OpenGL stack, running on top of Apple's kernel driver on macOS! But there was still no Linux kernel driver... time to help out with that part! ✨

The Mysterious GPU Firmware

In April this year, I decided to start trying to figure out how to write an M1 GPU kernel driver! Scott Mansell had already done a bit of reconnaisance work on that front when I got started... and it was already clear this was no ordinary GPU. Over the first couple of months, I worked on writing and improving a m1n1 hypervisor tracer for the GPU, and what I found was very, very unusual in the GPU world.

Normally, the GPU driver is responsible for details such as scheduling and prioritizing work on the GPU, and preempting jobs when they take too long to run to allow apps to use the GPU fairly. Sometimes the driver takes care of power management, and sometimes that is done by dedicated firmware running on a power management coprocessor. And sometimes there is other firmware taking care of some details of command processing, but it's mostly invisible to the kernel driver. In the end, especially for simpler "mobile-style" GPUs like ARM Mali, the actual hardware interface for getting the GPU to render something is usually pretty simple: There's the MMU, which works like a standard CPU MMU or IOMMU, and then the command processor usually takes pointers to userspace command buffers directly, in some kind of registers or ring buffer. So the kernel driver doesn't really need to do much other than manage the memory and schedule work on the GPU, and the Linux kernel DRM (Direct Rendering Manager) subsystem already provides a ton of helpers to make writing drivers easy! There are some tricky bits like preemption, but those are not critical to get the GPU working in a brand new driver. But the M1 GPU is different...

Just like other parts of the M1 chip, the GPU has a coprocessor called an "ASC" that runs Apple firmware and manages the GPU. This coprocessor is a full ARM64 CPU running an Apple-proprietary real-time OS called RTKit... and it is in charge of everything! It handles power management, command scheduling and preemption, fault recovery, and even performance counters, statistics, and things like temperature measurement! In fact, the macOS kernel driver doesn't communicate with the GPU hardware at all. All communication with the GPU happens via the firmware, using data structures in shared memory to tell it what to do. And there are a lot of those structures...

  • Initialization data, used to configure power management settings in the firmware and other GPU global configuration data, including colour space conversion tables for some reason?! These data structures have almost 1000 fields, and we haven't even figured them all out yet!
  • Submission pipes, which are ring buffers used to queue work on the GPU.
  • Device control messages, which are used to control global GPU operations.
  • Event messages, which the firmware sends back to the driver when something happens (like a command completing or failing).
  • Statistics, firmware logs, and tracing messages used for GPU status information and debugging.
  • Command queues, which represent a single app's list of pending GPU work
  • Buffer information, statistics, and page list structures, used to manage the Tiled Vertex Buffers.
  • Context structures and other bits that let the GPU firmware keep track of what is going on.
  • Vertex rendering commands, which tell the vertex processing and tiling part of the GPU how to process commands and shaders from userspace to run the vertex part of a whole render pass.
  • Fragment rendering commands, which tell the rasterization and fragment processing part of the GPU how to render the tiled vertex data from the vertex processing into an actual framebuffer.

It gets even more complicated than that! The vertex and fragment rendering commands are actually very complicated structures with many nested structures within, and then each command actually has a pointer to a "microsequence" of smaller commands that are interpreted by the GPU firmware, like a custom virtual CPU! Normally those commands set up the rendering pass, wait for it to complete, and clean up... but it also supports things like timestamping commands, and even loops and arithmetic! It's crazy! And all of these structures need to be filled in with intimate details about what is going to be rendered, like pointers to the depth and stencil buffers, the framebuffer size, whether MSAA (multisampled antialiasing) is enabled and how it is configured, pointers to specific helper shader programs, and much more!

In fact, the GPU firmware has a strange relationship with the GPU MMU. It uses the same page tables! The firmware literally takes the same page table base pointer used by the GPU MMU, and configures it as its ARM64 page table. So GPU memory is firmware memory! That's crazy! There's a shared "kernel" address space (similar to the kernel address space in Linux) which is what the firmware uses for itself and for most of its communication with the driver, and then some buffers are shared with the GPU hardware itself and have "user space" addresses which are in a separate address space for each app using the GPU.

So can we move all this complexity to user space, and have it set up all those vertex/fragment rendering commands? Nope! Since all these structures are in the shared kernel address space together with the firmware itself, and they have tons of pointers to each other, they are not isolated between different processes using the GPU! So we can't give apps direct access to them because they could break each other's rendering... so this is why Alyssa found all those rendering details in the macOS UAPI...

GPU drivers in Python?!

Since getting all these structures right is critical for the GPU to work and the firmware to not crash, I needed a way of quickly experimenting with them while I reverse engineered things. Thankfully, the Asahi Linux project already has a tool for this: The m1n1 Python framework! Since I was already writing a GPU tracer for the m1n1 hypervisor and filling out structure definitions in Python, I decided to just flip it on its head and start writing a Python GPU kernel driver, using the same structure definitions. Python is great for this, since it is very easy to iterate with! Even better, it can already talk the basic RTKit protocols and parse crash logs, and I improved the tools for that so I could see exactly what the firmware was doing when it crashes. This is all done by running scripts on a development machine which connects to the M1 machine via USB, so you can easily reboot it every time you want to test something and the test cycle is very fast!

At first most of the driver was really just a bunch of hardcoded structures, but eventually I managed to get them right and render a triangle!

This was just a hacked up together demo, though... before starting on the Linux kernel driver, I wanted to make sure I really understood everything well enough to design the driver properly. Just rendering one frame is easy enough, but I wanted to be able to render multiple frames, and also test things like concurrency and preemption. So I really needed a true "kernel driver"... but that's impossible to do in Python, right?!

It turns out that Mesa has something called drm-shim, which is a library which mocks the Linux DRM kernel interface and replaces it with some dummy handling in userspace. Normally that is used for things like shader CI, but it can also be used to do crazier things... so... what if I stuck a Python interpreter inside drm_shim, and called my entire Python driver prototype from it?

Could I run Inochi2D on top of Mesa, with Alyssa's Mesa M1 GPU driver, on top of drm-shim, running an embedded Python interpreter, sending commands to my Python prototype driver on top of the m1n1 development framework, communicating over USB with the real M1 machine and sending all the data back and forth, in order to drive the GPU firmware and render myself? How ridiculous would that be?

It's so ridiculous that it worked! ✨

A new language for the Linux kernel

With the eldritch horror Mesa+Python driver stack working, I started to have a better idea of how the eventual kernel driver had to work and what it had to do. And it had to do a lot! There's no way around having to juggle the more than 100 data structures involved... and if anything goes wrong, everything can break! The firmware doesn't sanity check anything (probably for performance), and if it runs into any bad pointers or data, it just crashes or blindly overwrites data! Even worse, if the firmware crashes, the only way to recover is to fully reboot the machine! 😱

Linux kernel DRM drivers are written in C, and C is not the nicest language to write complicated data structure management in. I'd have to manually track the lifetime of every GPU object, and if I got anything wrong, it could cause random crashes or even security vulnerabilities. How was I going to pull this off? There are too many things to get wrong, and C doesn't help you out at all!

On top of that, I also had to support multiple firmware versions, and Apple doesn't keep the firmware structure definitions stable from version to version! I had already added support for a second version as an experiment, and I ended up having to make over 100 changes to the data structures. On the Python demo, I could do that with some fancy metaprogramming to make structure fields conditional on a version number... but C doesn't have anything like that. You have to use hacks like compiling the entire driver multiple times with different #defines!

But there was a new language on the horizon...

At around the same time, rumours of Rust soon being adopted officially by the Linux kernel were beginning to come up. The Rust for Linux project had been working on officially adding support for several years, and it looked like their work might be merged soon. Could I... could I write the GPU driver in Rust?

I didn't have much experience with Rust, but from what I'd read, it looked like a much better language to write the GPU driver in! There are two things that I was particularly interested in: whether it could help me model GPU firmware structure lifetimes (even though those structures are linked with GPU pointers, which aren't real pointers from the CPU's perspective), and whether Rust macros could take care of the multi-versioning problem. So before jumping straight into kernel development, I asked for help from Rust experts and made a toy prototype of the GPU object model, in simple userspace Rust. The Rust community was super friendly and several people helped walk me through everything! I couldn't have done it without your help! ❤

And it looked like it would work! But Rust still wasn't accepted into mainline Linux... and I'd be in uncharted territory since nobody had ever done anything like this. It would be a gamble... but the more I thought about it, the more my heart told me Rust was the way to go. I had a chat with the Linux DRM maintainers and other folks about this, and they seemed enthusiastic or at least receptive to the idea, so...

I decided to go for it!

Rust beginnings

Since this was going to be the first Linux Rust GPU kernel driver, I had a lot of work ahead! Not only did I have to write the driver itself, but I also had to write the Rust abstractions for the Linux DRM graphics subsystem. While Rust can directly call into C functions, doing that doesn't have any of Rust's safety guarantees. So in order to use C code safely from Rust, first you have to write wrappers that give you a safe Rust-like API. I ended up writing almost 1500 lines of code just for the abstractions, and coming up with a good and safe design took a lot of thinking and rewriting!

On August 18th, I started writing the Rust driver. Initially it relied on C code for the MMU handling (partially copied from the Panfrost driver), though later I decided to rewrite all of that in Rust. Over the next few weeks, I added the Rust GPU object system I had prototyped before, and then reimplemented all the other parts of the Python demo driver in Rust.

The more I worked with Rust, the more I fell in love with it! It feels like Rust's design guides you towards good abstractions and software designs. The compiler is very picky, but once code compiles it gives you the confidence that it will work reliably. Sometimes I had trouble making the compiler happy with the design I was trying to use, and then I realized the design had fundamental issues!

The driver slowly came together, and on September 24th I finally got kmscube to render the first cube, with my brand new Rust driver!

And then, something magical happened.

Just a few days later, I could run a full GNOME desktop session!

Rust is magical!

Normally, when you write a brand new kernel driver as complicated as this one, trying to go from simple demo apps to a full desktop with multiple apps using the GPU concurrently ends up triggering all sorts of race conditions, memory leaks, use-after-free issues, and all kinds of badness.

But all that just... didn't happen! I only had to fix a few logic bugs and one issue in the core of the memory management code, and then everything else just worked stably! Rust is truly magical! Its safety features mean that the design of the driver is guaranteed to be thread-safe and memory-safe as long as there are no issues in the few unsafe sections. It really guides you towards not just safe but good design.

Of course, there are always unsafe sections of code, but since Rust makes you think in terms of safe abstractions, it's very easy to keep the surface area of possible bugs very low. There were still some safety issues! For example, I had a bug in my DRM memory management abstraction that could end up with an allocator being freed before all of its allocations were freed. But since those kinds of bugs are specific to one given piece of code, they tend to be major things that are obvious (and can be audited or caught in code review), instead of hard-to-catch race conditions or error cases that span the entire driver. You end up reducing the amount of possible bugs to worry about to a tiny number, by only having to think about specific code modules and safety-relevant sections individually, instead of their interactions with everything else. It's hard to describe unless you've tried Rust, but it makes a huge difference!

Oh, and there's also error and cleanup handling! All the error-prone goto cleanup style error handling to clean up resources in C just... vanishes with Rust. Even just that is worth it on its own. Plus you get real iterators and reference counting is automatic! ❤

Joining forces

With the kernel driver on the right track, it was time to join forces with Alyssa and start working together! No longer bound by the confines of testing only on macOS, she started making major improvements to the Mesa driver! I even helped a little bit ^^

We gave a joint talk at XDC 2022, and at the time we ran the entire talk on an M1 using our drivers! Since then we've been working on adding new features, bug fixes, and performance improvements to both sides. I added support for the M1 Pro/Max/Ultra family and the M2 to the kernel side, as well as more and better debugging tools and memory allocation performance improvements. She's been steadily improving GL conformace, with OpenGL ES 2.0 conformance practically complete and 3.0 conformance at over 96%! She also added many new features and performance improvements, and today you can play games like Xonotic and Quake at 4K!

And since the GPU power management is handled by the firmware, all that just works. I tested Xonotic at 1080p inside a GNOME session, and the estimated battery runtime was over 8 hours! 🚀

What about Vulkan support? Don't worry... Ella is working on that! ✨✨

What's next?

There is still a long road ahead! The UAPI that we are using right now is still a prototype, and there are a lot of new features that need to be added or redesigned in order to support a full Vulkan driver in the future. Since Linux mandates that the UAPI needs to remain stable and backwards compatible across versions (unlike macOS), that means that the kernel driver will not be heading upstream for many months, until we have a more complete understanding of the GPU rendering parameters and have implemented all the new design features needed by Vulkan. The current UAPI also has performance limitations... it can't even run GPU rendering concurrently with CPU processing yet!

And of course there is still a lot of work to do on the userspace side, improving conformance and performance and adding support for more GL extensions and features! Some features like tesselation and geometry shaders are very tricky to implement (since they need to be partially or fully emulated), so don't expect full OpenGL 3.2+ for quite a long time.

But even with those limitations, the drivers can run stable desktops today and performance is improving every week! Wayland runs beautifully smoothly on these machines now, just like the native macOS desktop! Xorg also works well with some improvements I made to the display driver a few days ago, although you can expect tearing and vsync issues due to Xorg design limitations. Wayland is really the future on new platforms! 💫

So where do you get it? We're not quite there yet! Right now the driver stack is complicated to build and install (you need custom m1n1, kernel, and mesa builds), so please wait a little bit longer! We have a few loose ends to tie still... but we hope we can bring it to Asahi Linux as an opt-in testing build before the end of the year! ✨✨

If you're interested in following my work on the GPU, you can follow me at @[email protected] or subscribe to my YouTube channel! Tomorrow I'm going to be working on figuring out the power consumption calculations for the M1 Pro/Max/Ultra and M2, and I hope to see you there! ✨

If you want to support my work, you can donate to marcan's Asahi Linux support funds on GitHub Sponsors or Patreon, which helps me out too! And if you're looking forward to a Vulkan driver, check out Ella's GitHub Sponsors page! Alyssa doesn't take donations herself, but she'd love it if you donate to a charity like the Software Freedom Conservancy instead. (Although maybe one day I'll convince her to let me buy her an M2... ^^;;)

Asahi Lina · 2022-11-29




All Comments: [-] | anchor

fareesh(10000) 3 days ago [-]

I've never played games on my M1 Macbook - what are some popular reasonably graphics intensive games that it would support? Could it run Dota2 for example?

rpastuszak(10000) 3 days ago [-]

Disco Elysium, Hades and CIV VI run really well on my MBA m1 (using a 4K display). These games are not as resource heavy as Dota2 AFAIK but I'm comparing them to my maxed out 16inch MBP from 2020 which acted more like a cursed semi sentient toaster than a hi spec laptop.

tingol(10000) 3 days ago [-]

Dota 2 runs pretty great.

drooopy(10000) 3 days ago [-]

Resident Evil Village recently came out and it performs surprisingly well even on the low end MacBook Air M1 with only 7 GPU cores. What's even more impressive is that the game is playable (low gfx settings, 30fps) when running that machine on low power mode.

helf(10000) 4 days ago [-]

The fact so much hardware these days is running a full real-time OS all the time annoys me. I know it is normal and understandable but everything is such a black box and it has already caused headaches (looking at you, Intel).

ramraj07(10000) 4 days ago [-]

Every cell in your body is running a full blown OS fully capable of doing things that each individual cell has no need for. It sounds like this is a perfectly natural way to go about things.

babypuncher(10000) 4 days ago [-]

This isn't even that new of a thing. The floppy disk drive sold for the Commodore 64 included it's own 6502 CPU, ROM, and RAM. This ran its own disk operating system[1]. Clever programmers would upload their own code to the disk drive to get faster read/writes, pack data more densely on the disk, and even copy protection schemes that could validate the authenticity of a floppy.

1: https://en.wikipedia.org/wiki/Commodore_DOS

jbverschoor(10000) 4 days ago [-]

I sometimes wonder about how fast tings could go if we ditch the firmware, and also just bake a kernel / os right into the silicon. Not like all the subsystems which run their own os/kernels, but really just cut every layer, and have nothing in between.

hinkley(10000) 4 days ago [-]

It's only a problem if they're closed source. We should be working on that.

comprambler(10000) 4 days ago [-]

Technically it is not a real time OS. There are very few OSs that have this moniker (vxworks, qnx, etc)

dist1ll(10000) 4 days ago [-]

There's this great USENIX talk by Timothy Roscoe [1], which is part of the Enzian Team at ETH Zürich.

It's about the dominant unholistic approach to modern operating system design, which is reflected in the vast number of independent, proprietary, under-documented RTOSes running in tandem on a single system, and eventually leading to uninspiring and lackluster OS research (e.g. Linux monoculture).

I'm guessing that hardware and software industries just don't have well-aligned interests, which unfortunately leaks into OS R&D.

[1] https://youtu.be/36myc8wQhLo

stock_toaster(10000) 4 days ago [-]

It's interesting that microkernels didn't 'win' at the OS layer, but they kind of seem to have 'won' one layer down.

GekkePrutser(10000) 4 days ago [-]

The real-time nature is not what makes it closed though. It's simply that it's been designed to be closed.

For example, Intel's ME could be a really useful feature if we could do what we want with it. Instead they lock it down so it's just built-in spyware.

liuliu(10000) 4 days ago [-]

I don't know. This sounds very computer-sciency-ish. We build smaller tools to help build big things. Now the big things are so good and versatile we can replace our smaller tools with the big things too. With the more powerful tools, we can build even bigger things. It is just compiler bootstrapping happening in hardware world.

the__alchemist(10000) 4 days ago [-]

Same. It's not about the principle, but that generally these OSes increase latency etc. There's so much you can do with interrupts, DMA, and targetted code when performance is a priority.

kajaktum(10000) 4 days ago [-]

Half the comments here are talking about the vtuber herself. Who cares. It's been talked before. Just imagine if half the thread is discussing what gender she is. What I am interested in is the claims here https://asahilinux.org/2022/11/tales-of-the-m1-gpu/#rust-is-.... (what is it called if it comes with a proof?).

The resident C/C++ experts here would have you believe that the same is possible in C/C++. Is that true?

aulin(10000) 3 days ago [-]

I must admit that for me this was a pleasant change of format from the youtube videos.

SXX(10000) 4 days ago [-]

I'm not exactly C or Rust expert so better to check @dagmx comment for that, but I know some C++ and worked with networking enough to know some pitfalls.

Talking of C++ it can be really solid to work with your own data structures where you control code on both ends. Using templates with something like boost::serialization or protobuf for the first time is like magic. E.g you can serialize whole state of your super complex app and restore it on other node easily.

Unfortunately it's just not the case when you actually trying to work with someone else API / ABI that you have no contol over. Even worse when it's moving target and you need to maintain several different adapters for different client / server versions.

dagmx(10000) 4 days ago [-]

In C? No, not unless you write your own scaffolding to do it.

In C++? Maybe, but you'd need to make sure you stay on top of using thread safe structures and smart pointers.

What Rust does is flip this. The default is the safe path. So instead of risking forgetting smart pointers and thread safe containers, the compiler keeps you honest.

So you're not spending time chasing oddities because you missed a variable initialisation, or you're hitting a race condition or some kind of use after free.

While there's a lot of people who say that this slows you down and a good programmer doesn't need it, my experience is even the best programmers forget and (at least for me), I spend more time trying to reason about C++ code than rust, because I can trust my rust code more.

Put another way, Rust helps with reducing how much of the codebase I need to consider at any given time to just the most local scope. I work in many heavy graphics C and C++ libraries , and have never had that level of comfort or mental locality.

saagarjha(10000) 3 days ago [-]

Possible? Definitely. Easier? Probably not. At least for the most part, there are a couple things which C(++) can sometimes be more ergonomic for and those can be isolated out and used independently.

mort96(10000) 3 days ago [-]

I have a lot of experience in C, a lot of experience in C++, and some experience with Rust (I have some projects which use it). My opinion is that it's true, and the other comments are good explanations of why. But I want to point out, in addition to those: There's a reason why Rust was adopted into Linux, while C++ wasn't. Getting C++ to work in the kernel would almost certainly have been way less work than getting Rust to work. But only Rust can give you the strong guarantees which makes you avoid lifetime-, memory- and concurrency-related mistakes.

dimator(10000) 4 days ago [-]

watching a virtual persona stream their development of their M1 GPU drivers is one of the most cyberpunk things I've ever seen! it's easy to forget that this world is looking closer and closer to those dreamed up by Gibson, Stephenson, etc. what a time to be alive.

anxiously(10000) 4 days ago [-]

Can someone explain this vtoon trend to me? It doesn't seem to be driven by anonymity because their real name is easily findable, so I assume it's something else? It seems very common, especially in certain communities.

Mistletoe(10000) 4 days ago [-]

I like your optimism, but it seems more like a Phillip K. Dick novel to me.

>In 2021, society is driven by a virtual Internet, which has created a degenerate effect called 'nerve attenuation syndrome' or NAS. Megacorporations control much of the world, intensifying the class hostility already created by NAS.

from Johnny Mnemonic

What can we do to make it more utopian?

Aaargh20318(10000) 4 days ago [-]

> watching a virtual persona stream their development of their M1 GPU drivers is one of the most cyberpunk things I've ever seen!

What would really push it into cyberpunk territory is if it turns out this is not an actual human but an AI-controlled virtual person.

segmondy(10000) 4 days ago [-]

Which is what Zuck is trying to get us all into with the Meta, but the world is not ready yet.

punnerud(10000) 3 days ago [-]

Not one comment here about the "GPU drivers in Python". I like the idea of iteration speed, over pure speed.

And the coprocessor called "ASC" also have similarities with Python, where the GPU is doing the heavy lifting, but the ASC (like Python) interact using shared memory. The same Python is doing with a lot of its libraries (written in C/C++)

saagarjha(10000) 3 days ago [-]

> And the coprocessor called "ASC" also have similarities with Python

It's a processor, not a programming language :) The team has essentially strapped the API into something that you can poke with Python instead of with a native driver.

rafale(10000) 4 days ago [-]

Loved reading this. About the triangle/cube screenshot, they were taken on Linux on a physical Mac OS computer? How were you able to deploy your driver, does the M1 GPU have a basic text/console mode allowing you to start and work with Linux?

Awesome job.

puyoxyz(10000) 3 days ago [-]

Displaying to the screen and stuff was already working, you can already use Asahi Linux and have a GUI and everything, it's just that it's all rendered by the CPU right now

calrizien(10000) 4 days ago [-]

Great work! Seems like Apple could make this a whole lot easier by giving even the slightest support...

sneak(10000) 3 days ago [-]

It is irksome to me given how much Linux is used inside Apple (board bringup, debugging, etc). You benefit from these gifts, Apple, give back a teensy bit in return. Everybody wins.

adultSwim(10000) 4 days ago [-]

Keep it up! I'm sticking with Thinkpads until Linux support

GekkePrutser(10000) 4 days ago [-]

Also they have matte screens and real keyboards.

The 2015MBP one was the last one that was passable for me, what came after is horrible. Even the new MBP that has real ports again is still not as good as the 2015 in terms of keyboard.

2OEH8eoCRo0(10000) 4 days ago [-]

Apple should upstream their drivers.

Who is Asahi Lina? Is that an actual person?

worldsavior(10000) 4 days ago [-]

Asahi Lina is a maintainer in Asahi Linux project. She is now much known because of the achivement she earned, programming the Asahi Linux GPU driver for MacOS.

sliken(10000) 4 days ago [-]

Their = apple? Their = Asahi Lina's?

Asahi Linux has been upstreaming, but of course it's ongoing. The GPU driver in particular depends on some rust inside the kernel bits which aren't in the mainline kernel, yet. The 6.1 kernel has some Rust bits, 6.2 will have more, but I don't believe that will be enough for the GPU driver ... yet.

wincy(10000) 4 days ago [-]

I think it's a vTuber anime persona of a very talented programmer or something?

nicoburns(10000) 4 days ago [-]

> Apple should upstream their drivers.

Apple don't have linux drivers. It would be great if they wrote some, but it's never going to happen.

> Who is Asahi Lina? Is that an actual person?

The virtual persona of an actual person who has chosen to remain anonymous (hence the name which would be a crazy coincidence otherwise).

dboreham(10000) 4 days ago [-]

> Apple should upstream their drivers.

To what upstream project?

tiagod(10000) 4 days ago [-]

>Asahi Lina, our GPU kernel sourceress. Lina joined the team to reverse engineer the M1 GPU kernel interface, and found herself writing the world's first Rust Linux GPU kernel driver. When she's not working on the Asahi DRM kernel driver, she sometimes hacks on open source VTuber tooling and infrastructure.

yjftsjthsd-h(10000) 4 days ago [-]

> Apple should upstream their drivers.

Apple's drivers are upstreamed, in Darwin. I'm not aware of any reason to believe that Apple has any Linux drivers that they could upstream.

nicoburns(10000) 4 days ago [-]

The m1n1 hypervisor specialised for debugging is a pretty genius idea. Is anyone aware of anyone else taking a similar approach? Seems like it would be a pretty generally applicable technique and would make OS/hardware driver development a lot more approachable.

gjsman-1000(10000) 4 days ago [-]

Not that I know of. m1n1 originated from software that (IIRC) was used initially for reverse-engineering the Wii.

ngcc_hk(10000) 3 days ago [-]

Idea wise actually S/360 run on hardware microcode and all these idea of virtual machine and hypervisor came from an unauthorised development called CP67 or later VM. IBM used it for development MVS etc. as some hardware is yet to be built for certain features.

But the modern day these development is crazy.

How can yo manage a 100+ structure in a language you just learnt (Rust) for a secret GPU the vendor does not share info.

bri3d(10000) 4 days ago [-]

Even before true CPU-supported 'hypervisors,' there was shim software like SoftICE that worked similarly to m1n1 in that you would run an OS underneath and then use a supervisor tool to trace and debug the OS under inspection.

More recently, it's fairly common to use a hypervisor or simulator for kernel debugging in device driver development on Windows via Hyper-V.

A lot of Linux driver development is done using qemu as well, although this is usually more targeted and isn't quite the same 'put a thin shim over the OS running on the hardware' approach.

The flexibility and I/O tracing framework in m1n1 are pretty uniquely powerful, though, since it was built for reverse engineering specifically.

hedgehog(10000) 4 days ago [-]

Some developers used user mode Linux for driver development, and I think some development has happened on the NetBSD rump kernel more recently. I find the work that goes into building this kind of tooling all pretty impressive.

averne_(10000) 4 days ago [-]

The nouveau project used a kernel module to intercept mmio accesses: https://nouveau.freedesktop.org/MmioTrace.html. Generally speaking hooking onto driver code is one of the preferred ways of doing dynamic reverse engineering. For userspace components, you can build an LD_PRELOAD stub that logs ioctls, and so on.

psychphysic(10000) 4 days ago [-]

What an interesting tale. But I feel like an anime character described it to me.

I'm totally fine with it (I'm grateful the story is being told at all), but it is surreal tone for technical writing.

qrio2(10000) 4 days ago [-]

pretty much the appeal(?) of asahi lina. it's been a weird ride to follow for sure.

amelius(10000) 4 days ago [-]

> But I feel like an anime character described it to me.

Mario Brothers would make more sense though. Whoever created this is a plumber par excellence.

noveltyaccount(10000) 4 days ago [-]

The quantity of exclamation points lol. I assume I'm just too old to get it...I'm okay with that, and I'm damn impressed with the results, so more power to Lina, whatever works for her.

IshKebab(10000) 4 days ago [-]

Not as surreal as an actual presentation by an anime character. WTF is going on here?

https://youtu.be/SDJCzJ1ETsM?t=1179

How can people watch this?

treesknees(10000) 4 days ago [-]

There are 149 exclamation points on that page (!)

rychco(10000) 4 days ago [-]

I'm actually very happy about the rise of VTubers/live avatars. I imagine that there are a lot of people that would love to interactively share their knowledge/skills on youtube/twitch but avoid doing so because they're not conventionally attractive or just too shy.

tiahura(10000) 4 days ago [-]

Is anyone porting the GPU driver to Windows?

raverbashing(10000) 3 days ago [-]

MS or Apple can port it themselves, but I think there's not so much interest on the Apple side

noveltyaccount(10000) 4 days ago [-]

Keep an eye on this project: https://github.com/amarioguy/m1n1_windows

monocasa(10000) 4 days ago [-]

I think there's larger barriers to getting windows running on Apple Silicon that would need to be addressed first.

For one example, Windows ARM kernels are pretty tied to the GIC (ARM's reference interrupt controller), but Apple has its own interrupt controller. Normally on ntoskrnl this distinction would simply need hal.dll swapped out, but I've heard from those who've looked into it that the clean separation has broken down a bit and you'd have to binary patch a windows kernel now if you don't have source access.

yjftsjthsd-h(10000) 4 days ago [-]

You can always just run NT in a VM under Linux:)

masklinn(10000) 4 days ago [-]

Porting what?

gfody(10000) 3 days ago [-]

is there an easy way to unvoicemod the streams?

pulvinar(10000) 3 days ago [-]

Turn on closed-captioning, and mute the sound.





Historical Discussions: Guide to FFmpeg (November 28, 2022: 978 points)

(978) Guide to FFmpeg

978 points 5 days ago by buss_jan in 10000th position

img.ly | Estimated reading time – 94 minutes | comments | anchor

In this guide, we'll go through the hot topics of FFmpeg. But before that, we'll cover some base ground to help you understand basic media concepts and FFmpeg. Feel free to skip the parts that are already trivial for you!

Introduction to FFmpeg

FFmpeg.org's definition is the following: 'FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge. No matter if they were designed by some standards committee, the community or a corporation.'

I think of FFmpeg as the go-to application for audio/video manipulation in an automated or scripted manner.

When you need to implement a service that manipulates video, or just have 300 media files that need to be converted into a different format, FFmpeg is your - nerdy - friend.

FFmpeg can do large chunks of the basic functionalities of a modern Non-linear (NLE) video editors, e.g., Davinci Resolve Studio or Premiere Pro. But, it does not have a graphical interface in that sense as those behemoths do, and unarguably it is way less friendly.

In a general NLE, you might do things like these:

  1. Click to import a file
  2. Drop it into the timeline
  3. Trim and Cut
  4. Add an overlay image
  5. Crop that overlay
  6. Add vignette
  7. Add some color changing effects, e.g. change the hue
  8. Add an extra audio track to the mix
  9. Change the volume
  10. Add some effects, e.g.: echo
  11. Export into various formats
  12. Export into a deployable video format
  13. Export the master audio in wav

Or, to achieve the exact same thing, you could also execute this command:

ffmpeg -y  \
    -ss 20 -t 60 -i bbb_sunflower_1080p_60fps_normal.mp4 \
    -i train.jpg \
    -ss 4 -i voice_recording.wav \
    -filter_complex '[0:v]hue=h=80:s=1[main] ; [1:v]crop=w=382:h=304:x=289:y=227[train] ; [main][train]overlay=x=200:y=200,vignette=PI/4[video] ; [2:a]volume=1.5,aecho=0.8:0.9:100:0.3[speech] ; [0:a][speech]amix=duration=shortest,asplit[audio1][audio2]' \
    -map '[video]' -map '[audio1]' -metadata title='Editor's cut' bbb_edited.mp4 \
    -map '[audio2]' bbb_edited_audio_only.wav

Yes, it isn't friendly at all, but it is very, very powerful once you become friends with FFmpeg.

Check out this comparison of the original and the edited one:

If you want to try this command out, get the example files and see it for yourself!

Installing FFmpeg

FFmpeg is available for most common and even uncommon platforms and architectures. You can be on Linux, Mac OS X or Microsoft Windows, and you'll be able to run or link to FFmpeg.

Installing FFmpeg is easy on most platforms! There is no installer, usually just a compressed archive you need to get for your platform and architecture.

In the case of Linux, most distributions include a pre-built FFmpeg in their software repositories. Therefore, you can install FFmpeg from those even more quickly.

FFmpeg history

The project was started in 2000 by the awesome Fabrice Bellard. The name is a concatenation of 'FF' meaning 'fast-forward' and MPEG, the name of a video standards group. It has been very well, active and alive since then, releasing a new release about every three months.

FFmpeg supported codecs and formats

The default FFmpeg shipped with my Ubuntu Linux distribution supports about 460 codecs and 370 formats.

See it for yourself:

ffmpeg -codecs
ffmpeg -formats

Compilation of FFmpeg

Keep in mind that the supported codecs and formats (and filters, demuxers, muxers, input and output methods, etc.) are highly dependent on the so-called compilation flags.

This means that the above number only represents the fact that it supports at least this many codecs and formats. Still, there are even more that the package builders excluded for various reasons, e.g.: licensing, architecture, size considerations, etc.

Since FFmpeg is open source, you can compile FFmpeg for yourself at any time.

Suppose for example, that you care about your layer's size (therefore the bootstrap speed) in AWS Lambda. In this case, you can compile an FFmpeg binary that only contains the mp3 encoder for example, and nothing else.

Also, you might not want to run into licensing issues and leave out stuff that would cause problems for your use case. Therefore you choose to leave out particular codecs/formats. I highly recommend checking out the '--enable-gpl', '--enable-nonfree' and '--enable-version3' compilation flags in this case, as well as this.

Or you might want to have a standalone FFmpeg binary in your project (e.g.: embedded, or some cloud instance), that does not depend on any operating system libraries. Then you want to make a so-called static build, that compiles in all the libraries into a single binary file, and does not depend on your OS' libraries and the runtime loading of other FFmpeg libraries. Search around for '--enable-static' in this case.

Finally, you can find pre-built static FFmpeg builds right here too.

FFmpeg's strengths

FFmpeg reads and writes most video and audio formats that matter for most of us. It is a very capable and high-performance tool for converting and manipulating these formats.

But FFmpeg can do even more!

Filtering

FFmpeg has vast amounts of filters for audio and video. Therefore, video manipulation is also a key feature of FFmpeg.

Hardware acceleration

It does support many kinds of hardware accelerations! Video encoding is a very resource-intensive operation, and you might come across quite a few hardware devices or features that might speed up your process!

Most notably, if you have an NVIDIA card, you can increase your H.264 or H.265 encoding and decoding throughput by multipliers compared to your CPU. But other things, such as VDPAU, VAAPI, or OpenCL, can be leveraged to boost your pipeline's throughput.

Learn more about the supported hardware acceleration methods here.

Versatile input/output methods

FFmpeg is also very capable when it comes to accessing input and output data.

Just to name a few: it can use your webcam, record from your microphone, grab your screen, or capture from your Blackmagic DeckLink. But FFmpeg can download directly from a web address, open all kinds of streams, read from a pipe, a socket, and of course, from files.

The same holds true for outputting the data. It can write to your webcam, play audio on your microphone... Just kidding:) It can output to files, streams, pipes, sockets and so on.

Running example commands

This article is full of FFmpeg commands that are working examples. The reason for that is that you could test these out for yourself! But the command line interfaces of different operating systems are slightly different, so the commands in this article are meant to be executed in a Linux bash shell.

To adopt these command lines to Microsoft Windows, you might need to:

  1. Change (cd) into the directory where you extracted the ffmpeg.exe. Alternatively, add that directory to the path to make it callable from anywhere.
  2. You might need to replace 'ffmpeg' to 'ffmpeg.exe'
  3. You will need to replace '\'-s (backslashes) at the end of the lines with '^'-s (hats)
  4. You'll need to replace the fontfile argument's value to something like this: fontfile=/Windows/Fonts/arial.ttf to get commands with the drawtext filter working.

MacOS users will need steps #1 and #4.

Now let's have a quick overview of media concepts. These concepts will be vital for us if we want to understand the latter sections of this article and FFmpeg's workings. To keep this section brief, it is a higher-level, simplified explanation of these concepts.

Audio

We'll briefly cover the following terms:

  1. Sampling rate
  2. Bitrate
  3. Channels

Sampling Rate

The sampling rate is the factor that shows how many times we measure/scan/sample the input data stream.

The image below shows the measurement windows (quantization) as gray bars.

Why does this matter? Because it is a balancing act. If we measure the signal less often, we'll lose more details (bad). Also, by having fewer samples, we'll have less data in the end. Therefore the file size will be smaller (good).

Here are some ballpark values:

  • 8 kHz (GSM - Low quality)
  • 44.1 kHz (CD - High quality)
  • 48 kHz (Very high quality)
  • 88.2 kHz (Insane - usually for production only)
  • 96 kHz (Insane - usually for production only)

There are no definite 'right answers' here. The question is what is 'good enough' for your use case? GSM focuses on speech, and not even quality but understandability and the least possible amount of data. Therefore, they found that 8 kHz is enough (there are quite a few more tricks), for their purposes.

The 'CD quality' aimed for high quality. Therefore they chose 44.1 kHz, that number has some history in it, but the main reason for aiming above 40 kHz lies in physics and how the human ear works.

There were two very smart guys whose theorem basically says that if you want a quite good signal representation, you have to sample it at twice the speed as its original frequency. Human hearing generally works up until about 20 kHz, so if you want 'good quality', you should aim for at least 40 kHz. And 40 kHz + some headroom + some more physics + historical reasons = 44.1 kHz! :)

As for the higher rates, those are only used when very high-quality audio editing is needed.

Bitrate

Bitrate represents the amount of data per second that results from our transcoding/quantization process. If it is 1411 kbit/s, that means that for every second of audio data, about 1411 kbit of output data will be produced.

Therefore, you can say that 1 minute of audio with 1411 kbit/sec will require:

(1411 kbit / 8) kbyte * 60 second = 10582 kbyte = 10.33 mbyte

Now, it is only easy like that with raw audio data and with a few simple codecs, e.g. PCM in WAVs.

Codecs compressing hard might throw your numbers around a little, as input data might be compressible with different rates. Variable bitrate is usually happening to save space. The encoder might output a lower bitrate if the data is 'simple' and does not require high precision.

Here are some ballpark values:

  • 13 kbits/s (GSM quality)
  • 320 kbit/s (High-quality MP3)
  • 1411 kbit/s (16bit WAV, CD quality, PCM)

Channels

Inside of most audio formats, you can have more audio channels. This means multiple, separated audio streams can be in the same file.

Many times, multiple channels have their own name:

  • If you have a single microphone, you will most probably record it into a single channel called Mono.
  • General music from the FM radio or streaming services usually has two channels in a so-called 'Stereo' configuration.

With stereo, there could be several methods how the audio 'image' can be made richer by leveraging audio panning, time and phase-shifting and much more. There is a special recording technique too, called Binaural recording, which is super awesome. Wear headphones for this, and don't be scared:)

For example, here are Big Buck Bunny's audio waveforms in Audacity:

You can see that there are two lines of waveforms and also that they are pretty similar. That is normal, as you usually hear the same thing with your two ears, but the matter is in the subtle differences between the two. That's where directionality, richness, and all kinds of other effects lie.

But why stop at two? The list continues:

  • 2.1, as it is often called, means three channels: 2 for stereo and one for the LFE ('low-frequency effects' a.k.a.: 'bass').
  • 5.1 is similar, with five directional channels (2 front, 1 center, 2 rear) and the LFE.

So channels are just separate 'recordings' or 'streams' of audio signals.

Image properties

For images, there are quite a few parameters, but we'll check out only these:

  • Resolution
  • Bit-depth
  • Transparency

Resolution

An image consists of pixels, single points that have a single color. The resolution of an image determines how many columns and rows of pixels are in an image. In other words: an image has a width and a height.

This image shows the first 10 pixels in the first row.

Here are some ballpark values for resolution:

  • 'HD' or 'Full HD' or '1K' or '1080p' means 1920x1080 pixels.
  • '4K' could mean a few values, but it should be about 3840x2160 pixels.
  • A regular 16mp photo you make of your cat is about 4608x3456 pixels.
  • General social media image posts are about 1080x1080 pixels.

Bit-depth

Bit-depth represents the number of bits used for storing a single pixel's color value. This is the same balancing game, and you need to decide between quality or file size.

General ballpark values for bit-depth:

Bits Colors Notes
1 2 Black & White
8 256 B/W or Limited color palette
24 16.7m 3x8 bit for R-G-B 'True color'
30 1073m 3x10 bit for R-G-B 'Deep color'

These last two sometimes are referred to as '8 bit' or '10 bit' respectively, especially when talking about videos. That means 8/10 bits per single color channel.

Transparency

Some image formats support an additional channel together with the red, green, and blue components: the alpha channel. The alpha channel determines how transparent a single pixel is, and it can have different bit-depths, it is usually either 1, 8 or 16 bits.

If the alpha channel is 1 bit, then the format can encode a pixel to be either transparent or non-transparent. If it is 8 or more bits, then the format can encode 256 or more steps of transparency.

Video properties

Video data is built by single images shown right after each other. This brings in most attributes of images and a few more!

So a video has a resolution that is its width and height.

Then the first obvious parameter of a video is the framerate, which defines how many images are shown in a second. Common values for this are 24, 25, 30, or 60.

A video file also has a codec assigned to it, which is the format describing how all those images were compressed into this video file. There are many more attributes of videos, but this is a good start.

Video codecs

Compression is a super important thing when it comes to video because you have thousands of images to keep together. If you aren't doing it in a smart way, then the resulting video will be very, very large.

Just imagine a 2-minute video, with 30 fps. That means it will have 60 s * 2 * 30 fps = 3600 frames! I have just taken a screenshot of an HD video, which was 730 kbyte in JPEG format. Now 3600 frame * 730 kbyte equals 2.5 gigabytes!

Can you imagine that? I hope not, and that's because compression brings that way, way down, to the level of tens of megabytes. These days a video of that size is quite high quality and about 2 hours long. Also, don't forget, that JPEG is already compressed, a single frame would be 6 mbyte when uncompressed. Now that 2-minute video would be 21 gigabytes if we'd store it uncompressed.

Standard codecs such as H.264 and H.265 are doing very clever and complex operations to achieve high compression ratios with good quality.

Just think about that, most frames in a video are quite similar, only containing small differences. So if we could only store that little difference between frames, we'd won a huge bonus! And that's just one of the many tricks codecs do.

Codec designers are also exploiting the weaknesses and features of the human eye. Such as the fact that we are more sensitive to light intensity changes than color changes (say hello to YUV). And they can get away with lower quality details for parts that are moving fast, and so on.

Because why lose precious bits for things that you can't even notice?!

There are many codecs out there, with different goals in mind, although the majority focus on keeping the file size low.

  • H.264, H.265: These are the most common ones, with the widest support in browsers, phones, players, etc. It focuses on small file sizes with good quality. (At the cost of resource intensiveness.)
  • Apple ProRes, DNxHD: These are common formats for production. They focus on quality and ease of processing and not on file size.

Audio codecs

The goal of audio codecs is the same as what we saw with the video codecs. It is just harder to demonstrate it as audio does not consist of single image frames but audio frames/packets. So an analog audio signal is of an almost infinite, or at least very high quality if you think of it.

At the lowest level, the speed and amplitude resolution is very high. We could say 'atomic', as we need to measure and store the speed and direction of atoms. So if you want to store that exactly, that will require a super high-quality measurement, which will also result in a very high bitrate data stream.

Thankfully, the sound is at least not propagating with light speed so we can save quite a lot just by that fact. (There's no need for an extreme sampling rate.) Then our hearing is very limited if we take the previous paragraph as a scale, so we win there again. We don't need most of that high precision that is there.

But still, if we take our hearing capability and want to store raw audio data with about 44.1 kHz of sample rate with about 1 Mbit/sec bitrate, we'd still get quite a lot of data. Check the calculations in the audio bitrate section above.

So raw audio can be compressed further, which is what many popular codecs do. They also exploit the human senses, but this time the human ear. We started with the basics that the human ear has a limit on the frequencies it can detect. Therefore, we can save a lot by cutting out the range of frequencies outside our hearing range. Unless you are a bat, you are fine between 20-20khz! :)

But there are other tricks, for example, auditory masking. That means that the presence of one frequency can affect your capability to detect a different frequency. From the codec's viewpoint, it can skip encoding a few frequencies if it is smart enough to know which ones you'll not notice. I'm sure there are a lot more tricks, let me know if you know about a few more interesting ones!

Here is a list of common codecs:

  • MP3, AAC, OGG: These are common lossy audio formats.
  • PCM (e.g. in a WAV container), FLAC: These are lossless formats.
  • MIDI: It is a funny format. It is like a music sheet that might sound different on different players or settings. It is usually not made from real audio data, but from recording a digital keyboard or as an output from an audio composing software.

Containers

Now we got through the fundamental building blocks, the image, the video, the video codecs, and the audio codecs, and we reached the top of this iceberg: the containers.

A container is a format specification, that combines all these streams into a single file format. It defines how to put all these data together, how to attach metadata (e.g. author, description, etc), how to synchronize these streams, and sometimes a container even contains indexes to aid seeking.

So, for example, a MOV container can contain an H.264 video stream and an AAC audio stream together.

Common containers:

  • MOV
  • MP4
  • MKV
  • WebM
  • WAV (audio only)

Example Material

I will use these example materials as inputs in the following parts of this article. If you'd like to follow along, save these files for yourself!

n

And we will make our own audio file by extracting the audio from the Big Buck Bunny movie! We'll use this file as an example, so after downloading the video file, please execute this:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 bbb_audio.wav

By the middle of this article, you'll understand this command, but for now, just make sure to have the WAV file next to your video file to test out the commands later in the article.

We'll use these files in the following parts of this article. Therefore make sure to get them!

FFplay and FFprobe

FFmpeg is the name of the main binary and the project itself, but it is shipped together with two other binaries, ffplay and ffprobe.

Let's check them out quickly, right in the command line!

FFplay

FFplay is a basic video player, that can be used for playing media. It's not a friendly video player, but it is a good testing ground for various things.

To execute it, just simply supply a media file:

ffplay bbb_sunflower_1080p_60fps_normal.mp4

If you want to test this exact command, you'll need to get the example files.

For example, it can be used to preview filters (we'll discuss those later), but let's see an example:

ffplay -vf 'drawtext=text='HELLO THERE':y=h-text_h-10:x=(w/2-text_w/2):fontsize=200:f

FFprobe

FFprobe, as its name implies, is a tool for getting information about media files.

This command:

ffprobe bbb_sunflower_1080p_60fps_normal.mp4

Will return us some general information about the video file:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':
  Metadata:
[...]
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
[...]
  Stream #0:0[0x1](und): Video: h264 [...]
[...]
  Stream #0:1[0x2](und): Audio: mp3 [...]
[...]
  Stream #0:2[0x3](und): Audio: ac3 [...]

I have abbreviated it heavily, as we'll check this out later.

But FFprobe is way more powerful than just this!

With the following command, we can get the same listing in JSON format, which is machine-readable!

ffprobe -v error -hide_banner -print_format json -show_streams bbb_sunflower_1080p_60fps_normal.mp4

The explanation of this command is the following:

  • '-v error -hide_banner': This part hides extra output, such as headers and the default build information.
  • '-print_format json': Obviously, this causes ffprobe to output a JSON.
  • '-show_streams' is the main switch that requests the stream information.
{
  'streams': [
    {
      'index': 0,
      'codec_name': 'h264',
      'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10',
      'width': 1920,
      'height': 1080,
      'bit_rate': '4001453',
      'duration': '634.533333',
      '############################': '[~50 lines removed]'
    },
    {
      'index': 1,
      'codec_name': 'mp3',
      'channels': 2,
      'bit_rate': '160000',
      '############################': '[~40 lines removed]'
    },
    {
      'index': 2,
      'codec_name': 'ac3',
      'channels': 6,
      '############################': '[~20 lines removed]'
    }
  ]
}

In this output, you can see three streams of data in this video file. The first (index: 0) is a video stream, that is an HD video with an H.264 codec. Then we have two audio streams, the first (index: 1) is a simple mp3 stream with stereo audio, and the second (index: 2) is an ac3 stream with 6 channels, most likely in an 5.1 configuration.

I have removed quite a lot of output for brevity, but you can get way more information out of these streams, e.g. fps for the video stream and so on.

Other than -show_streams, there are 3 more: -show_format, -show_packets and -show_frames. Unless you are really deep in the rabbit hole, you'll not need the last two, but -show_format could be useful:

ffprobe -v error -hide_banner -print_format json -show_format bbb_sunflower_1080p_60fps_normal.mp4
{
  'format': {
    'filename': 'bbb_sunflower_1080p_60fps_normal.mp4',
    'nb_streams': 3,
    'nb_programs': 0,
    'format_name': 'mov,mp4,m4a,3gp,3g2,mj2',
    'format_long_name': 'QuickTime / MOV',
    'start_time': '0.000000',
    'duration': '634.533333',
    'size': '355856562',
    'bit_rate': '4486529',
    'probe_score': 100,
    'tags': {
      'major_brand': 'isom',
      'minor_version': '1',
      'compatible_brands': 'isomavc1',
      'creation_time': '2013-12-16T17:59:32.000000Z',
      'title': 'Big Buck Bunny, Sunflower version',
      'artist': 'Blender Foundation 2008, Janus Bager Kristensen 2013',
      'comment': 'Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net',
      'genre': 'Animation',
      'composer': 'Sacha Goedegebure'
    }
  }
}

This is an overview of 'what is this file'. As we see, it is a MOV file (format_name), with three streams (nb_streams), and it is 634 seconds long. Also, there are some tags where we can see the title, the artist, and other information.

FFmpeg concepts

Here is a quick intro to how FFmpeg actually works!

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

FFmpeg opens the file, decodes it into memory, then encodes the in-memory packets back and puts them into some container: some output file. The term 'codec' is a mix of the words 'coder & encoder'. Those are the magic parts before and after the 'decoded frames'.

The decoded frames are uncompressed images in-memory, e.g. the most basic pixel format for video frames is called 'rgb24'. This just stores red, green, and blue values right after each other in 3x8 bits, or 3x1 byte, which could hold 16m colors.

The importance of this is that other than a few exceptions, you can only manipulate or encode the decoded frames. So when we get to different audio/video filters or transcoding, you'll need the decoded frames for all that. But don't worry, FFmpeg does this automatically for you.

Inputs

So you see and probably guessed, that FFmpeg must access the input data somehow. FFmpeg knows how to handle most media files, as the awesome people who develop FFmpeg and the related libraries made encoders and decoders for most formats available!

Don't think that it is a trivial thing. Many formats are reverse engineered, a hard task requiring brilliant people.

So although we often refer to input files, the input could come from many sources, such as the network, a hardware device and so on. We'll learn more about that later on in this article.

Many media files are containers for different streams, meaning that a single file might contain multiple streams of content.

For example, a .mov file might contain one or more streams:

  • video tracks
  • audio tracks (e.g. for the different languages or audio formats such as stereo or 5.1)
  • subtitle tracks
  • thumbnails
  • ...

All these are streams of data from the viewpoint of FFmpeg. Input files and their streams are numerically differentiated with a 0-based index. So, for example, 1:0 means the first(0) stream of the second(1) input file. We'll learn more about that later too!

Important to note that FFmpeg can open any number of input files simultaneously, and the filtering and mapping will decide what it will do with those. Again more on that later!

Streams

As we have seen in the previous section, streams are the fundamental building blocks of containers. So every input file must have at least one stream. And that's what you can list by the simple ffmpeg -i command for example.

A stream might contain an audio format such as MP3, or a video format such as an H.264 stream.

Also, a stream, depending on the codec, might contain multiple 'things'. For example, an mp3 or a WAV stream might include various audio channels.

So the building block hierarchy, in this case is: File → Stream → Channels.

Outputs

Of course, an output could be a local file, but it doesn't need to be. It could be a socket, a stream and so on. In the same way as with inputs, you could have multiple outputs, and the mapping determines what goes into which output file.

The output also must have some format or container. Most of the time FFmpeg can and will guess that for us, mostly from the extension, but we can specify it too.

Mapping

Mapping refers to the act of connecting input file streams with output file streams. So if you give 3 input files and 4 output files to FFmpeg, you must also define what should go to where.

If you give a single input and a single output, then FFmpeg will guess it for you without specifying any mapping, but make sure you know how exactly that happens, to avoid surprises. More on all that later!

Filtering

Filtering stands for the feature of FFmpeg to modify the decoded frames (audio or video). Other applications might call them effects, but i'm sure there is a reason why FFmpeg calls them filters.

There are two kinds of filtering supported by FFmpeg, simple and complex. In this article we'll only discuss the complex filters, as it is a superset of the simple filters, and this way, we avoid confusion and redundant content.

Simple filters are a single chain of filters between a single input and output. Complex filters can have more chains of filters, with any number of inputs and outputs.

The following figure extends the previous overview image with the filtering module:

A complex filter graph is built from filter chains, which are built from filters.

So a single filter does a single thing, for example, changes the volume. This filter is quite trivial, it has a single input, changes the volume, and it has a single output.

For video, we could check out the scale filter, which is also quite straightforward: it has a single input, scales the incoming frames, and it has a single output too.

You can chain these filters, meaning that you connect the output of one to the input of the next one! So you can have a volume filter after an echo filter, for example, and this way, you'll add echo, and then you change the volume.

This way, your chain will have a single input, and it will do several things with it and will output something at the end.

Now, the 'complex' comes in when you have multiple chains of these filters!

But before we go there, you should also know that some single filters might have multiple inputs or outputs!

For example:

  • The overlay filter puts 2 video streams above each other and will output a single video stream.
  • The split filter splits a single video stream into 2+ video streams (by copying).

So let's discuss a complex example from a bird's eye view! I have two video files, I want to put them above each other, and I want the output in two files/sizes, 720p and 1080p.

Now, that's where complex filtering will be faithful to its name: to achieve this, you'll need several filter chains!

  • Chain 1: [input1.mp4] [input2.mp4] → overlay → split → [overlaid1] [overlaid2]
  • Chain 2: [overlaid1] → scale → [720p_output]
  • Chain 3: [overlaid2] → scale → [1080p_output]

As you see, you can connect chains, and you can connect chains to output files. There is a rule that you can only consume a chain once, and that's why we used split instead of the same input for chains 2 and 3.

The takeaway is this: with complex filter graphs (and mapping), you can:

  • build individual chains of filters
  • connect input files to filter chains
  • connect filter chains to filter chains
  • connect filter chains to output files

FFmpeg's command line system

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

FFmpeg CLI

Finally, we arrived at FFmpeg, and trust me, we'll execute it quite a lot of times! Let's see how FFmpeg's command line options are organized, as that is the first tricky part we need to understand!

FFmpeg mostly thinks about input and output files and their options together with global options. You specify input files with the '-i' flag followed by a file name. For the output file, specify it as-is without any preceding CLI (command line interface) flag.

Specifying an input file

Let's specify just an input file:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 

The following image helps to understand the output:

  1. First, you get the 'banner', where you see the build information and lib versions. If you watch closely, you'll see the compilation flags, starting with --, e.g. --enable-shared.
  2. Then you get the same output as we have seen with ffprobe earlier.
  3. And then you get a complaint that there is no output file(s) specified. That's fine for now.

You can remove the banner here with '-hide_banner', but for brevity's sake I'll not include that anymore in the commands here, and I will leave it out from the outputs too.

Now, let's get brave, and specify an output file!

Specifying an output

As I've said earlier, the output file is understood by FFmpeg as it is just a filename. But more specifically, it is after the input(s) specifications, and it is not a value of any other switches.

Don't be confused for now, but yes, FFmpeg can have as many inputs and outputs as you'd like. We'll cover that in more detail soon!

This command line specifies a single output file:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

Before taking a look at the output, let me congratulate you! You have just converted a video file into an audio file, by keeping just the audio content!

This is how you transcode! Of course, you'll want to specify more parameters later on.

So, here is the output:

Let's analyze it!

(1) First, we have our input metadata printing, which we saw many times already.

(2) Then we have something called 'stream mapping'. We forced FFmpeg into a decision situation, as we specified an input file with 1 video and 2 audio streams. We said we wanted an audio output (guessed from the .wav extension). But we didn't specify which audio stream we wanted, so let's see what FFmpeg decided:

  • 'Stream #0:2' means 'The first input file's third stream' or 'input file index 0's stream with index 2.' This is the input.
  • '-> #0:0' means the first output file's first stream. This is the output.
  • Here you can learn more about how FFmpeg decide this.
  • Later on, we'll manually override the mapping.
  • Summary: FFmpeg decided to convert the third stream in the input file (the ac3 5.1 audio) into the first stream of the output file.

(3) Then we have our output metadata information. This reveals what FFmpeg will output. It usually copies most of the metadata, and here you also see the container/format information too.

(4) And then we see the output summary. For example, the transcoding was 181x faster than the playback speed. Nice!

Understanding the command line order

Before going further, let's understand FFmpeg's command line arguments from a bird's eye view!

In the manual, you'll see this:

ffmpeg [global_options] {[input_file_options] -i input_url} ... {[output_file_options] output_url} ...

(Parts in [...] are meant to be optional, and parts in {...} are meant to be specified 1 or more times.)

This is the general outline of how to specify inputs, outputs, input options, output options, and global options. The order matters, but it is easy to remember: global options, inputs and outputs. Also, i/o options come BEFORE the i/o specification.

Let's put these into pseudo command line options, to understand it better:


ffmpeg -i input1.mp4 output1.wav
ffmpeg -i input1.mp4 -i input2.mp4 output1.wav
ffmpeg -i input1.mp4 -i input2.mp4 output1.wav output2.mp3
ffmpeg [input1 options] -i input1.mp4 [output2 options] output1.wav
ffmpeg [input1 options] -i input1.mp4 \
       [input2 options] -i input2.mp4 \
       [output1 options] output1.wav \
       [output2 options] output2.mp3

As for the global options, these are the ones you might care about:

  • -hide_banner: To skip printing the banner.
  • -y: To overwrite the output even if it exists.

For example, you can run this as many times as you want:

ffmpeg -y -hide_banner -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

And it will overwrite the output and be less verbose than earlier.

Without explaining the options themselves, let's just see some real-world examples with options:

And here it is with two inputs and two outputs:

Mapping files

We saw above that this command:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

... will result in an audio file that contains one of the audio streams from the input video chosen by FFmpeg. This automatic stream selection is usually handy when it is trivial. For example, when you have one stream as input and one output file, you don't need to specify any mapping manually.

But in cases where it is not so trivial, you are usually better off manually specifying what you really want to do.

The following image summarises what our current situation is:

The video stream was not matched, as the output format was an audio file (.wav). But then FFmpeg chose Stream #2, because it has more channels.

So what if we'd like to get the stereo track instead? That is where mapping comes in! The mapping is a parameter of the OUTPUT file. Therefore the mapping arguments should come right before our output file definition!

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav

The argument -map 0:1 means, that in the output (since we specify it as an output option) we'd like to have Input #0's (the first input file) Stream #1!

Let's see the relevant parts from the output!

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':
[...]
Stream mapping:
  Stream #0:1 -> #0:0 (mp3 (mp3float) -> pcm_s16le (native))
  
[...]
Output #0, wav, to 'stereo_audio_only.wav':
  Metadata:
[...]
    Stream #0:0(und): [...] stereo [...]

The 'Stream #0:1 -> #0:0' part means that we have successfully overridden the mapping, to get the mp3 stream (0:1) into our output! Also, the output metadata reveals that we'll get a stereo result instead of the 5.1 earlier.

Multiple outputs

You can have multiple outputs from a single input, let's see when that might be useful!

Let's say, we want to extract BOTH audio streams into two separate WAV files! It is super easy:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav -map 0:2 ac3_audio_only.wav

See? I have just specified two output files with two mapping specifications! Also, I have sneaked in the '-y' to have it overwrite our previous file!

Let's check out the relevant parts of the output!

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':
[...]
Stream mapping:
  Stream #0:1 -> #0:0 (mp3 (mp3float) -> pcm_s16le (native))
  Stream #0:2 -> #1:0 (ac3 (native) -> pcm_s16le (native))
[...]
Output #0, wav, to 'stereo_audio_only.wav':
    Stream #0:0(und): [...] stereo
    
[...]
Output #1, wav, to 'ac3_audio_only.wav':
    Stream #1:0(und): Audio: [...] 5.1(side)

Now the mapping reveals two lines, as we have two outputs! And indeed, you'll get two .wav files as the output, one is stereo, and one is 5.1!

There might be several other reasons why you'd want to get multiple outputs. Let's briefly check out a few!

Different formats:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 stereo_audio_only.wav  stereo_audio_only.mp3 

Wow, did you catch that? We just created a WAV and an mp3 in a single command line! I've reverted to the automatic stream selection for brevity's sake.

A bit closer to real-life needs, you might want different output qualities:

ffmpeg -y  -i bbb_sunflower_1080p_60fps_normal.mp4  \
-map 0:1 -b:a 320k stereo_audio_only_high_quality.mp3 \
-map 0:1 -b:a 64k  stereo_audio_only_low_quality.mp3 

Here -b:a 320k means 'bitrate of audio should be around 320 kbit/sec'. So I have requested FFmpeg to make two mp3s for me, from the stereo stream of the input.

Checking on the files, this is what we got:

 25Mb stereo_audio_only_high_quality.mp3
4,9Mb stereo_audio_only_low_quality.mp3

One more common reason for having multiple outputs or using mapping is when we introduce filters into our pipeline, but that will be discussed later!

Now you understand the foundations of how to communicate your basic requirements to FFmpeg via its command line! Great job! Now we can dive even deepert.

Hands-on with FFmpeg

In this section, we will discover and even try out some common features of FFmpeg!

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

Inputs

Let's see the common ways FFmpeg is fed with different data!

File

Of course, you have already seen that if you have a local file on your filesystem, FFmpeg is happy to read it!

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav

This command which is exactly the same as one of our previous ones just reads a local file. Really, that's it.

Network

Did you know, that FFmpeg can open a file directly on the network?!

ffmpeg -t 5 -i http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4 bbb_first_5_seconds.mp4

The command above opens the file directly from the network and saves the first 5 seconds into a local file!

I wanted to spare bandwidth for these awesome guys over renderfarming.net, so I added the duration flag: -t 5. FFmpeg doesn't even download the full video for this operation. Isn't that wonderful?!

Webcam

FFmpeg can also open your webcam!

This is an example command for Linux:

ffmpeg -f v4l2 -framerate 25 -video_size 640x480 -t 10 -i /dev/video0 10seconds_of_webcam.webm

This would record 10 seconds of your webcam!

Accessing the webcam happens differently on different platforms. Also specifying parameters is different for each platform, so for this reason, if you'd like to access your webcam with FFmpeg, please refer to the documentation:

Microphone

Let's record some audio directly from your microphone!

List microphones:

Start 10 seconds of recording:

ffmpeg -f alsa -i hw:0,0 -t 10 out.wav

This command was meant to work on Linux, but you can check out how to do that on Microsoft Windows or macOS.

Pipe

Finally, FFmpeg can read from a pipe, and also output to a pipe.

On Linux, you could do something like this:

cat bbb_sunflower_1080p_60fps_normal.mp4 | ffmpeg -i - -f wav pipe:1 | pv > output.wav
cat bbb_sunflower_1080p_60fps_normal.mp4 | ffmpeg -i - -f wav pipe:1 > output.wav

This command would use the cat program to simply read in the video file and output it to its standard output. Then this output is piped INTO FFmpeg, through its standard input. The combination '-i -' means 'read from standard input'. By the way, standard input would be your keyboard otherwise, if we wouldn't use any redirection here.

Then we specify the required output format for FFmpeg, with '-f wav'. This is needed because now we'll have no output file name, and FFmpeg will not be able to guess the format. Then we specify 'pipe:1' as an output, meaning we'd like FFmpeg to output to its standard output.

From then, we pipe the data into a program called 'pv', it is just a metering tool, that dumps information on the throughput (from its stdin to its stdout). Finally, we redirect pv's output into a WAV file.

You might ask why we'd want to do that, why we talk about this. Piping can be useful if you build a complex pipeline from different programs or if you want to spare reading and writing to a local file.

For example, the node package fluent-ffmpeg can leverage this functionality by supplying input and output streams. For example, you can read from an S3 bucket and write to one directly.

But be warned, hell is awaiting you on that road. No kidding. You need to research the limitations of this technique. For example, many formats can not be streamed in this manner, as they need random access to the output data to write the indices at the beginning of the file after processing.

Outputs

FFmpeg can output into many protocols, from local file storage and ftp to message queue protocols all the way to streaming protocols.

For more information, check out the documentation here.

Transcoding audio with FFmpeg

In this chapter, we'll be going to see how to transcode into audio with FFmpeg!

The general formula is:

ffmpeg -i {input audio or video file with audio} [output options] output_audio.ext

Choosing a format

FFmpeg is quite smart, and by the extension, it can determine which codec to use. If you specify 'audio.wav' or 'audio.mp3' for example, FFmpeg will use the appropriate codec to do the encoding.

It is perfectly guessing most of the time. But if you want to specify the format manually, then the '-f' flag is your friend.

For this, you might want to consult the list of formats:

So, these three commands will do exactly the same, but the last two requires the -f flag.


ffmpeg -i bbb_audio.wav bbb_audio.mp3
ffmpeg -i bbb_audio.wav -f mp3 bbb_audio
ffmpeg -i bbb_audio.wav -f mp3 pipe:1 > bbb_audio

Setting the bitrate

In most cases. you want to specify the target bitrate you expect from your codec to output. If you are unsure what bitrate is, please read this article's audio bitrate section.

To specify the audio bitrate, use the '-b:a' option with a corresponding value, e.g.:

  • -b:a 320k: For the mp3 codec this is considered high quality.
  • -b:a 128k: Lower quality.
  • -b:a 64k: Low quality.

For example:

ffmpeg -i bbb_audio.wav -b:a 320k bbb_audio_320k.mp3

Setting the sample rate

You may want to specify the sample rate to ensure quality or low output file size. Half the sample rate could mean half the output file size. If you are unsure what the sample rate is, please read the 'audio sample rate' section of this article.

To specify the audio sample rate, use the '-ar' option with a corresponding value, e.g.:

  • -ar 48000: For high quality.
  • -ar 44100: For CD quality (still high).
  • -ar 22500: A bit of a compromise, not recommended for music, but for speech, it might be enough.
  • -ar 8000: Low quality, e.g. if you only want 'understandable' speech.

For example:

ffmpeg -i bbb_audio.wav -ar 44100 bbb_audio_44100khz.mp3

Setting the channel count

Setting the channel count can be useful, for example, if you have a stereo recording of a single person's speech. In that case, you might be content with just a mono output half the size of the original recording.

If you are unsure what an audio channel is, please read the 'audio channels' section of this article.

To specify the channel count use the '-ac' option with a corresponding value, e.g.:

  • -ac 1: For mono
  • -ac 2: For stereo
  • -ac 6: For 5.1

For example:

ffmpeg -i bbb_audio.wav -ac 1 bbb_audio_mono.mp3

Complete command line for converting audio with FFmpeg

This is how you produce a high-quality output:


ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.mp3
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.m4a
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.ogg

Check out this documentation about good quality audio transcoding too!.

Lossless formats

If you want to convert audio into a lossless format, here are a few choices for you:


ffmpeg -i bbb_audio.wav -compression_level 12 bbb_audio_lossless_12.flac 
ffmpeg -i bbb_audio.wav -compression_level 5 bbb_audio_lossless_5.flac   
ffmpeg -i bbb_audio.wav -compression_level 0 bbb_audio_lossless_0.flac   
cp bbb_audio.wav bbb_audio_lossless.wav 
ffmpeg -i any_audio.ext bbb_audio_lossless.wav

It's good if you know that flac results in a smaller file than WAV, as WAV doesn't actually compress by default:

117M bbb_audio.wav
52M  bbb_audio_lossless_0.flac
45M  bbb_audio_lossless_5.flac
43M  bbb_audio_lossless_12.flac

WAV is generally thought of as a lossless format, but keep in mind that the WAV container can contain lossy content too, but by default FFmpeg uses the pcm_s16le format, which is the 16 bit PCM, that could be understood as lossless.

Learn more here and here.

Transcoding video with FFmpeg

In this chapter, we'll be going to see how to transcode a video file into the two most common formats!

Converting to H.264

H264 is one of the most popular video codecs. Most devices, browsers and video players understand how to play it. It is efficient in storing video content, but as with most advanced video codecs, it is a resource intensive-process to encode and decode.

A complete command line for a high-quality H.264 transcoding with high-quality AAC audio is the following:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 \
-c:v libx264 -preset slow -crf 22 \
-profile:v main -g 250 -pix_fmt yuv420p \
-map 0:0 -map 0:1 \
-acodec aac -ar 44100 -b:a 320k bbb_transcoded_h264_HQ.mov

Make sure to understand this command and to customize it to match your needs.

To help you do that, let's dissect this command!

Global options:

  • -y: Overwrite the output.

Input options:

  • -i bbb_sunflower_1080p_60fps_normal.mp4: The input file.

Output options:

-c:v libx264: Set the codec to libx264.

-preset slow: libx264 has a lot of variables that you can be tune, and most of them balance the coding speed and the resulting file size. To make your life easier, there are presets by which you can easily declare what you need: small size or speed.

-crf 22: This is the constant rate factor, the main option for setting image quality. It is a number between 0-51, where 0 is lossless, and 51 is the worst quality. Generally, you want something between 17 and 28. This is the option to tune the balance between image quality and file size. Check my comparison video here.

-profile:v main -g 250 -pix_fmt yuv420p: These are advanced options, guaranteeing you a quite backward compatible result. (See this, this, and this.)

-map 0:0 -map 0:1: You might not need this: these options are selecting the correct video and audio streams. In our case, we have two audio streams, and we need the stereo one to avoid some issues with our aac stream.

-acodec aac: Select the AAC (Advanced Audio Coding) codec for the audio in the output. We need to be more specific than just -f for the format. We need to specify the audio codec here manually.

-ar 44100: Set the audio sampling rate (learn more about that in previous chapters of this article).

-b:a 320k: Set the audio bitrate (learn more about that in previous chapters of this article).

30seconds_of_bb.mkv: The output file name. All the options since the last -i (or the last output file) considered to be a modifier for this output.

Let's see the output:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':
[...]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (mp3 (mp3float) -> aac (native))
[...]
Output #0, mov, to 'bbb_transcoded_h264_HQ.mov':
    Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 60 fps, 15360 tbn, 60 tbc (default)
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, 5.1(side), fltp, 320 kb/s (default)
[...]
frame=38074 fps= 35 q=-1.0 Lsize=  324855kB time=00:10:34.51 bitrate=4194.1kbits/s dup=2 drop=0 speed=0.58x 

From this, we understand that FFmpeg chose the mp3 stream from the input file because we told it to do so. (Remember, it has two audio streams in it, a stereo mp3 and a 5.1 ac3.) We also see that my machine could transcode with 35fps (0.58 times the playback speed), and our settings resulted in an average video bitrate of 4200 kbit/s.

The video bitrate is an interesting question in this mode. With the CRF option, we specify the 'constant visual quality' we want. To reach a constant visual quality, the encoder works hard to guess how much it can compress certain parts of every frame, and the result of that guess defines the final average video bitrate.

If you want even better results with H.264, and you can afford a bit more processing time and a bit more complicated process, check out the 2-pass encoding instead of the constant rate factor method introduced above.

To learn more about these two different rate control methods, read the awesome Understanding Rate Control Modes article. And to learn more about the intricacies of H.264 encoding, check out the H264 encoding guide.

Finally, later on, I will show you a comparison video that shows how different CRF values perform!

Converting to H.265

H.265 is the successor of H.264, according to the official FFmpeg manual, it offers 25-50% bitrate savings while retaining the same visual quality.

A complete command line for a high-quality H.265 transcoding with high-quality AAC audio is the following:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 \
-c:v libx265 -preset slow -crf 27 \
-profile:v main -g 250 -pix_fmt yuv420p \
-map 0:0 -map 0:1 \
-acodec aac -ar 44100 -b:a 320k bbb_transcoded_h265_HQ.mov

And the result is:

...
encoded 38074 frames in 3384.84s (11.25 fps), 1720.32 kb/s, Avg QP:35.29

H.265 also has multiple rate control algorithms, I used the CRF method here. If you want to use a different rate control algorithm, then you may check out the H.265 encoding guide. Also, check out the next section, where I'll reveal how different CRF values perform!

This command is almost the same as what we used in the H.264 example above, so please refer to that section to understand the arguments.

If we compare H.264 and H.265 with our commands above, taking into account this 10-minute long video on my system, these are the results:

  • H.264 is 3 times faster (35 fps vs 11 fps)
  • H.264 produces a 2 times larger file (318 mb vs 156 mb)

Comparing CRF values with H.264 and H.265

I have created a video for your convenience, that shows the different crf values in action. The selected frame had some movement on it with the leaves in the bunny's hand. Movement is important with video codecs, as usually that's where quality losses are first visible.

This video shows how the different CRF values perform, from 0-51 with the H.264 and H.265 formats!

H.264 & H.265 CRF comparison video

(Can you guess which program I was using to make this?:))

Basic editing with FFmpeg

In this section, we'll achieve basic editing tasks by using FFmpeg only!

We'll just get a basic mp4 with default settings in these examples to keep things simple. But to encode the result in a proper, high quality way, please check the earlier sections where we learned how to encode into H.264 and H.265!

Trimming from the beginning of the clip

It is possible to specify an in-point for a media file. By doing that, you essentially cut off the specified amount from the beginning of the input file. Therefore, FFmpeg will skip the first part of the file and only transcode the remainder!

For this, you need the '-ss' flag! The value can be specified in seconds (5 or 5.2) or as a timestamp (HOURS:MM:SS.MILLISECONDS).

To get the outro only, we could seek all the way to the end of the video! (It is 00:10:34.53 or 635 seconds long!)


ffmpeg -y -ss 631 -i bbb_sunflower_1080p_60fps_normal.mp4 last_4_seconds.mp4
ffmpeg -y -ss 00:10:30.53 -i bbb_sunflower_1080p_60fps_normal.mp4 last_4_seconds.mp4

Seeking can be a bit tricky, so you may want to learn more about seeking here.

Trimming from the end of the clip

You can also set an out-point for an input file, therefore shortening it. There are two options for this:

  • -t: This sets the duration.
  • -to: This sets the timestamp where the input video should stop.

These two are mutually exclusive, and also they do the same if no -ss is specified. The value can be specified in seconds (5 or 5.2) or as a timestamp (HOURS:MM:SS.MILLISECONDS).

Let's experiment with them!


ffmpeg -y -t 30 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4
ffmpeg -y -t 00:00:30.0 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4
ffmpeg -y -to 30 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4
ffmpeg -y -to 00:00:30.0 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4

All four above commands result in exactly the same video. (For nerds: even the md5sum is the same.)

But let's see how they perform when we introduce seeking!


ffmpeg -y -ss 10 -t 30 -i bbb_sunflower_1080p_60fps_normal.mp4 part_between_10_and_40.mp4
ffmpeg -y -ss 10 -to 30 -i bbb_sunflower_1080p_60fps_normal.mp4 part_between_10_and_30.mp4

The first command will result in a 30 second long video, while the second command will be 20 seconds long only!

The figure below shows the difference:

Editing without reencoding

FFmpeg can do something I'm not aware of in any other popular NLE: it can edit videos without reencoding them!

The usual workflow is to decode the data frames (a/v) into memory, modify them as much as we like and then encode them into a new video file. The problem with this is that unless you work with raw or lossless codecs, you'll lose some quality in the process. Another issue with this approach is that it is computationally intensive.

For certain operations, you can configure FFmpeg, to keep the data frames intact, and this way, you can avoid decoding and encoding them! This is incredibly faster than regular transcoding, usually hundreds of times faster.

The 'certain operations' are those that don't need to modify the data frames themselves. For example, you can cut and trim this way. Also, you can manipulate streams while keeping others, like you can replace the audio track without touching the video frames.

All this is a bit of magic, and there are caveats you need to prepare for, but it is good if you know about this, as it is often handy!

The trick lies in two options:

  • -c:v copy: The 'copy' video codec
  • -c:a copy: The 'copy' audio codec

Let's see a few examples!

Remove audio while keeping the video without reencoding

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -c:v copy -an copied_video_only.mp4

Here, we used the '-an' option, which removes all audio streams. I remembered it as 'audio no', but that is just my mnemonic:)

Let's see how fast it was:

frame=38072 fps=20950 q=-1.0 Lsize=  310340kB time=00:10:34.51 bitrate=4006.7kbits/s speed= 349x

So It processed the whole 10 minutes of video in 2 seconds, 349x faster than playback, with 20950 fps!

Remove video while keeping the audio without reencoding

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -c:a copy -vn copied_audio_only.wav

Here, we used the '-vn' option, which removes all video streams. I remembered it as 'video no'.

Let's see how fast it was:

size=   24772kB time=00:10:34.14 bitrate= 320.0kbits/s speed= 776x 

776x faster than playback, finished in about a second, not bad!

Cut and trim without reencoding

ffmpeg -ss 10 -t 10  -i bbb_sunflower_1080p_60fps_normal.mp4 -c:a copy -c:v copy part_from_10_to_20_copied.mp4

There could be precision issues with seeking while you do this, so you may want to learn more about seeking and copying here.

Replace audio on video file without reencoding

We have removed audio and video already, but what if we want to swap them?

ffmpeg -y \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-i voice_recording.wav \
-map '0:v' -map '1:a' \
-c:v copy -c:a copy \
bbb_with_replaced_audio.mov

There is quite a lot going on in here, so let's explain the parts!

First, we have two inputs (-i), meaning we are better off manually specifying the mapping. The command would work without the '-map' options, but it would ignore our second input.

-map '0:v' -map '1:a' means that please use the first file's (first) video stream and the second file's (first) audio stream.

With -c:v copy -c:a copy, we require FFmpeg to copy the already encoded data packets without touching them. Therefore FFmpeg's work is mostly really just copying bytes, no decoding, no encoding.

Not surprisingly, that's what we see in the stream mapping too:

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=38072 fps=9750 q=-1.0 Lsize=  320645kB time=00:10:34.51 bitrate=4139.7kbits/s speed= 162x  

And since it is just copying, it was crazy fast, 162x of the playback speed, or almost 10k frames per second!

But!

Execute the exact same command, but with 'bbb_with_replaced_audio.mp4' (.mp4 container instead of .mov) as an output file! You'll get this:

Could not find tag for codec pcm_s16le in stream #1, codec not currently supported in container

The message is quite clear. You can not have a pcm_s16le (raw WAV, say that 10 times:)) stream in an MP4 container. I'm not sure if it is FFmpeg's or the container's lack of support, but we need to solve this. If you run into this situation, you might consider two solutions:

  1. Change the container: I've just tried MOV, and it worked.
  2. Encode the audio: We still copy the video data, and encoding audio isn't that painful.

I just showed you option #1, so let's see option #2:

ffmpeg -y \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-i voice_recording.wav \
-map '0:v' -map '1:a' \
-c:v copy \
-c:a aac -b:a 320k -ar 44100 \
bbb_with_replaced_audio_aac.mp4

This copies the video frames and encodes our WAV into a supported codec to be held in the mp4 container. You can refer back to the audio encoding section if you want to learn more about that.

Here is the output:

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Press [q] to stop, [?] for help
...
frame=38072 fps=2176 q=-1.0 Lsize=  313058kB time=00:10:34.51 bitrate=4041.8kbits/s speed=36.3x 

'Only' 36x faster than playback, 2176 fps, still not that bad!

Filtering overview

FFmpeg supports many audio and video filters. Currently, there are 116 audio and 286 video filters, but there are a bit more if we count the hardware accelerated ones too.

So how do we leverage them?

There are two ways to define filters, but I'm going to explain the complex filter, as the difference is not much, but it is more versatile. So there is a global option for FFmpeg, called: -filter_complex. With quite a weird syntax, you can specify all your filters and their parameters right after this option.

You can imagine the process with the following image:

Basically, your filter graph can access all the inputs (-i a.mp4 -i b.mp4 -i c.mp4), and it can produce as many outputs as you like (-map might be needed).

Basic syntax

Let's take a look at a simple, basic example:

ffmpeg -y  -t 5 \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'drawtext=text='HELLO THERE':y=20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf' \
filter_complex1.mp4

Although -filter_complex is a global option, I like to put it after the inputs and before the outputs as it is a bit easier to overlook the whole command that way. Thankfully the command line parser of FFmpeg is smart enough, and it works.

The command above produces a 5-second-long video, where the text 'HELLO THERE' is overlaid on the intro of Big Buck Bunny.

Let's understand the weird format for specifying filters!

We'll go bottom-up, and we'll build it from there. So the most basic format is this:

FILTER_NAME=ARGUMENT1=VALUE1:ARGUMENT2=VALUE2

For example:

drawtext=text='HELLO THERE':y=20:x=30

The first thing before the first equal (=) sign is the filter's name, which is the drawtext filter in this case. Then we have our first argument, 'text' and its value ''HELLO THERE''. Right after that, separated with a colon (:) comes the next argument, 'y' with a value of '20'.

You can guess what each of the text, y, x, fontsize and fontfile arguments do, as it is quite self-explaining. But especially for the first time, you'll heavily rely on the filtering documentation to understand every filter and every argument.

Also, several characters are reserved, such as: , : = and a few others depending on your environment, so sooner or later you need to learn about escaping too.

To recap, our pipeline looks like this now:

Multiple filters in a chain

This previous command is a single filter chain that consists of a single filter only, but you could have more filters put right after each other! It means that the output of one filter will be the input for the next! The way to do this is by separating them with a comma!

Let's draw two boxes with the drawbox filter!

ffmpeg -y  -t 5 \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex '  drawbox=x=10:y=10:w=100:h=100:color=red  ,  drawbox=x=200:y=200:w=100:h=100:color=blue  ' \
filter_complex2.mp4

See? The output of the first filter is passed to the output of the second filter!

Let's visualize our pipeline again:

Input and output pads

Now, we have skipped something this far, because for simple uses FFmpeg is smart enough to do it for us. And this is the specification of a chain's input and output pads!

Let's draw just a single rectangle for now:

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex 'drawbox=x=10:y=10:w=100:h=100:color=red' filter_complex3.mp4

FFmpeg sees that the input for our filter chain is a single video file, and the output is a single output video file. Therefore, it safely assumes that we want that single input as the input of our single filter chain. And that single output should be the single output of our single output chain.

That's really nice, as, in simple situations like this, we don't need to assign and map inputs and outputs manually! But when we get more inputs, filter chains, or outputs, it is no longer possible. Therefore, we need to understand how to assign inputs and outputs!

First of all, let's compare the following two command lines. They result in exactly the same result, but the second one represents what FFmpeg does internally (roughly):

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex 'drawbox=x=10:y=10:w=100:h=100:color=red' filter_complex3.mp4
ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex '[0:v]drawbox=x=10:y=10:w=100:h=100:color=red[out_link_0]' -map '[out_link_0]' filter_complex3.mp4

Do you see the difference? Before our filter chain, an 'input pad' is defined: [0:v]. The expected format between the square brackets is documented in the stream selection section of the official documentation, and this article already covered it.

But, a quick summary:

  • 0:v: This means the first video stream of the first input file.
  • 0:v:0: Means exactly the same thing but in a long form.
  • 0:0: Means the first stream of the first input file (not recommended, as it could be anything in theory. It could be a subtitle stream, a thumbnail, a video or an audio stream...)
  • 0:a: This means the first audio stream of the first input file.
  • 0:a:0: Means exactly the same thing but in a long form.
  • 0:a:1: Means the second (index #1) audio stream of the first input file.

So we can specify which input file should be connected to which input of the filter graph!

Also, something similar is going on at the end! Do you see, the [out_link_0] output pad definition at the end of our filter chain?

The naming here is easier, as basically you can specify any arbitrary name in here. It roughly means, 'please store the output data under this name'.

And when you specify your output file, you can or need to map it by selecting one of your filter graph outputs! Therefore, we must add the -map '[out_link_0]' option before our output file.

This map option means this: 'Please save the data stream with this name into the following output file.'

This is how you can visualize this input/output mapping:

Multiple chains

Coming from the previous sections, you are now ready to see and understand an even more complicated configuration, which has multiple input files, output files, and filter chains!

ffmpeg -y  \
-i train.jpg \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex '[0:v]drawbox=x=10:y=10:w=100:h=100:color=red[train_box] ; [1:v]drawbox=x=10:y=10:w=100:h=100:color=red[bbb_box]' \
-map '[train_box]' filter_complex4_train.jpg \
-map '[bbb_box]' filter_complex4_bbb.mp4

Let's see the output (two files next to each other):

We had two inputs, and we got two output files, an image, and a video, with a red rectangle on them, with a single command!

Are you still here? I hope! Let's understand what happened in that crazy command! We have two input files:

  • -i train.jpg: A simple image file
  • -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4: Our video file, but to make it quick, just the first five seconds of it

Then the first thing to note is that we have two filter chains! They are separated with a ';'.

Our first filter graph is this: [0:v]...[train_box]

  • This requests the first input file as an input
  • Draws a red box
  • Saves the output into the 'train_box' output pad

Our second filter graph is this: [1:v]...[bbb_box]

  • This requests the second input file as an input
  • Draws a red box
  • Saves the output into the 'bbb_box' output pad

And finally, we got two outputs, each mapping to one of the outputs of the filter graph:

  • -map '[train_box]' filter_complex4_train.jpg
  • -map '[bbb_box]' filter_complex4_bbb.mp4

Here is the same thing visually:

If you are thinking about making it even more complex and making filter graphs that combine multiple inputs into one for example, you are on the right track! It is possible, and we will get to that!

This was the introduction to the filtering system and its syntax.

Editing video

Now let's get to know a few filters and make some interesting stuff!

Resizing or scaling

The scale filter is a simple one, yet it is quite powerful!

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'scale=width=600:height=-1:force_original_aspect_ratio=decrease' \
filter_complex5_scaled1.mp4

The arguments speak for themselves, but a few things:

  • Specifying -1 to either width or height means rescaling while keeping the aspect ratio.
  • 'force_original_aspect_ratio' can be increase, decrease. Meaning it will increase or decrease the image to fit the specified bounding box while keeping the aspect ratio.

Adding text

We have already covered this a little, so let's dive deeper!

This is what we used earlier:

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'drawtext=text='HELLO THERE':y=20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf' \
filter_complex1.mp4

Now let's discover how to align the text!

Many filters, including drawtext, support variables in some of its argument's values. If you scroll down in the documentation of drawtext, you'll find this:

'The parameters for x and y are expressions containing the following constants and functions: '

And after this part, you'll see many variables which you can include in your x and y variables!

Let's see:


ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'drawtext=text='HELLO THERE':y=h/2-text_h/2:x=w/2-text_w/2:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf' \
filter_complex6_center.mp4
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'drawtext=text='HELLO THERE':y=30:x=w-text_w-20:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf' \
filter_complex6_right.mp4
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'drawtext=text='HELLO THERE':y=h-text_h-20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf' \
filter_complex6_bottom.mp4

And this is what we'll get in the end:

I need to mention one good trick that might not be obvious at first. So the text_h variable is a tricky one, because different text will be of different height! E.g.: '____' and 'WWW' will result in a different height.

For this reason, you do not always want to use text_h or even just a constant y=value expression but rather, you need to align text by its baseline. So just remember to use the 'ascent' variable whenever you need to align text vertically!

Check out these two examples! Each has two drawtext filters printing '_' and '_H':


ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'drawtext=text='_':y=200:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf,drawtext=text='_H':y=200:x=500:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf' \
filter_complex7_bad_text.mp4
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex 'drawtext=text='_':y=200-ascent:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf,drawtext=text='_H':y=200-ascent:x=500:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf' \
filter_complex7_good_text.mp4

Now let's compare the difference:

See? This is the difference between aligning the 'top left' or the 'baseline' of the text!

Adding an overlay

Overlaying is a very interesting thing to do with FFmpeg. Let's jump right in!

Basic

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex 'overlay' \
filter_complex8_overlay1.mp4

Easy as that!

Of course, the overlay filter has a ton of options, but I wanted to demonstrate the easiest possible command line. We don't even need to mess with input/output pads, as FFmpeg automatically understands the situation: two inputs for the overlay filter and its single output into a single output.

But just to exercise, we could have executed it like this:

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex '[0:v][1:v]overlay[output]' \
-map '[output]' filter_complex8_overlay2.mp4

And this would result in the same output! Check it out, now I have specified the two inputs for the overlay: [0:v][1:v]!

Aligned

Let's align the smiley into the center!

As we have seen with the drawtext, the overlay filter's arguments also support a few dynamic variables. We'll use those to achieve what we want!

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex 'overlay=x=main_w/2-overlay_w/2:y=main_h/2-overlay_h/2' \
filter_complex8_overlay3.mp4

Preprocessing the input for overlay

Let's get a bit creative!

I want to make it smaller, and I also want to blur it!

Now pause for a minute, and think about it, how you'd do that?!

...

Ready?

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex '[1:v]scale=w=200:h=-1,gblur=sigma=3[smiley] ; [0:v][smiley]overlay=x=100:y=100' \
filter_complex8_overlay4.mp4

For this we needed to have two filter graphs!

The first one is this: [1:v]scale=w=200:h=-1,gblur=sigma=3[smiley]

  • Scales the input image (the smiley).
  • Then the scaled output is also blurred.
  • Then the output is saved into the output pad named 'smiley'.

Then, we have our second filter graph: [0:v][smiley]overlay=x=100:y=100

  • This takes as input the first input file (the video).
  • This also takes as input the output pad named 'smiley'. (We are connecting two chains this time!)
  • Then the overlay filter does its overlaying thing, and we trust FFmpeg to pair the unnamed output with the single output file we specified.

Reusing content

Let's do one more, a really complicated one!

Let's have the outro overlaid over the intro!

ffmpeg -y \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-t 5 -ss 00:09:40 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-filter_complex ' [1:v]scale=w=1920/2:h=-1[outro]; [0:v][outro]overlay' \
filter_complex8_overlay5.mp4

We could have achieved it in several ways, e.g. we could use the trim filter, but to keep it easy, we just open the same file twice and seek/trim them.

  • -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4: Open the video, and keep the first five seconds of it.
  • -t 5 -ss 00:09:40 -i bbb_sunflower_1080p_60fps_normal.mp4: Open the same video again, but seek to the end and keep five seconds from there.

Then we have two filter graphs again, one scales down the outro, and the second is just an overlay.

Are you excited?:) I hope these made-up examples opened up your eye for the possibilities, and I hope you'll create very creative stuff with this knowledge!

Chroma keying, green screen, blue screen

In this section, we'll use chroma keying to remove the background from Big Buck Bunny's intro, and then we will put the transparent logo over the original video, as if it would be some kind of a logo overlay!

ffmpeg -y \
-ss 0.5 -t 2 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-ss 10 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-filter_complex ' [0:v]chromakey=color=0xfdfdfd:similarity=0.1:blend=0.2 , scale=w=-1:h=300 , loop=loop=-1:start=0:size=120[intro] ; [1:v][intro]overlay=x=-40:y=-40' \
-t 10 filter_complex9.mp4

So just to recap, Big Buck Bunny's first few seconds are like this:

And this is the result:

Also, the butterfly moves its wings repeatedly!

Let's examine the command!

  • -ss 0.5 -t 2 -i bbb_sunflower_1080p_60fps_normal.mp4: We read in the intro from 0.5 to 2.5 seconds.
  • -ss 10 -i bbb_sunflower_1080p_60fps_normal.mp4: We read in the video, starting from the 10th second.

Then we have two filter graphs, the first being this:

[0:v]chromakey=color=0xfdfdfd:similarity=0.1:blend=0.2 , scale=w=-1:h=300 , loop=loop=-1:start=0:size=120[intro]

As we see, we have three filters in here!

  • chromakey: This one takes a color and a few parameters as input, and outputs transparent frames. The specified color + the blended areas will be the transparent sections. In our case we replaced the white-ish (#fdfdfd) background color with transparency.
  • scale: We resize the full 1080p image into something around 300px high.
  • loop: With the loop filter, we repeat all the 2 seconds worth of 120 frames (60*2) over and over again, to have the butterfly move its wings continuously.

And then, finally we have the second filter graph:

[1:v][intro]overlay=x=-40:y=-40

Nothing fancy, just an overlay of the original video and our chrome keyed intro.

What else?

You might want to check out a few more filters, that I didn't cover here.

Here are just a few interesting ones:

Audio manipulation

In this chapter, we'll be going to check out some audio manipulation techniques with FFmpeg!

First of all, let's see our example file:

It is a voice recording, and it is intentionally... well, quite bad.

From the waveform, it is obvious that there are very different volume ranges in it. This is an example recording where each sentence was read in different strengths: 'normal', 'whisper' or 'powerful', this is why you see repeating patterns of amplitude ranges on the image.

It isn't visible, but it has some noise too, and of course, it is not normalized or enhanced in any way. Yet.

Please note that there are different scenarios, requirements, and ways to enhance audio. This is a simplified method to show the outline of the process in this article. I'm not an audio engineer, although I have some experience in the area. So if you know it better, feel free to fine-tune it for yourself even more, or contact me and recommend improvements!

I'm showing an example here with a very rough input, one that you'd just reject in real life as it would be useless due to its quality. But it is an excellent example to show the different steps of the enhancing process and to see what can be done to it!

The following steps are built upon each other, and we'll reach the complete command at the end!

Don't forget that these settings are specific to this voice recording. Sadly this can not be generalized too much.

Gate

Let's start with the gate filter!

A gate is like a switch that opens only if the signal is stronger than the threshold. So if the signal level is lower than the threshold, it cuts to complete silence. Although you might soften or delay this cut with the knee, attack, and release arguments.

We'll use this filter as a basic noise reduction method now! This helps us remove the noise between words and sentences by cutting it to silence. It doesn't remove noise in any other way, e.g. it doesn't touch the static on the voice itself.

Check this out!

ffmpeg -y \
-i voice_recording.wav \
-filter_complex 'agate=threshold=0.01:attack=80:release=840:makeup=1:ratio=3:knee=8' \
gate.wav

Let's hear it: gate.wav

And let's see it:

As you can see, the 'silent' parts were attenuated heavily, while the above-the-threshold parts remained similar. Those parts were still affected by the knee, attack, and release arguments determining how hard (knee) and quick (attack/release) the cut is.

I've left a quite high release timeout here to avoid sudden dips in the amplitude.

This is where we are right now:

The silent parts are more silent than before, but still, the amplitude range or the dynamic range is quite high. You must change your volume levels to hear everything and void blowing your speakers/brain out.

Equalization

Before fixing that, let's do a bit more housekeeping. Let's do some equalization and frequency filtering!

We'll use these filters:

ffmpeg -y \
-i gate.wav  \
-filter_complex 'highpass=f=100:width_type=q:width=0.5 , lowpass=f=10000 , anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1' \
gate_eq.wav

Let's hear it: gate_eq.wav

This command gradually attenuates frequencies below 100hz, as there are not much valuable content in there, but it can really lower the clarity of the speech.

Then we do the same, but for frequencies above 10 kHz. This is mostly needed because we have a lot of high-frequency noise, so this is a workaround for those. Also, a male voice is generally deeper than a woman's, so you might want to pay attention to how low you can put the bar.

Then comes anequalizer, which has a crazy an exceptional way of setting its arguments:

This: anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1 means:

  • at 250hz with a width of 100hz boost by 2 db, with Chebyshev type 1 filter on channel 0.
  • at 700hz with a width of 500hz attenuate by 5 db, with Chebyshev type 1 filter on channel 0.
  • at 2000hz with a width of 1000hz attenuate by 2 db, with Chebyshev type 1 filter on channel 0.

I agree. You might have used a friendlier equalizer in your life than this one:)

Those values are based on experimentation and common recommendations for voice. Feel free to tune it for your own needs!

Let's compare the frequency plots before and after:

Tip: To see the frequency plot in Audacity, open a file, select all, and choose Analyze → Plot spectrum!

Compression

The compressor filter applies dynamic range compression on the incoming audio data. To simplify this, the compressor varies the attenuation based on the incoming signal level. Basically, when you watch a badly mastered movie, this is what you are doing. When it is way too loud in some action scene, you reach for the remote control or mouse to lower the volume, but in the next moment, you will not hear what your heroes are saying, so you increase it back again.

Dynamic range compression roughly does the same. You may set it up in a way so that it would attenuate louder parts, therefore keeping the overall volume range relatively small.

It often happens that performers on the stage use a high dynamic range. Many performers will shout at one moment and then whisper in the next to increase drama or keep the attention. If you want to avoid manually adjusting the volume in real-time (while blowing off your speakers and pulling your hair out), then a compressor will save you in these situations!

This is why our example audio consists of different speaking strengths, so that we could see the dramatic effect of this filter.

ffmpeg -y \
-i gate_eq.wav \
-filter_complex 'acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6' \
gate_eq_comp.wav

Let's hear it: gate_eq_comp.wav

And let's compare the result of this with the original waveform!

Original:

Result:

Quite dramatic, isn't it?:)

Let's analyze this: acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6

First, level_in=6 sets the input gain. It is 1 by default, but since our example, audio is extremely silent at places, we boost up the whole thing before processing.

Then threshold=0.025 defines that everything above 0.025 should be attenuated.

Based on the image below, I've decided to cut at this point, as this is above most of the whispering, which cuts hard pops and 's'-es even in the 'whisper zone'.

Then ratio=20 means 1:20 in attenuation ratio, which means that if the level rises 20 dB above the threshold, it will be only 1 dB above the line after the attenuation. Basically, this is a very strong compression ratio, it is almost a limiter.

This far, we boosted the signal, then turned down everything that was above our 'whisper line' with a quite strong ratio, and now, everything is basically at the whisper level, even the parts that are shouting.

Finally, with the makeup=6 we just bring back everything to the level where the 'normal' parts were before.

Let's take a look back now, to understand why we used the gate and did the equalization before the compressor.

Generally, you want to remove unneeded parts and frequencies before compression, as the compressor will likely increase those too! So by removing most of the noise in the gaps, we avoided level_in=6 to increase them too! And the same goes for the high- and lowpass filtering.

Changing the volume

Now, if we want to make the result a bit louder, we could increase the previous step's makeup argument, or leverage the volume filter.

While we are at it, let's cut the first 4 seconds too with -ss 4.

ffmpeg -y \
-ss 4 -i gate_eq_comp.wav \
-filter_complex 'volume=1.1' \
gate_eq_volume_comp.wav

Let's hear it: gate_eq_volume_comp.wav

Let's make audio gate again

Excuse me for that title:)

So as I've described earlier, compression can amplify the noises, so you might want to run the result through a gate again:

ffmpeg -y \
-i gate_eq_volume_comp.wav \
-filter_complex 'agate=threshold=0.1:attack=50:release=50:ratio=1.5:knee=4' \
gate_eq_volume_comp_gate.wav

Let's hear it: gate_eq_volume_comp_gate.wav

In this case, I've used a softer gate, with ratio=1.5. Because of this, I could use shorter attack and release delays too, as the attenuation is not that strong, it isn't causing hard dips in the audio.

Putting it all together

Just a single command could have achieved all the steps above:

ffmpeg -y \
-i voice_recording.wav \
-filter_complex 'agate=threshold=0.01:attack=80:release=840:makeup=1:ratio=3:knee=8 , highpass=f=100:width_type=q:width=0.5 , lowpass=f=10000 , anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1 , acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6 , volume=1.1 , agate=threshold=0.1:attack=50:release=50:ratio=1.5:knee=4' \
gate_eq_volume_comp_gate_together.wav

I just copy-pasted all the filters right after each other with a comma between them.

Isn't it beautiful? Yeah, it isn't, but it is very practical:)

For the last time, check out the difference:

It has less noise, more clear voice, and a small volume range. Therefore it is easy on your ears!

What else?

You might want to check out a few more filters that I didn't cover here.

Here are just a few interesting ones:

Documentation

For your convenience, let me list the most important documentations that might be important for you! Most of these were already linked many times in this article.

If you got this far from top to bottom, then you are a true hero! I hope you enjoyed this, and I also hope that it inspired you to create something awesome with FFmpeg!

Please consider donating to FFmpeg they are awesome.




All Comments: [-] | anchor

scoutt(10000) 5 days ago [-]

A guide like this is always welcome. My biggest struggle with FFmpeg was to find clear documentation, tutorials and examples about API usage. All the guides out there (including this one) is for the command line version, or if the tutorial focuses on the API, it's probably outdated. Every time I have to go back and do something with FFMPEG I have to just plainly read the code of FFPlay to 'catch-up' with the workflow.

And don't get me started on that time I had modify the FFmpeg libs to provide support and implement an H264 encode/decoder on a custom architecture!!

onion2k(10000) 5 days ago [-]

My biggest struggle with FFmpeg was to find clear documentation...

I've found the ffmpeg documentation does a good job as a technical reference. The problem is that most people reading it are actually expecting a 'what options will fix my video' how-to guide.

blueflow(10000) 5 days ago [-]

This is a brilliant way for people to avoid reading the official ffmpeg manpages.

moffkalast(10000) 5 days ago [-]

One day the imperium of man will be finally defeated.

ranting-moth(10000) 5 days ago [-]

Not to criticize the guide, it's brilliant! If asked for improvements, I'd suggest adding a table of contents.

Slightly off topic, but the guide does suggests reading time. It's interesting that people keep using read estimates for technical/scientific/professional documentation. This one says it's a 58 minute read. Not 1 hour, not 59 minutes, but spot on 58 minutes.

Now, I'm not a novice in using ffmpeg and I think I'm at least an average person. I can tell you that it would take me a _lot_ longer to read this guide in a meaningful manner.

But, a brilliant guide. I'm definitely going to use it to expand my ffmpeg knowledge.

christiangenco(10000) 5 days ago [-]

Perhaps you'd like a word count instead?

I believe the way all of these reading time estimates are calculated is a simple `wordCount / averageWordsPerMinute`.

madeofpalk(10000) 5 days ago [-]

Estimate reading time from words per minute. Here's an NPM package that'll do it for you! https://www.npmjs.com/package/reading-time

flumpcakes(10000) 5 days ago [-]

There kind-of is a table of contents.

If you're on a desktop computer then the top right has a floating grey 'On this Page' which lists the section headers.

There's no numbering or sigils, which might make it hard to use effectively as a TOC.

llampx(10000) 5 days ago [-]

Its reading time, not understanding time :P

tambourine_man(10000) 5 days ago [-]

Your criticism seem extremely polite for a ranting moth :)

zerop(10000) 5 days ago [-]

Is it possible to have FFmpeg in browser (A JS port)?

xenodium(10000) 5 days ago [-]

While, I've been collecting useful one-liners for some time, there was always friction to apply them: search my notes, copy, paste, edit, etc.

As of recently, I finally landed on a solution I feel very comfortable with:

- Write templated one-liners (could be longer scripts too), automatically replacing which files to apply them to.

- Wrap the one-liners with a tiny elisp function with a meaningful name.

- From Emacs, I can now easily apply these wonderful one-liners to either the current file or a list of them, without having to remember details nor tweak the commands. Fuzzy searching to apply commands works great.

You can see an ffmpeg example in a recent post https://xenodium.com/seamless-command-line-utils

sbt567(10000) 5 days ago [-]

I'm using navi[0] for this exact purpose and very happy with it. Now it is one of indispensable tool for my workflow

[0] https://github.com/denisidoro/navi

miles(10000) 5 days ago [-]

Thanks so much for crafting and sharing this guide.

Echoing some of the cookie comments[0], another issue with the popup is that in macOS Safari 16.1 with an adblocker extension enabled, the page looks like this: https://imgur.com/a/0SZRSTw . As noted in the screenshot, the five various buttons to accept, reject, save, etc. do not work to dismiss the notice.

[0] https://news.ycombinator.com/item?id=33774040

andirk(10000) 5 days ago [-]

Put less important things at the bottom! It's that simple. Most text-heavy content should work without Javascript. Also helps search engines understand that your site isn't selling batches of cookies :)

zerop(10000) 5 days ago [-]

Is there a good FFmpeg front end?

imbnwa(10000) 5 days ago [-]

ff·Works if you're on macOS[0]

[0]https://www.ffworks.net/

apocalyptic0n3(10000) 5 days ago [-]

Handbrake uses ffmpeg in the background (a forked version if I recall correctly).

pessimizer(10000) 5 days ago [-]

I'm slowly starting to realize that almost all video tools on linux and plenty elsewhere are ffmpeg front ends. ffmpeg is too big for one frontend.

hugs(10000) 5 days ago [-]

'The term 'codec' is a mix of the words 'coder & encoder'. Those are the magic parts before and after the 'decoded frames'.'

I thought 'codec' was a portmanteau of coder & decoder. A coder and encoder are the same thing.

Trixter(10000) 3 days ago [-]

codec is coder/decoder. Not sure where you're getting your other definition. 'codec's are required to both create, and play back, a file with a certain compression algorithm.

cyborgx7(10000) 5 days ago [-]

This is all very interesting, but I don't want to know all of this information, and I don't want to have to know it. What I need is some kind of wrapper with sane defaults, such that when I perform some kind of operation on a video file, I only have to specify the parameters that matter to me. The last time I tried to work like that, the bitrate was unusable afterwards. I don't want to have to think about the bitrate to create a clip of a video.

staindk(10000) about 2 hours ago [-]

I'm late with this reply but I've found LosslessCut[1] works quite well for what I need (cutting gameplay footage to upload to youtube). It doesn't have very many features though.

[1] https://github.com/mifi/lossless-cut

ythrowawa123457(10000) 5 days ago [-]

There was a decent wrapper tool called 'transcode' which I really liked. Unfortunately went unmaintained and seems to have been forgotten by the internet (webpage has went away and I can only find some traces of its existance in some wikis).

b3lvedere(10000) 5 days ago [-]

Maybe Handbrake ( https://handbrake.fr/ ) ?

leokennis(10000) 5 days ago [-]

While I agree with you that ffmpeg is hard to fully utilize in its current form, how do you design a 'wrapper' for a CLI tool with 68,915 (I could be off by a few) parameters?

There's only one way to manage that, and that is to hide some options and functionality. Then it stops being an ffmpeg GUI, it starts becoming a 'video transcoding application' (or an 'audio grabber application 'or a 'subtitle insertion application') with ffmpeg as its backend.

See a similar thing for openssl here: https://smallstep.com/blog/if-openssl-were-a-gui/

Maybe someone can build a very specific AI that generates ffmpeg commands for you, based on its man page and what you input as free text?

johnnymorgan(10000) 5 days ago [-]

There are GUIs for ffmpeg out there, I haven't used one in awhile but they exist

narag(10000) 5 days ago [-]

The ultimate guide for me would include how to extract a short clip from a longer video specifying the start time in (minutes:)seconds and the duration or, alternatively, stop time.

Actually I created a GUI to do exactly that, calling the ffmpeg executable.

So? It doesn't work as it should. Depending in what point you start, you will find a very noticeable interval with black video. I do understand why it happens, but that's not the point.

pessimizer(10000) 5 days ago [-]

-ss <start time> AND (-t <duration> OR -to <end time>)

edit: https://shotstack.io/learn/use-ffmpeg-to-trim-video/

hnarn(10000) 5 days ago [-]

handbrake-cli does this well but the syntax for begin and end timestamps is pretty confusing when combined

sylware(10000) 5 days ago [-]

GUI frontend to command line is 'why' the command line output should be machine parseable, I think this is true for most of relevant ffmpeg output.

Wowfunhappy(10000) 5 days ago [-]

Are you using -codec copy? If so, this happens because (most) frames in modern codecs are not discrete. They rely on information from previous frames to construct the image.

If you cut a stream in the middle, it takes many frames before there's enough information to be displayed.

(The solution would be to re-encode up until the first discrete 'keyframe'... and I don't know how to do that with ffmpeg.)

buss_jan(10000) 5 days ago [-]

FFmpeg is powerful technology, yet approaching it can be very daunting. So we decided to create an approachable guide for intermediate devs. Feedback is very much appreciated.

rogerz46(10000) 4 days ago [-]

Thank you so much for sharing this and really help my daily tasks. I can automate them in CLI instead of GUI now.

vesinisa(10000) 5 days ago [-]

Think you might have a editorial content left under the headline 'Installing FFmpeg Check'. There are instructions for installing FFmpeg (first I thought it was maybe some tool called 'FFmpeg Check' to 'check' the output of an FFmpeg command line, like in the above screenshot, but this turned out not to be the case :)

vovi-z(10000) 3 days ago [-]

I'm learning ffmpeg to do some audio/visual manipulation for a backend service and was looking for the basics. This is perfect.

As far as feedback goes, I noticed when running through the guide that the following command is missing an -i flag: ffplay -vf 'drawtext=text='HELLO THERE':y=h-text_h-10:x=(w/2-text_w/2):fontsize=200:f

I've already learned a lot from the ffmpeg concepts section. Appreciate it.

throw7(10000) 5 days ago [-]

An example on how to seek and cut video exactly on it's 'I-frames' would be nice. The few times I've searched, I've never really found what I'm looking for. Usually I just use avidemux to seek and cut to keyframes.

dontbenebby(10000) 5 days ago [-]

Agreed - I've been looking for a one liner to turn a series of stills into a gif for a while and buried in my todos is a note to read the documentation so thanks, I'll read this when I'm done w/ a few other todos this morning.

Someone should do a gui for common tasks it can do line there used to be the Zenmap gui for nmap.

(Learning how to do those sorts of simple guis quickly in python of something is another holiday todo, if anyone has suggestions on that - all my tools thus far are gui based and I need to make sure as many folks as possible use them.)

antihero(10000) 5 days ago [-]

This is absolutely fantastic, thanks!

jpmattia(10000) 5 days ago [-]

It's a nice guide, and it will be very useful to a large number of folks.

> Feedback is very much appreciated.

Nothing about captions that I could find quickly, which seems like an oversight.

bambax(10000) 5 days ago [-]

Excellent guide, thanks!

Shameless plug: a while ago I wrote a simple Python script to remove silent parts from video using ffmpeg, which AFAIK it doesn't do out of the box:

https://github.com/bambax/Remsi

It might be helpful to others.

coldpie(10000) 5 days ago [-]

Thank you. Please get rid of the cookie popup. If a lawyer tells you you need it, find a better lawyer.

chewz(10000) 5 days ago [-]

Thank you...

nubero(10000) 5 days ago [-]

That article is a far longer joke than the two word classic "Desktop Linux". Seriously though: good software has a user interface.

nubero(10000) 4 days ago [-]

Just to be clear: That the article itself exists is no joke and certainly helpful to the people who use it. that being said, I work with video every day and neither I nor my filmmaking friend use it. It's a tool for the same kind of people who like to spend sundays in their garage, fixing their cars. In that example, the people who need to get actual work done are the ones who take their cars out and drive them...

shantnutiwari(10000) 5 days ago [-]

Sorry, I didnt find the guide very useful-- too much info, it looks like its written for PhD students rather than averge users.

Sometime ago I bought a book 'FFMPEG Zero to Hero' by Nick Ferrano-- I recommdend that instead. The book contains more practical info tahts immeditely useful, but not always easy to find online

mickael-kerjean(10000) 5 days ago [-]

Wow are you the guy from: 'Interview with FFMPEG enthusiast in 2022': https://www.youtube.com/watch?v=9kaIXkImCAM ? :)

kcsaba(10000) 5 days ago [-]

No, but that guy is awesome!:))

djha-skin(10000) 5 days ago [-]

I'm on a project right now where I'm using FFMPeg.

It turns out that for the Apple stuff it loses a lot of color information. Compressor captures this color information by using ICC profiles that it generates from (the quick time headers of?) a .mov file in non obvious ways. Gamma, contrast, and chroma are lost in transit.

This is perhaps not a huge deal since we're dealing with a proprietary file format anyway, but it does mean I had to stop using FFMPeg in the project, which was a shame.

dangerlibrary(10000) 5 days ago [-]

I worked with ffmpeg for a few years about 6-7 years ago. If I learned anything from that time period, it's that whenever I started a sentence with 'ffmpeg doesn't do ...' I was wrong. It may not do it out of the box, and the way to do it may not be documented anywhere, but I was always wrong.

Melatonic(10000) 5 days ago [-]

Doubtful. Just probably not out of the box.

This will take longer obviously but a two step process where you export to an image sequence (like say 16 bit tiff) is a pretty fool proof way to retain colour information. And true pro industries like the VFX industry pretty much exclusively use image sequences (because they are just superior in everyway). Although they would be using either DPX or EXR sequences.

aargh_aargh(10000) 5 days ago [-]

There's a myriad of ffmpeg examples on the web so I wish there was something like https://explainshell.com/ specifically for ffmpeg (including parsing filters). I can imagine it could even include visualization, e.g. a two-way mapping between CLI parameter and on-screen rectangle for picture-in-picture/watermark removal coordinates, cut marks in video timeline, tree of input/output streams, tree of filters... so a CLI parser+generator.

asicsp(10000) 5 days ago [-]

Closest I can think of is https://ffmpeg.guide/ (discussed last month: https://news.ycombinator.com/item?id=33383023)

DrBazza(10000) 5 days ago [-]
sorenjan(10000) 5 days ago [-]

That's really cool! I wonder if you could stream to a Chromecast?

rurtrack(10000) 5 days ago [-]

I'm using ffmpeg.wasm in the browser and let me tell you it is indeed something else.

Like you, it used to be just a handy tool to convert stuff.

But being able to process video straight inside the browser literally blew my mind

Yeahsureok(10000) 5 days ago [-]

Could I recommend a table of contents?

It looks great, bookmarked.

defrost(10000) 5 days ago [-]

A ToC is there as an On This Page block just under the first image near the top .. the section title lines following are links to deeper in the page document.

bool3max(10000) 5 days ago [-]

Slightly off-topic: Recently I needed a way to add chapter information to 200 or so local MP4 video files of varying codecs (though mostly H264). It wasn't hard to get the job done with FFmpeg, though I was very surprised that adding such simple metadata to an MP4 container, with `-codec copy`, takes a surprisingly long time (around 5 or so minutes for an ~8GB file). I don't know why that is, as I am unfamiliar with the MP4 container format's internals, though I do assume that no re-encoding/transcoding was done (again, due to -codec copy). And it most certainly wasn't a simply copy-paste of the video data into a new file either, as that would've been way quicker. Nevertheless I just let the script do its thing overnight.

cm2187(10000) 5 days ago [-]

Were you working on files on a NAS? Often faster to copy locally, run ffmpeg, copy back to the NAS for some reason. Faststart flag transcodes the file twice. Otherwise can't think of any reason.





Historical Discussions: Front end developers: stop moving things that I'm about to click on (November 27, 2022: 834 points)

(834) Front end developers: stop moving things that I'm about to click on

834 points 6 days ago by ingve in 10000th position

medium.com | Estimated reading time – 3 minutes | comments | anchor

Frontend developers: stop moving things that I'm about to click on

Please. It's an insult to my brain. Like, you put the button in one place, and then you're like "nah, let's move it somewhere else." Here are some examples that have annoyed me the most lately:

Lyft Bike "Scan" button

This one sucks as I'm often opening the Lyft app just to ride bikes. So I immediately hit the bike button and I'm trying to hit the "Scan" button as quickly as possible. But no! Depending on how fast my internet is at the current moment, that banner will appear and push the Scan button up, which means I end up hitting the banner instead 😟; this is often a really frustrating start to my daily commute.

Notion search results

ignore the page titles for your own sanity

I use Notion as a power user at this point. When I'm trying to navigate to a page, I'll press CMD+K to open the doc finder, and then type in some prefix of what I'm looking for. To pick a result, I'll be using CTRL+P or N to go up and down, and often press enter in under a second. What's crazy about this is it first returns one set of results, and then a pretty different set of results. IIRC rarely even the first result will change 😕.

Why it happens

Some ideas:

  • Loading something and inserting it in a way that moves around other elements
  • More complicated queries that would reorder the results. e.g. it appears the Notion search first returns title searches, and then actually searches the contents of documents which ends up in result reordering.
  • What I call "UX fragmentation". When personas of users differ so much or there are so many experiments running, that engineers and designers aren't fully aware of what end-users are seeing. You've probably seen this with features being A/B tested. Sometimes, things are loaded in one order or another order depending on a plethora of variables. And also the timing of network responses.

Do better

When you make a change to the screen, why not just commit to the where it should be? Do the users and the metrics get benefit from "pseudo-responsiveness"? If the answer is yes just to the latter, I think it'd be fair to deem this a capitalistic-UX-anti-pattern. Get it out.

If anyone's put thought into this or is annoyed by this, please lmk or send me resources and further readings so I can get more annoyed. Thanks:)




All Comments: [-] | anchor

pulvinar(10000) 6 days ago [-]

This is also a problem with seemingly stationary UIs, where a button is only shown when the pointer gets close to it. Especially when that button is within a clickable area, such as a thumbnail image.

The ideal solution to all of these would be for the UI to keep track of drawing events and disable any click that occurs in proximity to a just-redrawn area, and beep instead.

ptx(10000) 6 days ago [-]

The recently redesigned Gmail UI is a good example of this: Chats now show up as minimized icons, and moving the mouse pointer over the icon to click it causes a close button to materialize under the pointer.

saberworks(10000) 5 days ago [-]

Apple maps enrages me for doing this. When you first open the app it like shows a screenshot of what you were looking at last time while it loads. Sometimes that screenshot is retained as the active view (for example if I have a search up), but other times after a couple of seconds the whole app just resets. This causes me to hit the 'dictation' button quite often instead of the X button that was on screen when I start tapping (X button to close previous search result is replaced with a view reset and now the dictation button is under my thumb).

A screenshot of the app shouldn't be a loading screen!! It seems to depend on how long it's been since the app was last opened and/or how the internet connection is at the time. Bank apps show a proper loading screen.

techsupporter(10000) 5 days ago [-]

> Apple maps enrages me for doing this. When you first open the app it like shows a screenshot of what you were looking at last time while it loads.

This has started to really irritate me since it feels like every application in Apple's mobile ecosystem does this. My device has SIX GIGABYTES OF MEMORY[0]. Why do you need to swap everything out of RAM almost instantaneously?

What grinds my gears the most is that state isn't preserved. Apple Maps and Music are the absolute worst for this. Music will show the same screenshot and then just unceremoniously dump me to the 'Listen Now' or 'Browse' screen (it seems random which) and completely disregard whatever playlist or album I was in. The last song I played and track position are kept but good damn luck getting back to the spot I was in--particularly which playlist--before the app got put on ice.

That screenshot is just a point of 'hahaha nope' frustration.

0 - Yes, I mean actual Random Access Memory, not 'storage capacity.'

computerfriend(10000) 5 days ago [-]

This joke UI for the Hawaii nuclear missile alert system is my favourite depiction of this.

https://gfycat.com/queasygrandiriomotecat

smcl(10000) 5 days ago [-]

This is what The Guardian suffers from. It is slightly less-bad than it used to be, scrolling through an article on mobile would send you jumping around as new adverts spawned where you were reading - for the time being it appears to be within the first ~N seconds of load.

Steam suffers from it too - while clicking through your 'Discovery Queue' it often displays a link to a video of someone streaming the game it recommended. Not that it matters now, Discovery Queue is hopelessly broken for me - only shows some very low-quality games i'm not interested in

nonameiguess(10000) 5 days ago [-]

Even funnier that there is a control bar for gfycat itself that gets covered by a cookie acceptance popup that shows up after a second or so. Try to pause this and you'll get blocked halfway through your own click.

chrislan815(10000) 5 days ago [-]

loool this is a serious problem

track16(10000) about 13 hours ago [-]

That was a very scary day in Honolulu for all of us who live here. Ads will really be the death of us all, someday...

joezydeco(10000) 6 days ago [-]

And stop asking me to do a survey on your website when I've only been there for 5 seconds.

smeej(10000) 6 days ago [-]

Or sign up for your email list.

I don't know if I want to hear from you ever again yet!

etothet(10000) 6 days ago [-]

The current version of the Amazon Prime Video app (and maybe also the HBO Max app?) on Apple TV has one of the worst examples of this I've ever experienced. In their featured listings, they should cards of shows/movies in what is basically a portrait view. When one of the cards becomes active, after a second or two it becomes wider and its content and background image changes and the cards to its right are pushed out further to the right. So as you're looking at a card and you begin to read it, its entire contents is modified! It's so maddening.

bombcar(10000) 6 days ago [-]

I HATE all the video apps I use - the one that drives me up the wall the most is that I know they have a horizontal UI - I can see it on my TV - but on the phone you HAVE to browse in vertical, and play videos in horizontal.

WHYYYYYYYYY

wizofaus(10000) 6 days ago [-]

I always assumed it was done deliberately to trick users into clicking on ads...

AdvancedSimian(10000) 6 days ago [-]

It happens so often that I, too, assume it is done for precisely the same reason.

29athrowaway(10000) 6 days ago [-]

The gold standard for the web should be: Craigslist

Fast, functional, responsive.

Oxidation(10000) 6 days ago [-]

Or McMaster Carr. RS, with it's laggy backend (I assume, as their recent UI shinification didn't change it: it was bad before) could learn a lot from them.

Then again, McMaster Carr want you to use an app on mobile and the new RS site works on mobile quite well, with the lag more hidden by the inefficiencies of mobile browsing in general.

ianai(10000) 6 days ago [-]

And as far as we know no tracking/propping up anything for attention regardless of ramifications.

hnick(10000) 6 days ago [-]

whirlpool.net.au is so fast here in AU sometimes I refresh and I'm not sure it did anything.

fleddr(10000) 6 days ago [-]

All the examples are a case of unfortunate timing: asynchronous things doing late UI replacement that happen to conflict with the time window in which you click.

That's amateur stuff. Our car does this intentionally. As your finger is in mid-air, on its way to hit something on the touch screen, it's detecting this motion and then shows a context bar in the top and bottom by the time you land your finger. Which often overlays whatever you were going to touch.

It's so comically bad that I think a team of engineers is laughing back at the factory: 'look! he's trying to touch something again!'.

The other prank they included is traffic alerts. You're sitting in a traffic jam for 10 minutes already and then my music stops, only to hear: slow traffic ahead.

WaxProlix(10000) 6 days ago [-]

Some phones are great at this - if you're on a call and want to look up your account number to give to a service rep, getting close enough to tap an app button turns off your screen, even in speakerphone mode. Pull back your hand to tap the power button to turn on the screen? Well the screen detected your finger left and turned on just as you were hitting it. Now you're in a third state, intentionally off screen, and just moving your finger around the sensors won't change that. You have to tap the same power button again, and then start the process all over.

Don't forget you've been talking to a service rep this whole time, or more likely sitting going 'uhhhhhhmmm...' like a dipshit.

Normille(10000) 6 days ago [-]

Way back when I first started learning to build web pages [when HTML4 was just a glint in Tim Berners-Lees eye], it was conidered bad form not to prvide a 'width' and 'height' attribute for an image you were placing on the page.

Because if you provided these, the browser would know how much space to leave for the image to occupy after it had downloaded [this was back in the dial-up modem days. So images usually loaded gradually after the rest of the page] and so the layout of the page would not change. Conversely, if you didn't provide 'width' and 'height' attributes, the browser would only leave a default small icon sized space for the image and then you'd have the page content move and shuffle about as each image downloaded and the rest of the page content got elbowed out of the way, to accommodate it.

It's funny how such basic concepts of usability seem to have fallen by the wayside in our new modern Web3 [or whatever version buzzword we're on now] in favour of moving content, modal overlays and the like. And, since so many sites these days just aggregate content from countless other external sources, even that basic notion of pre-defining your image sizes is often neglected too.

at-fates-hands(10000) 6 days ago [-]

One of my first jobs as a front-end dev, I was told repeatedly if you released a site and had forgotten to put height, width on images, it was seen as a defect and would affect your year end metrics.

By my second or third job, nobody seemed to care about it and when I brought it up, it was brushed aside as a 'browser issue' and not something we devs needed to worry about.

You make a great point, and something I'm seeing a lot more of now that I'm in accessibility.

deckard1(10000) 6 days ago [-]

> basic concepts of usability seem to have fallen by the wayside

Sadly, the tech may have changed but both our ideas on usability and the incentives at play have remained the same. Which is to say, the devs I know of all know that popup videos, banners, modals, etc. are all trash much like the popup ad banners of 1997. But we do them anyway because the people that pay us have financial incentives that have been proven via misguided A/B testing that they work. And by misguided I mean... we're talking about manipulation, right? That's the whole shell game of A/B testing. You're testing your way to what works best for you and not your users. It's Tinder knowing the perfect match for you but not showing you that person because you'll quit the app if you ever met them.

I can't even take a11y or people that advocate a11y seriously today. You built modals and carousels with frustrating timeouts that slide content before I'm done looking at it! You're a circus clown.

frereubu(10000) 6 days ago [-]

This - 'Cumulative Layout Shift' - is part of Google's Core Web Vitals - https://support.google.com/webmasters/answer/9205520?hl=en - so if people are doing that, they're going to be penalised by Google. (Although probably only a bit, given the number of other factors that go into search rankings).

TheAceOfHearts(10000) 6 days ago [-]

Related to this, here's a pet peeves of mine. When YouTube is buffering a video, it shows a spinner and you can't press play or pause. Usually if something is buffering a lot I'm in a place with spotty internet, so I'd like to pause until I'm somewhere with better Internet. If I say pause, you should not continue playing the video under any condition. I have made my intent clear.

thewebcount(10000) 6 days ago [-]

Oh man, so many video-related websites do this shit. It's even worse in some cases where if you hit pause, then drag the playhead back to the beginning (say because you want to wait and show the whole thing to someone), it starts playing again as soon as you let up on the playhead. Like, I want to pause, put the playhead somewhere and still be paused. Is that so fucking hard to understand?

zestyping(10000) 5 days ago [-]

This has always bugged me too.

The _desired state_ (playing or pausing) and the _actual state_ are two different things.

Whenever these two things are different, they should have two separate representations.

The problem is that they are mashed into one ambiguous representation.

kjkjadksj(10000) 5 days ago [-]

Back in the day you used to be able to buffer the entire video too. So if I had a bad connection, I'd open a dozen youtube links, walk away for 5 mins, then come back and watch whatever with zero hiccup. No longer possible now that content only buffers a portion of the way, not just on youtube either. If you are on a spotty connection its therefore better to just pull the video with youtube-dl.

aendruk(10000) 6 days ago [-]

Reminds me of my Dell monitor. Years old muscle memory is:

  1. Unplug laptop from monitor
  2. Power off monitor
but step one triggers some automatic input detection mode for about 10 seconds during which the monitor ignores its power button.
jiggawatts(10000) 5 days ago [-]

State Machines!

It's not something you had to learn just to pass a CS exam and then promptly purge from your memory.

The buffering state is a state, not an arrow!

User interface states should respond to inputs, including cancellation, retry, etc...

pull_my_finger(10000) 6 days ago [-]

Honestly, everyone knows this. If you're creating sites that do this in 202X you're just being willfully ignorant/lazy. There are plenty of component libraries for rendering placeholder components in the popular frameworks, react has the suspense API etc etc for exactly problems like this.

layer8(10000) 6 days ago [-]

Still delay-inserted content like the Wikipedia donation banners are a regular occurrence.

Tiddles-the2nd(10000) 6 days ago [-]

When it comes to ad links I've always wondered if it's ignorance or done deliberately to make you click on the ad.

ldjb(10000) 6 days ago [-]

Something that really frustrates me is when an alert pops up on my screen just as I'm about to tap something, so my tapping then dismisses the alert before I've had a chance to read what it said.

Hopefully the contents of said alert weren't important, but in most cases I'll never know...

kadoban(10000) 6 days ago [-]

If this is for Android, there's a notification history you can look in. You may have to turn it on first, can't remember if it defaults on or off.

colanderman(10000) 5 days ago [-]

In addition to this, macOS has the following bizarre behavior with notifications.

If any one application emits multiple notifications (say, several new calendar items), they pop up in a group -- but this is hard to see, since they're all stacked on top of one another. There is an X at the top-left corner, just like all other notifications. When you go to click the X, the group suddenly expands, and the X changes into a 'Clear All' button!! 80% of the time this results in me closing a large stack of notifications without realizing they even existed.

ceejayoz(10000) 5 days ago [-]

Yes; I've long wished there'd be a 500ms or so 'no interaction' delay on these.

bondarchuk(10000) 6 days ago [-]

I have googled the letter 'n' countless times just trying to visit news.ycombinator on firefox android...

zinckiwi(10000) 5 days ago [-]

There's probably a bit of the algorithm that is dutifully noting your long-time, consistent interest in n, and making sure to reward you for your dependability by canonising it ever-stronger in your profile.

bombcar(10000) 6 days ago [-]

And now you're the world's leading expert on n

delecti(10000) 5 days ago [-]

I've likewise googled 'i' countless times trying to go to imgur on firefox windows for the same reason.

AndrewDucker(10000) 6 days ago [-]

Latest one to catch me out on is when receiving calls on my Pixel phone it will pop up 'screen call' as an option where the 'accept call' button had been half a second earlier, and not let you override it to actually accept the call.

derekp7(10000) 6 days ago [-]

That one caught me the other day (shortly after an Android OS update). I only saw it once, with the button down at the bottom of the screen. I can't seem to get it to do that again when I try calling my phone from another phone.

Also, your report is the only instance I've seen to confirm that this happens -- for whatever reason no one believes me when I post something about this to any Android group. Just like I seem to be alone in trying to figure out how to get the Contacts (when launched from the dialer) to show only specific groups (such as contacts with phone numbers). I can do this from the Contacts app, but not the Contacts tab inside the call dialer app.

lnauta(10000) 6 days ago [-]

Google with its 'related searches' that pops up because I clicked on a link and I am almost never interested in this extra information. When trying to filter the relevant javascript it breaks some other stuff on the page (last time I tried a few years ago). Grrmbl

ricardo81(10000) 5 days ago [-]

Yes. Has to be the dumbest injection of 'stuff' onto a page ever. I wonder if anyone non-accidentally clicks on that stuff.

drewbeck(10000) 6 days ago [-]

Instagram search is my daily frustration. 100% of the time I search I'm looking for a user's account. Type type type, see the user come up in results, go to tap but now they've loaded in 10 possible autocomplete options for my search, pushing the thing I want off the screen. And of course now I'm in the "explore" page with a search term I don't want, taking a bunch more taps to get where I need to go.

I agree with others that I think the incentives are against the user here. In the IG case they don't care if I find my friend quickly. It does not help them in any way. But hey they got a click on the autocomplete suggestions (win for that PM) AND another view on the explore page. Clearly I love these features!

lame-robot-hoax(10000) 6 days ago [-]

This pisses me off more than it should. I've slowly gotten better at not immediately clicking on the account (which ends up with me clicking on the search result that shows there instead) but every time I forget I get unreasonably mad.

h0l0cube(10000) 6 days ago [-]

Anyone else try peeping the chat in Zoom?

low_key(10000) 6 days ago [-]

It's awful. I absolutely hate it. First click: show chat. Second click: record meeting.

schipplock(10000) 5 days ago [-]

Also: someone sends you a direct message, you reply, but you send the message to everyone :).

exabrial(10000) 5 days ago [-]

While we're at it... Front end developers: stop using JavaScript for _fucking everything _.

Sohcahtoa82(10000) 4 days ago [-]

I'd settle for them to stop fucking with scrolling.

Scrolling behavior is user-configurable. There's no need to decide 'Fuck your preferences, one scroll-wheel click is going to be 1 line of text' and override my configuration. There's no need to implement 'momentum' to scrolling, or make it bounce, or try to implement smooth scrolling since my browser does it anyways.

jspash(10000) 5 days ago [-]

At $dayjob we have a pretty simple product page (e-commerce site) that has the requirement of 2 or 3 dropdown controls. In the early stages of the build I suggested we use the native browser <select> boxes. The idea was widely panned but I did it anyhow because we were short of time. It took me about 3 1/2 minutes to implement and another 2 hours to join it up with the back-end and write a few simple specs.

Fast forward to 6 months after launch. We've now replaced the native control with an 'all singing/all dancing' javascript version written in Vue. Thing is, it doesn't know all the lyrics and it couldn't do a box-step if it was stepping off a box. We get bug reports literally every few days because it doesn't act like the native control. And we'll _never_ actually stop the bug reports because people on different systems expect it to act like the native one. Something that is literally impossible to do when writing it on your own.

So now, even with all the evidence pointing to it being a bad idea, nobody want to rip it out because the integration is too muddy and could cause further bugs.

:facepalm

sunjester(10000) 5 days ago [-]

This is a problem for people who are very proficient when using the internet, 'computer experts', and not for many others.

However, this has been a problem in Google search results for a while now (and you know what I mean, I don't even have to say it), I figured they have would fixed it by now... I have even made a custom Chrome Extension that kills these annoying ads and shifty DIV's. It would be nice to see a big list of these crappy shift DIV's and UX fails.

djaychela(10000) 5 days ago [-]

> This is a problem for people who are very proficient when using the internet, 'computer experts', and not for many others.

I'd disagree. My mum is 83 and has complained about this in apps on her phone and when using websites.

mcv(10000) 6 days ago [-]

My biggest frustration for this is the typing completion on Android. I'm typing, and while I'm typing another character for the word, I see the word I want listed above the keyboard. But just as I hit the word, the suggestions update and my word is replaced by another. And often my word is still listed, but in a different place, which js incredibly annoying. Or sometimes the word disappears, despite it still being a valid completion for the word I'm typing. Why suggest it and then remove it?

bhauer(10000) 5 days ago [-]

Same thing happens all the time to me on the iOS virtual keyboard. The root problem, in my opinion, is that word suggestions lag far too much. They are refreshed at least a few hundred milliseconds after your last keystroke.

I know Windows Phone is long dead, but its virtual keyboard was consistently superior at providing suggestions that were well-aligned with user intent and not awkwardly delayed such that options shifted while the user's finger was in-flight.

linsomniac(10000) 6 days ago [-]

A lot of the responses are talking about how apps should fix this, which I agree with, but I'd also like to see it at a higher level...

I'd like to see frontends (browsers, Android, iOS?) remember their widget hit points for ~400ms, and if you've touched somewhere that had a different widget hit point within that time, use the old widget.

rasapetter(10000) 5 days ago [-]

Or simply ignore clicks on elements that haven't been visible for longer than a few hundred milliseconds. I'd be happy to click again.

Godel_unicode(10000) 6 days ago [-]

Or at a minimum don't go to the new widget but just have a UI indication that you registered the touch/click.

neilv(10000) 6 days ago [-]

This problem first occurred in the mid-'90s, and people saw it was a problem, and that's why we specified the image dimensions in HTML (before the GIF or JPEG header was loaded), so the layout wouldn't shift during load, when people were already reading and navigating.

Since almost the beginning, graphic designers were trying to layout Web pages like an advertising brochure, lots of bad ways, rather than as hypertext. When, in a sense, they finally did take control of the Web, HCI (usability in service of user) had evolved into 'UX' (advertising-like graphic design in service of non-users).

There's often disincentive for things like 'user can understand what the controls are and what the state is', 'user can focus on the information they want', etc. UX won't care about these things until some time after users are so aware of the problems that it affects engagement metrics and eventually A/B testing figures out why.

I'm imagining a eureka moment, during which a UX researcher, trying to find a new, improved dark pattern, to screw over hapless users harder, accidentally lets slip in some old-school good usability, which wasn't even supposed to be in the test sets, but discovers that this strangely resonates with users.

noduerme(10000) 5 days ago [-]

It would be really nice if modern browsers just waited to load all the images (or at least their sizes) before rendering any text on a page. Yeah, it would be slow as hell! It would force web designers to start compressing their images again and specifying sizes in the layouts. Obviously this isn't really possible anymore because of the range of screens we're dealing with. Okay, then here's an alternative:

A browser plugin that reads the image heights and page heights on each page you visit, along with your window size, and reports that back. Then pre-sizes the height of the window and the elements for other users with the same plugin based on what it recorded.

solardev(10000) 6 days ago [-]

In my experience, it was usually higher-ups and not the UX people who wanted to implement dark patterns. The UX people and devs wanted to make the enduser experience better (believe it or not), but were frequently overridden by some higher business priority (we NEED this popup as soon as the page loads! opt them in to all the newsletters by default! highlight the highest-margin products).

Some companies just don't respect their users or customers, and/or try to optimize for short-term metrics that make some individual or agency or department look good at the expense of long-term loyalty. But I've never seen a UX person willingly suggest a dark pattern... all the ones I've worked with were forced to implement them, despite much protest, because some manager wanted it there.

nyanpasu64(10000) 5 days ago [-]

Imagine if the people designing the Discord, Windows 10+, Telegram, and Amazon Prime cancellation UI to manipulate you into paying for services, were in charge of designing airliner cockpits and industrial plants. User interfaces can and should do better in preventing unintentional user error, and people should refuse to implement, use, or accept dark patterns.

DonHopkins(10000) 5 days ago [-]

Sun's tragic X11/XView based 'xbugtool' pioneered a lot of this misbehavior and worse in the early 90's, especially when you foolishly tried to use it to file a bug against itself, which triggered some extremely dark patterns indeed, like scrollbar spoofing, passive-aggressive input queue blocking, and keyboard focus locking:

http://www.art.net/~hopkins/Don/unix-haters/x-windows/xbugto...

For a more philosophical approach to user interface design, see 'Jojo on UI':

http://www.art.net/~hopkins/Don/unix-haters/x-windows/jojo-o...

_nalply(10000) 5 days ago [-]

This problem is even older, but was not as prevalent as now.

Early 1990 I experimented with Mosaic, and I experienced it the first time: I wanted to click on something but in the few hundreds of milliseconds between my decision to click and really clicking an alert popped up and I accidentally dismissed the alert and couldn't read the message.

This does not only happen on the web. All GUI software suffer from this problem. I know that Office products also have caused me to misclick because something just appeared or disappeared.

I have been thinking about a tweak to the GUI: Let things appear or disappear but still accept click events for about 200 milliseconds longer. As if the change did not happen yet. Perhaps combine this tweak with a soft but quick blending-in of the change, and only enable the new GUI after the blending-in finished.

This is different from the problem that GUI buttons move but at least related. Perhaps with this tweak moving GUI buttons won't annoy as much, because we can react. This is not ideal, but not as bad.

resonious(10000) 5 days ago [-]

I think this kind of thing is super easy to introduce with the modern practice of async everything.

When I visit a modern webpage, it feels like I'm watching a golem take form from the debris around it. It starts out as this weird ugly contentless frame, and then things start popping in one at a time. It's quite hard to know when it's done.

Makes me miss the days when servers would serve me a full webpage. Browser shows me that it's loading, and once it's done it's done. It's also easier to develop sites like that, as you don't have to code your own loading spinners or make those weird 'empty placeholder content' boxes that kind of show you how the site is laid out.

seer(10000) 5 days ago [-]

What an apt analogy! I always thought of it as a transformer unspooling from its compressed "car" form.

But thats what we get when multiple teams are responsible for the same web real estate.

Jedd(10000) 6 days ago [-]

Spotify (on Android) is the worst offender for this.

The playlist shortcut shows six playlists, but 2-5 seconds after rendering these, about the same time it takes you scan over them and see if the playlist you want is offered in this set and get your finger into position, it re-renders them with different content & different order.

The full playlist selection does the same thing, using some hybrid and opaque MFU / MRU algorithm -- I'd be delighted with a pure alphabetical ordering, rather than having to visually scan down an unsorted list that's re-ordering itself as I read.

Mind, the Spotify UI can be held up as a shining example for basically every user-hostile anti-pattern, so I guess there's some consistency in their design.

andrewflnr(10000) 5 days ago [-]

Interesting, I've encountered a lot of problems with Spotify on Android, but not that one.

I did have to do some work to track down the alphabetical ordering feature for my playlist listing, after they unset it and moved it, but it is possible to set it, at least for the main listing (I find the MRU thing helpful for the 'Add to playlist' menu, so I don't mind that I don't know if it's possible to set it there).

bschwindHN(10000) 6 days ago [-]

Yep, Spotify immediately came to mind after reading the title. Happens on iOS too.

kichik(10000) 6 days ago [-]

For me it's Twitter for Android. It keeps loading new tweets to replace tweets I'm actively reading in the timeline.

graycat(10000) 6 days ago [-]

I agree and don't like Web pages that are complicated, require time, clicks, and effort to learn to use, have page content jumping around, cover the page with images requiring the user to stop what they are trying to do, change the window size and magnification, find a way to get rid of the image and do so, return the window size and screen magnification to what they were, and get back to the reason they came to the Web site.

In my startup, uh, I'm on good terms with our front end developer and we agree on Web page design techniques:

So, no popups, pull downs, overlays. Nothing on the screen jumps around or moves. Large, relatively bold fonts and high contrast.

Always have both vertical and horizontal scroll bars.

No warnings or requests to permit the site to use cookies -- the site makes no use of cookies.

Hmm, maybe someone somewhere, maybe the EU, will make a big stink that we have not asked users to let us use cookies. Sounds like a good source of publicity until finally we end the stink explaining that we use no cookies and are not required to get approval for NOT using cookies!

The site HTML makes nearly no use of JavaScript.

Users don't login.

Users are not asked to spend money or up click on some popularity counter.

The server cores generate each Web page quickly, and each page sends for only about 40,000 bytes. When we convert the storage for our SQL database from rotating disk to solid state memory, the page generation will be significantly faster. Net, the site is responsive.

The pages and user interface are based on just the HTML controls with one line text boxes, push buttons, links where each link is a single word in the Roman alphabet in English with a simple meaning -- page design techniques very familiar to likely 2+ billion people.

Ah, we don't expect 2 billion devoted users right away and will be pleased with 1+ billion!

dredmorbius(10000) 4 days ago [-]

Thank you, in case you don't hear that enough.

InCityDreams(10000) 6 days ago [-]

You're doing everything right regarding cookies, but have somehow managed to either not understand how the EU would regard your non-use of them, or your site does not exist.

jodrellblank(10000) 6 days ago [-]

> 'Hmm, maybe someone somewhere, maybe the EU, will make a big stink that we have not asked users to let us use cookies. Sounds like a good source of publicity until finally we end the stink explaining that we use no cookies and are not required to get approval for NOT using cookies!'

The EU does not require you to ask permission to use cookies if you do not use cookies. Encouraging web developers to not use cookies, and as a consequence have no banners, is one of the positive incentives of the cookie legislation.

romellem(10000) 6 days ago [-]

I agree with [this][1] idea:

> an autocomplete has one chance to reorder per keystroke. if you got it wrong and you have a better ordering a bit later you must "swallow the sadness" (as per the original author of this wisdom) but never change already displayed items

[1]: https://twitter.com/dan_abramov/status/1470751551568363530

devnullbrain(10000) 6 days ago [-]

I want foo.

- Type 'f'

- foo appears in the search.

- Type 'o', because I type with 10 appendages and getting them to work together requires pipelining.

- foo disappears from the search

This should never happen.

fbdab103(10000) 6 days ago [-]

Can someone please tell the Windows Start Menu designers? Have completely ruined the experience as it constantly juggles items as its heuristics are seemingly updated. I especially enjoy how Bing results get prioritized over local applications and files.

krick(10000) 6 days ago [-]

Oh, it gets better. On Android (at least, LG-flavored one — never used any other android phone) I have an issue with inter-app 'Share' button popup. It somehow manages to re-shuffle all the suggestions (apps/recipients) exactly in the same amount of time that is natural for me to spend before clicking on the desired contact. Which several times made me send something to a wrong person, which resulted in extremely awkward situations. I don't know how common that is, since it was never fixed after all the updates, but I still have to be very careful and to make an unnatural pause waiting for UI to update every time I use this feature.

dredmorbius(10000) 4 days ago [-]

Also the fact that there are choices presented which I have never selected and never will.

I've seen that share-selection-reordering on numerous devices, including Onyx BOOX, which seem to otherwise mostly provide their own Android interface. So whatever's driving that seems to be deep in the environment.

brokenmachine(10000) 6 days ago [-]

The inconsistency of the UI is ridiculous too.

There's countless examples, but the one that really annoys me every day is when the alarms go off.

I have alarms to wake up in the morning, and on the alarm screen you have to swipe to dismiss, but to snooze you have to tap the other button.

Sleepy me has a lot of trouble remembering which one's which, even though I've done it probably a thousand times at this stage.

And even though it's a big 6.7' or whatever it is screen, they make the buttons absolutely microscopic.

zestyping(10000) 5 days ago [-]

Share is the worst. It is incredibly slow to appear and update.

meibo(10000) 5 days ago [-]

This still happens and it's the absolute worst, it gets worse the more apps you have. It's also just incredibly strange how it choose who to show in the share menu and there's no way to exclude apps.

No, I never want to share anything ever at all to #company-all on Slack. Do not show it in my YouTube share menu when I want to send a dog video to my partner.

rrrrrrrrrrrryan(10000) 5 days ago [-]

The Android share menu was trash for many years, but it's definitely been getting some love lately. (If you don't have a Pixel you might still be rocking an older Android version.)

Perhaps a year ago they drastically improved load times, but the recommendations became much more stupid, and in recent builds I feel they've actually struck a good balance.

macawfish(10000) 5 days ago [-]

This is so so so so bad. It's always such a risky situation too. The instant-send shortcuts pop up right as I'm about to tap something else, causing me to send something directly to the wrong person with no intermediate step. What if I'm sharing something confidential or sensitive? Way too much room for error.

brokenmachine(10000) 6 days ago [-]

I'm on Samsung and it doesn't re-shuffle as you're looking at it, but it's in a different order every time.

It's so irritating if you're doing the same operation many times, ie sending the same picture to multiple people separately.

Also the share menu has a different layout according to what app you're using. I understand that it's up to the app developer to get fancy with it, but it would be really nice if you could have a setting for 'use standard share menus', or something like that. Or even better if there was some way you could make it be the same every time, like choose your favorite apps to share to.

It's unbelievable that these billion dollar companies can be satisfied letting this garbage represent their product.

I'd make sure to fix these if it was my hobby project, let alone a 'finished' product that's on sale by a giant multinational.

bmitc(10000) 5 days ago [-]

Apple does this, a lot. The number of times I have incorrectly called somebody on FaceTime because I clicked on a contact at the same time FaceTime decided to update the list should be embarrassing for them. I have zero idea how this doesn't get fixed. It should be a trivial change to make sure the list is updated however they'd like before showing it to the user. As it is right now, given that it's slower or about as fast as me clicking something on the screen, it seems updating the list is the very last thing they do.

jer0me(10000) 5 days ago [-]

I'm always nervous about AirDropping things—the list shuffles around in real-time and there's no confirmation or way to rescind a file.

FollowingTheDao(10000) 5 days ago [-]

Right? Just click on Safari's address bar to see this in action!

TulliusCicero(10000) 5 days ago [-]

Alternatively, if the list is actually updated with new information, block input to the list for 300ms or so.

gwd(10000) 5 days ago [-]

I came here to complain about Apple's autocompletion feature: probably 30% of the time when I start to type a long word, the exact word I want comes up in the suggestion box at the top; only by the time I click the box, the word has changed to something else. It would have been faster to type all 7 characters out with my thumbs than to delete and re-type everything again.

nikolay(10000) 5 days ago [-]

Same with Windows, macOS, and Linux - when an input has the focus, new windows popping up should not steal it from it (unless it's a child window) - I often have parts of my password entered somewhere else as while I'm typing, another window steals the focus and my input with it! Not only is this annoying, but it's a security flaw as well! Regarding web apps, you can also do destructive actions if the scroll moves the wrong button underneath your mouse pointer!

tacker2000(10000) 5 days ago [-]

Same issue: when im typing something and then a popup appears out of nowhere and then I accidentaly hit "OK" or whatever the default button is by pressing space on MacOS! Wtf!?!?

lewantmontreal(10000) 5 days ago [-]

Sometimes I chuckle to myself wondering why people pay 36 bucks a year for 1Password. Then I launch my $10/year Bitwarden desktop app. I'm halfway through typing my master password and a popup launches capturing the rest of my keystrokes: "Would you like to update Bitwarden?"

snazz01(10000) 5 days ago [-]

I once clicked into a corporate login window and typed my domain password, it completely ignored my click and I ended up typing my password to a coworker in Slack.

climb_stealth(10000) 5 days ago [-]

This! If anyone has a solution against the window popping up in the foreground on MacOS please speak up.

I think one of the build-an-OS-from-scratch projects had this as a feature. You create a window first for a program before starting it and it is not able to pop up outside of that or change its focus. Don't remember which project it was but it always stuck in my mind as an awesome feature.

ajjenkins(10000) 6 days ago [-]

I agree that this is extremely annoying. It happens to me all the time with the iOS keyboard suggestions.

For websites, Google calls the Cumulative Layout Shift (CLS) and it's one of the Core Web Vitals they use for measuring website performance.

https://web.dev/cls/

bhauer(10000) 5 days ago [-]

Same here. All the time with the iOS keyboard. It's infuriatingly slow with suggestions. They always refresh well after my finger has already started moving to select one.

mclightning(10000) 5 days ago [-]

Ironically, Android's core UI elements are horrid in this regard.

heavyset_go(10000) 6 days ago [-]

Which is ironic considering Google search results do exactly this with suggestions after the page finishes loading.

NikkiA(10000) 6 days ago [-]

A particular bane of mine is the self-oscillating UIs, youtube is particularly bad at it these days - if the mouse pointer is in 'just the right place' (which is bigger than it sounds) then you get the seek-bar preview frames popup, which moves things just enough that the mouse pointer is no longer over the area that triggers it, so it vanishes, and the whole thing starts again.

Oxidation(10000) 6 days ago [-]

This can happen in native UIs as well. I've had to add hysteretic behaviours to prevent the appearance of a scroll bar somehow obviating the need for the scroll bar, so it vanishes and now it needs a scroll bar, repeat as fast as the toolkit event loop allows.

leetbulb(10000) 6 days ago [-]

I've always wondered, but figured that there must be, others who find this type of UI quirk (and find it humorous). 'Self-oscillating' I like it. Thanks.

devmor(10000) 5 days ago [-]

Twitter has an issue like this with its video player - clicking on the vertical volume bar to adjust it and moving past the seek bar that it intersects with causes the volume interaction to drop off in favor of the seek interaction.

soulofmischief(10000) 6 days ago [-]

A general golden UX rule is to never cause layout shifts on hover events, preventing this sort of thrashing.

Unfortunately, Google/YouTube and others have lost the plot and no longer employ good UX professionals in their flagship products.

klazutin(10000) 5 days ago [-]

I've recently come across a perfect example of just that: [1]. Just try to slowly move the mouse over any of the round red year marks at the center of the page.

[1]: https://www.computerhistory.org/timeline/ai-robotics/

vidanay(10000) 6 days ago [-]

YouTube also has the problem that there is almost no dead space to 'park' your cursor. You have to carefully place it in between autoplaying previews.

ianai(10000) 6 days ago [-]

That sort of stuff is outright user abuse.

toast0(10000) 6 days ago [-]

Google cloud console does this kind of thing without even needing to have the mouse anywhere in particular. I just close my eyes and click on where I hope the target will remain.

ptato(10000) 6 days ago [-]

Where is that spot? That has never happened to me, but I'm curious to see what it looks like.

krautsourced(10000) 5 days ago [-]

Netflix did that for a long time with the very right item in the suggestion list where I could never see the details because it would instead scroll to the next page, making the item I was interested in disappear to the left :) Seems like they fixed that though

princevegeta89(10000) 6 days ago [-]

Slightly unrelated, but all these media streaming apps and services like Netflix/Prime Video/Youtube etc. are so horrible by automatically playing whatever the fuck I am currently focused on, although I am just scrolling to somewhere to find my content. There is no way for users to focus 'out' of the thumbnails, other than focusing on a different thumbnail, and it keeps playing the nonsense. All I would do is go back all the way to main menu or where I started

hnick(10000) 6 days ago [-]

Following the same theme of liminal UI areas, another thing that peeves me is touch screens like my local supermarket self-checkout which respond with a beep (indicating they know I hit the button) but do nothing, because apparently the beep-region and action-region and not the same. Or maybe I dragged my finger, in which case, please don't beep.

simulation_(10000) 5 days ago [-]

My favourite example of this is bank app I use very often, which comes with a VERY annoying twist. The most used button (one-time payment code) sometimes switches with 'Take a loan' button, and after clicking loan button, someone from bank will call you in a near future to talk about it.

grishka(10000) 5 days ago [-]

There's no confirmation?...

userbinator(10000) 5 days ago [-]

Front end developers: Stop making SPAs.

99% of the time I'm forced to use a site that turns out to be a horribly bloated SPA that I only get to see after enabling JS, and which will do irritating things like what is described in the article, among others, it's to do something that could've easily been done with a simple HTML form and perhaps a bit of JS enhancement.

alvah(10000) 5 days ago [-]

What is a SPA? It's not as if Googling this word will enlighten me.





Historical Discussions: Goodbye, data science (November 29, 2022: 813 points)

(814) Goodbye, data science

814 points 4 days ago by sonabinu in 10000th position

ryxcommar.com | Estimated reading time – 13 minutes | comments | anchor

This is more of a personal post than something intended to be profound. If you are looking for a point, you will not find one here. Frankly I am not even sure who the target audience is for this (probably "data scientists who hate themselves"?).

I had been a data scientist for the past few years, but in 2022, I got a new job as a data engineer, and it's been pretty good to me so far.

I'm still working alongside "data scientists," and do a little bit of that myself still, but most of my "data science" work is directing and consulting on others' work. I've be focusing more on implementation of data science ("MLops") and data engineering.

The main reason I soured on data science is that the work felt like it didn't matter, in multiple senses of the words "didn't matter":

  • The work is downstream of engineering, product, and office politics, meaning the work was only often as good as the weakest link in that chain.
  • Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you'd get nearly the same regards in either case.
  • The work was often very low value-add to the business (often compensating for incompetence up the management chain).
  • When the work's value-add exceeded the labor costs, it was often personally unfulfilling (e.g. tuning a parameter to make the business extra money).

Shitty management & insane projects

Management was by far my biggest gripe. I am completely exhausted by the absolute insanity that was the tech industry up to 2021. Companies all over were consistently pursuing things that could be reasoned about a priori as being insane ideas– ideas any decently smart person should know wouldn't work before they're tried. Some projects could have saved whole person-years of labor had anyone possessed a better understanding of the business, customers, broader economic / social landscape, financial accounting, and (far too underrated in tech) any relevant subject matter areas.

Those who have seen my Twitter posts know that I believe the role of the data scientist in a scenario of insane management is not to provide real, honest consultation, but to launder these insane ideas as having some sort of basis in objective reality even if they don't. Managers will say they want to make data-driven decisions, but they really want decision-driven data. If you strayed from this role– e.g. by warning people not to pursue stupid ideas– your reward was their disdain, then they'd do it anyway, then it wouldn't work (what a shocker). The only way to win is to become a stooge.

The reason managers pursued these insane ideas is partly because they are hired despite not having any subject matter expertise in business or the company's operations, and partly because VC firms had the strange idea that ballooning costs well in excess of revenue was "growth" and therefore good in all cases; the business equivalent of the Flat Earth Society. It was also beneficial for one's personal career growth to manage an insane project (résumé lines such as "managed $10 million in top-line revenue," failing to disclose that their COGS was $30 million). Basically, there's a decent reward for succeeding and no downside for failing, and sometimes you will even be rewarded for your failures! So why not do something insane?

Also, it seems that VC firms like companies to run the same way their portfolios do– they want companies to try 100 different things, and if only 5 out of those 100 things work, then the VCs will consider that a success. On the ground floor, this creates a lot of misery, since the median employee at the company is almost certainly working on a product that is not destined to perform well, but the shareholders are happy, which is of course all that matters.

Shitty code & shitty data science

The median data scientist is horrible at coding and engineering in general. The few who are remotely decent at coding are often not good at engineering in the sense that they tend to over-engineer solutions, have a sense of self-grandeur, and want to waste time building their own platform stuff (folks, do not do this).

This leads to two feelings on my end:

  1. It got annoying not having some amount of authority over code and infra decisions. Working with data scientists without having control over infra feels like wading through piles of immutable shit.
  2. It was obvious that there is a general industry-wide need for people who are good at both data science and coding to oversee firms' data science practices in a technical capacity.

Poor mentorship

I don't want to be too snooty: in a sense, it's fine for data scientists to suck at coding! Especially if they bring other valuable skills to the table, or if they're starting out. And in another sense, bad code getting into production is a symptom of bad team design and management, more than any individual contributors' faults! By describing the median data scientist's coding skills as shitty, I'm just trying to be honest, not scornful.

The problem is that the median data scientist works at a small to medium-sized company that doesn't build their data science practices around a conceit that the data scientists' code will suck. They'd rather let a 23 year old who knows how to pip install jupyterlab run loose and self-manage, or manage alongside other similarly situated 23 year-olds. Where is the adult in charge?

23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college. So many careers are being ruined before they've even started because data science kids went straight from undergrad to being the third data science hire at a series C company where the first two hires either provide no mentorship, or provide shitty mentorship because they too started their careers in the same way.

Poor self-directed education

On the other hand, it's not just the companies' and managers' faults; individual data scientists are also to blame for being really bad at career growth. This is not contemptible for people who are just starting out their careers, but at some point folks' résumés starts to outpace actual accumulation of skills and I cannot help but to find that a teeny bit embarrassing.

It seems like the main career growth data scientists subject themselves to is learning the API of some gradient boosting tool or consuming superficial + shallow + irrelevant knowledge. I don't really sympathize with this learning trajectory because I've never felt the main bottleneck to my work was that I needed some gradient boosting tool. Rather the main bottlenecks I've faced were always crappy infrastructure and lacking (quality) data, so it has always felt natural to focus my efforts toward learning that stuff to unblock myself.

My knowledge gaps have also historically been less grandiose than learning how some state of the art language model works or pretending I understand some arXiv white paper ornate with LaTeX notation of advanced calculus. Personally, I've benefited a ton from reading the first couple chapters out of advanced textbooks (while ignoring the last 75% of the textbook), and refreshing on embarrassingly pedestrian math knowledge like "how do logarithms work." Yeah I admit it, I'm an old man and I've had to refresh on my high school pre-calc. Maybe it's because I have 30k Twitter followers, but I live in constant anxiety that someone will pop quiz me with questions like "what is the formula for an F-statistic," and that by failing to get it right I will vanish in a puff of smoke. So my brain tells me that I must always refresh myself on the basics. I admit this is perhaps a strange way to live one's life, but it worked for me: after having gauged my eyes out on linear regression and basic math, it's shockingly apparent to me how much people merely pretend to understand this stuff, and how much ostensible interest in more advanced topics is pure sophistry.

For the life of me I cannot see how reading a blog post that has sentences in it such as "DALL-E is a diffusion model with billions of parameters" would ever be relevant to my work. The median guy who is into this sort of superficial content consumption hasn't actually gone through chapters in an advanced textbook in years if ever. Don't take them at their word that they've actually grinded through the math because people lie about how well-read they are all the time (and it's easy to tell when people are lying). Like bro, you want to do stuff with "diffusion models"? You don't even know how to add two normal distributions together! You ain't diffusing shit!

I don't want to blame people for spending their free-time doing things other than learning how to code or doing math exercises out of grad school textbooks. To actually become experts in multiple things is oppressively time-consuming, and leaves little time for other stuff. There's more to life than your dang job or the subject matters that may be relevant to your career. One of the main sins of "data scientist" jobs is that it expects far too much from people.

But there's also a part of me that's just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I dunno, man.

Ultimately nobody really knows what they are doing, and that's OK. But between companies not building around this observation, and individuals not self-directing their educations around this observation, it is just a bit maddening to feel stuck in stupid hell.

Data engineering has been relaxing

These are the things I've been enjoying about data engineering:

  • Sense of autonomy and control.
    • By virtue of what my job is, I have tons of control over the infrastructure.
    • Data engineering feels significantly less subject to the whims and direction of insane management.
  • Less need for sophistry.
    • My work is judged based on how good the data pipelines are, not based on how good looking my slide decks are or how many buzzwords I can use in a sentence. Not to say data engineering doesn't have buzzwords and trends, but that's peddled by SaaS vendors more than actual engineers.
  • More free time.
    • I dunno, it feels like becoming a data engineer cured my imposter syndrome? I feel like I have more ability to dick around in my spare time without feeling inadequate about some aspect of my job or expertise. But this is probably highly collinear with not being a lackey for product managers.
  • Obvious and immediate value that is not tied to a KPI.
    • I like being valued, what can I say.
    • Ultimately the data scientists need me more than I need them; I'm the reason their stuff is in production and runs smoothly.
    • I have a sense that, if my current place of business needed to chop employees, that it would be a dumb decision to chop me over any data scientist.
  • Frankly, I feel really good at what I do.
    • As someone who has worked a variety of downstream data-related jobs, I have both a very strong sense of what my downstream consumers want, as well as the chops to QC/QA my own work with relative ease the way a good analyst would.
    • At my last company I had a lot of "I could totally do a better job at designing this" feelings regarding our data stack, and it has immensely fed my ego to have confirmed all of these suspicions myself.
    • This role gets to leverage basically everything I've learned in my career so far.

By far the most important thing here is the sense of independence. At this point it feels like the main person I should be complaining about is myself. (And self-loathing is so much healthier than hating a random product manager.) As long as my company's data scientists are dependent on me to get code into production, I have veto power over a lot of bad code. So if they are putting bad code in production, that ultimately ends up being my fault.

I think my career trajectory made sense– there was no way I was hopping straight into data engineering and doing a good job of it without having done the following first:

  • See a few data stacks and form opinions about them as a downstream consumer.
  • Get better at coding.
  • Pick up on the lingo that data engineers use to describe data (which is distinct from how social scientists, financial professionals, data scientists, etc. describe data), like "entity," "normalization," "slowly-changing dimension type 2," "CAP theorem," "upsert," "association table," so on and so on.

So, ultimately I have no regrets having done data science, but I am also enjoying the transition to data engineering. I continue to do data science in the sense that these roles are murkily defined (at both my "data scientist" and "data engineer" jobs I spend like 40% of the time writing downstream SQL transformations), I get to critique and provide feedback on data science work, and, hey, I actually did deploy some math heavy code recently. Hell, you could argue I'm just a data scientist who manages the data science infra at a company.

Anyway, great transition, would recommend for anyone who is good at coding and currently hates their data science job. My advice is, if you want to get into this from the data science angle, make sure you are actively blurring the lines between data science and data engineering at your current job to prepare for the transition. Contribute code to your company's workflow management repo; put stuff into production (both live APIs and batch jobs); learn how to do CI/CD, Docker, Terraform; and form opinions about the stuff your upstream engineers are feeding you (what you do and don't like and why). In fact it is very likely this work is higher value and more fun than tuning hyperparameters anyway, so why not start now?

Sorry, this post has no point, so it's ending rather anticlimactically.




All Comments: [-] | anchor

ChrisMarshallNY(10000) 4 days ago [-]

I have no axe to grind, irt data science/engineering, as I have no experience in either.

However, it seems this person's biggest gripe is with good old crap management; the bane of business for hundreds of years.

This line stood out:

> Companies all over were consistently pursuing things that could be reasoned about a priori as being insane ideas– ideas any decently smart person should know wouldn't work before they're tried.

That pretty much summarizes why I have been told that today's companies want only young people. Us 'olds,' are 'negative naysayers,' who say things like 'You know that the laws of physics forbid this, right?' or 'I tried that, a couple of years ago. It didn't work out, and here's why...'.

Apparently, young people are able to do the impossible, because they haven't been told it's impossible, and mixing 'olds' with them, spoils the soup, by telling them it's impossible (or maybe a lot more difficult that they imagine).

fritzo(10000) 2 days ago [-]

This resonates. I wish it didn't. I'm experimenting with ever more delicate communications to avoid spoiling the soup.

3pt14159(10000) 4 days ago [-]

> Meaning you could absolutely suck at your job or be incredible at it and you'd get nearly the same regards in either case.

Story time.

There was once a junior data scientist at Shopify that had learned Python and SQL and was tasked to figuring out how to fix their 'broken app store recommendation engine' but since they didn't know Ruby, they asked for my help in figuring out what was going on.

Well somewhere in the soup of math was a fuzz factor at the very top. Think of it like

    factor = 0.something  # Not 100% sure what the decimal portion was.
    some_complicated_math_that_maxed_out_at_one_pt_zero() + rand(factor)
Now the thing about ruby is that rand is basically broken for floats.

    Negative or floating point values for max are allowed, but may give surprising results.
https://ruby-doc.org/core-2.4.0/Kernel.html#method-i-rand

So basically what they thought they were doing was introducing a bit of randomness that would hinder others from reverse engineering their algorithm. What they actually did was make the recommendation algorithm fifty percent total noise. Yes, it's true. On every load half the recommended app scores were noise.

They fixed the bug and I'm sure a ton of balance sheets for businesses around the world are markedly different know because of it, but I never heard of it again.

This is one of the core problems with data science.

The lack of feedback.

mattkrause(10000) 4 days ago [-]

Ironically....

Having a subset of totally random recommendations wouldn't be a totally terrible idea---especially if you know which they were! It could help push the system out of local minima and it's the obvious benchmark to beat.

grepLeigh(10000) 4 days ago [-]

I'm reading a lot of sour grapes here, but I'd like to offer a more optimistic future snapshot (if you're a data scientist and happen to be reading).

I agree that many companies hire data scientists with only a vague idea about how to utilize them, but the same is true of software people in general. 'Software is eating the world' and so is the practice of extracting value from data.

The margin on software is high - often more than 95% - so there's a lot of room to screw up and 'figure it out' as a business. I think that's why there's a low bar for software and data management compared to, say, an automotive manufacturing line manager.

But that's where the opportunity is, if you're a budding data scientist:

- The business might not know how to effectively use/manage/train/mentor you.

- Upper management might have 20+ years of line of business experience, but will need your help to understand how your team can impact the business.

- You're going to need to seek out ways to impact the bottom line of the business.

All of the above is a recipe for leaping forward in your career. Since data science is a relatively new field, the demand for senior leadership FAR outstrips the available supply.

If you can learn how to effectively manage yourself, your team mates, and your function within the business - you have a ton of negotiating leverage and can name your price.

Source: I'm a data person who 'retired' in their early 30s. Now I do all the research and hard science I want. ;)

disgruntledphd2(10000) 2 days ago [-]

> the demand for senior leadership FAR outstrips the available supply.

This is so very, very true. Most of the 'bad' data science orgs I've spent time with, are bad because leadership is either bluffers or a data engineer/BI type person. It's generally hard for these types to run effective DS orgs as the skills needed are very, very different.

samvher(10000) 4 days ago [-]

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you'd get nearly the same regards in either case.

In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle.

A common one I've seen quite many times is people using a flawed validation strategy (e.g. one which rewards the model for using data 'leaked' from the future), or to rely on in-sample results too much in other ways.

Because these issues are subtle, management will often not pick up on them or not be aware that this kind of thing can go wrong. With a short-term focus they also won't really care, because they can still put these results in marketing materials and impress most outsiders as well.

whatever1(10000) 4 days ago [-]

Blame statistics for that. Wrong outcome? Well you were unlucky you fell into the 1% error range.

Correct outcome? You totally predicted it correctly.

There is literally no way you can screw up something in statistics and not being able to make up a story to defend your approach.

CapmCrackaWaka(10000) 4 days ago [-]

I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies.

I had one candidate who was in charge of a multi-armed-bandit project at their current company. I asked them how it worked, and how they settled on that. Their response was 'you know, I'm not really sure, the code was set up when I got there'. He had been there for over a year, and could tell me nothing!

> A common one I've seen quite many times is people using a flawed validation strategy (e.g. one which rewards the model for using data 'leaked' from the future), or to rely on in-sample results too much in other ways.

It's funny you mention this, we have a direct competitor who does this and advertises flawed metrics to clients. Often times our clients will come back to us saying 'XYZ says they can get better performance', the performance in this case being something which is simply impossible without data leakage or some flawed validation strategy.

etempleton(10000) 4 days ago [-]

I become wary any time someone utters the phrase, 'show me the data' or any variation there of. There is a specific type of leader who thinks that within the data lurks a magical solution just waiting to be discovered. There is also the leader who uses data as a trump card to win arguments and these folks are perhaps even worse. This is not new. The origination of the phrase, 'lies, damned lies, and statistics,' can be traced to the 1800s. I propose the following update:

There are three kinds of lies: Lies, damned lies, and data

I am being glib, I of course do not think all data is inconsequential, rather it is more often used from a place of ignorance or a place of ill intent it is rendered, on the whole, useless.

hcks(10000) 4 days ago [-]

> Because these issues are subtle, management will often not pick up on them or not be aware that this kind of thing can go wrong.

Management is not your teacher at school, it is not there to check up your results make sense.

Management mostly assumes you're competent at your job.

mmcnl(10000) 4 days ago [-]

This is somewhat captured in the article as well.

'Managers will say they want to make data-driven decisions, but they really want decision-driven data. If you strayed from this role– e.g. by warning people not to pursue stupid ideas– your reward was their disdain, then they'd do it anyway, then it wouldn't work (what a shocker). The only way to win is to become a stooge.'

In science, a good scientific result can be bad for business. There is often little appreciation for the 'science' in data science.

jmount(10000) 4 days ago [-]

Super point. I can't resist repeating it back. When incorrect work outperforms correct work in superficial evaluation, it is then selected for.

6gvONxR4sf7o(10000) 4 days ago [-]

This is the defining pain point for data science, in my experience. There's no simple ground truth to test competence against.

If someone tells you that the data says their work is good, the only real way to know if they're right or wrong is to look at what the data says yourself. If 99% of the work is building and 1% is checking something like latency, then you're likely to have more than one set of eyeballs on that 1%. But if 99% of the work is putting the data together and doing the analysis, then you're unlikely to have more than one person ever look at that part.

So incompetence goes unchecked (or worse, it is rewarded).

whiplash451(10000) 4 days ago [-]

You should pay the price for data leakage very quickly in production.

Does management look at slides or AB test dashboards?

jrochkind1(10000) 4 days ago [-]

> Approaches that are wrong from a statistics point of view

When OP talked about 'the main bottleneck to my work' in terms of areas he would need to learn more about -- I was expecting him to talk about facility with statistical methods and using them appropriately!

I'm not sure what to take from the fact that he never did! I would like to ask him what he thinks about that!

mumblemumble(10000) 4 days ago [-]

The problem is that nobody actually wants data science. They want data pseudoscience.

And for the same reason that people tend to want pseudoscience instead of science in any other domain, too. Science is slow, tentative, and messy, and usually responds to questions with even more questions rather than with answers.

Pseudoscience tends to be much more concerned with exuding confidence and providing clean-cut answers. It's what happens when a desire for science meets a need for instant gratification. Along the way, things like blinding and controls and watching for bias and validating assumptions tend to get dropped when they're inconvenient or difficult to explain. And they're always inconvenient and difficult to explain.

rgrieselhuber(10000) 4 days ago [-]

I've always disliked how data science was positioned within companies as well, it's outside the critical path of product and engineering, which means it becomes a mere abstraction to management (e.g. "throw that problem to the data science team and see what they come up with"), resulting in very vague and abstract requirements and, hence, deliverables. I think there is huge value in the discipline and technologies, but it unfairly gets relegated when not integrated to the whole product / engineering process. Hence, the title / concept of Data Engineer seems like a much better fit for this role within many companies.

dqpb(10000) 4 days ago [-]

> Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results.

This is to be expected from an information theory point of view. It's why 'fake news' will always be a thing.

chasely(10000) 4 days ago [-]

I've been pitched by many 'data-driven' vendors offering predictions. They often have very impressive accuracy metrics (RMSE, R2, etc). When I dive into the details these metrics are often reported using in-sample predictions.

I see this pointing to any of the following:

a) DS teams overpromising the accuracy of their approaches b) marketing driving the narrative and DS getting pulled along c) incompetence from the DS team

2devnull(10000) 4 days ago [-]

This gave me a chuckle. If you read the feature article you understand that this is also because management wants "decision driven data." They have an idea and use ds to provide charts and tables to support their idea. The harder the idea is to support, the greater value data science is able to provide.

I guess data science is inferior to research in this way. People care about research methods, rigor, etc... Maybe data scientists should adopt stricter standards, like actual scientists.

apohn(10000) 4 days ago [-]

>Meaning you could absolutely suck at your job or be incredible at it and you'd get nearly the same regards in either case.

One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build something amazing, or build a house of cards, nobody cares as long as the money people are happy, even if the business use case turns out to be wrong in the long run.

>With a short-term focus they also won't really care, because they can still put these results in marketing materials and impress most outsiders as well.

Forget Data Science, you see this in KPIs as well. Say a crappy metric has to be moved by Q2 next year and people will destroy the company to move it.

I feel like Data Science is just one of those areas where you are exposed to a wider range of people and get to feel the full crapola of the insanity of working in a corporation. For lots of roles (e.g. Engineering) you get to hide in a hole behind layers of people and not see some of this insanity.

boringg(10000) 4 days ago [-]

Coles notes: Data Engineer - more money, more clout, less analysis / interesting projects, more job security, more infra style work Data Science - less money, a lot of random projects (Sometimes totally overqualified for), more analytical, don't have as much clout / confusion & lots of people don't actually understand capabilities.

zhdc1(10000) 4 days ago [-]

Checks my CV:

50k+ lines of R, 10k+ lines of Julia, 5k+ in Python, C, and who knows what else. Most of it for what is, essentially, data engineering work.

Where do researchers with social science degrees fall on this scale? Less money, less clout. The projects are certainly interesting though (which is why I do what I do).

throwawaaarrgh(10000) 4 days ago [-]

I've never met a data scientist who could do anything more than basic statistics combined with the Python skills of a fifth grader (that's probably insulting to today's fifth graders tho). I honestly have no idea what they're supposed to be doing or why they're paid so much money. Pay a high school junior for the same and get better work. And where's the scientific method? Where's the experiments and rigor?

r-zip(10000) 4 days ago [-]

> where's the scientific method? Where's the experiments and rigor?

No time for that, buddy. CXOs need results, ASAP! (This was basically the attitude of my managers at the last place I worked).

euix(10000) 4 days ago [-]

I have to agree with a lot of this - I started my career as a data scientist right out of a STEM PhD back when the term just started coming into existence. At the time, anyone who wanted to get hired as a Data Scientist needed to be trained as a professional scientist, i.e. have a PhD - at first my expectation that the purpose of my job was to apply the scientific method to solve business problems by leveraging the companies own data as the empirical evidence - whether I did this using machine learning, excel tables or a chalkboard didn't matter. ML was a barely used term at the time, the first version of Tensorflow wasn't released until later that year.

But over time, the higher up I climbed the more I realized the job had marginal business impact. Usually a big company would hire a bunch of PhDs with fancy degrees and stick them in some 'Advance Analysis' department and leverage them as internal consultants, which just meant creating some models, writing a powerpoint deck, get a pat on the back from the execs - not a single model would ever see the day of light. I got all the way up to Director this way, before calling it quits this January, at the end I had basically nothing to do except work on 'corporate AI strategy', which meant writing presentations and white papers for upper management.

It was comparatively easy job, one could coast their entire life in some of these corporations - especially in government sanctioned oligopolies like banking.

wheelinsupial(10000) 4 days ago [-]

> the higher up I climbed the more I realized the job had marginal business impact

Do you have any observations why? I'm a pretty lowly business analyst, but my observation is if you don't own the decision making (usually by having profit and loss responsibility), you can't have much impact. Possibly it's the companies and industries I've worked at, but at the end of the day if the results don't meet expectations, it's the business owner that gets fired and not the people providing the recommendations.

ZephyrBlu(10000) 3 days ago [-]

Props for saying this, but you didn't realize the 'marginal business impact' bit until you were a Director..? Seems awfully convenient.

d23(10000) 4 days ago [-]

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you'd get nearly the same regards in either case.

This seems to be a problem with the industry as a whole. I'm speaking as a SWE, but I've observed similar things with PMs. I don't think it's impossible or even very hard to appreciate the right things, it just requires a bit of thought and the correct value system. Both of those seem to be a bit too far of a reach though.

r-zip(10000) 4 days ago [-]

I think a big part of the problem is that most PMs are non-technical, and at some point up the chain so are most managers. Data science, when done thoughtfully, requires you to appreciate minute issues from data collection and management all the way through feature engineering, model selection/design, and validation.

The biggest, hardest bridge to cross was an appreciation of the importance of metrics. For some reason, getting a business person to grok something as simple as precision/recall/F-beta is a near-impossible task. You can do multiple presentations on it (after having honed those presentations over years with multiple manager audiences), and it never sticks. It's always 'what's the accuracy?' It's impossible to do good work when your bosses insist on measuring and therefore optimizing for the wrong thing (which in my experience consulting for multiple Fortune 500 businesses, they always do).

Even worse, many organizations have such broken politics/cultures that the managers can't even tell you the big picture of what the project is trying to accomplish. Once you finally piece it together from the people who know their roles in-depth, it becomes clear that what they're trying to do is totally infeasible. At least that was my experience more than half the time.

salty_biscuits(10000) 4 days ago [-]

See a lot of pessimism here, and heard a lot of the same stuff by grumpy engineers over my career 'nobody knows how to do statistics properly', 'scientists are bad at coding' and 'managers don't care about quality and rigour'. I'm old enough to say it all pre-dates data science as a term. It all comes across as a conspiracy like everyone is bad on purpose.

There is a null hypothesis here, that the average person in role x is just average at that role. It is an extraordinary person who has high level skills across multiple domains like maths/science and coding, maybe so extraordinary that they wouldn't be working with you...

I'll admit that the article rings true, but I think there is an implied intentionality that I don't agree with. We are all just plodding along, doing our best with limited information and skills.

arcturus17(10000) 4 days ago [-]

> there is an implied intentionality that I don't agree with. We are all just plodding along, doing our best with limited information and skills.

Never got the feeling the author blamed the data scientists (he was one himself), but rather management, and not their bad intentions but their incompetence.

frgtpsswrdlame(10000) 4 days ago [-]

Read this yesterday and absolutely loved it. Especially feel the pain regarding working with management. I think there's an accountability that comes with evaluating management decisions with data that nobody really wants. I still have a lot of half-formed thoughts/opinions about this but it really feels like data-driven requires strict discipline but the data people who would be accountable for that discipline both 1. aren't empowered to wield it and 2. probably don't want to wield it anyways.

Also agree about the simple tools but it's really hard from a career perspective. If I deploy XGBoost in production and put it on my resume, I'm making double my salary next year. If I can find a simple ruleset or linear regression that performs 90%+ as well as the XGBoost and put it in production then nobody cares even though it feels like distilling the complex down to the simple is really where the value is.

oneoff786(10000) 4 days ago [-]

The thing is that anymore anything short of a deep net is equally difficult to implement as a linear regression, if not easier due to nulls and categoricals.

Also, modern tooling makes a lot of these models more than explainable enough for a lot of cases... 10% is a lot

disgruntledphd2(10000) 2 days ago [-]

Just as an FWIW, I've been interviewing data people for about a decade now, and I would definitely be far more impressed with the simpler approach, but I realise that people like me are a minority in the field.

csimon80(10000) 4 days ago [-]

It's so buzz word heavy. I had a manager that wanted me to solve a problem using the monte-carlo method when it fact the problem had a closed form solution...

r-zip(10000) 4 days ago [-]

Why do math on paper when you can write code and look cool (and not like a useless academic)?

CarbonCycles(10000) 4 days ago [-]

Now that's both funny and sad. Mostly sad....

drgiggles(10000) 4 days ago [-]

Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theory and at least a decent ability write passable code. With regard to engineering ability, certainly people exists with both skill sets, but its an awfully high bar. It is similar in my field (quant finance), the number of people that understand financial theory, valuation, etc and have the ability to design and implement robust production systems are few and you need to pay them. I don't see data science openings paying anywhere near what you would need to pay a 'unicorn', you can't really expect the folks that fill those roles to perform at that level.

kxc42(10000) 4 days ago [-]

That answer somehow reminds me of an article in logicmag: An Interview with an Anonymous Data Scientist [1].

[1]: https://logicmag.io/intelligence/interview-with-an-anonymous...

codeulike(10000) 4 days ago [-]

Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result.

I dont know anything about Data Science but as a bystander with a mathematical background thats what I assumed was going on so its kindof interesting to see it spelt out like that. Like you've put words to a preconception that I didnt even know I had.

manicennui(10000) 3 days ago [-]

I don't know many software engineers who have the ability to design and implement robust production systems.

layman51(10000) 3 days ago [-]

> Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result.

That's a good way of putting it. I remember in my first calculus-based probability+statistics class in college, I felt incredibly challenged by the theory. I wondered why there are so many probability distributions out there, why the standard stats formulas look like they do, what 'kernel density estimation' even is, etc.

On the other hand, my data science course did include some theory, but a big part of it was also learning how to type the right commands in R to perform the 'featured analysis of the week' on a sample data set. Something about these lab exercises felt off because it felt more like training rather than education. The professor expressed something along the lines that if we wanted to go far with this in the future, he would expect us to design the algorithms behind the function calls. I think the analogy he used was 'baking a cake from scratch rather than buying a ready made one at the store.'

jghn(10000) 4 days ago [-]

I worked adjacent to the data science field when it was in its infancy. As in I remember people who are now household names in the field debating what it should be called.

At the time I considered going down that path, but decided I did not have anywhere near the statistics & math knowledge to get very far. So I stuck with the path I had been on. Over time I saw a lot of acquaintances jumping into the data science game. I couldn't figure out how they were learning this stuff so fast. At some point I realized that most of them knew less than I did when I decided I didn't know enough to even begin that journey.

Of course, I was comparing myself against the giants of the field and not the long tail of foot soldiers. But it made for a great example to me of how with just about everything there's a small handful of people who are the primary movers, and then everybody else.

adamsmith143(10000) 4 days ago [-]

On the flip side you used to have statisticians writing code that is frankly unusable in a Production environment. You would weep at the R code I've seen and had to turn into something to actually produce business value.

ShredKazoo(10000) 3 days ago [-]

>Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge

An important thing people miss is that shallow statistical knowledge can cause subtle failures, but shallow software engineering knowledge can cause subtle failures too.

A junior frontend developer will write buggy code, notice that the UI is glitched, and fix the bug. A junior data analyst will write buggy code, fix any bugs which cause the results to be obviously way off, but bugs which cause subtler problems will go unfixed.

Writing correct code without the benefit of knowing when there is a bug is challenging enough for senior developers. I don't trust newbie devs to do it at all.

Context here is I used to work in email marketing and at one point I was reading some SQL that one of the data scientists wrote and observed that it was triple-counting our conversions from marketing email. Triple-counting conversions means the numbers were way off, but not so far off as to be utterly absurd. If I hadn't happened to do a careful read of that code, we would've just kept believing that our email marketing was 3x as effective as it actually was.

So, it's impossible to know how much of a problem this is. But there is every reason to believe it is a significant problem, and lots of code written by data scientists is plagued by bugs which undermine the analysis. (When's the last time you wrote a program which ran correctly on the first try?) Any serious data science effort would enforce stern practices around code review, assertions, TDD, etc. to make the analysis as correct as possible -- but my impression is it is much more common for data analysis to be low-quality throwaway code.

itissid(10000) 4 days ago [-]

I have never understood the what a good ML engineer couldn't do and a Data scientist could in _majority_ situations. When you need a decision to be made based on data its just common sense risk analysis added together with basic statistics.

I feel some good field training in statistics(Look up Andrew Gelman) a couple of good courses on Linear, Bayesian Regression is all you need, rest is just engineering skill.

The dichotomy between ML Engg and Datascience is as stupid as was between Systems Engg and Application Engg before Devops came along.

adamsmith143(10000) 4 days ago [-]

Can you make pretty graphs in Matplotlib and sound smart in front of Executives?

TrackerFF(10000) 4 days ago [-]

IMO a data scientist should also be a domain expert, in the same way analysts are.

But of course, too many view DS as some abstract skill where domain knowledge is not needed, and where the methodology will solve all problems / provide insight.

time_to_smile(10000) 4 days ago [-]

I think the qualifying term here is 'good'. I've worked with a surprising number of MLEs that don't really understand gradient descent or how most models really work under the hood. They certainly couldn't implement most things from scratch if they needed to (neither could most data scientists).

I used to think an MLE was a solid engineer who also had a strong quantitative and numerical computing background. The kind of engineer that always has a copy of Numerical Recipes handy, and if needed, could reimplement core components of statsmodels and sklearn in javascript.

I think after this current contraction in tech is over we'll see that most of the remaining 'data scientists/MLEs' will be the type of engineer I imagine an MLE to be.

civilized(10000) 4 days ago [-]

Great post. A few random comments...

> The median data scientist is horrible at coding and engineering in general. The few who are remotely decent at coding are often not good at engineering in the sense that they tend to over-engineer solutions, have a sense of self-grandeur, and want to waste time building their own platform stuff (folks, do not do this).

> It was obvious that there is a general industry-wide need for people who are good at both data science and coding to oversee firms' data science practices in a technical capacity.

The job of overseeing a crowd of stubborn self-important over-engineerers sounds pretty thankless.

> 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college. So many careers are being ruined before they've even started because data science kids went straight from undergrad to being the third data science hire at a series C company where the first two hires either provide no mentorship, or provide shitty mentorship because they too started their careers in the same way.

Startups are a low-paid job with a lottery ticket for a little dash of excitement. You get what you pay for.

> ...I live in constant anxiety that someone will pop quiz me with questions like "what is the formula for an F-statistic," and that by failing to get it right I will vanish in a puff of smoke. So my brain tells me that I must always refresh myself on the basics.

Focusing on the basics is better than pretending to understand fancy things, but even this level of 'continuous professional training' or whatever you want to call it is, to me, a bit off the mark. We can look up formulas whenever we want these days. We need more meaningful ways to test our understanding of things.

languageserver(10000) 4 days ago [-]

> 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college.

Ageism is disgusting and I cannot believe such blatant discriminatory language is seen as OK for a link posted to hackernews. How would you all say if he wrote that 40+ year old programmers should xx?

gpresot(10000) 4 days ago [-]

I realise the article was written for a specific audience for which this may be obvious, but what is the difference between data scientist and data engineer (in terms of what their job is)?

kingant(10000) 4 days ago [-]

Data scientist actually cook up and run the statistical/ML models on data and write reports about their 'findings'.

However, the data that data scientists want to use is often messy and comes from varied sources. Hence, data engineers do supporting infra work like cleaning/loading data from different databases, etc.

teraflop(10000) 4 days ago [-]

Generally speaking:

'Data engineering' means building systems that can manipulate data (e.g. storing, retrieving, and delivering it). There are usually fairly well-defined functional requirements about what the system is supposed to do, plus goals about performance and reliability that might be slightly more nebulous.

'Data science' means building systems that can draw conclusions from data. The functional requirement is usually some form of 'accuracy', as measured somehow against some kind of human evaluation of the same conclusion.

Concretely: a data engineer might be asked to build a system that can ingest every tweet posted to Twitter, and return the 10 most widely-used hashtags in the last hour. A data scientist might be asked to build a system that looks at a tweet and figures out what language it's written in, or whether it's spam, or whether an attached image is pornographic.

zmmmmm(10000) 4 days ago [-]

Feels like data science outside of deep learning is slowly edging towards the trough of disillusionment. Time to rebrand your CVs folks in line with the new hype, while waiting it out to get to the plateau of productivity ...

r-zip(10000) 4 days ago [-]

The funny thing is that it shouldn't be. There is plenty of potential, but all sorts of (mostly non-technical) factors are getting in the way.

jstx1(10000) 4 days ago [-]

As someone who also wants to move away from data science, data engineering is the last thing I would want to do. I think DE comes with many of the same problems and it's also a very ill-defined career track; I wouldn't recommend it to anyone. ML engineer or backend developer seem like much more appealing job profiles.

starwind(10000) 4 days ago [-]

I went data science to data engineering and been very happy. You may have trouble moving into backend development direct from data science because data scientists don't have a reputation for writing solid maintable code so data engineering could be a nice intermediate step cause it's heavy python

aulin(10000) 4 days ago [-]

would you highlight some of the biggest differences between ML engineering and data engineering? I believe they're sometimes used interchangebly especially if 'data' is 'datasets' for ML.

mynameisash(10000) 4 days ago [-]

My title is still software engineer, but I effectively do data engineering, and I work closely with data scientists.

I love a lot of it, but there's still plenty of bullshit to deal with. Just in the technical side, dealing with Python is a perpetual gong show, and most of my team's work seems to revolve around configuration of secrets and K8s.

I'm fortunate to be the guy that nerds out about performant code, so when something inevitably turns out to be a perf bottleneck, I can turn back into a regular old software engineer who trades in big data. Which I think is a better title/charge than data engineer, anyway.

I've talked with plenty of ML engineers, and they seem to immensely enjoy what they do. It seems that the periphery of data engineering is great; the core of it, not so much.

spaniard89277(10000) 4 days ago [-]

Correct me if I'm wrong because I'm on the receiving end of such models, but I feel that many times a couple of linear regressions, surveys and qualitative work with customers could land much better results.

I say so because I've had time to read some of the reports that DS teams produce to drive decisions in my BIGCORP and it makes very little sense most of the times.

And we suffer from it because we have direct contact with clients, but nobody cares about my department opinion, they will rather believe in some model where I can see insane dispersion in datapoints when they plot em in reports, conclussions by people who clearly has zero understanding of our business.

I'm forced to make decisions on how to treat certain customers, by entering data into some software and being given an output I can't challenge, which produces lots of insane and unfair situations.

Also, IDK how they clean and treat their data, but if they're relying on our ERP's data, good luck. Our CRM if full of BS because most employees rush to put whatever it allows to continue as they need to keep up with KPIs, so they aren't trying to make nice comments and check everything is ok.

SkyPuncher(10000) 3 days ago [-]

Actual, human based decisions will almost always win out.

Data is only helpful when it is directly and clearly tied to the problem.

* Good: 'Our customers are complaining of random drop-outs. We've noticed X% of requests to Y service take longer than Z time. We believe that's the problem'.

* Bad: 'Companies who are most successful on our platform upload X things in their first Z days. We must find a way for everyone to upload X things in Z days'.

webnrrd2k(10000) 4 days ago [-]

The word 'conclussion' is especially appropriate in this context -- it could easily be pronounced like a combination of 'conclusion' and 'concussion'.

Icathian(10000) 4 days ago [-]

This hit all the same high notes I was feeling when I quit Data Science to become a software engineer. It's an infinitely better gig and I encourage all my colleagues with enough chops to make the same switch.

sebastiansm(10000) 4 days ago [-]

How did you do the transition from DS to SWE?

_dain_(10000) 4 days ago [-]

yep, exact same feeling here. I had several years as a 'data scientist' and it was a an almost totally bullshit job. the org bought into the hype and hired a cohort of us straight out of university, but then couldn't find anything data-science-y for us to actually do. what I actually ended up doing 95% of the time was taping together dodgy excel-based workflows using python scripts. it gave me a visceral appreciation for Conway's Law. the other 5% was when I got to do some genuinely interesting mathematical work, but that wasn't 'data science' either, it was more like operations research. I lived for that stuff, but there wasn't enough of it.

so I jumped ship and became a software engineer. better pay and more interesting problems.

tgtweak(10000) 4 days ago [-]

I feel like in the near future there will be a more formal hybrid role between data engineering and data science - like devops or full stack developers. The best data scientists I have worked with (ML mostly) have been incredible data engineers as well - some of them former sysadmins, backend developers or DBAs themselves. They know where to get the data, how to set up pipelines and jobs, how to make sure they run properly, best practices for reading and writing to databases so they don't fall over, error reporting and logging, hosting their inference models on APIs, security by design... The amount of back-and-forth that gets cut out to go from raw data to product is huge when you compare it to a traditional siloed setup of business analysis/stakeholder, data science, infrastructure/security and systems engineering.

I know some people cringe (mostly infra) when they think of data scientists having direct access to databases and infrastructure but honestly you should have a level of understanding and responsibility to get there.

The data scientists that do data engineering are usually much more valuable to the company and definitely earn more.

rjtavares(10000) 4 days ago [-]

I agree, up to a point. I feel that companies up to a few billion dollars in stable revenues in a non data intensive business don't need teams of Data Scientists, but would benefit from some people having Data Science skills (both Data Engineers and Business Analysts).

agomez314(10000) 4 days ago [-]

'Managers will say they want to make data-driven decisions, but they really want decision-driven data'

Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors?

phkahler(10000) 4 days ago [-]

>> 'Managers will say they want to make data-driven decisions, but they really want decision-driven data' Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors?

Find me some evidence of WMDs in Iraq! Yessss Sir!

A4ET8a8uTh0(10000) 4 days ago [-]

That one stood out to me as well, but, to be fair, this predated current 'fashionable trend' for data driven decisions. It is, sadly, not a new development, but something to still be overcome.

Reimersholme(10000) 4 days ago [-]

Rather than wanting to confirm priors, I believe this usually is a problem with neither the PM nor the data scientist ensuring that the problem formulation is good enough before diving in. I.e., what data would be needed to actually test the hypothesis? Do we have that data or not? Is the hypothesis even formulated in a way to be falsified in theory?

I've seen so many analysis tasks where data scientists without questioning went away for a few weeks to crunch data and come back with some random graphs and statistics that are completely useless as decision support.

yamtaddle(10000) 4 days ago [-]

I've found this to be the rule, not the exception. Pointing out extremely-obvious (to me? Maybe I'm just unusually good at it? I don't even have much formal science training, and hell, barely any math training by the standards of HN folks, though) damning errors in experimental construction that should invalidate the whole thing won't earn you any friends, even if you do it before the work is undertaken, and even if you're telling the person who's claiming to want good data and useful results. Everyone seems to just want a veneer of science to what they're doing, not actually good efforts at it. As long as you have a paper-thin layer of justification that falls apart if anyone looks at it long enough, that's considered good enough and people will sit around in meetings nodding along.

Of course, in many situations the business totally lacks what it needs to correctly do the 'data-driven' stuff they want to, and it'd take a good deal of up-front effort by competent people to get it, amounting to entire new projects or deep modification of existing projects.

So, given the choice between: going without that stuff and acknowledging that a lot of what they're doing is guesswork and gut decision making, or simply arbitrary; putting a smaller but still-large amount of work into finding out what they can glean from what's available; spending the time and money to collect what they need, the right way, to do the data-driven decision making they claim to want to do; and insisting they're doing things 'data driven' but having all their data hopelessly ruined by e.g. selection bias and comically-bad experimental construction that can't possibly be yielding reliable results, so they can cheap out and get no actual 'data-driven' benefits aside from falsely claiming that's what they're doing—they tend to go with that last option, nearly every time!

lajosbacs(10000) 4 days ago [-]

This especially sucks if you are the middle manager. You know that what you are asked to do is a complete BS but you have to somehow communicate it to you underlings (who see through the BS) without using sarcasm or snarky remarks.

cobbzilla(10000) 4 days ago [-]

So true. This article accurately describes DS at many companies.

The preceding sentence is a hilariously cynical zinger:

"Those who have seen my Twitter posts know that I believe the role of the data scientist in a scenario of insane management is not to provide real, honest consultation, but to launder these insane ideas as having some sort of basis in objective reality even if they don't."

scifibestfi(10000) 4 days ago [-]

Calling it Data Science was a tell. Have you noticed how non-scientific things add 'science' to the name to make it sound like it has scientific rigor?

Data Science, Political Science, Social Science, Scientology

Compare to Physics, Biology, Math.

r-zip(10000) 4 days ago [-]

Computer science...

lajosbacs(10000) 4 days ago [-]

I too am critical of data science, but I think that this is a bit unfair. The scientific part could be called just 'statistics'.

xyzelement(10000) 4 days ago [-]

// it was often personally unfulfilling (e.g. tuning a parameter to make the business extra money).

He lost me here. Something I've always loved about being an engineer (and now in product) is that something small we do/tweak can have big impact.

If you tuned a parameter and that actually had tangible impact on the business, that's like the best case scenario and should be celebrated (vs doing some cool rocket science stuff that ends up unused and doesn't matter)

cmmeur01(10000) 4 days ago [-]

And all that extra profit is hovered up by the people above you that had nothing to do with it. Validated engineering cost savings should be treated like sales, the engineer gets a percentage.

adamsmith143(10000) 4 days ago [-]

If I work my ass off and my model recommends a few extra units, I don't see another dime, so where's the motivation?

starwind(10000) 4 days ago [-]

I made this same transition from data science to data engineering about 18 months ago and I've never looked back.

I hated working with bad code and dealing with arrogant phds who don't value good code. I've seen so many terrible Jupyter Notebooks just copied and pasted into VS Code and the data scientist just washed their hands of it calling it 'production ready.' Here's a conversation I've had multiple times:

Me: have you ever considered not making every variable global scope

Them: that's just software engineering. We do machine learning

Me: if it's just software engineering, then why can't you do it?

Meanwhile, automated data science tools are getting halfway decent. If you know what algorithm to pick and you don't need to run millions of records through the model every minute, your standard business analyst could probably get a solid model going--at least as well as most data scientists for all the reasons the article mentions.

And I like that I know I can do data engineering. With data science you can never really know if you can hit your target metrics given the data you have. So data scientists end up encouraged to fudge their results or make sloppy decisions. With data engineering I can say 'yes this is doable or no that's not' and people believe me.

My prediction: there's value in the massive volume of data but most of it can be had through standard dashboards, some summary statistics, a graph network, or maybe a linear/logistic regression. Most data science is BS and companies aren't getting the return they need to pay for these guys. (And good God, you almost certainly don't need a neural network.) Meanwhile, data engineering will get integrated into software development, and machine learning—by virtue of its proliferation through academia—will just become another tool for software developers. Data scientists won't get laid off enmass but they will go the way of the webmaster: either pick up new skills and evolve or move on til they end up with new titles

srajabi(10000) 4 days ago [-]

This resonates with me so much, I stumbled into data science out of University a decade ago. Left it to do SWE and came back to it in the last 3 years.

So many data scientists are full of themselves thinking they are magicians and software developers are blacksmiths who are beneath them.

Incrementally at my company the SWE's have automated so much of the data scientists workflow that they end up just as you describe, using the tooling and being relegated to becoming analysts.

After 3 years coming back to this field, I see the writing on the wall: In the 90's most models were created by software developers, in the 2030's most models will be created by software developers.

wodenokoto(10000) 3 days ago [-]

I don't get it. As a data engineer aren't you the one who has to deal with the DS code? Whereas a DS someone else has to deal with your code?

jmhammond(10000) 4 days ago [-]

> But there's also a part of me that's just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I dunno, man.

This is true everywhere. As a professor, every semester I'm baffled by students who aren't curious. But I've come to terms that there is a difference between those who will graduate and go on to be readers of hacker news and write this kind of article, and those who won't.

oneoff786(10000) 4 days ago [-]

Seems pretentious to me. I've never bothered to look through many things I use. I look extensively at how to use them and what the API offers. I have a good intuition for how most models work. I don't really care about the specifics of the implementations.

I have more important things to do. The hacker mentality, imo, is about identifying what's useful for you to explore to accomplish whatever you need. Often that's a lot of glue between things that other people built. Other times it's tweaking the internals to do something a bit different.

strikelaserclaw(10000) 4 days ago [-]

curiosity is good but there is so much stuff to learn out there that for many fields learning things deeply is much less important than learning a lot at 25-35% depth.

nicoburns(10000) 4 days ago [-]

I'm pretty curious, but I wonder whether I would have come across that way that my college professors. I felt like college stifled my curiosity. Undergraduate courses rarely care about original or creative work, or about students pursuing their individual interests. They more or less want students to learn what the authorities in the field think.

I did student representation while I was at college, so I had quite a bit of contact with teaching staff around discussing the learning process. There were a lot of complaints from their side that students weren't engaging with the course and were rote learning answers for exams.

My perspective was that most of the courses were badly taught (students were given little guidance and struggled to learn the basics) AND badly examined (you had to guess at what the professor wanted in order to score well - it wasn't actually assessing learning accurately). The courses where you found truly curious students were the ones that taught the basics in a way that other professors would consider hand holding (which meant they could get passed that onto more advanced material), and gave clear advice on what was expected in and how to approach the exam (so that students didn't have to worry about that and could focus on learning and their interests).

You'll always get some students who just aren't interested (perhaps they picked the wrong course, or simply aren't that academic), but you'll also find that the same students respond dramatically differently to different environments.

boringg(10000) 4 days ago [-]

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished.

I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre-determine if I think the time spent pursuing that rabbit hole has any value to it. Granted you never know the outcome - it is alway a gamble.

mschuster91(10000) 4 days ago [-]

Not everyone is wired that way. Personally, I have taken apart and reassembled most of the tech stuff I have at home simply because it interests me how things work (and broke and repaired a non-negligible amount of them in the process, to add), I've dabbled in repairing cars, gas boilers, do my own electricity work... but in my social circle, I'm pretty much the only one. And as I grew older, managed to land myself an s/o, I kind of get why - two other parts come into play:

The first issue is a lot of acquiring broad-spectrum knowledge involves risking quite an amount of money. A good DSLR cam can easily rack up a few thousand euros, a fully spec'd Mac Pro or larger drones cross the five digits without blinking. Messing around with gas and electricity can kill you, messing with water pipes can cause immense water damage. It takes a lot of ... let's say recklessness to even think about dealing with this if you're not a professional, and you have to have the resources in the first place.

But the real issue is time. Students, at least here in Europe, don't have the luxury of taking six or seven years for their basic diploma - 'thanks' to the Bologna reforms, you're fucked if you can't make it in the designed timeframe as you won't be eligible for most kinds of financial aid. That means you simply cannot afford 'wasting' a week to get that deep level of knowledge, you simply are happy enough if it runs well enough to get a passing grade. And once you've entered the workforce, it becomes even harder to have actual hobbies. It's one thing if you live alone, no one will bat an eye if you pull in an all-nighter on a weekend with just yourself, a crate of beer and a laptop and that's assuming you're not completely drained from your average 40 hours work week, 10 hours of getting to the workplace, and another 10 hours on domestic chores. When you live together with another person, the game completely changes: they also want time and attention from you - bonus points if your s/o has roughly the same interests that you have (which is why I suspect so many people meet their s/o at work). And with children... forget about hobbies of any kind if you don't have enough resources for either yourself or your s/o to be a stay-at-home parent.

This is why I so strongly advocate for a four-day and six-hour work week, a proper minimum wage and government-subsidized affordable housing for everyone. Just imagine what useful things people could run as side projects if they actually had the time to pull them off, not to mention the obvious physical and mental health benefits of not having to struggle with survival every single day. Add to that the elimination of 'bullshit jobs' and an end of wasting the best minds of the world on financial bullshit (i.e. HFT, 'quant investment funds') or advertising... or getting rid of racism and other discrimination. We as humanity could make so much more progress if we were not so hell-bent on exploiting each other.

warinukraine(10000) 4 days ago [-]

> But there's also a part of me that's just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I dunno, man.

Because there's a lot of things out there which are also interesting, and you don't have time to do all of them, so you choose. And different people choose differently.

angry_moose(10000) 4 days ago [-]

In a recent past life, I was a HPC (high performance computing) administrator for a mid size company (just barely S&P400) who was in the transportation industry, so I had a lot of interactions with the 'data science' team and it was just a fascinating delusion to watch.

Our CTO did the 'Quick, this is the future! I'll be fired if I don't hop on this trend' panic thing and picked up a handful of recent grads and gave them an obscene budget by our company's standard.

The main problem they were expected to solve - forecasting future sales - was functionally equivalent to 'predict the next 20 years of ~25% of the world economy'. Somehow these 4 guys with a handful of GPUs were expected to out-predict the entirety of the financial sector.

The amazing part was they knew it was crap. All of their stakeholders knew it was crap. Everyone else who heard about it knew it was crap. But our CTO kept paying them a fortune and giving them more hardware every year with almost no expectation of results or performance. It was a common joke (behind the scenes) that if they actually got it right, we'd shut down our original business and becomes the world's largest bank overnight.

At least it finally gave the physics modelers access to some decent GPUs which led to some breakthrough products, as they finally were able to sneak onto some modern hardware.

fnands(10000) 4 days ago [-]

Honestly, I can't tell you how many jobs ads I saw where I was wondering: 'What would they expect me to bring to the table here?'

Some companies just don't have the data, or heck even the need, for data scientist yet try and hire them anyway.

Give smart people a fundamentally ill-posed problem and they won't get anywhere anyway.

quickthrower2(10000) 4 days ago [-]

Not the first HN comment I have seen where $real_useful_department borrows resources off overfunded $bullshit_department to get the job done inspite of management.

mrtksn(10000) 4 days ago [-]

In retrospect, maybe they made the right call for themselves when the money was pouring. Probably everyone involved was paid very well for the charade. The ethics can be questionable but maybe its some kind of wealth redistribution, after all the people with money are trusted to make the calls and them falling for this maybe simply means the money is beter off somewhere else.

nerpderp82(10000) 4 days ago [-]

This blog post is directed at me, personally. Thanks W.D. This isn't just Data Science, I'd say that the gripes of the author are valid for about 60% of activity in tech companies. Not saying we can just eliminate 60% of it, but a lot of it supplies non-quantitative value that is driven by fashion (subset of politics) and direct politics.

There are a lot of naked emperors walking around with lots of folks standing as close as they can to shield them from the cool winter wind.

jrochkind1(10000) 4 days ago [-]

Not just tech companies either. Go bigger. It's possibly 60% of activity in US white collar economy. (I can't speak for those who work with their bodies, could be or not).

fnands(10000) 4 days ago [-]

Tangentially, I also think the term data scientist has been so abused as to almost be meaningless at this point. When I was applying for jobs it could range from anything from 'knows how to use MS Excel' to 'Can train large language models at scale'.

Personally I went for ML Engineering. My company at some point hired people as data scientists (some of my more senior colleagues still have the title, despite doing the same work I do), but started hiring people as ML engineers, i.e. people who can do half-decent SW engineering and also do ML. Just a filtering thing I guess.

I have a suspicion the term will start to fall out of fashion as things become more specialised.

tomrod(10000) 4 days ago [-]

Agreed.

I've run a 'data science consultancy' in some form or fashion for three years now.

When people say 'data science' they mean one of three things:

(1) MLE

(2) Data Management

(3) Data Analysis or Business Intelligence (applications of the same skillsets).

(1) has a lot of ongoing innovation, be it in MLOps, autoML, mapping frontier ML to business cases, etc. Innovation is expensive if the investment strategy is unprincipled. (2) is a critical and essential part of making data a usable asset. Management is expensive if it exists solely as a control process and gatekeeps access and use. (3) is core and will never get away from the adhocs and the standard flows, but the inferences are often dubious or not logically justifiable and requires depth of statistical knowledge (rare) to do well -- and courage to call out BS.

Very few people have the depth to do all three. What I have found is that many businesses hope for capacity in all three, plus some basic SWE, in the hope that they can decrease labor expenses. Not an irrational hope, to be frank, but ultimate the iron law of business holds: you can have it good, fast, or cheap -- pick two and be happy with one.

My core observation (and one I see validated based on client interest and experience) is that this is not new and has happened before -- it is the hype cycle in action. The digitization process (including moving to digital and then moving to Web) had a similar cycle. When you treat 'data science' like its a silver bullet it will generally fail to do anything but suck budget. When you embed it with your technology teams and treat it as an iterative add, as useful as devops, etc., you have a better chance for value add.

usgroup(10000) 4 days ago [-]

There are good research jobs in industry which are serious and mathematical. However they also require you to be serious and mathematical. I'd venture to say at this stage that most "data scientists" are either self taught segues from adjacent fields or have a shallow relevant background.

The serious places don't want you... so you end up at the place that can't tell the difference, and the self fulfilling prophecy begins.

r-zip(10000) 4 days ago [-]

Can you recommend a few such places? :)





Historical Discussions: Photography for geeks (November 27, 2022: 812 points)
Photography for Geeks (October 15, 2022: 7 points)
Photography for Geeks (September 17, 2022: 3 points)
Photography for Geeks (WIP) (September 18, 2022: 1 points)
Photography for Geeks (June 10, 2022: 1 points)

(812) Photography for geeks

812 points 6 days ago by excite1997 in 10000th position

lcamtuf.coredump.cx | Estimated reading time – 34 minutes | comments | anchor

Photography for geeks

Billions of people carry cameras in their pockets and use them to document their lives. Yet, despite the democratization of photo hardware, the knowledge of photographic techniques remains elusive. Countless books, webpages, and YouTube videos purport to offer advice, but tend to dwell on topics of little consequence to most hobbyists - such as shopping for gear or memorizing made-up rules of composition that seldom make for a good shot.

I'm not a pro, but I dabbled in photography for more than two decades - and after making countless mistakes, I have gotten fairly good. This page is an impassioned contrarian take on what it takes to snap great photos, along with a set of simple experiments that can be repeated at home.

Part I: Light

It is said that composition is the most important aspect of a photograph. I disagree. An intriguing interplay of light and shadows can make a discarded candy wrapper look profound. An unflattering light makes even the most expertly framed scenery look pedestrian and dull.

Monitors and photographic paper are capable of faithfully reproducing only a tiny portion of the luminance range our eyes can perceive. It follows that in a photograph or a video, shadows and highlights are not just a mild distraction: they control what can and cannot be seen. Good photographers exploit this property to accentuate what matters and conceal what does not. Inexperienced hobbyists abdicate the responsibility to an algorithm in the camera. They often end up with competent results, but seldom with what they had in mind.

Most novices think that illumination is entirely out of their control, but it isn't so. A solid grasp of the fundamentals allows one to leverage the environment. When shooting outdoors, you might ask your subject to stand near an exterior wall to eliminate the harsh shadows that plague most vacation shots. Indoors, you might move toward a nearest window or stay clear of recessed lights.

To understand how these tricks can help, let's talk about some of the key properties and types of light.

Point and area lights

One of the most consequential ways to classify light sources is by their apparent size. Some sources are so small or so distant that they behave as if all the light rays emanated from a single point in space. A well-known example is the sun on a clear day. Inside your house, the same can be said of a bare lightbulb or a flashlight, especially if placed some distance away. The shadows cast by such sources are intense and sharp-edged. The transition between light and shadow is abrupt because a point source instantenously disappears from view when an opaque item gets in the way.

Other sources behave as if the light emanated from a larger surface, be it because of the nature of the emitter or due to some apparatus that scatters the rays. A familiar mega-scale area light is the sky on an overcast day. Indoors, a common example is a bedside lamp equipped with a shade. Area lights produce soft shadows because there are many possible intermediate states between the source being fully visible and fully occluded. The softness is proportional to how large the source is and how closely it is situated.

Point sources have their uses in photographic work, but more often than not, they get in the way. In landscape photography, they create jarring contrast, particularly when snapping buildings or trees. In portrait work, they produce unflattering and distracting shadows, especially around the nose. The phenomenon can be seen in the following picture:

For simplicity and consistency, I am using a porcelain figurine as a stand-in for a model. In both of these examples, I placed the source a bit to the right of the camera. The problem with point lights gets more severe if the source is positioned overhead. In such circumstances, eyes appear dark and sunken, and your model may end up springing a Hitleresque mustache. Again, such lighting can be sometimes employed for dramatic effect - but it's more common for it to ruin the shot:

As hinted earlier, the remedy can be simple. Outdoors, a nearby reflective surface, such as a light-colored wall, can provide diffuse illumination that makes the shadows less evident. Indoors, staying away from recessed lighting is a good habit. Finding a night lamp or a window on the shady side of the house can also help.

Floodlights and spotlights

Another noteworthy quality of a light source is its beam shape. A bare lightbulb is omnidirectional and unfocused. It illuminates all nearby items fairly evenly, but the intensity falls off fast when you take a couple of steps back. A flashlight, in contrast, produces a focused beam that lights up only a tiny slice of your field of view, but can carry up to a hundred yards.

Wide-angle lights ('floodlights') are the bread and butter of photographic trade. In addition to providing uniform and predictable illumination, they also excel at isolating subjects from their backgrounds. If your subject is placed near the light but some distance away from the background, and if you set the exposure right, light falloff ensures that all the background clutter neatly fades from view.

Narrow-beam spotlights, on the other hand, tend to create extreme contrast and usually need to be offset by a floodlight to make the picture work. In particular, if you want to create a night scene of someone holding a flashlight or standing in the headlights of a car, you need to provide additional 'fill' illumination from a more uniform source - or shoot an underexposed photo at dusk.

It follows that accidental spotlights, such as recessed lighting or sunbeams coming through windows, are best avoided unless you have a way to offset their undesirable effects. That said, a well-employed spotlight can add value. The following picture uses a gentle spotlight to bring out the face while keeping the rest of the figurine in a subtle shade:

Illumination angle

The next light property to pay attention to is the angle of illumination. We already discussed the issues with overhead lights. The other undesirable extreme is a light source placed in about the same axis as your lens, for example when using an on-camera flash. This type of illumination prevents the formation of natural shadows that convey information about the curvature of three-dimensional objects, making faces look round and flat:

Once again, creative light positioning can be used for dramatic effect; for example, a light shining from below makes your model look sinister and out of this world (perhaps emerging from the depths of hell). That said, in most situations, the extremes are best avoided. Placing the light slightly to the side, perhaps at 20-30 degrees from the axis of the lens, is often the best starting point.

Color temperature

The final light quality we're going to discuss is color temperature. In the olden days, most man-made lighting - from candles to incandescent lightbulbs - had an inherent orange-yellow hue. Today, this is no longer a physical constraint, but most CCFL and LED lamps still mimic the effect, aiming for a color temperature between 2,700 K ('soft white' or 'warm white') and 3,500 K ('neutral white'). This is not just a force of habit: this type of illumination is easy on the eyes and nicely complements skin tones. Higher color temperatures with a stronger blue cast are reserved for factory floors, workshops, and other areas where maximum light intensity and superior color rendition matter more than good looks.

In outdoor settings, color temperatures dip to around 2,000 K during sunrise and sunset; together with lower light intensity and favorable illumination angles, these 'magic hours' are some of the best times for outdoor photography. During the day, the temperature will usually go up to about 5,500 K, and if the sky is overcast, it can reach 6,500 K or so. Moonlight isn't any bluer, but nights may appear this way when viewed through the windows of a warmly-lit home. Because of that, it's a habit of most photographers and filmmakers to add a dark blue cast to night scenes - and audiences have come to expect it as a visual cue.

You can see an approximation of the difference between incandescent ('tungsten') and daylight color temperatures below:

It's not that one color temperature is inherently better than another; orange cast can be more flattering in portrait work, but a photographer can add and remove it at will. In the era of photographic film, one would place a piece of colored glass in front of the lens. Today, the adjustment is as simple as moving a slider in a photo editing program or selecting a particular white balance setting in the camera. That said, if no action is taken, an in-camera algorithm will try to remove any 'errornous' color cast - and the result might not be what you aimed for.

Another gotcha is what happens when two lights with different color temperatures illuminate a single scene. In such circumstances, automatic white balance algorithms will struggle, and if the illumination is harsh, you end up with a mess of alternating yellow- and blue-tinted shadows that don't look right. Still, if the light is diffuse and white balance is picked by hand, mixing color temperatures can pay off:

It must be said that the color temperature model is fairly simplistic: it represents a one-dimensional continuum, from red to blue, in a two-dimensional color space. Some light sources can exhibit other tints: for example, early fluorescent lamps had an unpleasant greenish hue that made skin appear unhealthy and pale. Today, Western filmmakers intentionally replicate this look when portraying the Soviet-era world behind the Iron Curtain; a good example is the acclaimed HBO miniseries 'Chernobyl'. In any case: on-camera controls might not offer enough flexibility to fix all color issues, but most photo editing software has an additional white balance slider on the continuum from purple to green.

Getting creative with light

As noted earlier, a skilled photographer doesn't need to haul thousands of dollars in lighting gear everywhere they go. It is often enough to take note of existing illumination sources and reason about how they affect the scene. It pays to remember that, owing to the nature of camera sensors and computer displays, seemingly small differences in illumination have a huge impact on the appearance of the photograph.

Whether you're using existing or added lights, it's best not to put too much faith in on-camera algorithms. Automatic exposure usually works to preserve detail in the highlights, but in a high-contrast scene, it might make everything else too dark. Some cameras attempt to compress the dynamic range ('HDR') to salvage both highlights and shadows, but this can produce an artificial look: rough skin, unnaturally bright shadows, and weird halos in the areas of high contrast. It's better to stay in the driver's seat. Even on devices without manual exposure controls, it's usually still possible to dial in exposure compensation, or tap on the screen to set metering priority. But if your camera supports manually setting exposure time, you might be surprised how easy it is to get used to.

For those who want to invest in photographic lighting, I recommend starting small - perhaps with a pair of battery-powered LED tubes such as Genaray Beacon or Nanlite PavoTube 15C. This accessory provides a neatly diffused light that's more than enough for taking portraits of people and pets. But above all, unlike most other studio gear, it does not require any setup. You can grab it at a moment's notice, hold it in one hand, and have some casual fun.

To illustrate the potential of such simple tools, let's have a look at this snapshot taken in a darkened living room. I put one cheap LED tube upright behind my wife's back, pointed toward a bookshelf on the wall. I held another tube in front, above the camera, to exaggerate the curvature of cheekbones and eyebrows. The darkness in between did the rest to achieve an unsettling, low-key look:

For the next shot, we moved into the hallway. Another light pointed at her back introduced a subtle glowing silhouette - a flattering look that separates the subject from the background. I placed the main light a bit lower than before to make facial features appear more neutral and to reduce shadows:

Some LED tubes offer not just white light, but can be adjusted to achieve a full spectrum of colors. Although it's best to master traditional lighting first, such color variations can create a wide range of interesting effects. For example, if you want to recreate the popular 'night club' vibe of some contemporary films and video games, you'd want to make the background blue or violet, and then bathe the foreground in magenta or red.

Of course, there are limits to what LED tubes can do; for one, they aren't particularly bright and can't produce a collimated beam to selectively illuminate a small portion of the frame. It follows that serious studio photographers may also benefit from continuous floodlights such as Westcott Solix, as well as focusable spotlights such as Genaray Torpedo.

Shopping for lights can be tricky. Modern cameras perform well in low-light conditions, so photographic lights usually don't need to be exceptionally bright, but they should be matched to the intended use. When it comes to light output, there are two units of measurement you might encounter in product brochures: the lumen (lm) and the lux (lx). Lumens tell you the overall luminous output of a source, no matter which direction the light goes; lux tell you brightly the lamp will illuminate a unit of surface area right in front of it.

The latter is more useful for photography, and it's good to memorize several points of reference. On a clear day at noon, you might get up to 100,000 lx in the sun and 20,000 lx in the shade; a cover of clouds can bring it down to 2,000 lx or less. Household lighting is quite variable, but measurements tend to hover around 50 to 300 lx. Urban stret illumination in pedestrian areas is around 5-10 lx. A typical candle scores about 1 lx from one meter away, hinting at the historical origins of this unit of measurement. Last but not least, moonlight is usually around 0.1 lx.

Pocket photographic lights, such as Genaray Powerbank 96A or Luxli Fiddle, deliver around 150-200 lx at 1 meter at maximum power. This makes them quite suitable for indoor close-up photography, but fairly useless outdoors, at least until dusk. Handheld LED tubes peak around 400-600 lx; that's enough to provide an outdoor fill light early or late in the day. Affordable AC-powered lamps, such as Westcott Solix or Genaray Torpedo, usually deliver around 3,000-6,000 lx at 1 m. Finally, if you absolutely need it, larger (and costlier) spotlights can pump out 50,000 lx or more.

Part II: The optical path

Right after illumination, the optical system of your camera has an outsized impact on the appearance of your photos and video clips. This does not mean you need to go broke and buy a bulky camera with an array of interchangeable lens; it suffices to know when to zoom in, when to zoom out, and how to control the focus within the scene. The answers to these questions may seem obvious, but there's more to it than meets the eye.

Focal length, aka zoom

'Focal length' is a fancy way of describing the camera's field of view. A wide-angle lens with a short focal length can capture everything that's happening in a crowded bar; meanwhile, telephoto optics let you fill the frame with a single soda can from across the room. In theory, increasing the focal length is the same as snapping a wide-angle photo and then cropping and magnifying the center part. In practice, without optical magnification, you quickly run out of pixels and end up with a blurry mess.

At first blush, focal length may seem uninteresting: after all, you want your subject to occupy a reasonable portion of the frame, and for many novices, this dictates the zoom level they choose. What they overlook is that the size of the subject can be also controlled by moving the camera closer or farther away; in effect, there are countless combinations of distance and focal length settings that preserve the scale of the primary object they're trying to photograph. Let's demonstrate this effect using another porcelain figurine against the backdrop of a cluttered living room:

I captured this sequence by taking a step back every time I increased the focal length, so the size of the figurine's face didn't change. Yet, the pictures differ in many ways. For one, at wide angles, distant objects appear much smaller, and a good portion of the room ends up in the frame. As the focal length increases, the perspective becomes less pronounced, and field of view narrows all the way down to a single leaf of a household plant.

But there's something else happening too: in the first picture, the figurine appears to have a slender face, pouty lips, and a relatively large nose. As the sequence progresses, the face takes on a more rounded shape. This is another manifestation of the same optical phenomenon: at short focal lengths, features closer to the camera look larger, even if the distance differential is a fraction of an inch. It's why cell phone selfies usually look off.

The slideshow is annotated with focal length numbers for a 35mm camera sensor, also known as 'full frame' (a throwback to the days of photographic film). To achieve a similar effect with a smaller sensor, the focal length would need to be proportionally less. Unfortunately, when buying gear, it's not always clear if the manufacturer is talking about '35mm equivalent' focal length or about nominal measurements of the optical path. The former is common in cell phones; the latter in prosumer cameras.

Either way, in the 35mm world, there are several reference points to memorize:

  • The length of about 20-22mm corresponds to the natural field of view of human vision. Lenses in this vicinity are ideal for photographing nearby crowds, snapping photos for real estate listings, or filming your kids playing on the beach. That said, any features in the periphery of the frame will appear distorted.

  • Focal lengths around 40mm project a 1:1 image onto the sensor, as if you looked at your surroundings through a rectangular viewfinder with no optics at all. This captures lifelike perspective with little distortion. For historical reasons, the most popular lens choice in this neighborhood is actually 50mm - a versatile if unassuming choice for indoor and outdoor scenes where you don't need to cram too much into the frame.

  • Focal length of 85mm is considered to be ideal for professional portrait work. It is long enough to virtually eliminate distortion, but short enough to allow the photographer to stand reasonably close the the model.

  • Focal lengths of 200mm and above are most commonly employed for nature photography, sports, and for spying on people from afar.

Smartphones with zoom optics usually feature focal lengths corresponding to about 20-80mm in the world of full-frame sensors. Past that point, the phone usually switches to digital magnification, and picture quality takes a nosedive.

Depth of field

Depth of field is a property of the optical system that determines what slice of the scene will appear sharp when you focus the lens at a particular distance. Shallow DOF is a wonderful tool for isolating subjects and hiding background clutter, as shown in the sequence below:

The depth of field in your photos will depend on three things:

  • The focal length of the lens. Telephoto optics tend to produce shallow DOF; wide-angle lenses keep almost everything in focus most of the time.

  • The distance to your subject. In close quarters, the DOF might be measured in fractions of an inch; toward the horizon, it can span a mile or more.

  • The aperture of the lens, aka the relative diameter of the optical path. Large apertures (numerically low f-values, around f/1.2 to f/1.8) exhibit extremely shallow DOF and create a pleasing blur. Smaller apertures, around f/4 and above, provide relatively little latitude for creative DOF work.

Although most people find shallow DOF quite agreeable, it can interfere with landscape shots and street photography. In such situations, the simplest solution is to reduce the aperture (increase the f-number). This is akin to squinting your eyes; in addition to making the picture sharper, it also reduces the amount of light hitting the sensor, so you need to crank up the exposure time to cancel out the loss.

The optics of cell phone cameras usually exhibit extremely wide DOF at short focal lengths and do not expose aperture controls, but selective blurring might be doable when the lens are zoomed in. Barring this, some phones employ algorithms that try to estimate depth and selectively apply software blur to portraits. This doesn't always work well, but the results might be acceptable every now and then.

Focus point

Even if you are not angling for a shallow DOF, the camera still needs to be focused on the subject to produce crisp-looking photographs. The actual task of adjusting the optics to maximize sharpness is almost always delegated to the camera's autofocus mechanism. That said, it's still up to the photographer to tell the AF mechanism which portion of the image to look at while making the adjustments; in the absence of this input, the system may default to a point in the center of the frame, lock in on a random high-contrast edge, or pick what looks like a face.

Almost every camera is equipped with a joystick or a touch interface to guide the AF system. In portrait work, it is customary to focus on the eyes; we're instinctively drawn to this part of the face, so any accidental focus errors tend to stand out:

If the model's head is titled, it's safer to pick the eye closer to the camera or that is otherwise more prominent; again, there's no one correct answer, but focusing on the less visible eye usually doesn't look right:

If the photographer wishes to draw attention to some other element of the composition while keeping a person's face in the frame, it's best to make sure the eyes are robustly out of focus, and not just slightly blurred:

Portraits aside: when setting up landscape shots, many novices pick the horizon as the focus point. That can be a mistake: it's better to select a point about one third of the way between the nearest and the most distant plane in the photograph. As noted earlier, DOF increases with distance, so this approach maximizes the odds that both the foreground and the background are going to look just right. The same one-third rule works for macro shots.

Part III: Color and tone

Color theory is a peculiar discipline that studies contemporary color preferences and design fads to develop a set of immutable principles that purportedly govern all uses of color in creative work. As with the 'ironclad' rules of composition, I recommend skepticism - especially if the author brings out color wheels and starts drawing squiggles between equidistant points.

At the same time, it must be said that color and tone can be what separates a second-rate photograph from a memorable one. To illustrate, let's look at the potential evolution of this vacation shot deliberately chosen for its mediocrity:

The final photo appears to be better illuminated and more closely conforms to the popular depictions of the mountainous desert terrain around the Hoover Dam. Indeed, some cell phones automatically apply similar tone mapping tricks to photos taken in full sunlight; this happens without the photographer's knowledge and sometimes produces unexpected results.

On-the-scene adjustments versus photo editing

Some color and tone tweaks can be taken care of when taking the photo, but many others are accomplished more easily in front of a computer screen. Pay no mind to purists: even in the era of photographic film, there was no such a thing as an 'unaltered' photograph. The choice of process chemicals, paper, and enlarger settings profoundly affected the appearance of the final print. Deceptive edits may be a no-no, but absolute purity was never a meaningful goal.

At this point, we must note that the range of postprocessing options at your disposal is far greater if you capture photos as 'raw' files (DNG, CR3, NEF, etc). Most cameras default to JPEG - a compact, 8-bit file format that keeps only the bare minimum of data needed to display the original image. In the Hoover Dam example shown earlier, JPEG compression would have destroyed the texture of the clouds and the detail in the blown-out highlight in my son's hair. Luckily, the raw file I captured contained enough information to bring out that extra detail when the correct slider was dialed down in a software tool.

Virtually all mid-range digital cameras have the ability to capture raw images, but the feature is not always available on mobile phones. That said, at least some Huawei, Samsung, and Apple handsets support raw images. The photos taken in this mode take up more storage space and can't be directly opened with some programs, but if you're serious about photography, you should give it a go.

There is a wide selection of programs for editing raw images. The gold standard used to be Adobe Lightroom, but after the app moved to an exploitative subscription model, perhaps the best alternative is Capture One ($300 for a perpetual license), followed by Darktable (free). Most other general-purpose image editors, such as GIMP or Affinity Photo, are also capable of working with raw images, but lack useful photographic workflow features, such as catalog management. This matters if you're trying to quickly sort through several hundred photos, or apply the same adjustments across the entire lot.

Shifting the color balance

In the world of digital cameras, color balancing is implemented in software: after the picture is taken, the readings from the sensor are shifted by an algorithm to obtain neutral whites or achieve some other algorithmic goal. When working with JPEG files, the results of this transformation are baked into the final image and cannot be fully undone; but when shooting raw, the original sensor data is preserved, and a full range of adjustments is possible down the line.

Whenever the in-camera algorithm misses the mark, the most convenient way of removing an undesirable color cast is to point the photo editor's white balance picker at a portion of the picture meant to have a neutral shade of gray. This may be a stainless steel appliance, a section of the pavement, a white wall, a piece of clothing, or a special calibration card. If the result is unsatisfactory or if no suitable target is found, color balance can be also adjusted with manual sliders: 'temperature' for moving from yellow to blue and 'tint' for purple to green.

Of course, one doesn't need to seek a perfectly neutral look. As noted earlier, flattering orange hues are sought after in portrait work; subtle orange grading is the secret behind the good looks of most TV anchors on the news. Outside of portrait work, we are primed to interpret a yellow-orange cast in a well-lit photograph as an indicator of a warm summer day; while a shift toward blues might signal moonlight, frigid cold, or a sterile high-tech environment. Finally, greenish tints give the impression of an inhospitable place; while low-key purples, violets, and reds are the pop-cultural stand-ins for danger or lust.

As a practical example, consider this sequence of edits, starting with a neutral white balance calibrated on the pillows, and then introducing warmer and cooler tints:

Exposure adjustments

Along with color balance, exposure is one of the most important postprocessing tweaks you can make. Dark, underexposed ('low-key') photos tend to have an ominous appearance, and - with the right lighting and color tint - can imitate nighttime takes. On the other end of the spectrum, overexposed low-contrast ('high-key') shots may have an angelic vibe or signal a futuristic, high-tech setting of some sort.

Strictly speaking, exposure settings can't be changed after the fact: the picture is taken with a predefined shutter time, lens aperture, and sensor amplifier gain (also known as the ISO setting, in another throwback to the era of film photography). It follows that gross exposure errors, such as shooting a completely overexposed frame, usually aren't recoverable. Luckily, when working with raw images, there's still some data captured in the highlights and in the shadows, giving you room to fine-tune the exposure down the line. In essence, photo editing software may shift pixel values up or down to simulate the light hitting the sensor for a bit more or a bit less time.

The usual viable range of such adjustments is plus / minus two 'stops'; a stop means a two-fold increase or decrease in exposure time, so the range translates to exposure times between 25% and 400% of the starting point. The following photo of a dog illustrates the effect of adjusting exposure +/- 1 stop:

In this instance, the underexposed photo is pretty interesting, perhaps reminiscent of a gray, rainy day in the fall.

Again, there are limits to what can be done in software: the electronics in the camera have their constraints, and the optical path can get in the way, too: especially with lower-grade lens, extreme highlights tend to produce blooming and chromatic aberrations (color fringing) that can't be fixed with a single click. Further, the latitude of possible adjustments is much lower when working with JPEG files; going beyond +/- 25% would be pushing your luck.

Selective color and tone tweaks

In addition to the whole-picture techniques discussed in the two preceding sections, most photo editing programs offer a handful of targeted tone adjustment tools. Chief among them are:

  • Separate luminance adjustments for shadows and highlights. In most programs, this is a standalone feature; in Capture One, it's a part of a confusingly-named 'High Dynamic Range' tool. The tool helps restore details in overexposed or underexposed areas without affecting the rest of the image. For example, in the Hoover Dam photo, I used it to bring out the texture of the clouds.

  • Three-way hue and saturation adjustments for luminance ranges (shadows, midtones, highlights). Most programs refer to this as 'color balance'. The tool helps correct background color casts when mixing incompatible sources of light. It's also useful for adding distinctive tints to backgrounds, provided that the background is darker or lighter than the main subject of your shot.

  • Hue, saturation, and luminance adjustment for a specific source color (common preprogrammed choices: red, orange, yellow, green, cyan, blue, violet, magenta). Also known as the HSL tool, the mechanism is invaluable for making the vegetation greener or the sky more blue; or for selectively correcting skin tones, which tend to be clustered in the red-orange-yellow zone.

Here's a practical example of using these tools to make subtle stylistic tweaks to a photograph and give it a distinctive look:

Of course, the adjustments can be bolder. The following sequence of edits to a casual photo of my wife demonstrates some of the possibilities:

The first variant uses the low-key orange-teal color scheme all-too-common in superhero action films; the second one employs a sickly, low-contrast greenish cast seen in HBO's 'Chernobyl'; and the final one is a dark, blue-shifted version that, despite ample illumination, emulates a photo taken at night.

Of course, some photos may require corrections to regions that can't be isolated based on their color or luminance. In such a case, the photographer may need to resort to manual retouching, either creating a mask layer or freestyling with a 'dodge' or 'burn' tool or a color brush. Some emerging photo editors, such as Luminar Neo, employ machine learning to infer the 3D structure of the scene, offering the ability to apply sophisticated lighting effects ('relighting') without fiddling with a brush. The algorithms are slow and clunky, but every now and then, they can save a botched shot.

Part IV: Framing

Along with color theory, the principles of photographic composition are the refuge of armchair philosophers who pen stuffy essays about concepts such as 'the golden ratio' or 'the rule of thirds'.

Naturally, aesthetics are to some extent a social construct: your audience might expect photographs to look in a particular way simply because that's what they're accustomed to. It follows that one shouldn't be a habitual contrarian; but it's about as counterproductive to get too hung up on dogma.

With this in mind, in the final chapter of this guide, let's have a quick look at several important composition choices - and let's do it without getting bogged down with ideology.

Clutter

By far the most serious and preventable error in composition is background clutter. It's not that backgrounds need to be tidy: in street photography, busy cityscapes are a part of the picture as much as the subject standing in front. But when clutter is poorly controlled, it distracts from what the photographer is trying to say.

The distractions can come in many forms: busy textures, clashing colors, intrusive shadows or highlights, awkwardly cropped detail, or elements that simply do not belong - say, a toddler throwing a tantrum in the background of a glamour shot. In the following photograph of a camping knife, survivalist-themed background items make the picture more interesting; removing the props and exposing a messy kitchen countertop ruins the shot:

Of course, are times when we want the background to simply disappear. We showcased three ways of doing this earlier in the guide: area lighting, wide apertures, and long focal lengths.

Placing the subject within the frame

Most novice photographers tend to place their subject smack dab in the middle, facing the camera, and filling most of the frame. There is nothing wrong with this approach, especially in portrait work - but it tends to create a static and clinical look. When the setting permits it, a wider crop can easily liven up the photograph:

More dynamism can be added by moving the subject to the left or to the right; this works equally well in landscapes and in portrait work. The resulting negative space provides interesting contrast. If the model's gaze or head is turned in that direction, the photograph has a more balanced appearance and will usually convey a playful or contemplative mood. If all the action is concentrated on a single side of the photograph, the effect is more unsettling, creating a sense of a disturbance happening just outside the frame:

Titling the camera is another underused trick. It usually doesn't help much in portrait work, as it distorts body proportions and just looks odd; but in landscapes, it is a godsend. In the plains, tilting the camera up lets you to capture more of the sky and less distracting details right next to the photographer. In mountainous terrains or next to large bodies of water, doing the opposite helps capture more architectural detail or more reflections of the sky.

For portraits, it's also good to pay attention to model's posture; the pictures tend to be far more flattering if the model stands up straight and raises their chin up a bit.

Epilogue

The approaches and tools described in this document are by no means exhaustive. There are many other photographic techniques worth learning; for example, there are several methods for conveying motion - timelapses, multiple exposures, and selective blur effects created by panning the camera in tandem with the movement of the subject of the shot. Or, there are multiple optical and postprocessing-based methods for correcting or exaggerating perspective, or for achieving a variety of non-photorealistic looks.

My goal wasn't to create a comprehensive list of every little trick in the book. Instead, I wanted to help the readers discover the joy of photographic experimentation, no matter whether they're carrying a smartphone or a $5,000 professional photo rig. Unlike some other authors, I also wanted to do this without bragging about my own best shots: the examples employed on this page were all shot in the backyard or around the home with minimal prep.

If you are interested in learning from others, I recommend joining a photo community, although I would avoid sites such as Instagram or Flickr. They are centered around view counts, 'like' counts, and algorithmic feeds, leading to extreme homogenization of content and toxic community dynamics. My favorite site right now is Glass. They charge a modest fee ($30/year), but have a friendly community and give every photographer an equal chance of getting noticed without making it feel that you're participating in a zero-sum game.

You can contact me at [email protected], follow me on Twitter, or subscribe on Substack. For other features, check out my homepage.

Your lucky number is: 21892065




All Comments: [-] | anchor

post_break(10000) 5 days ago [-]

Somewhat related but if anyone is looking to get into photography Fuji is putting out digital cameras that emulate their film stocks. The results are pretty incredible. Also they have manual dials on the top so you're not trying to program a computer to use it. It's like driving a manual transmission vs an automatic.

safeimp(10000) 5 days ago [-]

Any specific recommendations?

nop_slide(10000) 5 days ago [-]

I recently picked up an x-t20 and it was a lot of fun, but then I somehow stumbled on the elusive x100v in store so I bought that.

They are indeed very fun! Highly recommend either of the two.

throw0101c(10000) 6 days ago [-]

If anyone wants to move beyond using the 'auto' setting on their camera (or phone), I would recommend the book Understanding Exposure by Bryan Peterson, the first edition of which was published in 1990:

* https://www.goodreads.com/book/show/142239.Understanding_Exp...

The principles involved haven't changed much in the intervening decades; the current fourth edition was publish in 2016.

If all you have is a phone you don't have to get new equipment: just perhaps a third-party 'camera app' that allows you manual control of aperture, shutter speed, ISO/sensitivity.

Once you know how each of these settings alter the resulting photo you can use them to alter the composition of photos, which is a whole other craft.

Edit: seems recent smartphones have little-to-no adjustable camera settings.

_HMCB_(10000) 5 days ago [-]

I bought that book months back. Have yet to read it but thanks for the reminder.

geokon(10000) 6 days ago [-]

I think when you break down all the variables there is really very little to play with bc no phones have variable apertures.

ISO is basically a linear gain that's done on the sensor. As long as you aren't blowing out your photo and loosing information, it basically makes not difference if you do it in sensor or later while editing

So the only variable left is the shutter speed - which is basically directly dictated by the amount of light you have. You try to get as much light as you can without blowing anything out. This is how you get the most information. You can decrease it to get faster shots with less blur, at the cost of more noise

So it all boils down to basically one 'slider'/variable between blur and noise

gofreddygo(10000) 5 days ago [-]

Aperture Size, Shutter Speed and ISO. Just understand what they are. And their units.

Actually see pictures varying one and keeping others constant to get a hang of things.

Then go backwards, check professional pictures and guess the values. Professional photography forums all photos have these values published .

Night/Day photography, moving/still and background focus are the only 3 skills you need as an amateur photographer. They rely on the 3 settings above.

Beyond that lies the rabbit hole that, if you venture, speak not to any people whom you wish to keep friends. They hate when you try to tell them.

Don't spend over 2k on lenses.

Have printed photos on glossy, matte paper. Touch and feel them. Worth the time.

excite1997(10000) 5 days ago [-]

> The principles involved haven't changed much in the intervening decades.

My problem with many old-time tutors is that they refuse to recognize that photography has gotten a lot easier. We don't need to learn the craft the way they did.

For example, you don't need stuff like the 'sunny 16' rule of exposure if you have real-time previews in the camera. You use visual feedback, usually with better accuracy.

In the same vein, you probably don't need to learn about flash guide numbers when modern continuous LED illumination covers 99% of use cases without any guesswork.

Or, you don't need to learn about optical filters (perhaps except for the polarizer) when almost all their functions can be accomplished in software without loss in fidelity.

billfruit(10000) 5 days ago [-]

Is there any good book on photo editing.

kawera(10000) 5 days ago [-]

A quite technical but fascinating book on lighting and photography: 'Light Science & Magic' by Fil Hunter and Paul Fuqua (https://www.goodreads.com/book/show/290153.Light)

Terretta(10000) 6 days ago [-]

This is very well done for a new-to-photography audience. Will be sharing around to people who say all their things look like snapshots, what's up with that.

Great use of examples, except for one: kid on bridge.

> At the same time, it must be said that color and tone can be what separates a mediocre photograph from a memorable one. To illustrate, let's look at the potential evolution of this vacation shot deliberately chosen for its mediocrity...

Then the dynamism is removed by 'correcting' the dutch angle to horizon, the surprisingly good color balance is skewed off, and the whole thing gets that circa mid-2000s HDR look from Flickr and Shutterfly and the like where every photo got tone-mapped.

Underwhelming of an end result, especially compared to the later color and tone examples (e.g. kitchen superhero).

SamBam(10000) 5 days ago [-]

I had the exact same thought on that sequence.

The angle is a personal choice, though I think I agree that the mirroring angles of the two bridges was more interesting than having the horizon straightened. But the final 'Tone Curve Corrected' turned a fairly nice image into a typical HDR-ruined photo, where the eye can't focus anywhere because all the colors have the same value.

To add some praise with the criticism, I thought the explorations of light white the photos of the wife were well done. Pointing the rear light at the subject's back to create that subtle halo is nice and I had not considered it.

bambax(10000) 5 days ago [-]

Straightened and exposure adjusted is good. They should have stopped there. White balance tweaking is often a bad idea; what you want is to get as natural colors as possible.

excite1997(10000) 5 days ago [-]

In the mid-2000s, HDR was all about jacking up local contrast, giving you that unique look of gritty skin and halos cropping up all over the place. I'm talking stuff like this:

https://digital-photography-school.com/wp-content/uploads/20...

Less obnoxious tone mapping that compresses shadows and highlights is a more modern trend, I'd say post-2012. It's basically done by every cell phone today when shooting a high-contrast scene.

pdpi(10000) 5 days ago [-]

Seeing as this is photography for geeks: Colour temperature is exactly that, a temperature.

Things glow when hot, with the colour of the light depending on the temperature and the material. When you say 'a colour temperature of 6200 K' it means the light is the same colour as that emitted by an ideal black body at that temperature. It's no coincidence we perceive colour temperatures around the high 5k to low 6k as white, because the surface of the sun is around 5800 K.

zokier(10000) 5 days ago [-]

In practice vast majority of scenes are not illuminated (directly) by a blackbody source, so for white balance purpose its inherently crude tool to achieve a color look, and often there is no 'correct' color temperature setting for a scene

serko(10000) 5 days ago [-]

The author gave a nice general overview of the topic.

But the thing is, none of these technical things are essential.

Like a lot of people here (me included, but I am actively trying to get rid of it) the author has an engineering mindset, that was built (I am just guessing here) through years and years of engineering work.

And it is really cool and may give an advantage in some professions (and sometimes in life overall), but it is not the best approach to photography or in any other arty topic where one could not objectively measure pleasure and value, and where aesthetic perception is the main definition of something being excellent and desirable.

People often forget that photography is still mostly an art form. And in art, the most important aspect is provoking some kind of emotional response (folks mostly pursue pleasant ones, but it is not limited to that).

To understand lenses, focuses, and shutter speed how much time does one need? A couple of days? Weeks?

It is objectively easy to learn the rule of thirds, focal points, white balance, etc. But which white balance makes beautiful images? Which calculations would make your friends adore their faces in the photo? I have no answer to that, do you?

Should images be dark, moody, and sharp like Roger Dickens's cinematography to be likable? Or dark, blurry, and saturated like Wong Kar-wai movies? Or they should be bright, light, and symmetrical like Wes Anderson's fairytales?? Or grainy and geometrically precise like Henri Cartier-Bresson's works? Or maybe images need to be provocative, erotically charged black-and-white photos, fashionable and borderline pornographic like Helmut Newton's works?

There are a lot of cases then some person with a developed sense of beauty creates unimaginably stunning photography by using a smartphone build-in camera, but even more cases when someone with a pricey a-la Hasselblad, titan tripod, cinema-level lenses, 5-point professional lighting, and a long list of detailed photogear videoreviews making the most boring and forgettable images possible.

Feelings > Any technical aspect, rules, or calculations

I am not saying that knowing your tools is not important at all, but it definitely less important than the internet wants it to be. For example, I know a Magnum agency photographer who takes all his photos with any digital SLR in auto mode (p-mode). And he adores Instagram.

So for any person who wants to start into photography (but not photography-related technology), I suggest watching a lot of photo books and cinematographically superior movies (all by critically acclaimed authors), visiting classical art museums, and for all costs avoiding any online photography communities.

And you should take as many photos as possible every day. Not only on vacations or holidays but just as a visual diary with colors that you found beautiful, unusual patterns that you start to see around, and unexpected shapes that things around us are forming.

Aesthetic goes first. It should be like a tingling feeling on the tips of your fingers when you see something interesting. Then from it go lighting, composition, and color. And these three are codependent.

You need to start seeing light, feeling colors, and thinking in shapes. Need to develop your sense of beauty and your watchfulness/visual awareness/contemplation (sorry, I don't know how to translate this properly).

One cannot create beautiful photography if one does not know what beauty is.

autorun(10000) about 17 hours ago [-]

I've been reading this forum since years, and generally its perception of photography is rather technical and very proud to achieve perfect photos with the right color balance and all that. Sure it's nice to know all those technical concepts, so you can be conscious when destroying them, and have fun.

Concerning photography tech, it's great to talk about sensors, how mobile camera is being achieved nowadays, like explained in those Pixel blog posts, or even about analog photo development works.

Hey HN, enjoy bad photos, take pictures with the worst camera you can find. I've read in some comments here that you need to take at least 10k photos first until you get better, this is not a sport

jwr(10000) 5 days ago [-]

Can we all stop for a moment to appreciate this fantastic content?

Great writing, with well-picked and carefully crafted examples, with lots of good (and correct) advice. On a web page. That you can just read.

If you don't appreciate this, let's describe what this website is NOT: it is not an ad-ridden cesspool of misinformation with auto-generated text interspersed with ads. It does not smack you in the face with a SUBSCRIBE NOW popup. It does not drag you through the mud of misleading 'customize my cookie preferences' dialogs. It has zero trackers. And the information is not posted on Facebook, Tumblr, Blogger, Medium, or any of the other black holes where content goes to be stamped with 'SUBSCRIBE NOW LOGIN NOW' popups and eventually disappear and die.

Michał's writing is excellent (his 'Guerrilla guide to CNC and resin casting' at https://lcamtuf.coredump.cx/gcnc/ is a classic already), but it's not just the writing that makes this content so great.

When reading this, I am reminded of the early days of the web and it makes me happy.

nickpp(10000) 5 days ago [-]

Totally agree but now I wish the author would get some kind of small reward for his effort so that he's incentivized to keep writing this kind of content.

lxe(10000) 5 days ago [-]

This is helpful for my stable diffusion prompts.

lelandfe(10000) 5 days ago [-]

A lot from the article will not work, because the terms are too overloaded ('three point'). SD photography and lighting keyword dump I've tested for 1.5:

f1.8, dof, bokeh, soft focus, chiaroscuro, backlit, golden hour, soft lighting, natural lighting, warm colors [but not 'cool'], Polaroid, lens flare

'85mm', '50mm' etc all make the image realistic, but don't do anything specific.

Pro-tip: use 'windows' to get many of the outdoor lighting keywords available indoors.

yieldcrv(10000) 5 days ago [-]

I've been using photography and focal length prompts on stable diffusion since I first installed it, would recommend

ak_111(10000) 5 days ago [-]

Whats a good intro book to understand optics that is technical but doesn't completely overwhelm (bonus if it includes how vision works from a physiological level)?

camillomiller(10000) 5 days ago [-]

This article doesn't touch on the most important aspect of the trade, something that makes a shot into a photograph: the choice of a subject. You can make the most technically perfect shot, and yet it would still look bland and empty and boring if you don't have a subject. No amount of technical understanding will help you choose a subject, as it has to do with with curiosity for the world and the art of observation. A shot originates in your camera. A photograph originates in your brain.

Worth reading, in my opinion even before you pick up any technical book: Roland Barthes - Camera Lucida, reflections on photography.

WA(10000) 5 days ago [-]

Counter point, directly from the article:

> An intriguing interplay of light and shadows can make a discarded candy wrapper look profound.

And I agree. I recently saw 'good' photos of a mundane university bathroom, but they were intriguing.

atoav(10000) 5 days ago [-]

Longer focal length, more zoom, more perspective compression, more blur outside of the focus area.

Wider open arperture, more light on the sensor, more blur outside of the focus area.

Longer exposure time, more light on the sensor, more motion blur.

Bigger sensor sensitivity (ISO), more light collected per unit of time, more grain/noise.

Bigger sensor, bigger field of view, more blur outside of the area of focus.

Then you also have neutral density filters (ND filters), which allow you to modify the exposure time/aperture while keeping the ISO where it is (e.g. if you wanna blur the background in bright sunlight).

You become a good fotographer if these interconnections are so well understood you don't need to think about them anymore — and you can focus your effort on finding the right combination of camera position, angle and lense settings.

The position and angle is what makes and breaks a good picture. This is why a good photographer with a smartphone takes better pictures than an amateur with a Hasselblad.

jeromenerf(10000) 5 days ago [-]

> Wider open aperture, more light on the sensor, more blur outside of the focus area.

/ shallower depth of field

nerdponx(10000) 5 days ago [-]

> You become a good fotographer if these interconnections are so well understood you don't need to think about them anymore — and you can focus your effort on finding the right combination of camera position, angle and lense settings.

Much like any skill: music, computer programming, cooking, etc. Mastering the technical fundamentals liberates you from having to worry about the technical fundamentals.

muglug(10000) 5 days ago [-]

The main thing about photography is your first 10,000 photos will likely be bad. Modern technology makes it possible to get through 10,000 snaps much faster and much cheaper than when using film.

But taking those photos is not enough — it's in the editing of those first 10,000 photos that you will find the hidden photographer in you.

Photography is a lot like writing code: you can create something that does the job, but does not do it beautifully. You can keep on creating until creating beauty becomes easier.

dmitriid(10000) 5 days ago [-]

> it's in the editing of those first 10,000 photos that you will find the hidden photographer in you.

Instead of taking 10 000 photos, then sifting through them, and then editing them... Take 1000 photos, but take the time to take them: wait for the right moment, come closer, step to the right, anticipate a moment, find great in the mundane. This will make you an infinitely better photographer.

bradknowles(10000) 5 days ago [-]

I wouldn't say it's your first 10k photos. Because anyone can just hold down the button and take hundreds of photos. And those hundreds of photos don't really count as separate pictures in this regard.

I would say that it's more like your first 10k different photographic compositions/subjects.

From there, I agree that you really learn in the editing.

coolandsmartrr(10000) 5 days ago [-]

If you take a workshop or two, you'll better understand the thought process of a photographer, leading you to start taking descent photos within 1000 snaps.

I agree with the need to edit for improvement - meditating on your composition and lighting will ingrain better camera habits for the next round of snaps.

Personally, taking analog photos with an exposure meter taught me about good exposure. Measuring exposure takes time, but it's a great learning process.

emsixteen(10000) 5 days ago [-]

I have a heap of photos that I'm very happy with from my first 10,000 exposures, and I'm sure that if I went back and re-edited them I'd have more. I think it's only when you start to dial down into trying to achieve a specific look where it takes some time and effort.

nbzso(10000) 5 days ago [-]

Just sharing: If you want to learn photography, buy yourself a film camera and read those two books.

Ansel Adams:

The Camera https://amzn.to/3gMyCa5

The Negative https://amzn.to/3U88fsN

The rest is just practice.:)

anta40(10000) 5 days ago [-]

I think for learning photography today, using a film camera will very likely wasteful, unless you are pursuing a photography degree, for example.

Get a digital camera. Instant feedback and histogram are very useful for judging your exposures. Once you are comfortable with that, you may pick a film camera.

js-j(10000) 5 days ago [-]

I thought so - until film photography prices began to skyrocket. Then I bought a quality second hand DSLR body compatible with my 35mm film lenses, and haven't regretted the decision, not for a minute. The investment has since completely paid out. Great books otherwise...

fedeb95(10000) 5 days ago [-]

Good advice. A film camera forces you to learn, while digital will 'fix' things for you

WastingMyTime89(10000) 5 days ago [-]

Best pictures of the article: the first four.

Why are they the best: their composition is interesting with great use of vertical lines, background colors and well positioned windows.

First line of the core of the article: 'It is said that composition is the most important aspect of a photograph. I disagree.'

Well, let's agree to disagree then.

xiande04(10000) 5 days ago [-]

Those are the best photos because he took all the other photos for demonstration purposes.

mod50ack(10000) 5 days ago [-]

I've been taking pictures for a few years now. Not all of my photos are fit for Wikimedia Commons, but I've put a lot on [1], especially of baseball players and buildings (just what I happen to enjoy photographing).

I began originally with point-and-shoots, which I borrowed from my parents and basically used as they're designed: pointing and shooting. The quality of those early photos is not always great. But with time, I learned how to get a quality shot of a person or action (in baseball) and how to wait for the right moment and so on. This was without cheating (using burst mode). Of course, you don't get 100% of the moments that are interesting. Sometimes someone blocks you, sometimes it just doesn't happen at the right time, sometimes you're eating a snack. It's OK. The most important thing is to get a lot of practice taking pictures of the thing you like to photograph.

You might thing non-moving subjects are easier. They are in a way. But if you are photographing buildings, for instance, then you should care about straight lines and getting the right perspective. This can be infuriating if you know what to look for. People move, but at least we humans don't have straight lines or flat surfaces and so it's pretty easy to capture us from many different angles.

The point-and-shoots helped me develop my sense of timing. You don't need to use a point-and-shoot for this, though; a better camera will be perfectly good (and give you better image quality). You can get a used DSLR for pretty cheap. The switch to interchangeable-lens cameras (DSLRs, which are a bit older, or mirrorless cameras, which are newer) will give you the opportunity to try out different lenses, which make a huge difference for your photography. (By the way, DSLR lenses can generally be used on newer mirrorless cameras.) My first photography-changing DSLR lens was a Canon 70-300, which made my baseball photography much more credible. You don't need to practice to use a new lens, of course. But if you practice using some, then you can gain a better understanding of the importance of focal length (and aperture) to your ptohography.

The next step is getting out of auto mode and getting a full understanding of manual exposure and RAW. For me, for a long time, I shot mostly in a program mode on my DSLR because I only saw through the optical viewfinder and wanted my exposure to be right, especially as I was shooting JPEG on small SD cards. That's all very well. But to get to the level I am at now, I needed to start shooting in full manual (the mode I now use 99% of the time), and shooting in my camera's RAW format (which gives you MUCH more editing latitude in basically every way, as well as the full quality of your camera).

Using a contemporary mirrorless camera can actually make manual exposure significantly easier. You can now see your exposure in your viewfinder (generally this can be turned off, too). (On DSLRs from the last ~ten years, you can get this same feature by using live view --- i.e., shooting with the back screen rather than the viewfinder.) This can confirm your exposure visually and also get you more familiar with the different settings and how to 'eyeball' it even on a DSLR/without exposure simulation in the viewfinder.

At every step, practice is key. If you want to take pictures, go out and do it. It doesn't matter if they're bad. They will not all be good, especially not at the beginning. Find the things you like to photograph and keep at it until you are happy with what you're doing. (Or, if you find you don't like photography, that's OK, too.)

[1] Shameless plug: https://commons.wikimedia.org/wiki/Special:MediaSearch?type=...

tsqj(10000) 3 days ago [-]

thanks for sharing! those are really valuable thoughts.

one of the top takeaways :

> The most important thing is to get a lot of practice taking pictures of the thing you like to photograph.

fedeb95(10000) 5 days ago [-]

All good advices. I've been taking pictures for a decade and reading books really does help more than a lot of online stuff you find. Also, looking at masters' photographs and reasoning about how they were made. After some time when I thought composition was all that mattered, I realised that light is all that matters. I use only natural light, so that means I shoot only when the intersection of good light and free time is not empty, but this has improved my craft.

Loic(10000) 5 days ago [-]

Time. It takes time and preparation to wait for the right light at the right place. Waking-up early is oft very well rewarded.

qntty(10000) 5 days ago [-]

See also, Marc Levey (CS professor at Stanford specializing in computational photography) gave some good lectures at Google a few years ago on digital photography:

https://sites.google.com/site/marclevoylectures/home

mgraczyk(10000) 5 days ago [-]

Levoy!

keepquestioning(10000) 5 days ago [-]

Anything like this but for analog?

yieldcrv(10000) 5 days ago [-]

This is good, I would add that "made up composition rules" are instead "made up composition guidelines", that people should learn how to deviate from after constricting themselves with false constraints

trap_goes_hot(10000) 5 days ago [-]

Artists will not have a common answer to 'Are there any rules in art?'. Personally, I think there are patterns in art, and there are often rules that can describe those patterns. Oddly, I think the word rule works out OK. 'guidelines' is problematic in another way as it also implies that one is being guided towards a certain aesthetic/end result.

canbus(10000) 5 days ago [-]

good things to learn, even better to forget.

semireg(10000) 5 days ago [-]

The things I'm drawn to in life is where art meets science. In hindsight, so much of the secret is knowing how to avoid failure. Baking bread? Build the intuition over time and you'll realize baking is forgiving so long as you don't do these "5 bad things." Gardening/farming? Yeah, there's a big list of bad things. Brewing beer? Another list of things to avoid. The basic rules (rooted in science) are like guardrails and everything else is the art. I love this so much.

In my early 20s I had a week long mind meld knowledge transfer from a self taught photographer. It made me fall in love with photography. I'm still using it to this day to photograph new label printers (black plastic is terrible to photograph) and labels (oh god they are 2D!).

I'm doing an OK job. Room for improvement but fine for the initial launch. You can see them here: https://mydpi.com/products/professional-synthetic-direct-the...

In case you're like "why is this guy selling label printers?!"

I'm a solo software dev that wrote Label LIVE (electron) to design and print labels. Now I'm vertically integrating with a printer I've imported from China and labels made in the USA.

Business and entrepreneurship: just avoid these 9999 things and you'll be fine! Science and art...

causi(10000) 5 days ago [-]

Another list of things to avoid. The basic rules (rooted in science) are like guardrails and everything else is the art.

Too many guides and educational material in general fail to do this. I'd try a lot more new things if along with the 'how to replace your transmission' book there was 'the top ten ways you could screw up replacing your transmission and their consequences' section.

keepquestioning(10000) 5 days ago [-]

You might like this book

https://natureofcode.com

kak9(10000) 5 days ago [-]

wait what are the five bad things in baking bread?

bambax(10000) 5 days ago [-]

Re: focus. Most cameras focus on shutter button half-pressed. Learn to use this by focusing first (with a half-press), framing, and then taking the picture by finishing the press (not doing a complete press again as that would reset the focus point).

Some cameras let you have a focus button different from the shutter button. This can be super useful and can become natural after a little while.

yetihehe(10000) 5 days ago [-]

Or even use manual focus. Want to do 'artistic' photos? Get lens with wide aperture (below f/2), it has very low depth of field, so you can focus on your subject and have blurred background. After using 'normal' cameras all my life, I've recently (less than 3 weeks ago) bought cheapest mirrorless camera and bought f/1.2 lens (50mm without zoom, those are cheapest lens there are). First photos of my dog were so sweet I instantly fell in love with that camera and lens.

bambax(10000) 5 days ago [-]

Good advice and good writing. But the title is misleading: this is photography for aspiring photographers, not esp. geeks. I was expecting more geeky things.

For example, I have recently been toying with making a kind of photobooth / selfies machine for parties by connecting a DSLR with a Raspberry Pi; people take pictures of themselves with a remote (the excellent Yongnuo RF-603 which is radio, not infrared); the Pi downloads the images (with gphoto2), resizes them, prints the date, and uploads to a server.

People can see their images on their phones less than a minute after it was taken. It's a lot of fun.

SamBam(10000) 5 days ago [-]

Around 2012 I was asked to make a photo booth for a friend's wedding. I ended up managing to rig an old-fashioned doorbell to an Arduino to give me a 3-second countdown, attached to a DSLR for the photo, an eye-fi card for the wireless connectivity because I think the camera didn't have on-board WiFi, and an iPad attached outside the both that always showed the last photo and let you scroll back.

I think I'd have to recreate it from scratch if I had to do it again because I have no idea now.





Historical Discussions: Everything I wish I knew when learning C (November 28, 2022: 738 points)

(738) Everything I wish I knew when learning C

738 points 5 days ago by bubblehack3r in 10000th position

tmewett.com | Estimated reading time – 17 minutes | comments | anchor

Learning C was quite difficult for me. The basics of the language itself weren't so bad, but "programming in C" requires a lot of other kinds of knowledge which aren't as easy to pick up on:

  • C has no environment which smooths out platform or OS differences; you need to know about your platform too
  • there are many C compiler options and build tools, making even running a simple program involve lots of decisions
  • there are important concepts related to CPUs, OSes, compiled code in general
  • it's used in such varied ways that there's far less a centralised "community" or style than other languages

This page is a living collection of summaries, signposts, and advice for these broader points that made my journey with C and other compiled languages easier. I hope it's useful to you! (And if it is, make sure to subscribe for any updates.)

General resources

Good projects to learn from

Sometimes it's helpful to just read some small, self-contained C code to get to grips with how it looks.

  • Bloopsaphone, a Ruby library for synthesising sounds which has a small C module at its core. Has a small number of concepts and a good structure.
  • Simple Dynamic Strings (sds). Has one .c and .h file each, and is a good example of how you might do more complex resource management.
  • Brogue CE, a roguelike video game. This is larger and less perfected, at >30k LOC. But I maintain it, and many of our contributors have sharpened their C by working on it.
  • stb single-file libraries. These are small to medium-sized modules designed to be highly portable, including targetting embedded devices and games consoles.

Compilation, linking, headers, and symbols

Some basics on how C compilation works, because it will help other things make sense.

C code is written in .c source files. Each source file is compiled to a .o object file, which is like a container for the compiled function code in the .c file. They are not executable. Object files have inside them a table of symbols, which are the names of the global functions and variables defined in that file.

# compile to objects
cc -c thing.c -o thing.o
cc -c stuff.c -o stuff.o

Source files are completely independent of each other, and can be compiled to objects in parallel.

To use function and variables across files, we use header files (.h). These are just ordinary C source files used in a specific way. Recall above that object files only contain the names of global functions and variables—no types, macros, or even function parameters. To use symbols across files, we need to specify all this extra information needed to make use of them. We put these "declarations" in their own .h file, so other .c files can #include them.

To avoid duplication, a .c file will typically not define its own types/macros etc. and will just include the header file for itself or the module/component it's part of.

Think of a header file as a specification of an API, that can be implemented across any number of source files. You can even write different implementations of the same header, for different platforms or purposes.

When compiling a reference to a symbol that has only been declared (e.g. by an included header) and not defined, the object file will mark that this symbol is missing and needs to be filled in.

The final work of joining one or more objects together, matching up all symbol references, is done by the "linker" component of the compiler. The linker outputs complete executables or shared libraries.

# link objects to executable
cc thing.o stuff.o -o gizmo

In summary, we don't "include" other source files in C, like we do other languages. We include declarations, and then the code gets matched up by the linker.

Do not use these functions

C is old and tries to be highly backwards-compatible. As such it has features that ought to be avoided.

  • atoi(), atol(), and friends; they return 0 on error, but this is also a valid return value. Prefer strtoi(), etc.
  • gets() is unsafe as no bounds on the destination buffer can be given. Prefer fgets().
  • Unbounded strfmt(), strcpy(), etc., for the same reason. Prefer the ...n... variants like strncpy(). (To reconsider...)

Arrays aren't values

It's important to realise that C, as a language, deals only with known-size pieces of data. You could probably summarise C as "the language of copying known-size values."

I can pass a integer or a struct around a program, return them from functions, etc. and treat them as proper objects because C knows their size and hence can compile code to copy their full data around.

I can't do this with an array. The sizes of arrays are not known in any useful way to C. When I declare a variable of type int[5] in a function, effectively I don't get a value of type int[5]; I get an int* value which has 5 ints allocated at it. Since this is just a pointer, the programmer, not the language, has to manage copying the data behind it and keeping it valid.

However, arrays inside structs are treated as values and are fully copied with the struct.

(Technically, sized array types are real types, not just pointers; e.g. sizeof will tell you the size of the whole array. But you can't treat them as self-contained values.)

Essential compiler flags

Compilers have so many options and the defaults aren't very good. Here are the absolute essential flags you may need. (They are given in GCC/Clang style; syntax may vary on other compilers.)

  • -O2: optimise code for release builds
  • -g -Og: for debug builds; enable extra information for debuggers, and optimise for debugging
  • -Wall to enable many warnings (kind of like a linter). You can disable specific warnings with -Wno-...
  • -Werror to turn warnings into errors. I recommend always turning on at least -Werror=implicit, which ensures calling undeclared functions results in an error(!)
  • -DNAME and -DNAME=value for defining macros
  • -std=...: choose a standard. In most cases you can omit this to use your compiler's default (usually the latest standard). Those with great portability concerns may use -std=c89 to keep to "classic" C.

See also the full docs for the huge number of options GCC supports.

Three types of memory, and when to use them

  • Automatic storage is where local variables are stored. A new region of automatic storage is created for a function when it is called, and deleted when it returns. Only the return value is kept; it is copied into the automatic storage of the function which called it. This means that it is unsafe to return a pointer to a local variable, because the underlying data will be silently deleted. Automatic storage is often called the stack.

  • Allocated storage is the result of using malloc(). It survives until it is free()'d, so can be passed wherever, including upwards to calling functions. It is often called the heap.

  • Static storage is valid for the lifetime of the program. It is allocated when the process starts. Global variables are stored here.

If you want to "return" memory from a function, you don't have to use malloc/allocated storage; you can pass a pointer to a local data:

void getData(int *data) {
    data[0] = 1;
    data[1] = 4;
    data[2] = 9;
}
void main() {
    int data[3];
    getData(data);
    printf('%d\n', data[1]);
}

Naming conventions

C has no support for namespaces. If you're making a public library, or want a "module" to have a name, you need to choose a prefix to add to all public API names:

  • functions
  • types
  • enum values
  • macros

Additionally, you should always include some different prefix for each enum, so you know which enum type the value belongs to:

enum color {
    COLOR_RED,
    COLOR_BLUE,
    ...
}

There's no real convention about names, e.g. snake_case vs camelCase. Pick something and be consistent! The closest thing to a convention I know of is that some people name types like my_type_t since many standard C types are like that (ptrdiff_t, int32_t, etc.).

static

On a function or file-level variable, static makes it file-local. It won't be exported as a symbol for use by other source files.

static can also be used on a local variable, which makes the variable persist between calls to that function. You can think of this like a global variable that is scoped to only one function. This can be useful to compute and store data for reuse by subsequent calls; but remember, this comes with the usual caveats of global/shared state, such as thread safety, and clashing with recursion.

The struct method pattern

If you learned a more featureful language before C, you might find it hard to visualise how to translate that knowledge. Here's a common idiom which resembles object-oriented programming: the "struct method." You write functions which accept pointers to structs to alter them or get properties:

typedef struct {
    int x;
    int y;
} vec2;
void vec_add(vec2 *u, const vec2 *v) {
    u->x += v->x;
    u->y += v->y;
}
int vec_dot(const vec2 *u, const vec2 *v) {
    return u->x * v->x + u->y * v->y;
}

You can't extend structs or do anything really OO-like, but it's a useful pattern to think with.

const

Declaring a variable or parameter of type T as const T means, roughly, that the variable cannot be modified. This means that it can't be assigned to, and also that it can't be changed if T is a pointer or array type.

You can cast T to const T, but not vice versa.

It's a good habit to declare pointer parameters to functions as const by default, and only omit it when you need to modify them.

Platforms and standard APIs

When you pull in #include <some_header.h> it's hard to conceptualise what you're depending on. It will be from one of the following:

  • The standard C library (abbr. "stdlib"). Examples: stdio.h, stdlib.h, error.h
    • This is part of the language specification, and should be implemented by all compliant platforms and compilers. Very safe to depend on.
    • https://en.cppreference.com/w/c/header
  • POSIX, a standard for operating system APIs. Examples: unistd.h, sys/time.h
  • A non-standard operating system interface:
  • A third-party library, installed in a standard location.

It can be a good idea to interface with your more platform-specific code through a platform-neutral header file so it can be implemented in different ways. Lots of popular C libraries are basically just unified, well-designed abstractions over platform-specific functionality.

Integers

Integers are very cursed in C. Writing correct code takes some care:

Sizes

All integer types have a defined minimum size. On common platforms, some are larger than their minimum size, such as int, which is 32-bit on Windows, macOS, and Linux, despite being minimum 16-bit. When writing portable code, you must assume integers can never go above their minimum size.

If you want exact control over integer sizes, you can use the standard types in stdint.h, like int32_t, uint64_t, etc. There are also _least_t and _fast_t types.

Should you use these well-specified types everywhere you can? I must admit I'm torn on this question, but the more I think about it, the more I think you should—there are no downsides. The only reason you really shouldn't is when making an API which has to interface with very old C89 compilers which lack stdint.h. There's also an argument for considering what the type communicates to the reader and whether the size is actually important; however by using standard types like int you are still implicitly relying on a certain size. It's probably no worse, yet clearer, to use int16_fast_t or something over int. (However, typically no one does this, including me!)

Arithmetic in C is subject to many bizarre rules which can give unexpected or unportable results. Integer promotions are especially important to be aware of.

char signedness

All other integer types default to signed, but bare char can be signed or unsigned, depending on the platform. As such, it's only portable when used for strings; specify the sign too if you want a small/minimum-8-bit number.

Macros vs const variables

To define simple constant values, you have two choices:

static const int my_constant = 5;
// or
#define MY_CONSTANT 5

The difference is that the former is a real variable and the latter is a copy-pasted inline expression.

  • Unlike variables, you can use macros in contexts where you need a "constant expression," like array lengths or switch statement cases.
  • Unlike macros, you can get a pointer to a variable.

Having constants actually be "constant expressions" is very useful and hence they should usually be defined as macros. Variables are better for larger or more complex values like struct instances.

Macros vs inline functions

Macros can have parameters, which can then expand to C code.

Advantages over functions:

  • The code is pasted right in the surrounding code, instead of compiling function call instructions. This can make code faster, as function calls have some overhead.

  • They can be type-generic. For example, x + y is valid syntax for any numeric type. If we made that a function, we'd have to declare them as arguments and choose their type, i.e. size and signedness, in advance, which would make it only usable in some contexts.

Disadvantages:

  • Repeated evaluation of arguments. Suppose we have a macro MY_MACRO(x). If x is used multiple times in the definition, then the expression x will be evaluated multiple times, because it is simply copied and pasted. Compare that with a function, where expressions as arguments are evaluated once to values and then passed into the function.

  • They can be error-prone because they work at the source level. It is generally a good idea to use brackets gratuituously, always around the whole macro definition itself and any arguments, so expressions don't merge unintentionally.

    // Instead of:
    #define MY_MACRO(x) x+x
    // Do:
    #define MY_MACRO(x) ((x)+(x))
    

Unless you need to be type-generic, you can get the best of both worlds by defining a function as static inline. inline provides a hint to compilers that the code in the function should be compiled directly into where it is used, instead of being called. You can put static inline functions in header files, just like macros, with no issues.




All Comments: [-] | anchor

caerwy(10000) 5 days ago [-]

I'd recommend also reading Rob Pike's Notes on Programming in C, http://doc.cat-v.org/bell_labs/pikestyle

emmelaich(10000) 5 days ago [-]

Quote which is interesting re Go lang..

> I eschew embedded capital letters in names; to my prose-oriented eyes, they are too awkward to read comfortably. They jangle like bad typography.

DSMan195276(10000) 5 days ago [-]

I like it, but the array details are a little bit off. An actual array does have a known size, that's why when given a real array `sizeof` can give the size of the array itself rather than the size of a pointer. There's no particular reason why C doesn't allow you to assign one array to another of the same length, it's largely just an arbitrary restriction. As you noted, it already has to be able to do this when assigning `struct`s.

Additionally a declared array such as `int arr[5]` does actually have the type `int [5]`, that is the array type. In most situations that decays to a pointer to the first element, but not always, such as with `sizeof`. This becomes a bit more relevant if you take the address of an array as you get a pointer to an array, Ex. `int (*ptr)[5] = &arr;`. As you can see the size is still there in the type, and if you do `sizeof *ptr` you'll get the size of the array.

tmewett(10000) 5 days ago [-]

Interesting, I didn't fully realise that. That it's arbitrary is annoying, I clearly had tried to rationalise it to myself! Thanks for the comments, will get around to amending

Jasper_(10000) 5 days ago [-]

I really wish that int arr[5] adopted the semantics of struct { int arr[5]; } -- that is, you can copy it, and you can pass it through a function without it decaying to a pointer. Right now in C:

    typedef uint32_t t1[5];
    typedef struct { uint32_t arr[5]; } t2;
    void test(t1 a, t2 b) {
        t1 c;
        t2 d;
        printf('%d %d %d %d\n', sizeof(a), sizeof(b), sizeof(c), sizeof(d));
    }
will print 4, 20, 20, 20. I understand that array types having their sizes in their types was one of Kernighan's gripes with Pascal [0], which likely explains why arrays decay to pointers, but for those cases, I'd say you should still decay to a pointer if you really wanted to, with an explicit length parameter.

[0] http://www.lysator.liu.se/c/bwk-on-pascal.html

a1369209993(10000) 5 days ago [-]

> There's no particular reason why C doesn't allow you to assign one array to another of the same length

Actually, there is a particular (though not necessarily good) reason, since that would require the compiler to either generate a loop (with conditional branch) for a (unconditional) assignment or generate unboundedly many assembly instructions (essentially a unrolled loop) for a single source operation.

Of course, that stopped being relevant when they added proper (assign, return, etc) support for structs, which can embed arrays anyway, but that wasn't part of the language initially.

retrac(10000) 5 days ago [-]

> it's largely just an arbitrary restriction

Kind of. But the restriction is in keeping with the C philosophy of no hidden implementation magic. C has the same restriction on structs. That's the same question; an array of bytes of known size to the compiler it could easily abstract away. But assignment is always a very cheap operation in C. If we allow assigning to represent memcpy() that property is no longer true.

Same reason why Rust requires you to .clone() so much. It could do many of the explicit copies transparently, but you might accidentally pass around a 4 terabyte array by value and not notice.

derefr(10000) 5 days ago [-]

> There's no particular reason why C doesn't allow you to assign one array to another of the same length, it's largely just an arbitrary restriction.

IIRC C has an informal guarantee that no primitive syntax will ever cause the CPU to do more than O(1) work at runtime. Assignment is always O(1), and therefore assignment is limited to scalars. If you need assignment that might do O(N) work, you need to call a stdlib function (memcpy/memmove) instead. If you need an allocation that might do O(N) work, you either need a function (malloc) or you need to do your allocation not-at-runtime, by structuring the data in the program's [writable] data segment, such that it gets 'allocated' at exec(2) time.

This is really one of the biggest formal changes between C and C++ — C++ assignment, and keywords `new` and `delete`, can both do O(N) work.

(Before anyone asks: a declaration `int foo[5];` in your code doesn't do O(N) work — it just moves the stack pointer, which is O(1).)

danoman(10000) 4 days ago [-]

Return pointer to array of 4 integers:

  int32_t (* bar(void))[4] {
      static int32_t u[4] = {1, 0, 1, 0};
      return &u;
  }
Return a pointer to a function taking a char:

  void f(char a) {
      // ...
  }
  void (* baz(void))(char) {
      return f;
  }
Joker_vD(10000) 4 days ago [-]

Another one corner of the language where arrays actually being arrays is important is multidimensional array access:

    int arr[5][7];
    arr[3][5] = 4; // equivalent to *(*(arr + 3) + 5) = 4;
This works because (arr + 3) has type 'pointer to int[7]', not 'pointer to int'. The resulting address computation is

    (char*)arr + 3 * sizeof(int[7]) + 5 * sizeof(int) ==
    (char*)arr + 26 * sizeof(int)
That's also another reason why types like 'int [5][7][]' are legal but 'int [5][][]' are not.
dahfizz(10000) 5 days ago [-]

Another weird property about C arrays is that &arr == arr. The reference of an array is the pointer to the first element, which is what `arr` itself decays to. If arr was a pointer, &arr != arr.

torstenvl(10000) 5 days ago [-]

Decent article. A couple minor points:

c89 or c99, not c98.

Plain static variables aren't thread-safe by default, true, but there's also _Thread_local

gpderetta(10000) 5 days ago [-]

I haven't written actual C code in decades. Did C11 get magic statics with thread-safe initialization like C++11? Does C even have non-trivial initialization of statics local variables?

fsloth(10000) 5 days ago [-]

I would add to this: 1. Visual Studio debugger is really, really good and you should use it to step through your program (or equivalent IDE based workflow). 2. Learn to compile your code with memory safeguards and use the tooling around them. Specific depends on the platform. On POSIX address sanitizer is good I hear. On Windows you can use the debug CRT and gflags to instrument your binary really well.

rahen(10000) 5 days ago [-]

Besides Visual Studio (the Windows IDE, not VS Code), QT Creator also has an excellent debugger.

Probably the best open source and cross platform C/C++ IDE around.

Sohcahtoa82(10000) 5 days ago [-]

People should be using IDEs with modern debuggers anyways.

Being able to step through your code one line at time and instantly see the values of all your variables is going to be FAR more effective than adding a bunch of print statements, no matter what language you're using.

It always blows my mind to hear about how many engineers don't know how to use their IDE's debugger and don't know what a breakpoint is.

nadavision(10000) 5 days ago [-]

Thanks for submitting this. I'm teaching myself C so these high level overviews are super useful for improving my intuition. In the following example, shouldn't there be an asterisk * before the data argument in the getData function call? The way I understand it the function is expecting a pointer so you would need to pass it a pointer of the data object.

> 'If you want to "return" memory from a function, you don't have to use malloc/allocated storage; you can pass a pointer to a local data:

void getData(int *data) { data[0] = 1; data[1] = 4; data[2] = 9; }

void main() { int data[3]; getData(data); printf('%d\n', data[1]); } '

torstenvl(10000) 5 days ago [-]

No, it's correct. The asterisk is a little inconsistent, in that it means two opposite things. In the declaration it means 'this is a pointer.' However, in an expression, it means 'this is the underlying type' and serves to dereference the pointer.

    int a = 5;
    int *x; // this is a pointer
    x = &a;
    int c = *x; // both c and *x are ints
If it were *data, it would be equivalent to *(data + 0), which is equivalent to data[0], which is an int. You don't want to pass an int, you want to pass an *int.
seritools(10000) 5 days ago [-]

the local variable data effectively decays to int*.

*data would give you an int, &data would give you an int*.

nadavision(10000) 5 days ago [-]

Thanks for all the great answers. The inconsistency between pointer deceleration and dereference syntax was what got me. :)

unwind(10000) 5 days ago [-]

No, it's fine.

The name of an array decays to a pointer to the first element in various contexts. You could do `&data[0]` but it means exactly the same thing and would read as over-complicated things to C programmers.

limaoscarjuliet(10000) 5 days ago [-]

I was born in '74 so the last generation to start with C and go to other, higher-level, languages like Python or JavaScript. Going in this direction was natural. I was amazed by all the magic the higher-level languages offered.

Going the other direction is a bit more difficult apparently. 'What do you mean it does not do that?'. Interesting perspective indeed!

cbdumas(10000) 5 days ago [-]

Introductory programming courses at the University of Arizona were still taught in C when I was a freshman in 2008

untech(10000) 2 days ago [-]

This was my experience with learning programming as well, however, I am 2x younger :)

sbaiddn(10000) 5 days ago [-]

Im a decade younger and my university taught C for its intro to programing class.

Granted it was a disaster of a programing class.

lp251(10000) 5 days ago [-]

I was born in the late 80s and C was my first language, in a community college intro to programming class.

user3939382(10000) 5 days ago [-]

What was nice about C then was that, based on my study of CPUs at the time, you could pretty much get your head around what the CPU was doing. So you could learn the instructions (C) and the machine following them (the CPU).

When I got to modern CPUs it's so complex my eyes glazed over reading the explanation and I gave up trying to understand them.

Hadarai5(10000) 5 days ago [-]

I started coding with C and OCaml in 2019. Everything in between these two was so unnatural. With JavaScript as the worst of all

andrewmcwatters(10000) 5 days ago [-]

The older I got, the more I realized you seem to be heavily disadvantaged by not having some rudimentary understanding of assembly when using C.

When you do have such an understanding, some small details, like declaring stack variables at the top of functions in ANSI C, become obvious.

programmer_dude(10000) 5 days ago [-]

One does not follow from the other. You can declare your stack variable anywhere in the function and still have the same stack layout. It is upto the compiler/language designer (I am not talking about C per se).

nwellnhof(10000) 5 days ago [-]

I wouldn't recommend fixed-size integers in general. Most of the time you want size_t or the lesser-known, signed ptrdiff_t. More often than not, integers are array sizes or indices, so size_t is the best type to use. In other cases, int is practically the same as int32_t and long long the same as int64_t except for really exotic platforms.

throwawaaarrgh(10000) 5 days ago [-]

I've been burnt enough by varying sizes that I don't care if there's a performance impact anymore. Consistency and reliability are the main things I care about.

alexandreyc(10000) 5 days ago [-]

After more than a decade without writing any C code, I'm currently reading 'C Programming: A modern Approach' by K. N. King (http://www.knking.com/books/c2/) and I found it very good. I think it's a better modern alternative to the K&R.

Koshkin(10000) 5 days ago [-]

832 pages is not a 'better alternative.'

lebuffon(10000) 5 days ago [-]

' ... is often called the stack.' ' integers are cursed'

These statements and a few others made me uncomfortable. They imply, to me, that the author has too little knowledge of computer internals to be programming in C.

C does a wonderful job of looking like a high level language but it was designed to do low level stuff. There is an implicit assumption, to my mind, that the user has a deeper understanding of what is behind the 'curtain' that is created by the compiler.

It almost seems like the there should a pre-requisite course like 'Introduction to Computer hardware' before one is allowed to write a line of C.

Maybe I'm just too old...

Existenceblinks(10000) 5 days ago [-]

Year 1 in my university, they started with electron, I was impressed. All to way to diode and how logic gate was made. And then digital circuit design, and you got your one|two's complement from here.

jjav(10000) 5 days ago [-]

> There is an implicit assumption, to my mind, that the user has a deeper understanding of what is behind the 'curtain' that is created by the compiler.

It's also useful to remember that C is from a time when approximately everyone doing programming understood these things because you had to be exposed to lower level details.

hellfish(10000) 5 days ago [-]

> ' ... is often called the stack.'

What's wrong about the stack part? It seemed a little odd (and unnecessarily abstract) that the author said 'automatic storage' instead of just stack

is there a situation where there's automatic storage but no stack?

jesse__(10000) 5 days ago [-]

Unsolicited personal opinion/anecdote incoming:

As someone who started writing C with very little understanding of how the underlying hardware worked (or, indeed, programming in general), I support and disagree with parts of this comment at the same time.

On one hand, I support the notion that programming well (in any language) requires knowledge of hardware architectures.

On the other hand, I disagree that people should not be 'allowed to write a line of C' before they have that understanding.

I started writing C early on in my programming career (having already dropped out of 'Introduction to Computer Hardware'), and I'll admit, it was tough. I probably would have had an easier time if I had taken a year to study x86, win32 internals and graphics pipelines. That said, I was interested in learning to program graphics, so that's what I did, and I learned a tremendous amount while doing it. It was probably the long way round, but if the goal was learning, 'just doing it' was an effective strategy.

What I'm trying to say here is that for people that would drop out of 'Introduction to Computer Hardware', learning C is actually an extremely good supplement for learning about underlying hardware architectures because, in the long run, you have no choice if you want to be good at it.

pclmulqdq(10000) 5 days ago [-]

I think the 'automatic storage' terminology is technically correct. C can be used in places where there is no actual 'stack' and these still need a mechanism for local variables, so they specify a different kind of automatic storage.

Everyone uses a stack now, though, even very exotic processors.

lmm(10000) 5 days ago [-]

> There is an implicit assumption, to my mind, that the user has a deeper understanding of what is behind the 'curtain' that is created by the compiler.

With the behaviour of today's C compilers, what benefit would there be to such an understading? It would seem to mainly give the user a lot of ways to shoot themselves in the foot.

tmewett(10000) 5 days ago [-]

>C does a wonderful job of looking like a high level language but it was designed to do low level stuff. There is an implicit assumption, to my mind, that the user has a deeper understanding of what is behind the 'curtain' that is created by the compiler.

Lots of people say this, indeed several comments here talk about learning assembly before C being beneficial.

I actually think this is not true, today. In its original incarnation C may have mapped closely to instructions, but the details of e.g. x86 are IMO neither necessary nor particularly useful knowledge. Knowing what memory is like and how stuff is laid out is enough.

C is not just a super-thin layer to the bare metal. By 'integers are cursed,' I mean exactly that: C integers are cursed. The arithmetic behaviours that standard C defines for integers are error-prone and in many cases undefined. C is a layer and it has its own behaviour, it doesn't just expose your CPU.

nayuki(10000) 5 days ago [-]

> Everything I wish I knew when learning C

By far my biggest regret is that the learning materials I was exposed to (web pages, textbooks, lectures, professors, etc.) did not mention or emphasize how insidious undefined behavior is.

Two of the worst C and C++ debugging experiences I had followed this template: Some coworker asked me why their function was crashing, I edit their function and it sometimes crashes or doesn't depending on how I rearrange lines of code, and later I figure out that some statement near the top of the function corrupted the stack and that the crashes had nothing to do with my edits.

Undefined behavior is deceptive because the point at which the program state is corrupted can be arbitrarily far away from the point at which you visibly notice a crash or wrong data. UB can also be non-deterministic depending on OS/compiler/code/moonphase. Moreover, 'behaving correctly' is one legal behavior of UB, which can fool you into believing your program is correct when it has a hidden bug.

A related post on the HN front page: https://predr.ag/blog/falsehoods-programmers-believe-about-u... , https://news.ycombinator.com/item?id=33771922

My own write-up: https://www.nayuki.io/page/undefined-behavior-in-c-and-cplus...

The take-home lesson about UB is to only rely on following the language rules strictly (e.g. don't dereference null pointer, don't overflow signed integer, don't go past end of array). Don't just assume that your program is correct because there were no compiler warnings and the runtime behavior passed your tests.

saghm(10000) 4 days ago [-]

I recently dealt with a bit of undefined behavior (in unsafe Rust code, although the behavior here could similarly happen in C/C++) where attempting to print a value caused it to change. It's hard to overstate how jarring it is to see an code that says 'assert that this value isn't an error, print it, and then try to use it', and have the assertion pass but then have it be printed out as an error and then panic when trying to use it There's absolutely no reason why this can't happen since 'flipping bits of the value you tried to print' doesn't count as potential UB any less than a segfault, but it can be hard to turn off the part of your brain that is used to assuming that values can't just arbitrarily change at any point in time. 'Ignore the rest of the program and do whatever you want after a single mistake' is not a good failure mode, and it's kind of astonishing to me that people are mostly just fine with it because they think they'll be careful enough not to make a mistake ever or that enough of the time it happened they were lucky that it didn't completely screw them over.

The only reason we use unsafe code on my team's project is because we're interfacing with C code, so it was hard not to come away from that experience thinking that it would be incredibly valuable to shrink the amount of interfacing with C as small as possible, and ideally to the point where we don't need to at all.

bluecalm(10000) 5 days ago [-]

It's not insidious at all. C compiler offers you a deal: 'Hey, my dear programmer, we are trying to make an efficient program here. Sadly, I am not sophisticated enough to deduct a lot of things but you can help me! Here are some of the rules: don't overflow integers, don't dereference null pointers, don't go outside of array bounds. You follow those and I will fulfill my part of making your code execute quickly'.

The deal is known and fair. Just be a responsible adult about it: accept it, live with the consequences and enjoy efficiency gains. You can reject it but then don't use arrays without a bound check (a lot of libraries out there offer that), check your integers bounds or use a sanitizer, check your pointers for nulls before dereferencing them, there are many tools out, there to help you, or... Just use another language that does all that for you.

lolptdr(10000) 5 days ago [-]

As a curious FE developer with no C experience, this was very interesting. Thanks for writing the article!

photochemsyn(10000) 5 days ago [-]

This article on undefined behavior looks pretty good (2011?)

https://blog.regehr.org/archives/213

A main point in the article is function classification, i.e. 'Type 1 Functions' are outward-facing, and subject to bad or malicious input, so require lots of input checking and verification that preconditions are met:

> 'These have no restrictions on their inputs: they behave well for all possible inputs (of course, "behaving well" may include returning an error code). Generally, API-level functions and functions that deal with unsanitized data should be Type 1.'

Internal utility functions that only use data already filtered through Type 1 functions are called 'Type 3 Functions', i.e. they can result in UB if given bad inputs:

> 'Is it OK to write functions like this, that have non-trivial preconditions? In general, for internal utility functions this is perfectly OK as long as the precondition is clearly documented.'

Incidentally I found that article from the top link in this Chris Lattner post on the LLVM Project Blog, 'What Every C Programmer Should Know About Undefined Behavior':

http://blog.llvm.org/2011/05/what-every-c-programmer-should-...

In particular this bit on why internal functions (Type 3, above) shouldn't have to implement extensive preconditions (pointer dereferencing in this case):

> 'To eliminate this source of undefined behavior, array accesses would have to each be range checked, and the ABI would have to be changed to make sure that range information follows around any pointers that could be subject to pointer arithmetic. This would have an extremely high cost for many numerical and other applications, as well as breaking binary compatibility with every existing C library.'

Basically, the conclusion appears to be that any data input to a C program by a user, socket, file, etc. needs to go through a filtering and verification process of some kind, before being handed to over to internal functions (not accessible to users etc.) that don't bother with precondition testing, and which are designed to maximize performance.

In C++ I suppose, this is formalized with public/private/protected class members.

titzer(10000) 5 days ago [-]

> how insidious undefined behavior is.

Indeed. UB in C doesn't mean 'and then the program goes off the rails', it means that the entire program execution was meaningless, and no part of the toolchain is obligated to give any guarantees whatsoever if the program is ever executed, from the very first instruction. A UB-having program could time-travel back to the start of the universe, delete it, and replace the entire universe with a version that did not give rise to humans and thus did not give rise to computers or C, and thus never exist.

It's so insidiously defined because compilers optimize based on UB; they assume it never happens and will make transformations to the program whose effects could manifest before the UB-having code executes. That effectively makes UB impossible to debug. It's monumentally rude to us poor programmers who have bugs in our programs.

Waterluvian(10000) 5 days ago [-]

I haven't used C or C++ for anything, but in writing a Game Boy emulator I ran into exactly that kind of memory corruption pain. An opcode I implemented wrong causes memory to corrupt, which goes unnoticed for millions of cycles or sometimes forever depending on the game. Good luck debugging that!

My lesson was: here's a really really good case for careful unit testing.

halpmeh(10000) 5 days ago [-]

Was rewriting the stack due to undefined behavior or was it due to a logic error, e.g. improper bounds calculation?

dockd(10000) 5 days ago [-]

I would go one step farther: The documentation will say it is undefined behavior but the compiler doesn't have to. Here's an example from the man page for sprintf

  sprintf(buf, '%s some further text', buf);
If you miss that section of the manual, your code may work, leading you to think the behavior is defined.

Then you will have interesting arguments with other programmers about what exactly is undefined behavior, e.g. what happens for

  sprintf(buf, '%d %d', f(i), i++);
jhallenworld(10000) 5 days ago [-]

I've been programming in C forever, one advantage is that the language has not evolved much (especially compared with C++), but it has evolved.

There was the big K&R C to ANSI C function declaration transition. For portable code, you used K&R C well into the 90s (because older machines only had the K&R compiler), or used ugly macros to automatically convert from ANSI to K&R.

Another was the addition of 'const' to the language. It used to be said that const was a virus: once you start using it, you need to use it universally in your entire code-base.

A more recent big one is stdint.h (types like uint32_t). To correctly use these types you must also use the macros in inttypes.h for scanf and printf conversions. IMHO, they both should have been in the same header files, they go along with each other.

So in the old days, you would say: unsigned int x; printf('%u\n', x);

But now, you should tediously say: uint32_t x; printf('%'PRIu32'\n', x);

(Because uint32_t might be defined as a long even if it's the same size as in int, so you will get compiler warnings. You could use %lu, but then you get compiler warnings the other way.)

Another: On UNIX, we don't have to worry about 'wide' characters (use UTF-8 instead) and wide strings, but you certainly do on Windows. 'Wide' characters are obsolete but Windows is stuck with them since they are built into the OS.

suprjami(10000) 5 days ago [-]

> stdint.h / inttypes.h

Including inttypes also includes stdint.

You use stdint where you need the types only, like a header or non-IO module, and use inttypes where you need the print formatting.

I agree it's a bit weird but that's the way I understand the intended usage.

a1369209993(10000) 4 days ago [-]

> It used to be said that const was a virus: once you start using it, you need to use it universally in your entire code-base.

In order for const to actually work for what it's supposed to do, it does have to be viral in the direction of data flow. You should start by adding const to function arguments that point to data the function only reads (and doesn't pass the pointer to any subroutines) and expand from there. Eg:

  _Bool isurl(char /*const here*/* s) {
    while(isalpha(*s)) s++;
    return *s == ':';
    } /* s is never written through */
Then anything that passes pointers (only) to functions like isurl, and so on as is convenient.
hddqsb(10000) 5 days ago [-]

Some constructive feedback:

> Here are the absolute essential flags you may need.

I highly recommend including `-fsanitize=address,undefined` in there (docs: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...).

(Edit: But probably not in release builds, as @rmind points out.)

> The closest thing to a convention I know of is that some people name types like my_type_t since many standard C types are like that

Beware that names beginning with 'int'/'uint' and ending with '_t' are reserved in <stdint.h>.

[Edited; I originally missed the part about 'beginning with int/uint', and wrote the following incorrectly comment: 'That shouldn't be recommended, because names ending with '_t' are reserved. (As of C23 they are only 'potentially reserved', which means they are only reserved if an implementation actually uses the name: https://en.cppreference.com/w/c/language/identifier. Previously, defining any typedef name ending with '_t' technically invokes undefined behaviour.)']

The post never mentions undefined behaviour, which I think is a big omission (especially for programmers coming from languages with array index checking).

> void main() {

As @vmilner mentioned, this is non-standard (reference: https://en.cppreference.com/w/c/language/main_function). The correct declaration is either `int main(void)` or the argc+argv version.

(I must confess that I am guilty of using `int main()`, which is valid in C++ but technically not in C: https://stackoverflow.com/questions/29190986/is-int-main-wit...).

> You can cast T to const T, but not vice versa.

This is inaccurate. You can implicitly convert T* to const T*, but you need to use an explicit cast to convert from const T* to T*.

hddqsb(10000) 5 days ago [-]

UPDATE regarding '_t' suffix:

POSIX reserves '_t' suffix everywhere (not just for identifiers beginning with 'int'/'uint' from <stdint.h>); references: https://www.gnu.org/software/libc/manual/html_node/Reserved-..., https://pubs.opengroup.org/onlinepubs/9699919799/functions/V....

So I actually stand by my original comment that the convention of using '_t' suffix shouldn't be recommended. (It's just that the reasoning is for conformance with POSIX rather than with ISO C.)

nwellnhof(10000) 5 days ago [-]

> -fsanitize=address,undefined

In addition, I recommend -fsanitize=integer. This adds checks for unsigned integer overflow which is well-defined but almost never what you want. It also checks for truncation and sign changes in implicit conversions which can be helpful to identify bugs. This doesn't work if you pepper your code base with explicit integer casts, though, which many have considered good practice in the past.

tmewett(10000) 5 days ago [-]

Thank you so much! I will definitely be amending a few things. WRT no section on undefined behaviour - you're so right, how could I forget?

rmind(10000) 5 days ago [-]

Certainly yes, but for debug builds and tests. It can be heavyweight for production.

sedeki(10000) 5 days ago [-]

How can a (badly chosen) typedef name trigger _undefined behavior_, and not just, say, a compilation error...?

I find it difficult to imagine what that would even mean.

matthewaveryusa(10000) 5 days ago [-]

C spec:

>That shouldn't be recommended, because names ending with '_t' are reserved.

Also C spec naming new things:

>_Atomic _Bool

I'm glad to see the C folks have a sense of humor.

Joker_vD(10000) 5 days ago [-]

Why the hell 'potentially reserved' was introduced? How is it different from simply 'reserved' in practice except for the fact such things can be missing? How do you even use a 'potentially reserved' entity reliably? Write your own implementation for platforms where such an entity is not provided, and then conditionally not link it on the platforms where it actually is provided? Is the latter even possible?

Also, apparently, 'function names [...] beginning with 'is' or 'to' followed by a lowercase letter' are reserved if <ctype.h> and/or <wctype.h> are included. So apparently I can't have a function named 'touch_page()' or 'issue_command()' in my code. Just lovely.

MisterTea(10000) 5 days ago [-]

> I get an int* value which has 5 ints allocated at it.

Not crazy about this array explanation. Better wording would be: 'I get a memory address that points to the first byte of a chunk of memory that is large enough to hold 5 ints and tagged as int'

> Essential compiler flags

Nitpick, but this is only true if you are on a gcc/clang platform.

> If you want to "return" memory from a function, you don't have to use malloc/allocated storage; you can pass a pointer to a local data

It should be specified the memory must be passed by the caller for this to work. You can't create a var in a called function and return that (you can but you will get undefined behavior as that memory is up for grabs after the function returns).

> Integers are very cursed in C. Writing correct code takes some care.

Cursed how? No explanation.

Overall this article is not very good. I would add these to the lists:

General resources: Read the K&R book. Read the C spec. Be familiar with the computer architectures you are working on.

Good projects to learn from: Plan 9. Seriously. It's an entire OS that was designed to be as simple as possible by the same people who made Unix and the C.

pif(10000) 5 days ago [-]

> Be familiar with the computer architectures you are working on.

But keep in mind that C is not a portable assembly!

BeetleB(10000) 5 days ago [-]

The one thing missing from this list: Compiler optimizations can have undesirable effects.

A trivial example I ran into in an older job: I was compiling a program that relied on libm.so (the standard math library you get when including math.h). Now I wanted the code to use my own custom libm.so - not the one that was installed in /usr/lib or wherever, so I ensured it was dynamically compiled.

My code had some calls like:

    int x = sin(3.2);
During compilation, it computed sin(3.2) using the system libm. Notably, it would not use the sine function in my custom libm.so (and indeed, I needed it to!)

And IIRC, even -O0 was not enough to prevent this from happening. I had to go through the man page to figure out which compiler flags would prevent it.

(I don't recall if this was gcc or icc).

kccqzy(10000) 5 days ago [-]

C doesn't have namespaces so the compiler is certainly within its right to deduce that the sin() function is the one from the standard library.

Actually even in C++ after the compiler performs the unqualified name lookup if the result of the lookup is the standard sin() function it will make use of its internal knowledge about the function to do optimization.

Remember that the C or C++ standard doesn't deal with compilers and linkers; the standard deals with the 'implementation' as a whole.

unwind(10000) 5 days ago [-]

This looks decent, but I'm (highly) opposed to recommending `strncpy()` as a fix for `strcpy()` lacking bounds-checking. That's not what it's for, it's weird and should be considered as obosolete as `gets()` in my opinion.

If available, it's much better to do the `snprintf()` way as I mentioned in a comment last week, i.e. replace `strcpy(dest, src)` with `snprintf(dst, sizeof dst, '%s', src)` and always remember that '%s' part. Never put src there, of course.

There's also `strlcpy()` on some systems, but it's not standard.

0xbadcafebee(10000) 5 days ago [-]

Would it be a sin to use memcpy() and leave things like input validation to a separate function? I'm nervous any time somebody takes a function with purpose X and uses it for purpose Y.

mtlmtlmtlmtl(10000) 5 days ago [-]

>sizeof dst

Note that this only works if dst is a stack allocated(in the same function) array and not a char *

masklinn(10000) 5 days ago [-]

strncpy does have its odd and rare use-case, but 100% agree that it is not at all a "fix" for strcpy, it's not designed for that purpose, and unsuited to it, being both unsafe (does not guarantee NUL-termination) and unnecessary costly (fills the destination with NULs).

The strn* category was generally designed for fixed-size NUL-padded content (though not all of them because why be coherent?), the entire item is incorrect, and really makes the entire thing suspicious.

saagarjha(10000) 4 days ago [-]

snprintf is pretty slow, partly because it returns things people typically don't want.

tmewett(10000) 5 days ago [-]

Ah, that was one of my less considered additions - thank you for the feedback!

manv1(10000) 5 days ago [-]

The reason you use specific sizes for types (int8, int16, uint16, etc) is so you know how to read/write them when you move your data between platforms.

x86 is little endian. ARM apparently can be configured to be either.

In real code there should be readXXX and writeXXX functions that read/write data from disk/network and do the byte swapping in there.

You could also just convert everything to JSON, but you're trading space for complexity/time.

jesse__(10000) 5 days ago [-]

> ARM apparently can be configured to be either.

That's insane!!! TIL

user3939382(10000) 5 days ago [-]

The couple of times I tried to really learn C I ran into the same problems. I started by trying to answer two questions: what are the modern features of C that I need to learn to use it, and what style/rules should I follow.

What I found is a body of several C spec updates that are each defined in reference to previous C spec updates. So I'm supposed to... ? Learn C from the 70s and then study each version update and mentally diff everything to figure out what C is now?

Then in terms of rules and style, unlike when K&R C was published, I couldn't find any authority. Actually what I see is that even programmers who have been writing C for many years frequently debate what practices are correct vs not. You can see it in this very thread. Every language has this, but I've seen it much more with C than other languages.

Actually learning the material for me is hard when I can't even get a firm handle on what it is I'm supposed to learn.

tmewett(10000) 5 days ago [-]

Completely agree - that's exactly why I wrote this. Some of things I wrote even seem super obvious but with no real authority you wonder whether or not this is 'the way' to do things. If you ever want to get some practical experience, we'll help you out if you want to contribute to Brogue :)

Gigachad(10000) 5 days ago [-]

There are probably very few people learning C from scratch now vs just already knowing it from the 70s.

jmclnx(10000) 5 days ago [-]

Nice, slowly 'teaching' someone at work c for use on AIX. I will send him the link. I am not a very good teacher :)

donio(10000) 5 days ago [-]

Curious what the backstory is for this (someone learning C for the first time on AIX in 2022)

spacedcowboy(10000) 5 days ago [-]

' You can't extend structs or do anything really OO-like, but it's a useful pattern to think with'

That's not quite true. If you define 2 structs so that they start the same (eg: both with "int x; int y" in your example), pointers can be passed to functions with either struct type. You can use this to add fields (eg: int z) to structures, and extend a 2d vector into a 3d one...

With a bit of creative thought, and constructive use of pointer-to-functions, you can do quite a bit of OOP stuff in C.

hddqsb(10000) 5 days ago [-]

This technique is used in Berkley sockets: the `sockaddr` type (https://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys...) is the 'base' type for other types such as `sockaddr_ip` (for IPv4). But it's undefined behaviour according to ISO C due to the 'strict aliasing' rule: https://en.cppreference.com/w/c/language/object#Strict_alias...

leni536(10000) 4 days ago [-]

No, this breaks aliasing.

The defined way to do something like this is to have the smaller struct as the first member of the larger one. The first member is guaranteed to have the same address as the outer object.

The common initial sequence trick is guaranteed to work with unions in limited circumstances.

dahfizz(10000) 5 days ago [-]

You can also put function pointers in a struct, which is sometimes useful. The syntax is ugly, but you can do a lot of OO stuff that way too.

icedchai(10000) 4 days ago [-]

The Amiga used this technique extensively throughout its OS.

shultays(10000) 5 days ago [-]

I would be surprised if C guarantees those x and y to have same offsets

Plus you cant cast and dereference one type's pointer to the other. That would be UB

sylware(10000) 5 days ago [-]

C syntax is already too rich and complex.

typedef/enum/(_Generic)/etc should go (fix/cleanup function pointer type declaration). Only sized primitive types (u32/s32,u64/s64,f32/f64 or udw/sdw,uqw/sqw,fdw/fqw...). We would have only 1 loop statement 'loop{}', no switch. I am still thinking about 'anonymous' code blocks for linear-code variable sub-scoping (should be small compile-unit local function I guess). No integer promotion, no implicit cast (except for void* and maybe for literals like rust) with explicit compile-time/runtime casts (not with that horrible c++ syntax). Explicit compile-time/'scoped' runtime constants: currently it is only 'scoped' runtime, with on some optimization passes to detect if the constant would be compile time. extern properly enforced for function plz (aka write proper headers with proper switches), and maybe 'local' instead of 'static' for compile-unit-only functions since all functions in a compile-unit are 'static' anyway like global variables.

'Non-standard': ban attributes like binary format visibility (let the linker handle the fine details), packed (if your struct is packed and not compatible with the C struct, it should be byte defined, with proper pointer casts), etc.

Fix the preprocessor variable argument macro for good (now it is a mess because of gcc way and c++ ISO way, I guess we will stick to gcc way).

With the preprocessor and some rigorous coding, we should be able to approximate that with a 'classic' C compiler, since we mostly remove stuff. Was told that many 'classic' C compiler could output optional warnings about things like implicit casts and integer promotions.

In theory, this should help writting a naive 'C-' compiler much more easily than a 'classic' C compiler and foster real life alternatives.

I am sure stuff I said are plain broken as I did not write a C compiler.

I wonder how far rust is from this 'C-' as it is expected to have a much less rich and complex, but more constraint, syntax than C.

RustLove(10000) 5 days ago [-]

There's a lot in 'C', but I don't see anything in it that really isn't needed to serve its modern purpose.

In 2022 'C' is used as a portable assembly language. When you really need to control where and how memory is allocated, and represent data structures used directly by the hardware in a high-level language.

matvore(10000) 5 days ago [-]

I sympathize a bit with the dislike of enum. We could also probably get away with replacing typedef's with either #define (for simple types) or one-field structs (for arrays and function pointers; the latter usually need a void* to go along with them anyway).

There are reasonable low-level optimizations you can do that switch is needed for. You can have cases that start or end around blocks in non-hierarchical ways. This makes it similar to a computed goto.

einpoklum(10000) 5 days ago [-]

Half of the items the poster lists as 'wish I'd known' are things which were literally part of the curriculum when I taught C at my alma mater (Technion IIT, Haifa). And then, some things are better not know - like cppreference, which would just confuse you with a lot of C++, which is interesting, but not relevant. Also, there are some nitpicks to make, like about array decay etc.

More generally:

We can't expect to know a lot of context before learning anything. You have to learn things in _some_ order. It's better to know C when you study PDP assembly, and at the same time it's better to already know PDP assembly when you study C - as you have insight into the motivation of what's possible via the language syntax itself. Same thing for Calculus and Topology: The latter course helped me understand and generalize a lot of what we had done in the former, but without the former, I wouldn't have had proper motivation for the latter, which might have seemed like useless sophistry.

tmewett(10000) 5 days ago [-]

I didn't learn computer science at college/university - only via the internet, and C is quite different to learn that way vs other languages. That was my main motivation for writing :)

vmilner(10000) 5 days ago [-]

Is

  void main()
 Still non-standard in C?
zabzonk(10000) 5 days ago [-]

yes

fastaguy88(10000) 5 days ago [-]

I find this article very strange, perhaps because I started using 'c' so long ago. To the bullet points:

(1) In general, 'c' is always 'c' at the command line, regardless of the platform.

(2) yes, there are options and build tools, but cc my_program.c -o my_program works fine. I have a very hard time figuring out how to compile/run java.

(3) hard to see how this has anything to do with 'C', vs any other compiled language.

(4) so?? I would think I would be more concerned about how to use 'c' for my problem, without worrying about how to use 'c' to solve some other problem. It is hard for me to understand why a language that can do many things is more problematic than a language that only does a few things.

My sense is that reading this article makes things harder, not easier. Most people do not care whether an int is 32 or 64 bits.

I won't argue that copying things (that are not ints or floats) needs to be understood, but many popular languages (specifically python) have the same problem. Understanding the difference between a reference and a value is important for most languages.

There are different schools of thought -- those that can imagine lots of issues after reading the documentation, vs those that simply try writing code and start exploring edge cases when something breaks. I learn faster by trying things, and rarely encounter the edge-case issues.

msla(10000) 5 days ago [-]

Now I need to make a C dialect with better string handling built-in and call it 'C'

jklehm(10000) 5 days ago [-]

It's largely the same, aside perhaps from not shipping with the java compiler by default:

  * install java (includes compiler/runtime): `sudo apt install default-jre`
  * compile: `javac MyProgram.java`
  * run:     `java MyProgram`
_ache_(10000) 5 days ago [-]

> You can cast T to const T, but not vice versa.

Humm, actually, you can. But guess what ? Modification of the underlying data is an UB !

pif(10000) 5 days ago [-]

Which is the only possible logical consequence of modifying something that others asked you not.

const_cast is there for buggy (or legacy) libraries where const was not specified explicitly, but is implicit in the behavior.

LegionMammal978(10000) 5 days ago [-]

It's only UB if the original variable is declared with a const type. If you convert a modifiable T * into a const T *, then cast it back into a T *, then you can modify it without any issues.

quietbritishjim(10000) 5 days ago [-]

> Declaring a variable or parameter of type T as const T means, roughly, that the variable cannot be modified.

I would add '... cannot be modified through that pointer'. (Yes, in fairness, they did say 'roughly'.) For example consider the following:

    void foo(int* x, const int* y)
    {
        printf('y before: %d\n', *y);
        *x = 3;
        printf('y after: %d\n', *y);
    }
This will print two different values if you have `int i = 1` and you call `foo(&i, &i)`. This is the classic C aliasing rule. The C standard guarantees that this works even under aggresive optimisation (in fact certain optimisations are prevented by this rule), whereas the analogous Fortrain wouldn't be guaranteed to work.
pm2222(10000) 5 days ago [-]

*y in both printf, right?

vardump(10000) 5 days ago [-]

A small detail: you probably meant

  printf('y before: %d\n', *y);
pafje(10000) 5 days ago [-]

You already know this, but I would add that under strict aliasing rules, this is only valid because x and y point to the same type.

The most common example is when y is float* and someone tries to access its bitwise representation via an int*.

(Please correct me if I'm wrong)

https://gist.github.com/shafik/848ae25ee209f698763cffee272a5...

pratk(10000) 4 days ago [-]

One C idiom that I found pretty useful is Xmacros [1]. For instance, it's used extensively in GCC code-base (.def files).

[1] https://www.drdobbs.com/the-new-c-x-macros/184401387

rramadass(10000) 4 days ago [-]

Very Neat! Didn't know of this.

Have any more of 'C Idioms' collection?

nottorp(10000) 5 days ago [-]

C is easier if you first learn assembler for an architecture or two :)

jorgeleo(10000) 5 days ago [-]

Agree. Not only C, but almost any other language

wheelerof4te(10000) 5 days ago [-]

Everything is easier once you torture yourself with trying to learn Assembler :)

krylon(10000) 5 days ago [-]

When I first learned C - which also was my first contact with programming at all - I did not understand how pointers work, and the book I was using was not helpful at all in this department. I only 'got' pointers like three or four years later, fortunately programming was still a hobby at that point.

Funnily when I felt confident enough to tell other people about this, several immediate started laughing and told me what a relief it was to hear they weren't the only ones with that experience.

Ah, fun times.

EDIT: One book I found invaluable when getting serious about C was 'The New C Standard: A Cultural and Economic Commentary' by Derek Jones (http://knosof.co.uk/cbook). You can read it for free because the book ended up being to long for the publisher's printing presses or something like that. It's basically a sentence-by-sentence annotated version of the C standard (C99 only, though) that tries to explain what the respective sentence means to C programmers and compiler writers and how other languages (mostly C++) deal with the issue at hand, but also how this impacts the work of someone developing coding guidelines for large teams of programmers (which was how the author made a living at the time, possibly still is). It's more than 1500 pages and a very dense read, but it is incredibly fine-grained and in-depth. Definitely not suitable for people who are just learning C, but if you have read 'Expert C Programming: Deep C Secrets' and found it too shallow and whimsical, this book was written for you.

Sohcahtoa82(10000) 5 days ago [-]

One thing that helped me understand pointers was understanding that a pointer is just a memory address.

When I was still a noob programmer, my instructor merely stuck to words like 'indirection' and 'dereferencing' which are all fine and dandy, but learning that a pointer is just a memory address instantly made it click.

Pointers are a $1000 topic for a $5 concept.

harry8(10000) 5 days ago [-]

'The C Puzzle Book' is the thing I recommend to anyone who knows they want to have a good, working understanding of how to use pointers programming in C.

Many years ago I did the exercises on the bus in my head, then checking the answers to see what I got wrong and why over the space of a week or so. It's a really, really good resource for anyone learning C. It seemed to work for several first year students who were struggling with C in my tutorials as well and they did great. Can't recommend it highly enough to students and the approach to anyone tempted to write an intro C programming text.

nayuki(10000) 5 days ago [-]

From over a decade ago, I really enjoyed this clay animation on C pointers: https://www.youtube.com/watch?v=5VnDaHBi8dM , http://cslibrary.stanford.edu/104/

chasil(10000) 5 days ago [-]

Having basic experience in any assembly language makes pointers far more clear.

'Addressing modes,' where a register and some constant are used to calculate the source or target of a memory operation, make the equivalence of a[b]==*(a+b) much more obvious.

I also wonder about the author's claims that a char is almost always 8 bits. The first SMP machine that ran Research UNIX was a 36-bit UNIVAC. I think it was ASCII, but the OS2200/EXEC8 SMP matured in 1964, so this was an old architecture at the time of the port.

'Any configuration supplied by Sperry, including multiprocessor ones, can run the UNIX system.'

https://www.bell-labs.com/usr/dmr/www/otherports/newp.pdf

colonCapitalDee(10000) 5 days ago [-]

My experience with pointers was the inverse of yours. My first programming language was Java, and I spent many hours puzzling out reference types (and how they differed from primitive types). I only managed to understand references after somebody explained them as memory addresses (e.g. the underlying pointer implementation). When I later learned C, I found pointers to be delightfully straightforward. Unlike references in Java, pointers are totally upfront about what they really are!

mumblemumble(10000) 5 days ago [-]

I would highly recommend the video game Human Resource Machine for getting a really good understanding of how pointers work.

It's more generally about introducing assembly language programming (sort of) in gradual steps, so you'll need to play through a fair chunk of the game before you get to pointers. But by the time you get to them, they will seem like the most obvious thing in the world. You might even have spent the preceding few levels wishing you had them.

andrewla(10000) 5 days ago [-]

Pointers are by far the most insidious thing about C. The problem is that nobody who groks pointers can understand why they had trouble understanding them in the first place.

Once you understand, it seems so obvious that you cannot imagine not understanding what a pointer is, but at the beginning, trying to figure out why the compiler won't let you assign a pointer to an array, like `char str[256]; str = 'asdf'`, is maddening.

One thing I think would benefit many is if we considered 'arrays' in C to be an advanced topic, and focused on pointers only; treating 'malloc' as a magical function until the understanding of pointers and indexing is so firmly internalized that you can just add on arrays to that knowledge. Learning arrays first and pointers second is going to break your brain because they share so much syntax, but arrays are fundamentally a very limited type.

nayuki(10000) 5 days ago [-]

> C has no environment which smooths out platform or OS differences

Not true - C has little environment, not no environment. For example, fopen('/path/file.txt', 'r') is the same on Linux and Windows. For example, uint32_t is guaranteed to be 32 bits wide, unlike plain int.

> Each source file is compiled to a .o object file

Is this a convention that compilers follow, or are intermediate object files required by the C standard? Does the standard say much at all about intermediate and final binary code?

> static

This keyword is funny because in a global scope, it reduces the scope of the variable. But in a function scope, it increases the scope of the variable.

> Integers are very cursed in C. Writing correct code takes some care

Yes they very much are. https://www.nayuki.io/page/summary-of-c-cpp-integer-rules

trap_goes_hot(10000) 5 days ago [-]

On Windows you can directly access UNC paths (without mounting) with fopen. You can't do this on POSIX platforms. Also, not all API boundaries are fixed width so you're going to be exposed to the ugliness of variable width types.

I think the article is correct that one must be aware of the platform and the OS when writing C code.

LegionMammal978(10000) 5 days ago [-]

> Is this a convention that compilers follow, or are intermediate object files required by the C standard? Does the standard say much at all about intermediate and final binary code?

The standard only says that the implementation must preprocess, translate, and link the several 'preprocessing translation units' to create the final program. It doesn't say anything about how the translation units are stored on the system.

> This keyword is funny because in a global scope, it reduces the scope of the variable. But in a function scope, it increases the scope of the variable.

Not quite: in a global scope, it gives the variable internal linkage, so that other translation units can use the same name to refer to their own variables. In a block scope, it gives the variable static storage duration, but it doesn't give it any linkage. In particular, it doesn't let the program refer to the variable outside its block.

manv1(10000) 5 days ago [-]

fopen will/should fail on windows with the unix path syntax.

The reason it's indeterminate is because some stdc lib vendors will do path translation on Windows, some won't. I believe cygwin does (because it's by definition a unix-on-windows), but I'm pretty sure the normal stdclib vendors on windows do not.

I'm almost positive that MacOS (before MacOS X) will fail with unix path separators, since path separators are ':' not '/'.

pjc50(10000) 5 days ago [-]

The elderly 'C Traps and Pitfalls' http://www.literateprogramming.com/ctraps.pdf was an extremely helpful book when I was learning C, although I don't know how well it's held up.

nayuki(10000) 5 days ago [-]

I skimmed it and all the points made still appear to be true today. Namely, all the C language rules and the human psychological pitfalls.





Historical Discussions: Firefox Translations: Translate websites in your browser without using the cloud (November 29, 2022: 728 points)
Firefox Translations by Mozilla Firefox (June 03, 2022: 3 points)

(728) Firefox Translations: Translate websites in your browser without using the cloud

728 points 4 days ago by PaulKeeble in 10000th position

addons.mozilla.org | | comments | anchor

Firefox Translations provides automated translation of web content. Unlike cloud-based alternatives, translation is done locally, on the client-side, so that the text being translated does not leave your machine.

Languages supported:

Production Spanish Estonian English German Czech Bulgarian Portuguese Italian French Polish

Development Russian Persian (Farsi) Icelandic Norwegian Nynorsk Norwegian Bokmål Ukrainian Dutch

Firefox Translations was developed with The Bergamot Project Consortium, coordinated by the University of Edinburgh with partners Charles University in Prague, the University of Sheffield, University of Tartu, and Mozilla. This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825303.




All Comments: [-] | anchor

AdmiralAsshat(10000) 4 days ago [-]

I've just installed it, and I'm impressed so far. I've only run it against some sample German Wikipedia articles (https://de.wikipedia.org/wiki/Clan_of_Xymox), but it produces surprisingly readable text. I also particularly like the 'highlight potential errors' option to flag stuff that even the translation service thinks might be a bit off.

It's not nearly as speedy as Google Translate, but I'll take that happily if it means keeping it local.

jelmervdl(10000) 4 days ago [-]

To be fair, Wikipedia basically appears in its entirety in the training data. It's a good test to see whether the translation model and all the plumbing works well, but not whether the model generalises well.

skorpeon87(10000) 4 days ago [-]

It seems to work brilliantly with Russian. Less than 5 seconds to translate a medium length wikipedia article on my computer. This exceeds my hopes and expectations.

eager_noob(10000) 3 days ago [-]

It really is surprisingly readable. Does excellent work on german, spanish & italian f1 news articles.

joveian(10000) 3 days ago [-]

My experience with occasional use over the past four months or so is that newspapers, business documents, and other formal documents often appear to have essentially perfect translation, at least in terms of sounding plausable without understanding the language translated. For more casual stuff like twitter posts or song lyrics it makes plenty of obvious errors, although of course some of that stuff makes little sense as written. It seems to have different issues than Google Translate, although not much if any worse overall from my limited experience.

It sounds like after Bergamot funding ended there were some communication issues and the Translate Locally group that was working with the Firefox Translate group stopped working together and now have their own extension, as mentioned in another comment:

https://news.ycombinator.com/item?id=33795141

That one can be installed outside the browser and would likely give better performance, although I haven't tried that yet.

The other interesting open source translation software I've seen is Apertium:

https://github.com/apertium

I don't think they have a browser extension unfortunately but it is an entirely rule based translation rather than AI models. I haven't tried this one yet either but hope to soon. I did try their web interface a few times:

https://www.apertium.org

rstuart4133(10000) 3 days ago [-]

I used to use Android Chrome when overseas because auto translating of web pages is indispensable. It was a major concession for me, because I loath Google preventing ad blocking on Android Chrome. Still, it was less than perfect - constant popups asking you about the quality of the translation. The delay in sending it to Google wasn't noticeable when you had a good connection, but on the edge of mobile coverage it made web browsing even more miserable.

Then this came along. All those nits are gone. Personally I find the translations easier to understand than Google's. (When you're overseas easier to understand trumps grammaticality perfect every time.) I'm not a fan of the banner at the top - the could move it to a tool bar icon like Ublock Origin does, but apart from that - it's damned good.

Now we need a replacement for Google Lens. For all it's flaws, Lens seems near magical to me.

friend_and_foe(10000) 4 days ago [-]

Is there source code available for this?

GekkePrutser(10000) 4 days ago [-]

They just changed something though that breaks it for me.

Previously there was a manual translate button. Now you can only set up automatic translation with an exception list.

This is really annoying for me, I'm learning Spanish and I like to switch between the original and the translation. Now that option is suddenly gone.

skorpeon87(10000) 4 days ago [-]

Did you accidentally press the 'Translate this tab automatically as you browse' button? Maybe try in a new tab.

andrenatalbr(10000) 3 days ago [-]

This is definitely not the right behavior. Could you please file an issue with steps to reproduce or mail the team (address is in the addon page)

xeonmc(10000) 3 days ago [-]

So, do we call them 'FFT' then?

plaguepilled(10000) 3 days ago [-]

LOL yes, if their market share increases after rapid app improvements we could call it a Fast Firefox Transformation

_def(10000) 3 days ago [-]

This is nice for translating whole sites. As a non-native english speaker, what I often want is to translate single words. I hope this add-on can provide this feature as well in the future.

Vinnl(10000) 3 days ago [-]

You can do that using the toolbar button since a recent version.

marcodiego(10000) 4 days ago [-]

What is a good simple to use open-source app or lib that is able to do translations locally?

jelmervdl(10000) 4 days ago [-]

https://translatelocally.com/

The website isn't pretty but it's the same tech as the browser extension. It came out of the same research project.

But because its optimised native code, and not limited by single threaded wasm, waaay faster!

Edit disclosure: I worked on that project, and this software.

megablast(10000) 3 days ago [-]

> A CPU that supports SSE4.1 extensions is required for this addon to function properly. If it doesn't, an error will be displayed when the translation is being started.

Obviously. Replacing text is a huge CPU intensive task.

jelmervdl(10000) 3 days ago [-]

It is running a full transformer type neural network on your cpu, it needs all the speed it can get.

It's because the translation engine requires at least SSSE3.1 instructions[1]. These are translated to wasm SIMD instructions[2] which are only enabled by browsers if the underlying hardware is there to execute these at least somewhat efficient.

[1] https://github.com/browsermt/bergamot-translator/issues/418

[2] https://emscripten.org/docs/porting/simd.html

m_eiman(10000) 3 days ago [-]

FWIW it works on M1 Macs, I suppose they same something equivalent to SSE4.1.

727564797069706(10000) 3 days ago [-]

Translation is pretty much the only reason I still use Chrome. Great to see that Firefox tries to solve this blocker!

Downside is that the language I mainly need it for is not in the list yet.

aqme28(10000) 3 days ago [-]

That shouldn't have been a blocker. I've been using this extension for a while that does the same thing as Chrome https://addons.mozilla.org/en-US/firefox/addon/traduzir-pagi...

Mick-Jogger(10000) 2 days ago [-]

Does anyone have an alternative for Android phones? Missing translations is the only thing that annoys me daily-driving Firefox on Android.

m4rtink(10000) 2 days ago [-]

Yeah, I guess its still a no-Go due to the botched addon support in the regular Android Firefox build ? But maybe in the Nigtly/Beta using that annoying addon list trick ?

https://www.androidpolice.com/firefox-beta-107-android-insta...

eye_street(10000) 4 days ago [-]

I'd like this to be able to translate just a selection, like Safari can do.

Zuiii(10000) 3 days ago [-]

Please open a github issue and let them know.

sendfoods(10000) 3 days ago [-]

> Translate websites

Just installed it, but how does it translate 'websites'? The description states that is translates text input, entered manually. What am I missing?

trabant00(10000) 3 days ago [-]

You have a new button inside the address bar that translate the entire page.

voxelghost(10000) 4 days ago [-]

Any chance of support for Japanese in the nearish future?

andrenatalbr(10000) 2 days ago [-]

That's in the roadmap, yes.

hutzlibu(10000) 3 days ago [-]

Sadly not for firefox mobile. Is there a fundamental difference between FF desktop extensions and mobile extensions?

I would be fine using it, with ugly UI, not optimized for mobile ..

ciroduran(10000) 3 days ago [-]

if you're on iOS you're out of luck, Apple forces to use the same underlying browser, so there are no extensions there

Animats(10000) 3 days ago [-]

That's useful.

Can it be extended to Asian languages? Most of the supported languages have common roots.

vinay427(10000) 3 days ago [-]

Yes, all of the current and in-development languages listed have common roots except for Estonian.

beagle3(10000) 3 days ago [-]

Recent version also added a toolbar button with a box you can just copy text into and get a translation, which is awesome.

The first release translated endless twitter scroll inline perfectly. Unfortunately, this stopped working for me - either a Firefox update or a translations update - so I actually have to use this box often.

andrenatalbr(10000) 3 days ago [-]

> The first release translated endless twitter scroll inline perfectly.

This is the right behavior and it's still working for me. Could you please file an issue to: https://github.com/mozilla/firefox-translations/issues with your STRs?

camgunz(10000) 3 days ago [-]

I've been using this for a few months and I love it. I will take this opportunity to gripe though and say that it doesn't handle DOM changes, e.g. if you're going through a form and you click an expander, the contents of that expander won't be translated if it was dynamically inserted into the DOM. This is more or less any AJAX request, or React/Vue component with its own data calls, etc. As a result, I often find myself falling back to Chrome if I need translations (recently moved abroad).

Also, it would be great if the models were cached. If I don't click 'translate this tab automatically as I browse' I'll download the whole model on every page load. Not great if you're tethered to your phone :(

andrenatalbr(10000) 3 days ago [-]

DOM changes should be handled and it should be working still. Could you please file a bug https://github.com/mozilla/firefox-translations/issues with the proper steps to reproduce, please?

jelmervdl(10000) 3 days ago [-]

That must be a bug. There is a lot of code in the extension specifically to make it track DOM changes and push translations for those as they happen.

React pages are especially tricky to translate as the framework does not take kindly on external JavaScript swapping out text nodes. There's code in the extension to work around that.

jonesdc(10000) 3 days ago [-]

Can anyone recommend a good dictionary addon for Firefox?

Quenz(10000) 2 days ago [-]

Simple, recommended by Firefox [0], has light/dark theme [1], more customizable and pretty but not open source [2]. I've not used these extensively.

  [0] https://addons.mozilla.org/addon/dictionary-anyvhere/
  [1] https://addons.mozilla.org/addon/english-popup-dictionary/
  [2] https://addons.mozilla.org/addon/lumetrium-definer/
devit(10000) 4 days ago [-]

Wow, finally!

The lack of working translation was the biggest reason to use Chromium instead of Firefox.

And being local is great, and the quality even seems quite good.

pessimizer(10000) 4 days ago [-]

Translation by Google/Yandex already worked smoothly in Firefox. The reason this post is neat is because the translation is done offline.

Regular cloud translation: https://github.com/FilipePS/Traduzir-paginas-web

PaulKeeble(10000) 4 days ago [-]

Translates websites in your browser without using the cloud, a different approach to that of google, should protect privacy better. It doesn't handle mixed pages well unfortunately (so the other languages in my RSS feed are not being detected) and I can't force it to translate the pages but otherwise it seems to do a similar job.

Jap2-0(10000) 4 days ago [-]

> I can't force it to translate the pages

I see both an icon on the right end of the address bar I can click to manually translate pages, and the ability to click on the addon from the extensions list in the toolbar to open a popup to translate arbitrary text.

askhan(10000) 3 days ago [-]

Also Simple Translate, which uses Google so presumably less or no privacy, but it does handle mixed pages by translating any selected text

https://addons.mozilla.org/en-US/firefox/addon/simple-transl...

Great for a web-based RSS reader with feeds in multiple languages onscreen...

tomxor(10000) 4 days ago [-]

This is the future.

So many things can be unclouded, and provide a better user experience for it (which is how we win).

autoexec(10000) 4 days ago [-]

I wish I had your optimism. I suspect the opposite though.

A better user experience was never the goal of cloud services. Providing translation and other services in the cloud gives companies massive amounts of data about you that they can leverage to their advantage and gives them opportunities to control and shape what we're allowed to do or see. As long as that stays true there will always be a push for users to give up more control and become more dependent on third parties.

I want to live in a future where more things are done locally and independently, but things are headed in the other direction and there's a lot of money and power behind preventing the pendulum from swinging back. I'll do what I can to fight the trend though, and if this add-on really works as advertised I'll gladly use it.

jelmervdl(10000) 4 days ago [-]

I think doing machine translation in the cloud makes more sense commercially (easier to monetise) and is a lot less challenging (better hardware, model sizes less of an issue, fewer worries about exposing proprietary code or models).

But I agree that this local experience should be the future. It's good to have the control and independence.

vanilla_nut(10000) 4 days ago [-]

I use Librewolf as my main browser, instead of Firefox. I understand that Librewolf is a fork of FF, so it's not necessarily 100% API compatible with FF addons. But especially without network access, I wonder why Firefox Translate won't work in Librewolf. Maybe Librewolf disables some ML/AI features?

It's a shame; I'd like to use FF Translate instead of Google Translate. But Mozilla's telemetry and frequent style changes that break my Userchrome.css styling are a bit of a dealbreaker.

phil294(10000) 3 days ago [-]

I'm also using Librewolf, and I have no issues using Firefox translate. I must have changed several settings over time, including about:config flags, but apparently it definitely is possible.

Apart from that, I assume it's quite difficult for anyone to help you based on the error description 'doesn't work'.

jelmervdl(10000) 4 days ago [-]

It might be because Firefox Translate hooks into some private extension apis (for browser chrome and telemetry) that you can only use when your extension is signed by Mozilla itself. Maybe those apis have been removed from Librewolf, or it doesn't accept Mozilla's certificate as an exception.

I maintain a forked version of the Firefox extension[^1] that doesn't use any private apis (or telemetry). It is slightly different[^2], but uses the same libraries.

Disclosure: I work for one of the project partners and contributed to the Mozilla extension.

[^1]: https://addons.mozilla.org/en-GB/firefox/addon/translateloca...

[^2]: https://github.com/jelmervdl/translatelocally-web-ext#differ...

okasaki(10000) 4 days ago [-]

You can block telemetry with a hosts file[0], and update less often by running ESR.

0: https://github.com/MrRawes/firefox-hosts

andrenatalbr(10000) 2 days ago [-]

Telemetry can be disabled on: about:addons -> Firefox Translations -> Preferences and then disabling 'Report high-level user interaction' and 'Report errors'

sand500(10000) 4 days ago [-]

Amazing this can all fit in 3MB

> called Project Bergamot. The ultimate goal of this consortium was to build a set of neural machine translation tools that would enable Mozilla to develop a website translation add-on that operates locally, i.e. the engines, language models and in-page translation algorithms would need to reside and be executed entirely in the user's computer, so none of the data would be sent to the cloud, making it entirely private.

> In addition to that, two novel features needed to be introduced. The first was translation of forms, to allow users to input text in their own language that is dynamically translated on-the-fly to the page's language. The second feature was quality estimation of the translations where low confidence translations should be automatically highlighted on the page, in order to notify the user of potential errors.

https://blog.mozilla.org/en/mozilla/local-translation-add-on...

stlee42(10000) 4 days ago [-]

it doesn't fit in 3MB.

you can see the model sizes here: https://gist.github.com/jelmervdl/1a48816e4c3643ff5d9e1fd682...

they are like 15MB per language pair each way

stlee42(10000) 4 days ago [-]

previous discussion here: https://news.ycombinator.com/item?id=31596888

as noted in [0], the models are hosted on storage.googleapis.com and downloaded on demand. this still seems to be the case [1]

0: https://news.ycombinator.com/item?id=31604095

1: https://github.com/mozilla/firefox-translations/blob/main/ex...

GekkePrutser(10000) 4 days ago [-]

Weird. Why give your biggest competitor insight in your usage numbers??

autoexec(10000) 4 days ago [-]

I guess this won't work for me then. I'm not going to unblock googleapis.com for it.

ezconnect(10000) 4 days ago [-]

Too bad there's no Japanese or Korean translation :(

KVFinn(10000) 4 days ago [-]

>Too bad there's no Japanese or Korean translation :(

It's interesting that they have Icelandic, a language spoken by 350,000 people. I'm often surprised seeing Icelandic has a full audio dub on a Disney or Netflix movies, above languages spoken by 100X or more speakers. That can't be an economic decision.

Anyone know why Icelandic punches above its own weight? Or is it because it has so few speakers that people do dubs to help preserve it?

6yyyyyy(10000) 4 days ago [-]

Which is especially surprising when they have a Japanese character in the logo.

soganess(10000) 4 days ago [-]

I'm a long-time firefox user and very vocal hater. That said, I've had less and less to hate with each update. I used to hate ff mobile(too slow). I used to hate the browser on Linux(no Wayland, no hardware video, weak multi-process). But all that has been fixed. Plus, cookie isolation, auto cookie banner management, HTTPS preferred, and mobile extensions. Not to mention the promise to support content filtering extensions.

Add to that local translation! FF is the best it has ever been and somehow it is losing user share faster than ever. I used to get all the hate, but now at days... I'm just not sure. I'd love to know why.

autoexec(10000) 4 days ago [-]

> Add to that local translation! FF is the best it has ever been and somehow it is losing user share faster than ever. I used to get all the hate, but now at days... I'm just not sure. I'd love to know why.

I'm also a long time fan of firefox, but it does seem like there's always something I have to disable or fix with every major update.

Firefox has always claimed to be very pro-privacy, but they don't always live up to that with their actions (From a privacy perspective perhaps the biggest slap in the face was Pocket). Even here 'local translation' means that Firefox needs to repeatedly ping Google's servers in order to download the models. Still better than letting Google translate the contents of a page, but far from ideal from a privacy standpoint.

As it stands I have modifications to over 100 settings in about:config that have to be made with each fresh install of firefox to get it locked down properly. Firefox is still the best browser out there because it gives you the ability to disable all that stuff, but its still a pain.

User share is always going to suffer because for most people their phones come with something else already, their PCs and tablets do too, and they don't know any better. They don't see or understand the problems with using chrome so why look for a browser that gives them better solutions?

I use firefox because I care about my security and privacy. If I didn't care about that I'd use Chrome or whatever Microsoft is shipping these days too because I know sites will make sure their stuff works in those browsers. All the hate I have for firefox comes from a place of love. I care about firefox. I depend on it. I want firefox to be better. Often that means I'm calling them out on their bullshit, bitching about the extra work they're giving me after updates, or just mentioning how they could have done better.

Firefox users complain about firefox because if they didn't care enough to complain they'd just be using chrome.

Barrin92(10000) 3 days ago [-]

>I'd love to know why.

well as an individual data point, I recently threw in the hat because both my regional bank's website and my health insurance page stopped working at all on Firefox. Not really something I can do without and it seems to be getting worse for companies with smaller web presences.

geokon(10000) 3 days ago [-]

.. but the linked translation extension doesn't work on Firefox mobile.. (Fennec from FDroid)

Shorel(10000) 3 days ago [-]

It's because you were too noisy.

Basically, reputation builds at glacial speed, and it is rapidly destroyed.

Complaints and reports of bad user experience linger for years, if not decades, and reversing consumer trends requires humongous good will.

They say there's only one chance for a good impression. But there's also basically no chance at all for a second bad impression.





Historical Discussions: The last three years of my work will be permanently abandoned (November 30, 2022: 705 points)
A Long Expected Update (November 30, 2022: 2 points)

(705) The last three years of my work will be permanently abandoned

705 points 3 days ago by chubot in 10000th position

ericlippert.com | Estimated reading time – 5 minutes | comments | anchor

It's been almost two years since my last update here. A lot has happened. I hope you all are continuing to weather the ongoing multiple global pandemics and other anthropogenic crises.

Apologies that this is so long; I didn't have time to make it shorter.

Obviously blogs do not come with a service level agreement, but some explanation is in order for the long gap. It's pretty simple.

  • Facebook actively discourages people to blog about their work.
  • Working from home during the covid pandemic was enervating.
  • After thinking about programming languages for many hours a week for over 25 years, I simply didn't have the energy and enthusiasm to write much on the subject.

Blogging was an easy thing to drop in favor of pursuits that got me away from sitting in front of a screen in my home office. I've been spending my leisure time in the last couple years working on improving my nature photography skills and learning to scuba dive. Turns out you cannot catch covid when below 15 metres of seawater. And there are weird slugs in the Puget Sound!


Photos of the author and a golden dirona nudibranch by Amber, who convinced me to take up diving.


Today is the tenth anniversary of moving my blog to ericlippert.com on my last day at Microsoft, the fiftieth anniversary of my birth, and my last day at Facebook-now-Meta.

My team — Probabilistic Programming Languages — and indeed entire "Probability" division were laid off a couple weeks ago; the last three years of my work will be permanently abandoned.

The mission of the Probability division was to create small teams that applied the latest academic research to real-world at-scale problems, in order to improve other groups' decision-making and lower their costs. New sub-teams were constantly formed; if they didn't show results quickly then they were failed-fast; if they did show results then they were reorganized into whatever division they could most effectively lower costs.

We were very successful at this. The PPL team in particular was at the point where we were regularly putting models into production that on net reduced costs by millions of dollars a year over the cost of the work. We were almost ready to be spun off.

We foolishly thought that we would naturally be protected from any layoffs, being a team that reduced costs of any team we partnered with. In retrospect, that was a little naive. A team that reduces costs of other teams is not on anyone's critical path.

The whole Probability division was laid off as a cost-cutting measure. I have no explanation for how this was justified and I note that if the company were actually serious about cost-cutting, they would have grown our team, not destroyed it.

Speaking of cutting costs, the company is still pouring multiple billions of dollars into vaporware called "the metaverse". News flash: no one wants to wear VR goggles to spend any time in a digital heaven where the role of God is played by Mark Zuckerberg and you can do anything you can imagine, including "work" and "shop".

Ugh.

I would be happy to be shown to be wrong, wrong, wrong. Maybe there is a useful, engaging, fun, just, equitable, democratic, sustainable, novel VR experience where the avatars have legs, but Meta is $20 billion in and aside from the legs I don't see any evidence that any of the above is forthcoming.

Yes, I am a little vexed.


I have a great many people to thank for my time at Facebook: Erik Meijer for recruiting me and finding seven years worth of interesting problems for me to dig into. Peter Hallam, with whom I have now worked with on three compiler teams at three companies, for encouraging me to take that offer. Walid Taha, Michael Tingley, John Myles White, Joe Pamer and Satish Chandra for their leadership and mentorship. And to many, many coworkers too numerous to mention here. The quality of the people I worked with at Facebook was amazing. Everyone was kind, smart, dedicated, thoughtful, generous with their time and knowledge, and a genuine pleasure to work with. I learned so much from all of them. Leaving those teammates is the hardest part.


Lots of people have asked how they can help me and my team. I am so grateful and appreciative. Friends, colleagues, strangers on Twitter, just about everyone has been sympathetic and helpful. Most of my team has found other positions and I am hopeful that the rest will soon.

I am not looking for another position at this time.

Why's that?

I know I don't look it, but I'm beginning to feel it in my heart. I feel thin, sort of stretched, like butter scraped over too much bread. I need a holiday. A very long holiday. And I don't expect I shall return. In fact I mean not to.

I am very fortunate to have spent the pandemic thus far working safely from home, for a supportive team and for excellent pay. But after >26 years of thinking about programming languages for corporations, and the last three years of my work being thrown away, I need a good long corporate detox before I go looking again.


Coming up next on FAIC:

The work we did on Bean Machine, our embedded Python DSL for Bayesian inference, is quite interesting. In coming episodes I'll explain what it is, how it works, and what we learned. No one else is ever going to do this post-mortem analysis, so I might as well!

Like this:

Like Loading...




All Comments: [-] | anchor

moloch-hai(10000) 3 days ago [-]

To Management, you are either in a Cost Center or a Profit Center. In all advertising-supported monstrosities, only adtech and sales are profit centers. Literally everything else is a cost center. Everyone at Facebook and Google is in a Cost Center if they are not directly involved in landing advertising accounts, presenting ads, or billing for ads.

Never look for work in a Cost Center.

Come hard times, Cost Centers are cut first. Not because it is good for the business, but because cutting payroll impresses Wall Street, inflating stock valuation. To Wall Street, layoffs mean you are serious.

gknoy(10000) 3 days ago [-]

> billing for ads is in a Cost Center.

I've only briefly worked closely with a billing team, but my impression was always that billing is seen less as a cost center, but more as a critical 'without this team we get no money' team, which seems closer to a profit center. I'm not sure how far up the management team that perspective stays true, though.

squokko(10000) 3 days ago [-]

> Speaking of cutting costs, the company is still pouring multiple billions of dollars into vaporware called "the metaverse". News flash: no one wants to wear VR goggles to spend any time in a digital heaven where the role of God is played by Mark Zuckerberg and you can do anything you can imagine, including "work" and "shop".

The hedge fund guys would say the same about your team. I know you are emotional but this is uncalled for.

svnt(10000) 3 days ago [-]

Are... are you attempting to shame-discipline him? You non-seq a hedge fund and then appeal to morality? What a strange little comment.

samiam_iam(10000) 3 days ago [-]

Boohoo

dang(10000) 1 day ago [-]

We've banned this account for posting unsubstantive comments.

If you don't want to be banned, you're welcome to email [email protected] and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.

Kiro(10000) 3 days ago [-]

> News flash: no one wants to wear VR goggles to spend any time in a digital heaven where the role of God is played by Mark Zuckerberg and you can do anything you can imagine, including "work" and "shop".

I'm sure no-one here likes the idea of the Metaverse but from Meta's perspective I definitely think it's the correct bet. This doesn't make it less dystopian but that's exactly the point. If they can pull it off they win. Great video explaining it way better then I can: https://www.youtube.com/watch?v=CqkhjL3WvWQ

WorldMaker(10000) 3 days ago [-]

I haven't watched this particular video yet, but a similar explainer video that I came across this week has me feeling a lot like 'Those who have forgotten Second Life seemed doomed to repeat it (badly)'. I had a bit of a rough moment realizing that statistically most of the engineers at Meta are probably too young to remember Second Life. (And some of the ones that are old enough like Mark Zuckerberg were too busy in other parts of the internet to have learned the realest lessons, which were not technical but sociopolitical.)

wanderingstan(10000) 3 days ago [-]

By writing this post, does the author violate the non-disparagement clause of any severance agreement? Or perhaps they are set financially so can turn down the severance?

grepfru_it(10000) 3 days ago [-]

Or, more likely, op didn't get severance

kalkin(10000) 3 days ago [-]

'Everyone was kind, smart, dedicated, thoughtful, generous with their time and knowledge, and a genuine pleasure to work with.'

I know this is what everyone says after leaving a company if they don't want to burn bridges, and I bet it's even fairly true to Eric's experience. I'm also entirely sympathetic to his frustration with both the immediate experience of being laid off for any reason and the broader large-organization irrationality about costs.

And yet. We're talking about Facebook, a company whose impact on the world is very hard to see as net positive, from teen mental health to national politics. I just really wish it was the industry norm that people who are, in local ways, genuinely thoughtful and kind, and have many employment options, would also think seriously about what their work is ultimately building.

giraffe_lady(10000) 3 days ago [-]

This is something I still find as kind of a culture shock on HN honestly. The idea that pursuing work that is personally interesting and remunerative is at worst neutral is basically an in-built assumption here. There is often pretty intense policing of it if there's a whiff of deviation.

FWIW I don't think it's true either. A lot of engineers can be convinced to work on basically anything if it's 'a hard problem' in the right way. This is bad. We should all consider ourselves responsible for the end results, rather than entitled to the means.

ericlippert(10000) 3 days ago [-]

I'm 100% sincere in that praise of my colleagues.

Many people, myself included, had a lot of concerns about the products the company was building and their effects on the world. When you work on a team whose mission is to help other teams make better decisions at lower cost, the aim is to look at the whole system and improve the whole thing.

Let me give you an example. Most 'this content doesn't belong on FB' decisions are made by ML, but a great many go to human review. Imagine what that job is like. It's emotionally exhausting, it's poorly compensated, burnout is high.

My team had a model in production where we would use Bayesian reasoning to automatically detect when a particular human was likely to have made the correct decision about content classification, and therefore, if two humans disagreed, how to resolve that impasse without getting a third involved. (And in addition we get a lot more information out of the model including bounds on true prevalence of bad content, and so on.)

Does that save the company money? Sure. Millions of dollars a month. (And for the amateur bean counters elsewhere on this page: the data scientist who developed this model is NOT PAID MILLIONS OF DOLLARS A MONTH.) But it also (1) helps keep bad content off of the platform, so users aren't exposed to it, (2) lowers the number of human reviewers who come into contact with it, which is improves their jobs, and (3) frees up budget for whatever improvements need to be made to this whole workflow.

That's just one example; everything that we did was with an eye towards not merely saving the company money, but improving the ability to make good decisions about the products.

chubot(10000) 3 days ago [-]

(story submitter here) This is true and should be acknowledged. To be honest, I submitted it because like many others I learned something from the author's blog, and followed it over the years

I also submitted it under a title involving 'probabilistic programming languages', thinking 'it's cool that you can get paid to work on such a thing'

Though it's also true that I don't use Facebook and would probably not want it to be optimized any more than it is :-/

I also worked in Big tech and there is a lot of genuinely useful knowledge and practice 'locked up' there -- i.e. knowledge that is not in open source code. They have assembled a lot of expertise

The P programming language (for concurrency / state machines) is another example of that -- it's used inside Microsoft Windows and AWS, and is beyond the state of the art elsewhere.

I don't know what to do about that, but it should be acknowledged. We should also acknowledge that some big tech products are great and world changing in a largely positive way (even though in my personal opinion Facebook is the least of those, I can also see it argued the other way).

hunter-gatherer(10000) 3 days ago [-]

I've brought this up a few times and the response is usually against me. I feel the same way, although I understand that people need to work and people need to get paid. I've worked jobs that probably had little if any positive net impact on society. I'd contend that net-positive industries are actually a small minority. Just because something provides a consumed/demanded product doesn't mean that it is a net positive, and I think people generally aren't as good at diffentiating between the two.

tasuki(10000) 3 days ago [-]

I use Facebook to keep in touch with old friends and sometimes people organize events and they invite me on Facebook and it works and makes my life a tiny bit better.

musk_micropenis(10000) 3 days ago [-]

Sorry to hear about the lay-off. I can't stress enough how influential Eric Lippert was on my early career. His work on C# and .NET, but more importantly his openness and engagement with the community played a big part in me continuing on the Microsoft stack.

Just jumping into any random month[1] in his blog archive from my formative years is incredible nostalgia for me. It's not the kind of high-concept 'a monad is a monoid in the category of endofunctors' content that will make the front page of HN, but was a great pipeline of information for a junior .NET developer hungry to learn.

[1] https://ericlippert.com/2009/08/

smcl(10000) 3 days ago [-]

Not much I can add here, I think musk_micropenis really hit the nail on the head

ericlippert(10000) 3 days ago [-]

Thank you, that's kind of you to say.

UglyToad(10000) 3 days ago [-]

Strongly agree, I've not read much of his blog but every time I do read an article the quality and personality stand out.

Hoping he gets the time to recover and decompress.

rjbwork(10000) 3 days ago [-]

100% agree. Still primarily working in .NET (now core! oh wait, now just .NET again, lol) some 11 years on. Thanks Eric!

UncleOxidant(10000) 3 days ago [-]

> Most of my team has found other positions and I am hopeful that the rest will soon.

Wow, I guess I would've figured that it would take people with this kind of background a while to find another gig (working on similar things) in the current environment. So maybe things aren't as bad as they seem? (yet, anyway)

> But after >26 years of thinking about programming languages for corporations, and the last three years of my work being thrown away, I need a good long corporate detox before I go looking again.

I feel this. I'm about to finish up a contract working on a product that's about to be killed (before ever really seeing the light of day) and it's kind of hard not to feel like Sisyphus at this point. Not really interested in looking for something else for a while.

iLoveOncall(10000) 3 days ago [-]

> Wow, I guess I would've figured that it would take people with this kind of background a while to find another gig (working on similar things) in the current environment. So maybe things aren't as bad as they seem? (yet, anyway)

Finding another position doesn't mean finding another equivalent position.

I work for a FAANG right now. If I was getting laid off I would get the first job I could and then keep applying to other companies that are more suitable to the level that I had before.

dirheist(10000) 3 days ago [-]

It's the way the cookie crumbles.

eevilspock(10000) 3 days ago [-]

> Apologies that this is so long; I didn't have time to make it shorter.

A paraphrase of Blaise Pascal's famous line (that is commonly misattributed to Mark Twain: I would have written a shorter letter, but I did not have the time.

ericlippert(10000) 3 days ago [-]

Not the first time I've made that joke, but it's none the worse for having been used before.

0xbadcafebee(10000) 3 days ago [-]

I don't know who needs to hear this, but the Metaverse is literally just The Sims with a helmet. The only reason FB cares about it is it's a place to sell advertising. But nobody wants it. We wanted Facebook because we could be voyeurs, shout into the void, and receive little hits of dopamine from 'likes'. But who wants to be a voyeur of virtual people? People who play The Sims, that's who. I don't think The Sims will make billions of dollars, with or without a helmet.

hnick(10000) 2 days ago [-]

I've heard people make the case that MMOs were VR and Social Media combined, before Facebook etc existed. And I think it's true. I would log on to chat (FFXI in my case) with friends. We would occupy a shared space. People still run nightclubs and do other roleplaying in virtual spaces in these games.

The only thing really missing is the helmet but as far as occupying a virtual world, it's closer than anything else has been in terms of providing what people actually want.

dathinab(10000) 3 days ago [-]

As someone not in the topic the metaverse seems even worse, i.e. it looks like Facebook spent 20 Billion and didn't manage to add anything to VR other small companies haven't added years earlier...

I'm probably missing something, but it honestly looks worse then burning money, it looks like extremely inefficiently burning money.

Anyway sounded like an interesting work before the team was dissolved.

MikusR(10000) 3 days ago [-]

Good standalone headsets were a thing before Quest?

jfengel(10000) 3 days ago [-]

In theory you're missing the network effect of everyone already being on Facebook. So they can quickly spin up a vast virtual world with all the people you know already in it.

In practice, it hasn't happened, and nobody seems excited to do it. Whatever benefits they hope to get from the network effect seem unlikely thus far.

dibt(10000) 3 days ago [-]

> News flash: no one wants to wear VR goggles to spend any time in a digital heaven where the role of God is played by Mark Zuckerberg and you can do anything you can imagine, including "work" and "shop".

I could say the same thing about TikTok, Twitch, Instagram, Whatsapp, Youtube, WeChat, etc.

I'm not bullish on the metaverse. I live simply, with less tech than the average person. I don't expect to participate. With that said, I don't doubt it will be more successful than what graybeards expect. More than what Meta is dumping into it? No idea.

There will be things in the future you will not want to participate in. That's ok.

baobabKoodaa(10000) 3 days ago [-]

There's a difference between you personally not using WhatsApp and you saying 'no one wants to use WhatsApp'.

geraldwhen(10000) 3 days ago [-]

Anyone who currently works in a cost center: leave before it's too late.

If you don't directly produce revenue, you will get laid off. Not if, but when.

dohdhdaa(10000) 3 days ago [-]

"Too late" in this case is being paid not to work for 4+ months.

georgeburdell(10000) 3 days ago [-]

I'm probably not as talented as the author, but I can't relate to this feeling of giving up because some work won't be used. I have been working for ten years post-PhD and every single product I've ever worked on has been canned, sometimes very circuitously via acquisitions. My work is trade secret so I've never filed a patent, written a publication, nor given a talk. I have zero outwardly observable accomplishments. My resume and LinkedIn rolodex are the only testaments that I've done anything at all.

And yet I don't see myself retiring once I have enough money in a few years

300bps(10000) 3 days ago [-]

I can't relate to this feeling of giving up because some work won't be used

Everyone is motivated by different things. My strongest motivation and satisfaction comes from implementing technology to make drastic and lasting positive change in the work done by other people. Agile development methodology with iterative development and meaningful change every couple weeks suits me very well.

What you described as your work would not be fulfilling to me.

francisofascii(10000) 3 days ago [-]

I don't think Eric is giving up or retiring, just taking a much needed break. We should all look up from our keyboards from time to time to see the bigger picture.

> I need a good long corporate detox before I go looking again.

WorldMaker(10000) 3 days ago [-]

You may not always feel that way.

At some point you may start to wonder what your legacy on this planet is. At the very least: if you've made a good use of your limited time (and the scarce resource that is your labor). (Hard mode: if you've left the planet better off than you found it?)

The last few years have pushed a lot of people's 'burn out' buttons and the self-reflection of 'what have I accomplished with my time?' (and 'have I contributed more to good or to evil in this world?') are very easy burn out spirals to experience, so a lot of people are asking these sorts of questions now. (Including just about every day lately for months on 'Ask HN', in a million different unique individual ways, if you've not yet noticed.)

You sound like you are in a very fortunate place in your life that you aren't struggling with that right now. I envy you a little. I'm also glad for you and I hope it remains that way for you.

(I've spent too much time in the last few months worried that too much of my precious labor into finished projects and net revenue generation has been spent in service to the greater evil than the greater good of the world and have been struggling to figure out what that means or what I do with that cursed feeling.)

colineartheta(10000) 3 days ago [-]

How were you able to acquire a PhD without a publication?

bombcar(10000) 3 days ago [-]

There's a big difference knowing ahead of time, also.

If I am hired to do trade secret work I'm already understanding that it will never be 'known' even if the product or something associated with it DOES work - and many companies in the world will never be known anyway, let alone their products.

kevingadd(10000) 3 days ago [-]

I don't think it necessarily has to be about giving up, but it makes a lot of sense that if you already sort of hate your employer, them deciding to throw out a bunch of valuable work you did and lay you off is a good incentive to reconsider your current industry or at least take a break.

Personally I had an entire year worth of difficult sweng work thrown out due to politics, and it's impossible for that not to negatively impact my mood (or performance reviews)!

ska(10000) 3 days ago [-]

> but I can't relate to this feeling of giving up because some work won't be used.

People are fulfilled by different things. Some people are far more interested in their working having a meaningful (to them) impact to the 'outside' world than the specifics of the work.

woah(10000) 3 days ago [-]

> Speaking of cutting costs, the company is still pouring multiple billions of dollars into vaporware called "the metaverse". News flash: no one wants to wear VR goggles to spend any time in a digital heaven where the role of God is played by Mark Zuckerberg and you can do anything you can imagine, including "work" and "shop".

Sounds like this guy was totally opposed to the company's new focus, to the point of describing it in derisive terms. Seems like letting him go was the best thing for everyone, and maybe it was judged that his team wouldn't be able to continue successfully without him.

ericlippert(10000) 3 days ago [-]

I was regretted attrition. And very, very far from the most vital person on my team.

mikebenfield(10000) 3 days ago [-]

I have no data to back this up, but if I had to bet I'd guess that the large majority of Meta employees - along with shareholders - are also strongly opposed to the company's new focus.

blitz_skull(10000) 3 days ago [-]

Just staring at facts on paper... It sounds like Facebook has no interest in cutting costs since they cut a team that reduced the cost of every single team they interacted with. By a large multiple of whatever it cost to have that team employed.

Regardless of your take on the 'Metaverse', it's clear that this was in fact, not the best thing for everyone.

cleandreams(10000) 3 days ago [-]

This is sad. I find probablistic programming languages very interesting. This probably means that many of the most useful ideas are disappeared. Does anyone know any relevant papers that describe what they did?

aboodman(10000) 2 days ago [-]

I am late to this thread, and I don't know if you'll see this eric, but you've been a sort of hero of mine for a long time and I'm sad both on your behalf and the worlds' that your work is being abandoned. I followed your blog waaay back in msft days, when I was just getting started, and you were a small part of why I applied to Microsoft, got into .net, and became the programmer I am.

You've build a large number of wonderful things and I wish we could have all experienced the latest.

I hope after a long rest you find your way back to software and perhaps share what you were working on in some other way.

ericlippert(10000) 2 days ago [-]

Thank you, that's kind of you to say!

Satam(10000) 3 days ago [-]

Author mentions that his team's work saved 'millions' of dollars for Meta every year - let's assume that's $10 million. Meta's operating expenses are over $80 billion annually. That's barely one hundredth of a percent in savings for Meta.

I'm sure they were doing interesting work otherwise, but it make sense why the team would be considered for cuts if there weren't any breakthroughs on the horizon.

bfeynman(10000) 3 days ago [-]

This news didn't surprise me at all. Academics and research scientists on teams like this are very far removed from driving revenue and understanding the value you provide. It's just as likely that the teams they saved costs for are also hemorrhaging anyway and being shut down or reduced.

trenchgun(10000) 3 days ago [-]

Full quote:

'The mission of the Probability division was to create small teams that applied the latest academic research to real-world at-scale problems, in order to improve other groups' decision-making and lower their costs. New sub-teams were constantly formed; if they didn't show results quickly then they were failed-fast; if they did show results then they were reorganized into whatever division they could most effectively lower costs.

We were very successful at this. The PPL team in particular was at the point where we were regularly putting models into production that on net reduced costs by millions of dollars a year over the cost of the work. We were almost ready to be spun off.

We foolishly thought that we would naturally be protected from any layoffs, being a team that reduced costs of any team we partnered with. In retrospect, that was a little naive. A team that reduces costs of other teams is not on anyone's critical path.'

sp332(10000) 3 days ago [-]

If the team is costing less than $10 million/year, it still makes less sense to let them go.

trenchgun(10000) 3 days ago [-]

His other comment clarifies this: 'The team was all mathematicians. We did the math. I helped one of our data scientists put a model into production that saved $15M a year from that model alone, and we had a dozen people like that. We were working on signal loss models that had potential to save billions. I genuinely do not understand the logic of cutting this team to save costs.'

https://news.ycombinator.com/item?id=33806727

pronlover723(10000) 2 days ago [-]

> News flash: no one wants to wear VR goggles

This to me sounds similar to 'I think there is a world market for maybe 5 computers' (https://www.lexology.com/library/detail.aspx?g=164a442a-1b90...). That was wrong

or 'people don't want to lug a computer with them to the beach or on a train to while away hours' (https://qz.com/593329/choice-quotes-from-a-1985-new-york-tim...). Looking around the beach and especially the train, everyone is lugging around a portable computer (phone)

I don't think today's VR/AR is 'the thing' but I do think, in the same way people thought Palm Pilots and Windows CE devices were a small niche market for ~15 years and it wasn't until iPhone that the masses finally understood what a pocket device was good for VR/AR will eventually reach a version/device that will be more compelling than smartphones and similarly blow up

It might take until they get to a small dot like 'Striking Vipers' but I'm glad at least one company, if not 5, are pushing forward.

walnutclosefarm(10000) 2 days ago [-]

Sounds just as much like what critics said about the Segway, which was supposed to revolutionize pedestrian movement, and which ... well, let's just say the critics turned out to be right.

Which is to say, I don't think you can tell much about the stickiness of a technology from what it's proponents say for or against it. History is full of well-hyped failures, and not a few overperforming fringe ideas.

cvz(10000) 2 days ago [-]

The full line from the article appears to be more along the lines of (and I am paraphrasing here) 'no one wants Mark Zuckerberg's cynical and unimaginative interpretation of VR' and not a dismissal of the technology itself.

kaashif(10000) 2 days ago [-]

> News flash: no one wants to wear VR goggles

Cutting that sentence off there paints a slightly different picture from the full sentence:

> News flash: no one wants to wear VR goggles to spend any time in a digital heaven where the role of God is played by Mark Zuckerberg and you can do anything you can imagine, including "work" and "shop".

Plenty of people want to wear VR goggles, but it's hard to see how the metaverse specifically will take off.

midoridensha(10000) 2 days ago [-]

>This to me sounds similar to 'I think there is a world market for maybe 5 computers' That was wrong

>or 'people don't want to lug a computer with them to the beach or on a train to while away hours'

Exactly. It's also just like how 'people don't want to wear 3D glasses in a theater'. That was wrong too: every single movie today is in 3D.

Oh wait... No movies are in 3D now. They tried (for a 2nd time) and failed and gave up.

VR goggles might get some popularity for gaming, but the 'metaverse' thing is just dumb.

AR, however, makes a lot of sense if they can make it convenient. I would love to have some cycling glasses connected by BT to my phone, which show a simple moving map display as I'm riding in the city so I know where to turn, instead of having to stop every so often and pull out my phone to look at where I am.

kyleyeats(10000) 3 days ago [-]

So which one was it? Was the team very important or is the work being thrown away?

raydev(10000) 3 days ago [-]

Both.

trenchgun(10000) 3 days ago [-]

Facebook has become a clown car company.

ChrisMarshallNY(10000) 3 days ago [-]

I wish Mr. Lippe