Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

September 28, 2022 02:06



Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: Whisper – open source speech recognition by OpenAI (September 21, 2022: 1697 points)

(1698) Whisper – open source speech recognition by OpenAI

1698 points 6 days ago by _just7_ in 10000th position

openai.com | Estimated reading time – 3 minutes | comments | anchor

Read Paper
View Code
View Model Card

Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. We show that the use of such a large and diverse dataset leads to improved robustness to accents, background noise and technical language. Moreover, it enables transcription in multiple languages, as well as translation from those languages into English. We are open-sourcing models and inference code to serve as a foundation for building useful applications and for further research on robust speech processing.

The Whisper architecture is a simple end-to-end approach, implemented as an encoder-decoder Transformer. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and to-English speech translation.

Other existing approaches frequently use smaller, more closely paired audio-text training datasets,123 or use broad but unsupervised audio pretraining.456 Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper's zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.

About a third of Whisper's audio dataset is non-English, and it is alternately given the task of transcribing in the original language or translating to English. We find this approach is particularly effective at learning speech to text translation and outperforms the supervised SOTA on CoVoST2 to English translation zero-shot.

We hope Whisper's high accuracy and ease of use will allow developers to add voice interfaces to a much wider set of applications. Check out the paper, model card, and code to learn more details and to try out Whisper.




All Comments: [-] | anchor

FloatArtifact(10000) 6 days ago [-]

This would be a cool thing to integrate into Dragonfly https://github.com/dictation-toolbox/dragonfly

synkarius(10000) 5 days ago [-]

It would. I wonder how this compares with Kaldi, one of the two open source speech recognition engines that Dragonfly currently supports.

jfoster(10000) 6 days ago [-]

It seems like OpenAI are finally living up to their name for once with this release? Anything I'm missing?

From what I can gather:

1. Includes model weights. I can't find the URL, but they reference them enough and have a CLI tool, so I presume I just haven't found them yet.

2. Includes code: https://github.com/openai/whisper

3. Released under MIT License: https://github.com/openai/whisper/blob/main/LICENSE

thesausageking(10000) 6 days ago [-]

It's one model and in a non-strategic area where there are existing open source projects (Kaldi, DeepSpeech, ...).

For a company that raised $1B, that's not exactly living up to their name and original mission.

solarmist(10000) 6 days ago [-]

This kind of model is harder to abuse, so I guess it passed their internal checks much more easily.

I can understand not releasing GPT-3, even if I disagree with the decision.

no1youknowz(10000) 6 days ago [-]

This is awesome. But I really want the other way.

To be able to give it text and hear the speech. A TTS (text to speech).

As a language learner, the ability to create my own sentences (based on existing ones I have, in changing a word here or there). Would be amazing.

How long till we have this I wonder. I know I could use a service to do this currently. But having something running locally, I'd prefer.

Hopefully someone in the OpenAI team reads this. :)

freedomben(10000) 6 days ago [-]

Likewise, TTS is what I really want. My goal is to be able to create audio books from text. I've been using Amazon Polly and it's acceptable quality, but I would be ecstatic to be able to do it locally on my own hardware.

TaylorAlexander(10000) 6 days ago [-]

I suspect this is coming. I mean we do have decent text to speech systems already, but in this vein of "we used neural networks and now it's very very good" you can imagine that with something like GPT-3, to extend it they could use this speech to text system so you could speak to it for input, and then a natural progression is that it can use text to speech to return the output, so you just have a voice oriented conversational system.

So I think TTS is a logical part of the system. I also think that there are peculiarities of voice interaction that aren't captured in text training datasets, so they would need to do some fine tuning on actual voice conversation to make it feel natural.

All in due time I suppose.

noreally_(10000) 6 days ago [-]

A notebook is available to try with your microphone on Colab here: https://colab.research.google.com/drive/1nBZ-pDIaIi3N1DIIXvJ...

I'm surprised by the quality on non-English languages, given that 80+% of the training data is English, and the rest is split between tens of languages.

bambax(10000) 6 days ago [-]

Thanks! I played with this in French and posted the results as replies to this comment: https://news.ycombinator.com/item?id=32928643

It's sometimes close to perfect, and sometimes goes off the rail; I think that maybe the model tries to establish some sort of consistency for each sentence; if starts wrong for the first few words of a sentence, it can't build the rest properly.

But it's super fun.

berberous(10000) 6 days ago [-]

How do you get this to translate instead of just transcribe?

goffi(10000) 6 days ago [-]

Really interesting, I can see ton of potential uses.

2 questions:

1) how does it compare to state of the art FOSS solutions? I'm seeking about DeepSpeech or Vosk

2) would it be somehow possible to associate timestamp to the words recognized? That would be amazing for things such as audio editing or skipping to a particular location on a video

nshm(10000) 6 days ago [-]

You properly mentioned timestamps. There are many other important properties of good ASR system like vocabulary adaptability (if you can introduce new words) or streaming. Or confidences. Or latency of the output. Compared to Vosk models this model can not work in streaming manner, so not very suitable for real-time applications.

But in general the model is robust and accurate and trained on the amount of speech we never dreamed about in Vosk. We will certainly benefit from this model as a teacher (together with others like gigaspeech models). I recently wrote about it https://alphacephei.com/nsh/2022/06/14/voting.html

goffi(10000) 6 days ago [-]

> goffi

for 2), it's actually written in the description: 'phrase-level timestamps', so it should be possible (phrase level is neat for skipping to a special location on a video, but maybe not for audio editing).

IceWreck(10000) 6 days ago [-]

Is there a list of system requirements somewhere ? Can it run on cheaper low memory GPUs ? maybe CPUs ?

yjftsjthsd-h(10000) 6 days ago [-]

On my ancient desktop it happily fell back to running on CPU just fine.

StevenWaterman(10000) 6 days ago [-]

Their models range from 70mb to 3gb. The largest model is smaller than the optimised stable diffusion. Not sure what the inference speed is like, haven't tried it myself yet.

mewse-hn(10000) 6 days ago [-]

I know this isn't a tech support forum but maybe someone here knows. I'm attempting the sample python code from the github and almost get a transcription running on my work laptop without a GPU, but I run into this error message:

>>> result = whisper.decode(model, mel, options)

Traceback (most recent call last):

[snip]

RuntimeError: 'slow_conv2d_cpu' not implemented for 'Half'

It looks like a Torch error, is there some twiddling with 'options' I can do to get it to run?

mewse-hn(10000) 6 days ago [-]

I seem to have worked around it by tweaking the 'options' line from the sample code to this:

>>> options = whisper.DecodingOptions(fp16=False)

ignite(10000) 5 days ago [-]

I am running on work laptop not using GPU. (I'm running in docker). I just get

    warnings.warn('FP16 is not supported on CPU; using FP32 instead')
And it works.
jcims(10000) 6 days ago [-]

Did respectably with some mumble rap: https://controlc.com/d353dafb

(some NSFW words in the lyrics obv)

derangedHorse(10000) 6 days ago [-]

Whisper performed a lot better than I would've expected it to!

mmh0000(10000) 6 days ago [-]

Okay this is super impressive. I just downloaded Whisper and fed it a random flac file I had handy and it did a really good job. Also impressive that it works on my weak CPU:

A 3m07s flac took 5m to transcribe:

  $ whisper --device cpu 'BLACKPINK - BORN PINK/01 Pink Venom.flac'
  Detecting language using up to the first 30 seconds. Use `--language` to specify the language
  Detected language: korean
  [00:00.000 --> 00:10.000]  Blackpink
  [00:11.000 --> 00:14.000]  Kick in the door, wave in the coco
  [00:14.000 --> 00:16.000]  팝콘이는 친게 껴들 생각 말고
  [00:16.000 --> 00:19.000]  I talk to talk, run ways I walk walk
  [00:19.000 --> 00:21.000]  힘 감고 팝 팝 안 봐도 척
  [00:21.000 --> 00:24.000]  By one and two by two
  [00:24.000 --> 00:26.000]  내 손끝 두 하나에 타면 아지은 중
  [00:26.000 --> 00:30.000]  갓 자쇼 지금 화려해 T makes no sense
  [00:30.000 --> 00:32.000]  You couldn't get a dollar out of me
  [00:33.000 --> 00:38.000]  자 오늘 밤이야 눈톱을 품고
  [00:38.000 --> 00:41.000]  미혼을 뺏음 down
  [00:41.000 --> 00:43.000]  Look what you made us do
  [00:43.000 --> 00:47.000]  천천히 널 잠재울 파이어
  [00:48.000 --> 00:52.000]  잠이 날 만큼 아름다워
  [00:52.000 --> 00:53.000]  I bring the pain like
  [00:53.000 --> 00:57.000]  디스탑, 팽팽, 디스탑, 팽팽, 디스탑, 팽팽, 팽팽
  [00:57.000 --> 00:58.000]  Get em, get em, get em
  [00:58.000 --> 01:00.000]  Straight till you don't like
  [01:00.000 --> 01:01.000]  Whoa, whoa, whoa
  [01:01.000 --> 01:03.000]  Straight till you don't like
  [01:03.000 --> 01:04.000]  Ah, ah, ah
  [01:04.000 --> 01:05.000]  Taste that, pink venom
  [01:05.000 --> 01:06.000]  Taste that, pink venom
  [01:06.000 --> 01:08.000]  Taste that, pink venom
  [01:08.000 --> 01:09.000]  Get em, get em, get em
  [01:09.000 --> 01:11.000]  Straight till you don't like
  [01:11.000 --> 01:12.000]  Whoa, whoa, whoa
  [01:12.000 --> 01:13.000]  Straight till you don't like
  [01:13.000 --> 01:14.000]  Ah, ah, ah
  [01:14.000 --> 01:15.000]  Blackpink and Amo
  [01:15.000 --> 01:17.000]  Got it by the smack ram
  [01:17.000 --> 01:18.000]  But rest in peace
  [01:18.000 --> 01:19.000]  Please light up a candle
  [01:19.000 --> 01:20.000]  This the knife of a vando
  [01:20.000 --> 01:22.000]  Messed up and I'm still in saline
  ...SNIP...
lunixbochs(10000) 6 days ago [-]

Looks like it defaults to the model called 'small'.

I just ran some benchmarks - M1 Max, pytorch, with a 1.29 second flac (looks like the matrix math was running on a single thread):

    tiny
    146.522ms detect_lang
    549.131ms decode_one
    0.057ms tokenizer
    base
    354.885ms detect_lang
    1046.679ms decode_one
    0.011ms tokenizer
    small
    803.892ms detect_lang
    3194.503ms decode_one
    0.017ms tokenizer
    medium
    2279.689ms detect_lang
    10128.255ms decode_one
    0.023ms tokenizer
    large
    3656.478ms detect_lang
    17249.024ms decode_one
    0.016ms tokenizer
adgjlsfhk1(10000) 6 days ago [-]

For more benchmarks on an rtx 2060 (6gb), the 'small' model for me is roughly 10x real-time and the tiny model is 30x real-time.

manishsharan(10000) 6 days ago [-]

Oh this is a relief to have something opensource in this field. I had using Mozilla Deepspeech for transcribing my voice notes , often with hilarious to incomprehensible results. DeepSpeech is dead ; so I will be sure to check this out.

pabs3(10000) 6 days ago [-]

DeepSpeech got spun out of Mozilla to coqui.ai and they are continuing the open nature of the project.

dindindin(10000) 6 days ago [-]

I'm not in the Speech Recognition circles and am looking for open source speech recognition I can play around with - would this be the new state of the art?

mercurywells(10000) 6 days ago [-]

For me as a deaf person the current state of art (in terms of speed & usability) is the Recorder app on a Google Pixel phone (4a/6 Pro is what I've used)

StevenWaterman(10000) 6 days ago [-]

Yes

visarga(10000) 6 days ago [-]

Most probably

The5thElephant(10000) 6 days ago [-]

How is it Apple, Google, or Microsoft are not further ahead of the game on speech recognition like this? They have the resources to hire the best ML researchers and throw tons of computing hours at it, yet Siri, Google, and Cortana continue to struggle to get anywhere near this level of comprehension.

wongarsu(10000) 6 days ago [-]

Siri and Cortana have to run at least in real time, with reasonable compute resources. Probably faster than real time when the audio gets shipped off to the cloud and transcribed there. This model can't do that (in the 'large' version, which the examples use).

Also, you are comparing Whisper's highlight reel with everyday performance of other models. Nobody shows their weaknesses in their highlight reel.

Kuinox(10000) 6 days ago [-]

OpenAI is owned by Microsoft FYI.

fxtentacle(10000) 6 days ago [-]

This AI has a 30 second delay on the audio processing because it needs to be able to 'look into the future' to get these good results. That 30s delay would be unacceptable for Siri/Google/Cortana.

beastman82(10000) 6 days ago [-]

In my unmeasured empirical observation Google has amazing speech recognition

danso(10000) 6 days ago [-]

This is an astonishing package. Every AI voice-to-text model I've tried on 'The Wire's' famous 'fuck' scene [0] usually fails, because the youtube clip's audio quality is bad and it's a scene with virtually no dialogue except breathing and 'Fuck'. But Whisper returned impressive results [1]

[0] https://www.youtube.com/watch?v=DS6pE88Xg3s

[1]

    $ yt-dlp --extract-audio --audio-format mp3 -o wire-fuck.mp3 https://www.youtube.com/watch?v=DS6pE88Xg3s
    $ whisper --language en wire-fuck.mp3
    [00:00.000 --> 00:02.000]  Oh
    [00:13.260 --> 00:15.260]  Fuck
    [00:15.260 --> 00:31.260]  Motherfucker
    [00:50.700 --> 00:52.700]  Fuck
    [00:52.700 --> 00:58.700]  Oh
    [00:58.700 --> 01:10.700]  Fuck
    [01:28.700 --> 01:55.900]  Fuck
    [02:02.340 --> 02:03.700]  Motherfuck.
    [02:10.220 --> 02:11.220]  Oh, fuck.
    [02:11.780 --> 02:12.780]  Oh, fuck.
    [02:25.900 --> 02:27.900]  Fuck, fuck, fuck, fuck, fuck, fuck.
    [02:27.900 --> 02:28.900]  Motherfucker.
    [02:32.900 --> 02:33.900]  Oh, fuck.
    [02:34.900 --> 02:35.900]  Fuck.
    [02:35.900 --> 02:36.900]  Oh, fuck.
    [02:36.900 --> 02:37.900]  Oh, fuck.
    [02:37.900 --> 02:38.900]  Oh, fuck.
    [02:48.900 --> 02:49.900]  Motherfucker.
    [02:53.900 --> 02:54.900]  Fucking A.
    [02:54.900 --> 02:56.900]  Mm hmm.
    [02:56.900 --> 03:12.900]  Fuck.
    [03:26.900 --> 03:28.900]  Motherfucker.
    [03:28.900 --> 03:32.900]  Fuck me.
    [03:58.900 --> 04:01.900]  Oh.
    [04:28.900 --> 04:34.900]  Fuck.
AndrewKemendo(10000) 5 days ago [-]

I've been on HN since 2012 and this might be one of the best comments I've ever read

owenpalmer(10000) 5 days ago [-]

nsfw

andy_xor_andrew(10000) 6 days ago [-]

Hold on, it does not only speech recognition, but also language translation, in the same model?

What an interesting approach. What benefits does this have over having two dedicated models, one for speech-to-text, and another for translation?

It just seems so odd, given the problems of speech-to-text and Spanish-to-English seems so different from one another (in terms of the problem domain). Seems so unusual to have both handled by one model!

Does knowledge of speech-to-text carry over into knowledge of translation? Does knowledge of translation carry over into knowledge of speech-to-text? So weird.

ByThyGrace(10000) 5 days ago [-]

Judging from the chart in their github README, Whisper performs much better in parsing Spanish audio than any other language and that in particular blows my mind. I would have expected English to be at the top of any such model, it being such an IT lingua franca.

Now I wonder if it works equally well with Spanish from Spain (and its different regions) and Spanish from the New World (and in its myriads of different flavours).

newhaus1994(10000) 6 days ago [-]

My understanding is that multi-modal models are the primary focus of OpenAI right now, due to their stated goal of achieving AGI. This product is probably better thought of as an offshoot of their work to create a fully generalizable model, rather than a specific attempt to provide translation/transcription services.

beanlog(10000) 6 days ago [-]

It sounds useful to me because you can use tone information to help with the translation, which text-to-text translation can't do. But I'm not sure if that's how this model actually works.

TaylorAlexander(10000) 6 days ago [-]

It seems these days that language-oriented models are commonly becoming multilingual by default. There are a lot of common threads when understanding sentence construction between different languages. French and English have different rules but they will still have things like nouns, adjectives, subjects, prepositions, etc. It seems that by training models on many languages you get both a more robust understanding of language, and it saves you the trouble of having to make many more localized models for every language. I also believe that the other languages help the models construct sentences in languages which have very small training sets. If it has a few examples in a rare language as well as good translations to a better-known language, then it can provide good support for the rare language.

We also see in image generation models that multi-modal networks are more powerful than single purpose networks. As we move towards more advanced AI systems I suspect we will see more and more generalizable networks with distinct advantages over separate networks that get plugged together.

thuttinger(10000) 6 days ago [-]

I tried running it in realtime with live audio input (kind of).

If you want to give it a shot, you can find the python script in this repo: https://github.com/tobiashuttinger/openai-whisper-realtime

A bit more context on how it works: The systems default audio input is captured with python, split into small chunks and is then fed to OpenAI's original transcription function. It tries (currently rather poorly) to detect word breaks and doesn't split the audio buffer in those cases. With how the model is designed, it doesn't make the most sense to do this, but i found it would be worth trying. It works acceptably well.

kkielhofner(10000) 6 days ago [-]

Haven't tried it yet but love the concept!

Have you thought of using VAD (voice activity detection) for breaks? Back in my day (a long time ago) the webrtc VAD stuff was considered decent:

https://github.com/wiseman/py-webrtcvad

Model isn't optimized for this use but I like where you're headed!

minimaxir(10000) 6 days ago [-]

The model output can be tweaked to produce audio embeddings (akin to BERT for text embeddings and CLIP for image embeddings), which can lead to some interesting applications as the previous two examples have demonstrated.

FerociousTimes(10000) 6 days ago [-]

What do you mean exactly by audio embeddings?

lynguist(10000) 6 days ago [-]

How can I use this (or something similar) for live translation? I don't mind if there's a 30s delay.

As in I don't want to input a file, I want to input the microphone sound.

agnos(10000) 6 days ago [-]

Would also like to know this. It looks like they're processing the audio file in 30 second chunks, so a naive approach of keeping a buffer of 30-second input stream chunks and just continually writing to an output .mp3 could work...

blueberrychpstx(10000) 6 days ago [-]

Was wondering the same.

I really wish I would have been paying attention in Unix class...

Something like `microphone | chunk 3s | whisper | stdout` would be SO COOL!!! I think that's possible but too lazy to look more.

cercatrova(10000) 6 days ago [-]

Their Scottish accent example is pretty good, I'd like to see it work on some very strong English accents like this one: https://www.youtube.com/watch?v=nJ7QB3om-QY

homarp(10000) 6 days ago [-]

Detected language: english

[00:00.000 --> 00:05.400] Gordy and County Kerry are investigating the theft of up to 60 sheep on Mount Brandon.

[00:05.400 --> 00:10.400] One of the farmers is offering a reward for information leading to the return of the use,

[00:10.400 --> 00:12.200] which are worth thousands of euro.

[00:12.200 --> 00:14.200] Well, I'm fine with that.

[00:14.200 --> 00:15.200] That's right.

[00:15.200 --> 00:16.200] Do you own them?

[00:16.200 --> 00:17.200] Anyone can say it.

[00:17.200 --> 00:18.200] Fine with that.

[00:18.200 --> 00:22.720] Last Saturday, Mikey Joe O'Shea brought his flock of Scotch sheep down from the mountain

[00:22.720 --> 00:25.320] commonage ahead of lambing.

[00:25.320 --> 00:29.840] He discovered over 50 were missing, allowing for a number of deaths and

[00:29.840 --> 00:30.840] strays.

[00:30.840 --> 00:34.600] Mikey is convinced over 45 sheep have been stolen.

[00:34.600 --> 00:35.600] It was a good night.

[00:35.600 --> 00:36.600] It would be a full moon there.

[00:36.600 --> 00:37.600] It would be a good night.

[00:37.600 --> 00:38.600] It would be bright out.

[00:38.600 --> 00:40.600] There could be anyone going up in the mountains.

[00:40.600 --> 00:41.600] It would be a good night.

[00:41.600 --> 00:43.600] Well, that was 45 sheep missing.

[00:43.600 --> 00:49.600] Mikey and the lambs and everything in the sheep, they counted out a nice bit of money.

[00:49.600 --> 00:52.200] They've been doing the boat in Nassan.

[00:52.200 --> 00:53.200] It's a big one. [00:53.200 --> 00:54.200] It's a big one. [00:54.200 --> 00:55.200] It's a big one.

[00:55.200 --> 00:59.000] Mikey's next door neighbor says some of his sheep have also been stolen.

[00:59.000 --> 01:00.000] Come back. [01:00.000 --> 01:01.000] Come back. [01:01.000 --> 01:02.000] Come back.

[01:02.000 --> 01:03.000] I've been missing about 10 years.

[01:03.000 --> 01:04.000] It's not all that difficult.

[01:04.000 --> 01:06.320] All they've got to do is have a good dog.

[01:06.320 --> 01:10.560] Have a good dog and go at night, some moonshine night.

[01:10.560 --> 01:11.560] Just put the dog around him.

[01:11.560 --> 01:14.120] Put him on a trailer and walk him.

[01:14.120 --> 01:18.360] And then probably somebody else to pick him up.

[01:18.360 --> 01:29.960] Everybody's doing it north, but he's doing it.

mod(10000) 6 days ago [-]

Those are Irish.

angrais(10000) 6 days ago [-]

Are you sure? I just ran some of Kimmy's sketches through it and ... The results are garbage.

dom96(10000) 6 days ago [-]

This really makes me want to build a Amazon Echo/Google Nest/etc replacement that's open hardware, open source and most importantly recognises voice completely offline. I find that I don't use these smart devices for much more than setting timers anyway so this seems like an easy project.

I just wonder what system requirements Whisper has and whether there are open source voice recognition models that are specifically built for embedded devices.

solarkraft(10000) 6 days ago [-]

Are you thinking about reimplementing Mycroft?

The Mycroft has done a lot of cool and important work in the field to ship an actual personal assistant product (stuff like wake word detection).

suyash(10000) 6 days ago [-]

This is only one side of the coin, you still need really good models for Speech Synthesis and then be able to have it all working in almost real time, ideally locally on device.

MacsHeadroom(10000) 6 days ago [-]

I really want all this too. The smallest model is ~80mb and the largest is 3gb. Not sure about system requirements yet; but models that small suggest this may be doable locally on a single board computer.

Edit: According to this comment[0] the base model runs in real time on an M1 CPU. The tiny model apparently decodes an audio file twice as fast. These are promising results.

[0] https://news.ycombinator.com/item?id=32927360#32929739

eatsyourtacos(10000) 6 days ago [-]

Can this be used as a real-time transcription or is it too slow for that?

Curious what anyone is using these days for a real-time transcription. It doesn't have to be perfect, but just good enough.

My kids watch some youtube vidoes where people will make a mod where it converts them talking to text then look for keywords and spawn a boss in Terraria if you say the wrong keyword etc.

I made a clone of that with the .NET System.Speech.Recognition library. It... works.. but my biggest problem is that #1 it waits until you are done speaking to translate to text on the callback, so there was too much of a delay for it to be fun.. the point is that it will be checking a stream of chatter. #2 is the recognition is pretty crap, I mean it's nearly good enough for my silly purpose but it's still pretty bad.

blueberrychpstx(10000) 6 days ago [-]

If your family uses Apple devices, Apple offers free on-device speech recognition. Only caveat is that it needs to be restarted every minute due to whatever stupid limitation (or bug) they've introduced.

https://developer.apple.com/documentation/speech/recognizing...

Also, see `requiresOnDeviceRecognition`

jayavanth(10000) 6 days ago [-]
whimsicalism(10000) 6 days ago [-]

It might require too much work for what you are looking for, but the wav2letter library is the best real-time transcription OSS I have found by a considerable margin.

NaturalPhallacy(10000) 6 days ago [-]

I tried it out and it's way too slow on my machine that is no slouch (Ryzen 9 5950/GTX 3080).

It's doing seconds of translation per minute for me at least.

TaylorAlexander(10000) 6 days ago [-]

The base model seems to run faster than real time on my machine. The "medium" model is larger and runs more slowly - roughly real time or maybe slightly slower.

suyash(10000) 6 days ago [-]

Depends if you're trying to run it offline or over the cloud.

knaik94(10000) 6 days ago [-]

I got a super weird results with the 'medium' and language Japanese (with a --task translate). The song is False Sympathy by Mondo Grosso.

'[01:17.000 --> 01:32.000] Translated by Releska' when using the translate to english. That entire part of the song is instrumental. This line does not appear at all in the original transcribe only in the opus format rip.

It shows up in the yt rip in format 251 (opus), but not in format 140 (aac from youtube), nor the flac rip. All three are giving different results.

The translation quality is tied to bitrate. Same song converted to different words, the only difference being bitrates and formats. Converting my own rip with the same parameters as yt (opus @140 and then @130) didn't allow me to reproduce this error.

The model hung for a solid extra minute at the end when translating to english, the last 90ish seconds of the song took real time 60 seconds, while the entire rest took about 90. The same behavior was not observed with the transcribe.

Some of the english words are incorrect but that was expected. The first Japanese 'mistake' I found was '全ては二人の' instead of 'すべては ふたりの'. With the left being what whisper wrote. A single random word 'hey' was transcribed/translated to english even though it's the singer elongating the 園 while singing the 楽園. '落ちてゆく 二人で繋がれた二人のラグ HEY' instead of '落ちていく 鎖でつながれた 二人の楽園' .

I am using the official subtitles released on the youtube video.

It's a complex Japanese song with both japanese and english, and the original transcribe took about 20 real time seconds to start with the first line, 130 seconds for the whole song. It seems to be showing results in 20 second window increments, but this seems to depend on what it considers audio and what it is throwing away.

On my computer I wasn't able to use the large model because I ran out of VRAM, I have 8gb, not sure how much more it'd require. So I ran it with medium.

The song is False Sympathy by Mondo Grosso. The mv is suggestive, in case that matters. I grabbed a fresh audio rip from Youtube because I didn't want to take it out of my cd case.

https://www.youtube.com/watch?v=B6Y-WsgpzlQ

It is translating this version differently from the director's cut version. I ripped both as opus.

There is something weird about how it is handling the opus encoded version, as I find the same 'Translated by Releska' in a wav version transcoded from the opus.

adeptima(10000) 6 days ago [-]

Japanese output will produce lot of tiny mistakes. However the whole output is still good enough. Like 95% plus good enough.

Found lot mistakes in 3-4 characters kanji ... and I guess most native Japanese will do mistakes time to time too, and this is why they pop up lot of buzzwords on screen with all kind of highlighting to avoid double guessing.

amrrs(10000) 6 days ago [-]

Here's a live demo on Hugging Face Spaces if you want to try - https://huggingface.co/spaces/Amrrs/openai-whisper-live-tran...

coder543(10000) 6 days ago [-]

I've tried speaking to that demo several times... I used the built in feature to record from microphone, and I played back the samples to make sure they were audible and clear.

Sometimes it outputs the words 'thank you' (which I did not say), sometimes it outputs a period. It never once output anything I said. It seems completely broken.

EDIT: apparently something about the combination of Safari+HF+Whisper was not working. I tried another Whisper demo on HF and had the same results. Switching to Chrome made it work flawlessly... I have no idea what kind of codec incompatibility was happening.

clemnt(10000) 6 days ago [-]

this is amazing! got it working in French too

TaylorAlexander(10000) 6 days ago [-]

Hey this looks great! I like to record audio notes while driving in my car after work, to kind of decompress my thoughts from the day. But I never go back and listen as they can be long and meandering. Sometimes in the audio log I will sum up my thoughts, but this might be 20 minutes in and hard to find. I really wish I had transcriptions so I could easily scan the full contents. I have tried Mozilla Deepspeech (I don't want a cloud solution) and I was surprised to find that I could not get Deepspeech to reliably transcribe them. There is a bit of road noise, though I think for a human listener they are easy to understand. It looks like this one might actually do the trick!

EDIT: Tried it and it worked great! It is very easy to use. I just did the pip install line in the readme and was ready to go. You literally just run the one pip install line, and then you run the program in the format 'whisper my_audio.wav' and it goes. Really nice job OpenAI!

zhynn(10000) 6 days ago [-]

I do this too! I have been doing it for about a year now, and haven't ever run into someone else that does this kind of audio-journaling. Would you be up for comparing notes sometime about how it is working out for you? I am finding that it is extremely effective form of self-care, but with lots of personal caveats. I would be so interested to hear your experience.

Snitch-Thursday(10000) 6 days ago [-]

Google's recorder app for android will let you record audio files and make some transcriptions, right on the device.

petercooper(10000) 6 days ago [-]

I'll probably explore using this, but I've used an app called Just Press Record to do what you say. Runs on Apple Watch too, so you can tap a complication at any time in the day, speak, and you get a transcript on your phone, etc.

petercooper(10000) 6 days ago [-]

Just tested this on some developer podcasts which usually fail hard given they're full of technical jargon, brand names, etc. Whisper is a revolution! It's picking up terms like Heroku, DigitalOcean, GitHub, ECS, AWS, etc. and capitalizing properly - something nothing else did unless you provided a whole pile of guiding vocabulary.

ma2rten(10000) 6 days ago [-]

Did these podcasts have transcripts? You might be inadvertently evaluating it on data that it was trained on, which is basically cheating. Even if not, it might be trained on similar podcasts. Judging how good these kinds of models are is really hard.

Jnr(10000) 6 days ago [-]

Cool!

I am one of the top contributors to the tiny Mozilla Common Voice data-set for my language. The data-set is very small compared to other popular languages and none of the other mentioned data-sets contribute to that language to train the model of Whisper.

And even with so little data to train on it still works surprisingly well.

catfan(10000) 6 days ago [-]

[zalgo redacted]

archon1410(10000) 5 days ago [-]

Where do they mention what datasets they've used? I've tried looking at the paper but can't find it.

jdmoreira(10000) 6 days ago [-]

Looking forward to see if this works well with foreign accents

mminer237(10000) 6 days ago [-]

They have an example in the post with a very thick Scottish accent. You should listen to it. It's pretty impressive.

aidenn0(10000) 6 days ago [-]

For those on NixOS, here's a quick and dirty flake.nix that will let you make a venv in which to 'pip install''

Just put it in a flake.nix, and 'nix develop' followed by 'virtualenv ./venv; . ./venv/bin/activate; pip install git+https://github.com/openai/whisper.git'

    {
      description = 'Python 3.9 development environment';
      outputs = { self, nixpkgs }:
        let
          system = 'x86_64-linux';
          pkgs = import nixpkgs { inherit system; };
        in {
          devShells.${system}.default = pkgs.mkShell {
            buildInputs = [
              pkgs.ffmpeg
              pkgs.python39
              pkgs.python39Packages.pip
              pkgs.python39Packages.numpy
              pkgs.python39Packages.pytorch
              pkgs.python39Packages.virtualenv
            ];
          };
        };
    }
aidenn0(10000) 6 days ago [-]

This should, in theory, work with CUDA; my GPU doesn't have enough RAM to do it (it runs out at 2.9GiB allocated, I have 4GiB, but am running a compositing desktop, which chews up about 600MiB; not sure where the other ~400MiB went)

[edit]

I confirmed CUDA worked with the 'small' model, which used 3.3GB of GPU ram, and resulted in much poorer recognition than the 'medium' model on my CPU (but it ran at least two orders of magnitude faster).

    {
      description = 'Python 3.9 development environment';
      outputs = { self, nixpkgs }:
      let
        system = 'x86_64-linux';
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;
          config.cudaSupport = true;
        };
      in {
        devShells.${system}.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            cudatoolkit linuxPackages.nvidia_x11
            cudaPackages.cudnn
            libGLU libGL
            xorg.libXi xorg.libXmu freeglut
            xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib 
            ncurses5 stdenv.cc binutils
            ffmpeg
            python39
            python39Packages.pip
            python39Packages.numpy
            python39Packages.pytorch-bin
            python39Packages.virtualenv
          ];
          shellHook = ''
              export LD_LIBRARY_PATH='${pkgs.linuxPackages.nvidia_x11}/lib'
          '';          
        };
      };
    }
BasilPH(10000) 6 days ago [-]

Any opinions on what this means for speech-to-text companies like rev.ai and assmembly.ai ?

We've tested open source solutions for s2t, like kaldi, but the quality was not good enough. However, one of the main advantages of a service like assembly.ai to me was that they offer sentence splitting in form of punctuation and speaker detection, which Kaldi does not.

So I guess I answered my own question to some degree: A S2T service is more than just S2T. We already see assembly.ai add more and more features (like summarisation, PID redaction ect.) that are a value-add to plain S2T.

Still, curious to hear what your take on that is.

nshm(10000) 6 days ago [-]

You can apply public punctation model from Vosk on top of Kaldi output, you can also get speaker labels with existing open source software.

On quick video transcription test this model is more accurate than AssemblyAI and Rev AI. It will be harder for them to sell pure ASR now. Some more business-oriented applications will still be important though, for example ASR as part of callcenter analytics solution or as a part of medical ERP system.

The value of automatic summarization is small, without AI it is very hard to make it right, you need to be an expert in the field to understand what is important.

adeptima(10000) 6 days ago [-]

Japanese results looks pretty impressive!

Took マッコウクジラ14頭が海岸に打ち上げられる オーストラリア(2022年9月21日) https://www.youtube.com/watch?v=bZkNIzeRBk4

Extracted audio with youtube-dl -f bestaudio https://www.youtube.com/watch\?v\=bZkNIzeRBk4

Converted into [00:00.000 --> 00:13.000] オーストラリア南部の島で、真っ向くじら14棟が海岸に打ち上げられて死んでいるのが見つかり、専門家が調査のため原地入りしました。 [00:13.000 --> 00:25.000] 原地メディアによりますと、オーストラリア南部のキング棟で、19日、少なくとも14棟の真っ向くじらが海岸に打ち上げられて死んでいるのが見つかりました。 [00:25.000 --> 00:31.000] ほとんどが若いオーストを見られ、専門家が現場に重むき調査に当たっています。 [00:31.000 --> 00:41.000] くじらの死害は大きく運んだり埋めたりすることが難しいため、自然に分解されるのを待つ方針が検討されています。 [00:41.000 --> 00:52.000] また、死害を狙い、サメが海に集まる可能性があるとして、原地東局はサーファーなどに周囲に近づかないように呼びかけています。 [00:52.000 --> 01:02.000] 一方、21日にはタスマニア棟でおよそ230棟のくじらが浜辺に打ち上げられた状態で見つかりました。 [01:02.000 --> 01:07.000] およそ半数がまだ生きている模様で急助活動が進められています。 [01:07.000 --> 01:23.000] 見つかったのは、ゴンドーくじらの仲間と見られています。

knaik94(10000) 6 days ago [-]

Did you try translating them to english? I want to see if you get a similar error as me with a random phrase 'Translated by Releska' showing up.

gzer0(10000) 6 days ago [-]

Shocked at how good the results are, and how easy of an installation it is.

Here are the exact steps to follow to get it running on Ubuntu 22.04 via WSL and yt-dlp:

  1. pip install git+https://github.com/openai/whisper.git
  2. yt-dlp -f 'ba' -x --audio-format mp3 https://www.youtube.com/watch/?v\=bZkNIzeRBk4
  3. renamed the file to test.mp3
  4. whisper test.mp3 --language Japanese --task translate --model large
Note: the large model will download a ~3Gb file
mwlp(10000) 6 days ago [-]

Super impressive. I tested it on a Japanese streamer whose enunciation isn't exactly perfect and it did a decent job: https://www.youtube.com/watch?v=ROiOU1scaNA

  [00:00.000 --> 00:06.500]  Since the last one started, the number of times I've eaten has decreased.
  [00:06.500 --> 00:11.000]  If I get too carried away with the last one, I'll get hungry and do it.
  [00:11.000 --> 00:14.500]  I don't have time to eat.
  [00:15.500 --> 00:18.000]  I'm going to eat now.
  [00:20.000 --> 00:23.000]  It's going to take about 10 minutes from here.
  [00:23.000 --> 00:31.000]  It's been a while since I've had my last meal.
  [00:31.000 --> 00:36.000]  I feel like I'm losing my女子力.
  [00:36.000 --> 00:39.000]  I have to go back to my original self.
  [00:39.000 --> 00:44.000]  I have to get ready and go to bed.
  [00:44.000 --> 00:46.000]  It's not good.
  [00:46.000 --> 00:51.000]  I've been drinking a lot lately, so I'm going home.
  [00:51.000 --> 00:53.000]  I have to get my nails done this fall.
  [00:53.000 --> 00:54.000]  Halloween nails.
  [00:54.000 --> 00:57.000]  Halloween, Halloween, Halloween.
  [00:57.000 --> 00:59.000]  I'm going to the beauty salon today.
  [00:59.000 --> 01:02.000]  I'm going to get my nails done the day after tomorrow.
  [01:02.000 --> 01:10.000]  I used to look at a lot of clothes, but I stopped looking at them.
  [01:10.000 --> 01:12.000]  I'm going crazy.
  [01:12.000 --> 01:22.000]  My stomach's stopped in the middle of summer.
alach11(10000) 5 days ago [-]

How long until this gets implemented in Twitch? Real-time subtitles for any stream in the language of your choice?! That would be huge.

adeptima(10000) 6 days ago [-]

translation is not the strongest part. transcription looks very good.

magicalhippo(10000) 6 days ago [-]

It's struggling with Norwegian. Which I guess isn't shocking. The large model performs a fair bit better than the small, though neither is 'good'.

Though I assume the amount of Norwegian it has been exposed to is fairly limited, so in that light I'm actually impressed as well.

I tried it on a news segment from the radio[1], this is the large model output:

    [00:14.000 --> 00:17.200]  En skamløs krenking av FN pakten.
    [00:17.200 --> 00:24.000]  USAs president og verdensledere svarer på den russiske presidentens atomtrusler og krigsmobilisering.
    [00:25.500 --> 00:29.400]  Arbeidsklær som er ment til å være til begge kjønn, har det med å være tilpasset.
    [00:29.400 --> 00:33.400]  Men hvordan ville det gått, om det var motsatt?
    [00:34.100 --> 00:38.900]  Dyrevernsorganisasjon vil ha digital merking av regnstyr,
    [00:38.900 --> 00:44.900]  men næringen selv insisterer på den gamle tradisjonsrike måten med rissing av kniv.
    [00:45.600 --> 00:51.400]  Mange strømselskaper er positive til å tilby kundene fastpris på strøm, og det årevis.
    [00:51.400 --> 00:59.900]  Da risikerer de å måtte betale mye i nettopp åretsvis, sier aktører som aldri tilbyr fastpris.
    [00:59.900 --> 01:21.900]  Dette er onsdagens Dagsnytten. Jeg heter Espen Ås.
For reference, here's what he actually said, from the source[1] itself:

    * En skamløs krenking av FN-pakten. USAs president og verdensledere svarer på den russiske presidentens atomtrusler og krigsmobilisering.
    * Arbeidsklær som er ment å være til begge kjønn, er som regel tilpasset ... menn. Hvordan hadde det gått om det var motsatt?
    * Dyrevernsoganisasjon vil ha digital merking av reinsdyr, men næringen selv insisterer på den gamle tradisjonsrike måten med rissing av kniv.
    * Mange strømselskaper er positive til å tilby kundene fastpris på strøm - og det i årevis.
    - Da risikerer de å måtte betale mye i nettopp; årevis, sier aktør som aldri tilbyr fastpris
    Dette er onsdagens Dagsnytt 18 - jeg heter Espen Aas.
The translation didn't fare that well though:

    [00:14.000 --> 00:17.000]  A shameless violation of the UN treaty.
    [00:17.000 --> 00:24.000]  The US president and world leaders respond to the Russian president's nuclear threats and war mobilization.
    [00:24.000 --> 00:33.000]  Work clothes that are meant to be for both genders have to be suitable, but how would it be if it was the other way around?
    [00:34.000 --> 00:44.000]  The animal welfare organization will have a digital marking of reindeer, but the industry itself insists on the old traditional way of tearing a knife.
    [00:45.000 --> 00:51.000]  Many electricity companies are positive in offering customers fixed electricity prices, and that is annual.
    [00:51.000 --> 00:58.000]  Then they risk having to pay a lot in just a year, says an actor who has never offered fixed prices.
    [00:58.000 --> 01:20.000]  This is Wednesday's Dagsnytt 18. My name is Espen Ås.
For reference, here's Google Translate's attempt, which is pretty good:

    * A shameless violation of the UN Charter. The US president and world leaders respond to the Russian president's nuclear threats and war mobilization.
    * Work clothes intended for both sexes are usually adapted to ... men. How would it have gone if it had been the other way around?
    * Animal welfare organizations want digital marking of reindeer, but the industry itself insists on the old, traditional way of marking with a knife.
    * Many electricity companies are positive about offering customers a fixed price for electricity - and for years.
    - Then they risk having to pay a lot in precisely; for years, says a player who never offers a fixed price
    This is Wednesday's Dagsnytt 18 - my name is Espen Aas.
[1]: https://radio.nrk.no/podkast/dagsnytt_atten/l_5ce3e323-97a3-... (not sure if it's available outside of Norway)
kiwih(10000) 6 days ago [-]

Given this, are there good (and available/open source) models for text to speech? Last time I tried everything still sounded extremely robotic, and/or were a pain to set up and run. It would be fun to set up a pipeline where the two processes 'communicate'.

obscur(10000) 6 days ago [-]

Measuring performance in rounds of successful Chinese whisper

(irony)

pen2l(10000) 6 days ago [-]

Neat, https://github.com/openai/whisper - they have open-sourced it, even the model weights, so they are living up to their name in this instance.

The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I've seen. Will be super curious to see other folks trying it out and seeing if it's as robust as it seems, including when confronted with audio speech with natural tics and uhhh's and uhmm's and everything in-between.

I think it's fair to say that AI-transcription accuracy is now decidedly superior to the average human's, what the implications of this are I'm not sure.

anigbrowl(10000) 6 days ago [-]

It was already better. I edit a podcast and have > a decade of pro audio editing experience in the film industry, and I was already using a commercial AI transcription service to render the content to text and sometimes edit it as such (outputting edited audio).

Existing (and affordable) offerings are so good that they can cope with shitty recordings off a phone speaker and maintain ~97% accuracy over hour-long conversations. I'm sure it's been an absolute godsend for law enforcement other people who need to gather poor-quality audio at scale, though much less great for the targets of repressive authority.

Having this fully open is a big deal though - now that level of transcription ability can be wrapped as an audio plugin and just used wherever. Given the parallel advances in resynthesis and understanding idiomatic speech, in a year or two I probably won't need to cut out all those uuh like um y'know by hand ever again, and every recording can be given an noise reduction bath and come out sounding like it was recorded in a room full of soft furniture.

knaik94(10000) 6 days ago [-]

It seems far from good with mixed language content, especially with English and Japanese together. The timestamps are far from perfect. It's far from perfect. It's nowhere close to human for the more ambiguous translations that depend on context of word. It's far below what anyone that spoke either language would consider acceptable. Maybe it's unfair to use music, but music is the most realistic test of whether it's superior to the average human.

bambax(10000) 6 days ago [-]

The French version is a little contrived. The speaker is a native speaker, but the text is obviously the result of a translation from English to French, not idiomatic French.

I will try to put the code to the test, see how it goes.

Workaccount2(10000) 6 days ago [-]

Can't wait to see twelve new $49.99/mo speech parser services pop up in the next few weeks.

darepublic(10000) 6 days ago [-]

> Neat, https://github.com/openai/whisper - they have open-sourced it, even the model weights, so they are living up to their name in this instance.

Perhaps it will encourage people to add voice command to their apps, which can be sent to gpt3

pabs3(10000) 6 days ago [-]

Is the training dataset and code open too?

suyash(10000) 6 days ago [-]

More of this is welcome, they should live up their name and original purpose and share other models (code, weights, dataset) in the open source community as well.

Simorgh(10000) 6 days ago [-]

I've been experimenting with voice-interfaces where typing is replaced by talking, but I find it hard to transition users to voice - we 'seem' to prefer typing to talking.

I wonder if this will change.

ironlake(10000) 6 days ago [-]

Personally, I would rather type than talk when interacting with a computer. The only time I use voice interfaces are when the physical interface is so poor it's just easier to use voice. Apple TV devices are an example of this.

shpx(10000) 6 days ago [-]

We shouldn't call this open source. The model definition + the data is the source code. The model weights are a compilation artifact.

> The source code must be the preferred form in which a programmer would modify the program. [...] Intermediate forms such as the output of a preprocessor or translator are not allowed.

> https://opensource.org/osd

If I asked a programmer from OpenAI to modify the model to better support Japanese speakers from Hokkaido, their 'preferred form' of the model's source code would include the 680,000 hours of audio used to train the model.

Yes that means that there are almost no open source models and yes it's awesome that they released this and made the weights available. Just don't call it open source.

nl(10000) 6 days ago [-]

This isn't really true.

You can do a lot with weights and no training data - for example you can pull the end layer off it and use it as a feature extractor.

And to modify it for Japanese speakers you'd fine train the existing model on additional data. If you wanted to modify the model you can (sometimes, depending on what you want to do) modify an existing architecture by removing layers, adding replacements and fine tuning.

I don't quite know what the right analogy of trained data is. In many ways it is more valuable than the training data because the compute needed to generate it is significant. In other ways it is nice to be able to inspect the data.

> The source code must be the preferred form in which a programmer would modify the program.

As a machine learning programmer I'd much prefer the weights than the raw data. It's no realistic for me to use that training data in any way with any compute I have access to.

rvz(10000) 6 days ago [-]

Yes. It just like calling the release of compiled closed binary blobs as 'open source' even when the source of reproducing the compiled output is unavailable.

> If I asked a programmer from OpenAI to modify the model to better support Japanese speakers from Hokkaido, their 'preferred form' of the model's source code would include the 680,000 hours of audio used to train the model.

Precisely. These 'users' lifting the model can't do it themselves. You will still be contacting OpenAI for support or to add support for another language and they will be the ones able to modify the model.

> Just don't call it open source.

That is true, it is still closed source and already we are seeing the hype squad already apologising to OpenAI as they 'open sourced' a closed model that you can't modify yourself.

OpenAI is still business as usual and nothing has changed.

pabs3(10000) 6 days ago [-]

The Debian deep learning team's machine learning policy would call this a 'toxic candy' model:

https://salsa.debian.org/deeplearning-team/ml-policy

BTW, wouldn't you take the existing model and do additional Hokkaido Japanese speaker training on top of it, rather than retraining the model from scratch?

sergiotapia(10000) 6 days ago [-]

Does this work with multiple speakers?

I want to build a tool that takes a video and generates subtitles for it, then I want to index the subtitles and let people search for a specific quote to scrub to that part of the video using automatically generated urls.

This is for a specific fandom of a ton of content, lots of dirty audio mostly recorded in a gym setting with multiple people speaking.

867-5309(10000) 6 days ago [-]

pretty sure such a tool made HN front page a few months ago

samstave(10000) 6 days ago [-]

AI speech recognition FN scares the heck out of me...

for so many reasons.

But one that really pisses me off is not being able to turn it off on the iphone, and the fact that aside from 'hidden cameras in my airBnB' -- soon we will have to worry about secret listening machines EVERYWHERE

jfoster(10000) 6 days ago [-]

Also, based on their demo, this model seems like it might have comprehension well above the level of a typical human.

Anyway, it's out there now. No way to turn back.

ma2rten(10000) 6 days ago [-]

We will see an explosion of AI capabilities in the next couple of years. This will have a huge impact on our lives, much of it good but some of it also bad.

wongarsu(10000) 6 days ago [-]

'Secret listening machines everywhere' was a pretty big thing in East Germany. It's also the central theme of the movie The Lives of Others.

Of course, the ability to scale this more cheaply (throwing more compute at it, instead of more people) is somewhat scary, but it's not really introducing a new capability. Especially since you still have to do something with the transcript. An AirBnB landlord who reads the transcript of what you said could as well have listened to the recording.

aidenn0(10000) 6 days ago [-]

I just threw a random rock MP3 at it, and a first readthrough shows no transcription errors; this is quite good.

Now I just want OCR that's even 50% as good as this...

aidenn0(10000) 6 days ago [-]

Ran a few other songs through it and found one obvious mistranscription:

'He's the bedroom cosmic rocker' (should be 'He's the veteran cosmic rocker' in Veteran Cosmic Rocker by The Moody Blues)

I also noticed that it's a little on the conservative side for detecting speech; all songs were missing at least part of one line.

sjsdaiuasgdia(10000) 5 days ago [-]

I was comparing a batch of transcriptions between these models and vosk, and noticed that the medium.en model produces some weird results compared to the others. I've seen a number of loops with one word or a small sequence of words repeating several times. It seems more prone to output that reads like nonsense than the others.

More troubling is a short audio clip that got a few full sentences back, several times the text length that comes back from the other models or vosk. The content of the sentences is extremely far from the audio content. The best alignment I can find is the first word of medium.en's interpretation is somewhat phonetically similar to the audio.

The small.en model doesn't show these behaviors, at least in this data set.

nshm(10000) 5 days ago [-]

The whole value of this model is in 680 000 hours of training data and to reuse this value you need large model, not smaller ones. Smaller versions just don't have enough capacity to represent training data properly.

powera(10000) 6 days ago [-]

My first take: it is slow.

The 'base' model (supposedly 16x faster than the large one) takes more than the audiofile playback time on my machine to do transcriptions.

fitznd(10000) 5 days ago [-]

I'm seeing even worse. On my M1 Max 2021 macbook pro, I tried transcribing a 30 minute video file and left it on overnight and it was only half way through. I feel like something could be wrong with my setup but I'm only using the defaults.

StevenWaterman(10000) 6 days ago [-]

That example at the top of the page (speed talking) blew me away. He started talking, I was stunned for a minute, then realised yes, it really was English, and I just burst out laughing.

That's so, so far beyond the previous state-of-the-art, it's absurd.

NaturalPhallacy(10000) 6 days ago [-]

It's a micromachines ad from the '80s. He talked like that in all of them!

As for speed, to a computer we don't talk very fast, not even that guy.

I wonder if it could handle Rap God by Eminem....Let's find out!

toss1(10000) 6 days ago [-]

Like every model I've seen there is something like this:

>>A decoder is trained to predict the corresponding text...

Prediction of expected text in the context of the previous text.

While this is valuable in casual transcription, it can be extremely dangerous in serious contexts.

From personal experience, having given a deposition with an 'AI' transcription, it will literally reverse the meanings of sentences.

This is because it produces the EXPECTED output in a context, and NOT THE ACTUAL OUTPUT.

Like a speaker that clips the output, these types of systems 'clip' the really valuable information out of a transcription. Worse yet, this is a completely silent failure, as the transcript LOOKS really good.

Basic info theory shows that there is more information contained in 'surprising' chunks of data than in expected ones. These systems actively work to substitute 'expected' speech to overwrite 'surprising' speech.

The transcript I got was utter trash, multiple pages of errata I had to submit when the normal is a couple of lines. And as I said, some literally reversed the meaning in a consequential way, and yet completely silently.

This kind of silent active failure mode is terrifying. Unless it is solved, and I see no way to solve it without removing ALL predictive algos from the system, these types of systems must not be used in any situation of serious consequence, at least not without real redundancy and backup.

lunixbochs(10000) 6 days ago [-]

Do you have a demo audio clip for this? I'd be interested to see how it looks in practice.

Tomis02(10000) 6 days ago [-]

I've been saying this for years. Current 'AI' algorithm are fundamentally flawed because they rely on a statistical approach. This works moderately well for some use cases but it will rarely give you 100% confidence. Good luck with self-flying planes or self-running nuclear power plants.

hijp(10000) 6 days ago [-]

Anyone get it running on m1 mac?

I keep getting `ModuleNotFoundError: No module named 'setuptools.command.build'`

simmanian(10000) 5 days ago [-]

I got it working inside a docker container on my M1 MBP. FWIW, I'm having my $180 tinyminimicro PC run a translation task while my M1 MBP runs a transcription task with the same audio input. So far, the PC is actually outputting results a lot faster than the MBP. Interesting results.

kif(10000) 6 days ago [-]

I got requirements installed, but then when running the Python example, I get:

RuntimeError: 'slow_conv2d_cpu' not implemented for 'Half'

dceddia(10000) 6 days ago [-]

Yep, I had this too. `pip3 install -U pip setuptools` took care of it. (If you get an error about pip3, try `pip` instead)

Smaug123(10000) 6 days ago [-]

I'm still not successfully using the GPU, but it's working decently quickly (with the base model - it's incredibly slow to use the Large model) using just the CPU. I'm going to have to check what magic stable-diffusion is doing to enable the GPU :(

nik_s(10000) 6 days ago [-]

I just tested the model [1] using an RTX3090, trying to translate a french text I found here [2].

Some observations:

- The full translation of the 6:22 minute video takes about 22 seconds (17x real time)

- It recognizes the language by default (and did a good job to recognize it was french audio)

- MIT License [3]!

- The quality of the transcription is good, but not perfect.

- The quality of the translation (if you don't consider transcription errors as a translation error) is generally very good.

---

The transcription:

> Bonjour à tous, <error>j'suis</error> espère que vous allez bien, c''est ENTI. Et aujourd', <error>aujourd',</error> on se retrouve <error>un peu physique</error> pour parler de la termo dynamique. Vous ne vous inquiétez pas, ça va bien se passer. On va y aller ensemble, <error>être à par exemple,</error> je vous accompagne à travers une série de vidéos pour vous expliquer les principes de base en termo dynamique. Et bah, c''est parti, on va y aller tranquillement. Lidée, c''est vous puissiez comprendre la termo dynamique dans son ensemble. Donc, je vais vraiment prendre mon temps pour <error>couplisser</error> bien comprendre les notions,

The translation:

> Hello everyone, I hope you're doing well, it's NT and today we find ourselves a little physical to talk about the thermo dynamic. Don't worry, it's going well, we're going to go together and be the same. I'm going to accompany you through a series of videos to explain the basic principles in thermo dynamic. Well, let's go, <error>we're going to go quietly</error>. The idea is that you can understand the thermo dynamic <error>in sound together</error>. So I'm really going to take my time to understand the notions,

---

All in all very happy that OpenAI is publishing their models. If Stable Diffusion is any guide, people will hack some crazy things with this.

[1] https://github.com/openai/whisper [2] https://www.youtube.com/watch?v=OFLt-KL0K7Y [3] https://github.com/openai/whisper/blob/main/LICENSE

seszett(10000) 6 days ago [-]

> dans son ensemble

> in sound together

That's hilarious and honestly, incredibly bad. 'Dans son ensemble' is a very common idiom (meaning 'as a whole') while 'in sound together' has to be pretty rare. 'Son' means 'his/hers/its' as well as 'sound', and the former meaning is probably more common in general so I have no idea how this result could arise.

'Termo' also doesn't exist in French, it's 'thermo', so the transcript even makes orthographic errors.

And I forgot about 'couplisser' which is also a hilarious made-up word that sounds like it could mean something, but doesn't! Edit Google finds exactly one reference of this, in a patent with a typo on the word 'coulisser'.

I'm still impressed by the transcript quality since it covers many languages, but the translation part is quite poor.

StevenWaterman(10000) 6 days ago [-]

Was this with the `base` model? `large` is running ok on a P100 in colab, but is about 4% the speed of `base.en`. Certainly seems like some of these models will be fast enough for real-time.

NaturalPhallacy(10000) 6 days ago [-]

How did you get it to use the GPU?

I have it running right now and it's not touching the GPU.

joshcryer(10000) 6 days ago [-]

It also runs well on a CPU and seems to have proper memory management. Wonderful timing because I was using DeepSpeech for some audio recordings and it required me to script up a splitter to make the files into .wav and then do snippets of 10 seconds each. Everything about this just works out of the box. On a core i5 I'm getting about 30 seconds every minute. Transcriptionist jobs just turned into editor jobs. I love how it drops the inflections in the audio as well, because it was trained on transcription work, and that is one of the first things you learn to do (drop the uhs and ums and huhs etc, unless it is a strictly verbose transcription).

solarmist(10000) 6 days ago [-]

Is it translation or transcription? Or both?

Both, wow. This is really interesting.

gok(10000) 6 days ago [-]

Comparing this model's word error rates to the state of the art [1] on a few common test sets:

                           Whisper    SoTA
  LibriSpeech test-clean      2.7%     1.8%
  LibriSpeech test-other      5.6%     2.9%
  Switchboard                13.1%     4.9%
  CallHome                   15.8%     9.5%
The authors do explicitly state that they're trying to do a lot of fancy new stuff here, like be multilingual, rather than pursuing just accuracy.

[1] https://github.com/syhw/wer_are_we

lunixbochs(10000) 6 days ago [-]

I suspect Whisper is more robust than other 'SOTA' models, but this release is likely leaving a fair bit of accuracy on the table considering the amount of resources OpenAI is capable of throwing at training it.

Comparing the readily available test sets from the paper to some of my personal robust models (for the Talon models, this is greedy decoding, no language model):

                       Talon  Talon  Talon  Whisper  wav2vec 2.0
                       28M    300M   1B     Large    960h
    librispeech clean   3.21   2.52   2.40   2.7      2.7
    librispeech other   8.21   6.56   5.63   5.6      6.2
    common voice       13.88  11.65   8.86   9.5     29.9
    tedlium             7.51   6.55   5.47   4.0     10.5
I have a battery of more difficult tests on hand (including adversarial tests, and diverse accent-specific metrics). I'll look at running these tests on each of the Whisper model sizes and following up with a larger comparison.
StevenWaterman(10000) 6 days ago [-]

One of the things they point out is that the SoTA on e.g. LibriSpeech is only good at LibriSpeech, and doesn't generalise as well.

> Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper's zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.

liminalsunset(10000) 6 days ago [-]

I really wish I had this about half a year ago when I was building a tool to automatically turn online school lectures into searchable, clickable transcripts (kind of like YouTube or EdX transcripts).

I was originally using Adobe Premiere Pro's speech to text to do it, and wrote Python to convert its output to the Hyperaudio format on GitHub. With this, I can totally skip all of that step and this is fully open source, too.

App idea:

Build an app that takes a video and uses Hyperaudio or a similar project to add a clickable and searchable transcript (clicking in transcript seeks video)

resoluteteeth(10000) 6 days ago [-]

You could already do the speech recognition in a fully open source way with vosk easily, although Whisper may be more accurate

james-revisoai(10000) about 13 hours ago [-]

You still interested in this? I'd be keen to chat to you, worked on a searchable transcript provider for educational youtube videos (likewise, unfortunately pre-whisper, so I did a lot of work with sentence completion perplexity and rpunct to try and improve transcript quality from youtube automatic transcriptions). Can be contacted at revision.ai and demo what we were able to do till now, would be great to hear your thoughts.

synergy20(10000) 5 days ago [-]

is there a high quality text to speech equivalent project like this?

bergenty(10000) 5 days ago [-]

Seriously, when I first landed on the page without reading anything else I thought it was text to speech with the "micro machine" example and I was floored. The speech to text is obviously mind blowing too.

throwamon(10000) 6 days ago [-]

Is it feasible to use this for Talon-like voice-driven computer usage?

lunixbochs(10000) 6 days ago [-]

If the Whisper models provide any benefits over the existing Talon models, and if it's possible to achieve any kind of reasonable interactive performance, I will likely integrate Whisper models into Talon.

Talon's speech engine backend is modular, with Dragon, Vosk, the WebSpeech API, and Talon's own engine all used in different ways by users.

FloatArtifact(10000) 6 days ago [-]

Maybe, a number of speech recognition engines have been integrated into https://github.com/dictation-toolbox/dragonfly

emcq(10000) 6 days ago [-]

Be wary of using this model - the licensing of this model seems sketchy. Several of the datasets used for training like WSJ and TED-LIUM have clear non-commercial clauses. I'm not a lawyer but releasing a model as 'MIT' seems dubious, and hopefully OpenAI has paid for the appropriate licenses during training as they are no longer a research-only non profit.

pabs3(10000) 5 days ago [-]

Are there any AI/ML models that don't use sketchy licensed datasets? Everything seems to be 'downloaded from the internet, no license' or more explicitly proprietary. The only exception I can think of would be coqui/DeepSpeech?

nshm(10000) 6 days ago [-]

I think they didn't use WSJ for training, only for evaluation. Paper includes WSJ under 'Evaluation datasets'

jefftk(10000) 6 days ago [-]

This is a big dispute right now: OpenAI and other AI companies generally take the position that models learning from data does not make the output of the models a derivative work of that data. For example, GitHub Co-pilot uses all publicly available GitHub code regardless of license, and DALLE-2/StableDiffusion/etc use lots of non-free images. I don't think this has been challenged in court yet, and I'm very curious to see what happens when it is.

darkpicnic(10000) 6 days ago [-]

I just wrote a script with Hazel to automatically transcribe my voice notes to txt. It handles punctuation extremely well. What a wonderful contribution!

pbassham(10000) 5 days ago [-]

Exactly what I was planning to do! Want to share yours with me?

harry8(10000) 6 days ago [-]

Can you plug this into a computer on your premises to get speech recognition without amazon, apple or google's cloud (or any other cloud) involvement?

Right now I decline all speed recognition because I don't want orwellian listening devices in my house or pocket and haven't seen an answer. (Also haven't been too bothered about speech command interfaces to bother with a load of research - lazy me).

fragmede(10000) 6 days ago [-]

Yes, after the download of the model weights (from https://openaipublic.azureedge.net/) it's an entirely offline process.

abidlabs(10000) 6 days ago [-]

Here [1] is a video tutorial on building a web UI that accepts microphone input and runs it through Whisper for speech transcription

[1] https://www.youtube.com/watch?v=ywIyc8l1K1Q&ab_channel=1litt...

amrrs(10000) 6 days ago [-]

Thank you for sharing!





Historical Discussions: Nightdrive (September 23, 2022: 1433 points)
Nightdrive (September 21, 2022: 2 points)

(1435) Nightdrive

1435 points 5 days ago by GeorgeHahn in 10000th position

incoherency.co.uk | Estimated reading time – 10 minutes | comments | anchor

Nightdrive

Fri 16 September 2022 Tagged: software

I've made a JavaScript simulation of driving at night time on the motorway. It's hard to classify what it is. It's not a video, because it's generated dynamically. It's not a game, because you just watch. It's not a screensaver, because it's not the 90s. Maybe it's a 'demo'?

This is something I've been planning to make for years, but kept forgetting about. I would only remember it when in the passenger seat of a car on the motorway at night time, fascinated by the patterns of moving lights.

Here's a small embedded version, if your browser runs JavaScript:

For the full version (with music!) click here: Nightdrive.

Gina said it's a great way to find out how filthy your monitor is.

The entire scene is created purely by drawing circles on a HTML5 canvas. It actually works a lot better than I expected. The biggest flaw is that the cars are totally transparent, so you can see lights from distant cars even where they should be occluded by closer cars.

Motorway simulation

The core simulation is entirely 2-dimensional. The coordinate space looks like this:

The motorway is always parallel to the y-axis.

Each car has a 2d position vector and a y-axis velocity (cars on our side having positive velocity, and cars in the oncoming carriageway having negative velocity).

400 cars are positioned at 25 metre intervals, in random lanes, at initialisation time. They also have some slight variation in the size, position, and colour of the lights. Cars in the left-most lane travel at a constant simulated speed of about 40mph, in the middle lane at 70mph, and in the fast lane at 100mph.

When any car is 'behind' the viewer (lower y coordinate), and with a lower velocity (will get further behind) it is wrapped forwards by 10km. Similarly, when any car is 10km 'in front of' the viewer (higher y coordinate), and with a higher velocity (will get further ahead), it is wrapped backwards by 10km.

That's enough to give the illusion of an infinite road with infinitely-many cars, travelling in both directions, and at different speeds in different lanes.

Projection

A top-down view of a 2d world isn't the effect we're going for, so we need to first make our world coordinates 3-dimensional, and then project them on to the 2-dimensional screen.

To make the coordinates 3-dimensional, I just add a 'height' to every point in the 2d simulation. This works because all of the elements I need are at constant heights above the ground: car headlights, cat's eyes, and street lights.

My projection into screen space is not necessarily mathematically sound. I just fiddled with it until it looked how I wanted.

To turn a point from world coordinates into screen coordinates we first subtract the observer's position, to get a position (x,y,z) relative to the observer.

If the relative position vector has a negative y coordinate, then we know the object is behind the viewer and therefore not visible, so we don't draw anything.

Otherwise, we find the distance to the object by taking the magnitude of the relative position vector:

dist = sqrt(x2 + y2 + z2)

Then our coordinates in screen space are:

xscreen = xcentre + k * x / dist yscreen = ycentre - k * z / dist

Where k is a scale factor that sets the size on the screen, and (xcentre, ycentre) is the centre of the screen.

(We need to subtract for yscreen because in screen space positive y is down, but in our world space positive z is up).

I found it actually worked better if I omitted the x offset from the distance calculation (so just sqrt(y2 + z2)), otherwise objects near the edge of the field of view were weirdly distorted. But I wouldn't suggest doing that in the general case.

You can look at the actual projection function I used if you like.

Terrain

To make the road a bit more interesting we can add some hills. I wrote a function adding up some arbitrary sine waves to make something that seemed reasonable:

function terrain(y) {
    return 10*Math.sin(y/1000) + 5*Math.cos(y/527) + 2*Math.sin(y/219);
}

This takes a y coordinate in road space, and returns the height of the road at that point. This is then added to all z coordinates by the projection function.

Looking at the road from the side, the terrain it generates looks something like this:

(x on the graph is y in world-space, y on the graph is z in world-space, both in metres).

Road occlusion

At this point we have quite a glaring issue: we can see through the ground!

Everything the other side of the crest of the hill should be occluded by the hill, but since all we're doing is projecting positions of lights, nothing can ever get occluded by anything.

A point P is occluded by the road if there is any position, between the viewer and P, that has a screen-projected road height 'higher' (smaller value) than the screen-projected y coordinate of P. Because that means if we were to actually draw the road, it would get drawn over the top of P.

I sorted all rendered lights by distance from the viewer (closest first), and walked along the list, keeping track of the highest screen-space road height so far encountered, and occluding any points that fall 'below' that height. That means we'll occlude any points that fall behind the road position at any closer position that has a light on it. This is good enough, because we always have frequent-enough sampling of the terrain function due to the density of cars and street lights.

You can look at the implementation if you like.

Come to think of it, I expect there might be an analytical solution to determining whether any given point should be occluded by the road. Maybe that would be better! We just need to find out whether the straight line passing through the camera and P intersects the road at any point between the camera and P: if it does then P is occluded by the road.

Music

The background music is from this video which describes it as 'Royalty Free'.

I've been told that the audio quality on Nightdrive is bad. I did this on purpose to reduce the size of the mp3, and it doesn't sound bad to me. If you think it sounds bad then you should mute Nightdrive and play that YouTube video in a different tab.

More

There are a few more things that I think would be fun to do:

Bends: Instead of having a perfectly-straight road, it would be good to add some bends, so the patterns of lights would curve off into the distance, something like this:

(From jonnywalker on flickr)

Street light occlusion: Street lights aren't just magical floating orbs, they're held up by metal poles. When something passes behind the metal pole, it is momentarily invisible. This would not be too hard to do in Nightdrive. We'd just use a similar process to the 'road occlusion' to hide lights that are hidden by the poles.

Car occlusion: Cars also are not just magical floating orbs, they have volume and are opaque. This would be slightly harder than street light occlusion. Probably a first pass would be to render a black cuboid behind each car's lights, so that the cuboid blocks out anything that would be blocked by the car. This is inconvenient because currently we can only render circles.

Speed variation: Not every car in the same lane drives at exactly the same speed. Sometimes you catch up to the car in front, or the car behind catches up to you. Adding speed variation is easy, but comes with the problem that cars end up driving through each other, which for some reason feels unrealistic. This brings us on to...

Lane changes: When a car catches up to the car in front, it should indicate right and move over to overtake. When it has passed and there's a space on the left it should indicate left and move back over. We could give each car a sort of 'personality' which tells it how close it can get to other cars, how long it should indicate for before changing lanes, etc.

Rain shader: A rain effect might be too much trouble to do with just a canvas. Maybe it would need WebGL. But I would like to apply some sort of rain shader to the rendered image to give the effect of driving at night time. We can imagine having a collection of 'droplets' on the 'windscreen' which refract the light passing through them (dots that distort the pixels underneath). Periodically we could sweep a windscreen wiper across the field of view to clear the dots, and we can randomly add dots at some rate based on how heavily it's raining.

Game: Finally, I wonder how we could turn this into some sort of game? I'm especially not looking to make a driving game. I still want the car to basically drive itself. What game can we make where the premise is that you're a passenger on the motorway at night time? It shouldn't be a particularly taxing game, I think the main experience should still be that you're just enjoying watching the lights, but it would be cool if there was some interactivity and some sort of goal.

If you like my blog, please consider subscribing to the RSS feed or the mailing list:




All Comments: [-] | anchor

willhinsa(10000) 5 days ago [-]

This reminds me of the movie Drive (2011), in particular the first song on the soundtrack, Kavinsky - Nightcall

https://youtu.be/MV_3Dpw-BRY

jgwil2(10000) 5 days ago [-]

Seems to be a popular aesthetic for synthwave compilations on YouTube: https://www.youtube.com/watch?v=ICcFMBzOnYs

I wonder if Drive originated this aesthetic or if it's just coincidence.

pcthrowaway(10000) 5 days ago [-]

I was thinking of this song exactly before I turned on the music; thought it might even be the same song for a second. I wonder why the (visual) aesthetic is evocative of this song?

aleksiy123(10000) 4 days ago [-]

Real human bean. Love this movie.

keyle(10000) 5 days ago [-]

Wow Drive is from 2011? Man... I feel like it's a new movie. I loved it, maybe that's why.

jcynix(10000) 5 days ago [-]

Very nice! I instantly recognized the tune but from a completely different event, namely

https://youtu.be/wkF9w86XXKU

That's the cover by band London Grammar and TIL about the original one. Didn't know about the movie which now is on my todo list, thanks.

ElectroSlayer(10000) 5 days ago [-]

Great song! I personally hear this track from Kavinsky when watching that demo: https://www.youtube.com/watch?v=ErQNRwH-Hmk

7373737373(10000) 5 days ago [-]

I wanted to do the same for the train feeling: https://qewasd.com/train

TOGoS(10000) 4 days ago [-]

Hmm, some of those trees appear based on parallax to be farther away than the shoreline that they're painted over. Otherwise, pretty nice.

franciscop(10000) 5 days ago [-]

> 'It's hard to classify what it is [...] Maybe it's a 'demo'?'

Exactly, those are demos, and while I believe it's slightly different there's a whole culture around it that I've never been aware until recently called 'demoscene'! I even have a small 'Demos' section on my website with a bunch of those, not to the same level of quality though since for me it'd be a bit more like 'self-contained small experiments that resulted in something cool so I put it together as a demo':

- 'Zoom', hyperdrive-like effect: https://francisco.io/demo/zoom/

- 'Tree generation', specify a JSON/HTML structure and it'll generate a tree: https://francisco.io/demo/tree/ (disc: it was inspired by a broken demo I saw from someone else)

- 'Stereo Depth', calculate depth of a couple of stereo images using JS: https://francisco.io/demo/depth/

- 'Terminal player', specify a bunch of commands in plain text and they'll be 'played' like a video: https://francisco.io/demo/terminal/

Ruarl(10000) 5 days ago [-]

I would have classified it as a 'simulation'. Which is what the author calls it, but then tries to find other things to call it too. Simulation is fine.

bodge5000(10000) 5 days ago [-]

Im not sure the cars and lights need to be anymore than floating orbs, its quite a nice minimalist aesthetic with just the lights. I'm not sure it needs to be a game either, maybe the ability to steer the car would be good but any more than that might seem forced (and I'm not even sure it needs that). The only real problem I see is that cars pass through you from behind (the only time the floating orbs effect becomes an issue), but even that seems minor for what it is.

jstanley(10000) 5 days ago [-]

Yes, the 'self-driving cars' logic needs a lot of work. That's the part I am least fond of currently.

rezmason(10000) 5 days ago [-]

I'm amused that we share initials and a passion for this kind of project! Mine's https://rezmason.github.io/drivey .

One key difference: this guy built his demo from scratch, whereas mine's a port of someone else's work. It's great to see another implementation, with its own techniques and features.

JKCalhoun(10000) 5 days ago [-]

Was yours based on 'Drivey'?

EDIT: never mind, it looks like it was, very cool.

zxcvbn4038(10000) 4 days ago [-]

That is amazing, I remember seeing an early version of this on Windows, its so great to see it revived and with source!

mhd(10000) 5 days ago [-]

Oh, I think I remember the original Drivey and was following the dev blog for a while back then. But I think in the end, it didn't go anywhere (no pun intended).

smrtinsert(10000) 5 days ago [-]

This is beautiful.

zulu-inuoe(10000) 5 days ago [-]

That's really cool, but I wonder why on my machine is maxes out all the cores on my CPU and still runs at something like 1 FPS on high If I turn it to Low it runs okay but looks.. not great.

But I don't see anyone else having issues so, is it just me?

tomcam(10000) 5 days ago [-]

Gorgeous and hypnotic.

zxcvbn4038(10000) 4 days ago [-]

That is amazing, I remember seeing an early version of this on Windows and being impressed. It is great to see it revived and with source!

breck(10000) 5 days ago [-]

Wow that's gorgeous. Is there a link to the source?

flanbiscuit(10000) 5 days ago [-]

interesting that this is running very slow (like 1 frame per second) on Chrome, but runs very smooth on Firefox. I'm on the latest version of both for Mac.

sprkwd(10000) 4 days ago [-]

Yeah. That is awesome.

syndic8_xyz(10000) 5 days ago [-]

This person is really incredible. Just check out their work: https://www.rezmason.net/projects.html

jmiskovic(10000) 5 days ago [-]

Amazing to see you in comments; your version was inspiration for this project: https://twitter.com/j_miskov/status/1490358867104083972

The video didn't generate much interest so unfortunately I didn't release anything playable.

archon810(10000) 4 days ago [-]

Hmm, nothing happens for me on mobile (Chrome, Pixel 6 Pro). Just a black screen with some controls on the bottom.

justinlloyd(10000) 5 days ago [-]

I do not like how your city planners put pedestrian crossings on blind curves. That's really bad design that could lead to accidents.

P.S. It's a joke. This is a really well executed demo.

jstanley(10000) 5 days ago [-]

Neat! I like that this is kind of the opposite effect (obviously Drivey is way more polished).

Drivey shows objects as their silhouettes on top of a light background, whereas Nightdrive shows objects as their lights on top of a dark background.

latchkey(10000) 5 days ago [-]

Great work on this though! Super impressed. Would be fun to clean it up and convert to typescript.

neilparikh(10000) 5 days ago [-]

Would be neat to have eye gaze be literally controlled by where you look using webcam tracking! Not sure how accurate it would be though.

phist_mcgee(10000) 5 days ago [-]

As a fan of vaporwave this is

aesthetic 可ぷき

aerovistae(10000) 5 days ago [-]

> The biggest flaw is that the cars are totally transparent,

This seems easily fixable, no? Just make cars opaque black and assign progressively decreasing z-index to each spawning light. (Unlikely anyone will leave page open long enough to reach min value, and you could just reset at that point.)

rezmason(10000) 5 days ago [-]

Someone with your level of optimism has an advantage.

The tricky thing is, this project's renderer is currently a queue of circles to draw to a canvas. It's under 100 lines of JavaScript. So any increase in complexity will require substantial changes, like abstracting over types of drawables.

fiat_fandango(10000) 4 days ago [-]

Does anyone have good resources for creating 'generative assets' like this?

I have a decent handle on js but I have no idea where to start in terms of tooling. For now, I've been trying to build basic eye tracking to apply a 'block' of color over eyes with an animation in said block on top of a real-time video stream.

Elsewhere, automating mouth animations from audio / even moving a character in a random motion when audio input is provided.

Any advice where to start would be greatly appreciated!

BasilPH(10000) 4 days ago [-]

I've done fun things with p5.js, which might have the primitives you need to do what you described.

danparsonson(10000) 5 days ago [-]

Related: tutorial for writing something similar with ShaderToy - https://youtube.com/playlist?list=PLGmrMu-Iwbgs0H9va0DlopOGy...

mft_(10000) 5 days ago [-]

That was that I immediately thought of, but the shadertoy link is probably more gratifying: https://www.shadertoy.com/view/MdfBRX

altacc(10000) 5 days ago [-]

So realistic that it hogs the middle lane when it could move to the left ;)

AnthonyUK(10000) 5 days ago [-]

The red line indicates the hard shoulder but yes, even after the law change here in UK I have not seen much actual change on the road tbh regarding middle-lane hogging.

People are either ignorant, lazy or don't care as there is very little chance of being caught.

jcynix(10000) 5 days ago [-]

That might be realistic, cf. a female's view of the autobahn:

https://ww3.cad.de/foren/ubb/uploads/NoIdea/perscheid32.jpg

by the late cartoonist Martin Perscheid.

What irritated me a bit instead, where the cars passing on the right side, until I realized that it was made by someone from UK ;-)

chrisco255(10000) 5 days ago [-]

> What game can we make where the premise is that you're a passenger on the motorway at night time? It shouldn't be a particularly taxing game, I think the main experience should still be that you're just enjoying watching the lights, but it would be cool if there was some interactivity and some sort of goal.

If you wanted to make it interactive, maybe a Pokemon Snap like functionality where you try to capture photos of random environment features or creatures.

justinlloyd(10000) 5 days ago [-]

A man running alongside your car window, jumping over obstacles. Technically this game already exists, but it would be interesting to combine with this demo.

toss1(10000) 5 days ago [-]

Very cool. One 'flaw' that immediately strikes me as cool is that the cars are transparent — so, for instance, when a car passes you between you and the oncoming traffic, you see the taillights, but also the oncoming headlights/streetlights are not obscured by the car. So it's sort of disembodied headlights and taillights, which makes the effect even cooler... fun!

skrebbel(10000) 5 days ago [-]

Tbf I think that's the point

masswerk(10000) 5 days ago [-]

BTW, while we may think that driving is a universal thing, I instantly found it amazing how North American this is. From the integrated indicators, to cars passing on your right, to the road lights and even how the road undulates. Having said that, great project!

Edit: Well I got this entirely wrong and missed that this was meant to be lefthand traffic. (I caught a segment that looked more like there was an independent road passing an interstate crossing a city, rather than being opposite lanes.)

jstanley(10000) 5 days ago [-]

What do you mean by 'integrated indicators'? Don't cars have integrated indicators all over the world?

berkut(10000) 5 days ago [-]

Erm, the cars are driving on the left (red taillights on left side of motorway, white headlights on right)... wouldn't North America be the other way around, with cars overtaking on the left?

mbrameld(10000) 5 days ago [-]

I think it's from the UK or somewhere else that they drive on the left. Opposing traffic is on the right.

hugginn(10000) 5 days ago [-]

Isn't this left hand traffic though? As in, not what you'd find in America? It feels very Swedish to me, except for the traffic direction.

pavlov(10000) 5 days ago [-]

> 'Maybe it's a 'demo'?'

There's a long-standing tradition of people creating realtime graphics software on personal computers that doesn't offer any interactivity. They are indeed called demos and the community is called the demoscene. It goes back to the late 1980s and had a golden era in the mid-90s on Amiga and PC. (Pre-Internet, watching and making demos was one of the few socially and creatively oriented things you could do for free on a home computer.)

To be pedantic, this isn't a full-blown demo. Small demos are called intros, and a category of them is the size-constrained intro (e.g. 1kB or 4kB). So this could be either a small intro or an effect as part of a bigger demo.

It's worth looking up some of the small intros. People can squeeze entire GPU-raytraced universes with music into a few kilobytes.

danjoredd(10000) 4 days ago [-]

Man...I need to get into the Demo scene. I have been disillusioned with the tech world because of just how utilitarian everything feels. I have been looking for something like this forever

skrebbel(10000) 5 days ago [-]

IMO the author could very much submit this to a demoparty in the 'demo' category (assuming they'd substitute the music with a fitting original work).

These days, the only notable difference between demos and intros is file size. If it's bigger than 64kb, it's a demo. You can totally have a short, single-effect demo like this one, and there's plenty such demos out there.

Ergo, I disagree, I think this counts as 'full-blown' for any reasonable current definition.

soham(10000) 5 days ago [-]

Would love to see some small intros. Any links/sources you may suggest?

rippercushions(10000) 5 days ago [-]

This reminds me of Desert Bus, the legendary/notorious driving game by Penn & Teller. During the second half of the game, you're driving at night and the view is not dissimilar.

https://www.newyorker.com/tech/annals-of-technology/desert-b...

Of course, because Desert Bus is literally the worst game ever, the steering continually veers to the side and you have to keep nudging the wheel or you'll crash.

Mountain_Skies(10000) 4 days ago [-]

Wonder if anyone has built an AI to drive the bus and get that sweet single point.

rgovostes(10000) 5 days ago [-]

See also Martijn Steinrucken aka BigWings's The Drive Home:

ShaderToy: https://www.shadertoy.com/view/MdfBRX [WebGL]

Video: https://www.youtube.com/watch?v=WrxZ4AZPdOQ / Making of: https://www.youtube.com/watch?v=eKtsY7hYTPg

jstanley(10000) 5 days ago [-]

This is fantastic, thanks for sharing!

garaetjjte(10000) 4 days ago [-]

I feel this isn't complete without mesmerizing shadow patterns from streetlights on dashboard.

system2(10000) 4 days ago [-]

If he spends more time on this project I bet he will turn it into a full-fledged pc game.

stevenhubertron(10000) 5 days ago [-]

Very cool. Of course the music is retrowave.

Insanity(10000) 5 days ago [-]

Indeed, I very much enjoy the genre.

https://en.wikipedia.org/wiki/Synthwave

teddyh(10000) 4 days ago [-]

Reminds me very much of Night Driver, the arcade game from 1976:

https://archive.org/details/arcade_nitedrvr

chclau(10000) 4 days ago [-]

That's also the first thing I thought when I saw the title

imknewhere(10000) 4 days ago [-]

I remember playing this game one time when I was a kid! I still think about it, from time to time, for some reason.

I found the entire experience so disorienting. I remember my dad laughing at me, I did so terribly at it (TBF I was probably 6 or 7)

Keyb0ardWarr10r(10000) 5 days ago [-]

Please allow the lanes to be reversed, we drive on the right side

tjungblut(10000) 5 days ago [-]

somebody already opened a PR for that: https://github.com/jes/nightdrive/pull/2

spookierookie(10000) 5 days ago [-]

Saw it for a min. Nice.

Then I bumped into the car in front of me which was changing lanes. Not nice.

jstanley(10000) 5 days ago [-]

Yes, sorry about that. The lane switch planning is incredibly poor!

davidkunz(10000) 5 days ago [-]

As a German, it would have been more relaxing if cars on the left were faster.

Mountain_Skies(10000) 4 days ago [-]

That was my first thought too. I was going to post 'Tsk tsk, going slow in the middle lane, making people pass you on the right' but then noticed it was in the UK.

mortenjorck(10000) 5 days ago [-]

> It's not a screensaver, because it's not the 90s.

It may be 2022, but I would love a screensaver version of this.

jcynix(10000) 5 days ago [-]

It would definitely fit into xscreensaver's collection, yes.

jordemort(10000) 5 days ago [-]

I love this. I'd love an idealized version where nobody ever passes on the right even more.

gavmor(10000) 5 days ago [-]

Notice oncoming traffic is also on the right? It's UK traffic. For peace of mind, try tossing a `transform: scaleX(-1)` on the <canvas /> element.

svdr(10000) 5 days ago [-]

Is it legal to overtake on the right in the US? It's not in the country where I live (NL).

blauditore(10000) 5 days ago [-]

This was made in the UK, so overtaking on the right is the proper way. Note the opposite traffing passing on the right too.

93po(10000) 5 days ago [-]

Yes, you'd basically be stuck going well under the speed limit otherwise, because you will 100% always find someone going really slow in the left lane and oblivious to blocking tons of traffic. Even when there's sign after sign saying 'Unlawful to use left lane unless passing'

caeril(10000) 5 days ago [-]

> It's not a screensaver, because it's not the 90s

What's funny is that screensavers may end up coming back into vogue if OLED displays continue to have burn-in issues. Ray-traced flying toasters may be in our future.

danjoredd(10000) 5 days ago [-]

Im into it

warmuuh(10000) 5 days ago [-]

As German, the thing that annoys me after 5 seconds is that the driver is overtaken on the right side and does so himself ... Hate it

bnegreve(10000) 5 days ago [-]

The driver drives on the left, so over taking on the right is the proper way to do it.

kleiba(10000) 5 days ago [-]

I wouldn't use the word 'hate' but I also find it irritating, even though I'm not German.

Admit it, though: you dislike even more that in that app, you're not zooming by everyone else on the left-most lane, right? ;-)

Lukesys(10000) 4 days ago [-]

This gentleman made a great web game for the UK TV show, Countdown. Been using it for years...

https://incoherency.co.uk/countdown/practice/

wging(10000) 4 days ago [-]

Also more recently, he made Protohackers! https://protohackers.com/

BigJono(10000) 5 days ago [-]

Love it.

Might be a good thread to ask. A few years ago I found a couple of sites that were like shadertoy but for 2D canvas shit like this. One I think was codegolf.tk (which appears to have disappeared), and I can't for the life of me remember what the other was.

Does anyone know of any such sites?

aeno(10000) 5 days ago [-]

https://www.dwitter.net/ comes to mind

delronde(10000) 5 days ago [-]

> It's hard to classify what it is.

It's generative art :)

You should launch it on https://www.fxhash.xyz/

You might find these interesting:

https://lunarean.substack.com/p/solace

https://mirror.xyz/0xF7E15015D31e1Be374c21E6F1dE91147C8B5db8...

jstanley(10000) 5 days ago [-]

I am a cryptocurrency fan but never understood NFTs.

Can you please explain what anyone would get out of buying an NFT as opposed to just looking at the web page for free whenever they want?

Not trying to be dismissive, just trying to understand.

btbuildem(10000) 5 days ago [-]

I'd call this a 'demo' for sure -- harkening back to the mid-90's when people built things for fun / just because they could / to explore

skrebbel(10000) 4 days ago [-]

People still do this.

ffhhj(10000) 5 days ago [-]

This reminded me of Night Driver for Apple II:

https://youtube.com/watch?v=l0nkMGyfYO8

mdswanson(10000) 5 days ago [-]

Me too...though in the arcade and on the 2600.

coldtea(10000) 5 days ago [-]

>The entire scene is created purely by drawing circles on a HTML5 canvas. It actually works a lot better than I expected. The biggest flaw is that the cars are totally transparent, so you can see lights from distant cars even where they should be occluded by closer cars.

Hmm, why not just draw a black rectange around any pair of car lights? It should work for givimg the impression of a solid car within the context of the video...

Edit: hmm, he does say this 'This would be slightly harder than street light occlusion. Probably a first pass would be to render a black cuboid behind each car's lights, so that the cuboid blocks out anything that would be blocked by the car.'

jetbooster(10000) 5 days ago [-]

I somehow feel the simplicity of everything only being paired lights giving an _impression_ of something somehow adds to the charm

zxspectrum1982(10000) 5 days ago [-]

Nice but I usually turn my lights on when I drive at night :-)

pixl97(10000) 4 days ago [-]

I mean, unless they are riding their brakes then their lights are on.... It just happens that all the cars have their front bulbs out.

aerovistae(10000) 5 days ago [-]

> if your browser runs JavaScript:

I don't understand this remark. What browsers do people use that don't have javascript?

b0afc375b5(10000) 5 days ago [-]

I think it just means 'if you have JavaScript enabled'. But to give a technical example, Lynx doesn't support JavaScript.

rezmason(10000) 5 days ago [-]

Some people run their browsers with JavaScript turned off. Some folks disable CSS. Some even browse via Lynx, a text web browser. It seems like the author's met some of these folks.

jstanley(10000) 5 days ago [-]

I don't like to presume :)

Bakary(10000) 5 days ago [-]

For most people, this will mean Ublock origin settings. For a smaller number of people, stripped down greybeard-approved browsers

mdtusz(10000) 5 days ago [-]

This reminds me of this video: https://youtu.be/WrxZ4AZPdOQ

He does incredible work with shaders and explains them in very clear detail.

ciaron(10000) 5 days ago [-]

I watched this video a couple of years ago and was blown away by what can be done with shaders.

tsuujin(10000) 5 days ago [-]

Who else grew up playing Night Driver on the Atari 2600?

pronlover723(10000) 5 days ago [-]
axus(10000) 4 days ago [-]

First thing I thought of when I saw the title. Was using the paddle control less or more frustrating than the typical game?

ASalazarMX(10000) 4 days ago [-]

I was pleasantly surprised the article actually had something resembling Night Driver. Such a simple concept, but it was a game that stimulated your imagination. That and star Riders were brilliant for their time.




(1377) Someone is pretending to be me

1377 points about 10 hours ago by iBotPeaches in 10000th position

connortumbleson.com | Estimated reading time – 17 minutes | comments | anchor

On September 14, 2022 I got an email from someone that reeked eerily of a scam, but it turned out to be quite real. The full text is below.

Hi Connor, A few days ago, this person named Maris [redacted] found me on GitHub and reached out and asked me to be his senior software engineer where my priorities will be communicating with clients. I thought it was a tiny bit strange since I'm a college junior having a hard time to land a SWE intern offer. But I took it, I knew what I was capable of.

Ah, this is what he was doing: He finds contracting positions, pretends to be a real developer with experience matching that position, and wants me to interview as that real developer. In this case, he found a Laravel/React contracting position, researched and decided to become you. Furthermore, he sets up a fake email similar to yours (as attached in the document) for communicating with the client.

I doubt this happens very often since he pretends to be different developers most suited for the job. But, perhaps you should let [redacted] know what's going on (the client).

Take care, Andrew

So I was about to toss this in the trash, but it had a word document attached. Thankfully I'm not sitting on a Windows machine and can just preview the document via Google without a fear of infecting myself.

So I previewed the document and it was scary. It was a document intended for someone to have a cheat sheet for an interview on how to act as me.

  • It included a subset of my personal information.
  • It included my education history.
  • It included my employment history.
  • It included my certifications.
  • It included a fake cover letter.
  • It included a fake email/address that was 'near' mine.
  • It included information about the company interviewing for.

The more I read this document - the more creepy it felt. Someone went to the effort to extract tons of true information from my available sources and build a fake profile of me.

  • They created an email that just tossed a 2 at the end of my legitimate one.
  • They picked an address of a house that is for sale in Tampa.

So I emailed the guy back that sent this to ask for more information. The exchange went like:

Me: This behavior seems odd - you are reached out by a party and asked to fake your identity for a job, which in this random case was me?
Them: Yup, and they asked me to fake it because they aren't that fluent in English. Doing a voice call interview with a client will certainly expose them.

So I had all information to join this interview that the fake Connor Tumbleson would have joined. So I did.


The Zoom Interview

I joined the Zoom call early and the client thankfully admitted me quickly. I explained as quick as possible that while I was the real Connor - I was not applying for this job.

I tried to explain how the individual who was hired to impersonate me had a great set of morals and forwarded this all to me. I was asking for all the information this employer had on the fake me when the story got even weirder.

Another Connor Tumbleson joined the call and was stuck in the meeting room. The gentleman of this company was frankly amazing and allowed me to change my name, turn off my video and tweak my avatar and stay on the call.

So the interviewer admitted entry to a new person who then spoke with an accent I could not place. The interviewer kicked off the interview and asked this individual to explain a bit about himself.

This imposter for the next 2 minutes proceeded to read the same document, that I was leaked, saying things word for word as well as saying his name was Connor Tumbleson. This was a different person hired to impersonate me than who reached out to me. My guess the original person declined and this just got passed to a different person.

I could not stand this anymore listening to someone legitimately claim they were me while using my photo, name and hard earned achievements. So I turned on my camera, renamed myself back and asked the individual what the hell they were doing.

The fake Connor Tumbleson immediately left the Zoom call.

So I spoke a few more minutes with this interviewer thanking him for going along with it. I then learned this was a interviewee discovered through Upwork - so I asked for all the proof and continued my investigation.


Fake Upwork of Me

The company who my impersonator applied at reached out and provided all the images above as well as reporting this account to Upwork.

So with this discovery - I found the fake Upwork of myself. This was creepy as it included some jobs I worked and some that were made up. It was like my actual resume with some made up boosted traits. It did have my real Laravel certification so my guess is someone just exported a data-set of me from LinkedIn.

So it seemed like I was starting to understand the picture now.

  • A person/company sets up fake Upwork profiles of real people.
  • They apply to jobs in hopes to get an interview using that fake profile.
  • They find suspecting victims on GitHub who are willing to go along with this.
  • That person uses the identity of someone else to land the job.

I'm not sure how this actually pans out if it works. Can you really refuse to use video for the entire contract of the job? How do you get paid if you need to submit tax documents? Do you pretend to be me forever?

So it was time to investigate further. Who or what was behind this profile?


However, before I did that. I wanted to childishly email the address of the person impersonating me.

Email from me to fake Connor.

The response I got I'm not sure how to react - an apology for assuming my identity which is absolutely insane. You could just make up profiles and use AI generated images, but using my real photo and information and GitHub is far beyond 'sorry'.

This next part of this post is all in part thanks to the individual who passed on the details to me and it only gets crazier. I got written approval from Andrew (The Individual) to show his name/photo unredacted. He blogs at unfooling.com and may have a post up about this as well.

If you put on the shoes of Andrew. You would have been reached out to by a person using a fake profile, name and email. They would be looking for a business partner with a very generic document in hopes to attract you to join after a bit of small talk.

Hope you're doing well. I just came across your GitHub profile. Looks like you've got a lot of experience in software development. So, I just wanted to reach out to you and see if you're available for a part-time role

I'm currently looking for a senior software engineer who can work with me as a business partner and it will be fantastic if we can work together.

A bit about me... I have a remote development team and we mostly work with clients from North America.

We provide web and mobile app development services. Most of our team members are not native English speakers and we often face communication problems.

So, we need a native English speaker with software development experience to help us with client communications. The rate is between $60 and $80/hr depending on your experience.

The main responsibility will be taking job interviews on our behalf and talking to our potential clients. Please shoot me a message if you're interested in further discussion.

This individual was interested and that led to a document being sent along after considering to accept this position.

This document spells out some weird behavior between the lines and lacks any official business domain, email or logo. Especially paired with a generic Google email address and a phone number that resolves to the Embassy in Panama does not seem right.

So Andrew like most potential hires has some valid questions before jumping into this opportunity.

A potential 'fake' Connor talking to the presumed leader.

I'm redacting the sites provided for the sole reason that I can't confirm if they were actually developed by this company and don't want to loop them into something incorrectly.

Reached out to one client who PND claimed they built.

1/3 of the companies responded and claimed PND did not build their website nor had they ever heard of them. So I believe this is a never ending web of lies.

What we do learn from this email is the website of the company itself - PND Design. So we take a look at their website and it doesn't appear to look that great.

pnddesign (d0t) com

So I continue to go back in history on Archive Machine until I find a change to the site design.

December 2020 of pnddesign (d0t) com

This gives me an older company name of 'PND Developing' which finally gets me some results on the web. It seems because the website says Madison, Wisconsin and so does the LinkedIn page and the domain matches - we found the right page.

https://www.linkedin.com/company/pnd-developing/about/

So recapping - we have a business in Wisconsin, a phone number in Panama with a generic Gmail. However, we do have a profile associated with this company and that led on quite the discovery mission.

We can find Plamen Dimitrov who appears associated with PND Design on LinkedIn and the name of this individual looks pretty close initial wise with PND. This entire thing could be lies on lies - so maybe even that can't be trusted. However with the initials in the Slack channel being PND we might as well investigate. This individual has a website associated with himself and this leads me to see an eerily relation between the sites.

Same 'No Right Click' on all the sites.

This individual is associated with a personal site and a company site that uses the services of two other sites that all act with similar features and broken designs. It looks like the security enhancements of web browsers in the past few years have not been patched on these websites.

So we now have the following sites associated with this and I can say with pretty good gut certainty these are built/hosted/controlled by the same party.

You can reverse those sites to find a couple hundred of domains on that IP, then cross-check the list against matching Google Analytics IDs and find almost 50 domains.

I clicked on one of the domains and it actually loaded - which was a surprise after the four sites above not really doing too well.

The 'Anti Right Click' is back again.

It seems I've found the pattern of installing WordPress with an odd set of plugins. I'm not sure what the obsession with no right clicking is. So now we can continue to find more websites built by this company.

  • pdrecipes (d0t) com
  • pndstore (d0t) com
  • acupunctureherbalmd (d0t) com
  • francelavindesign (d0t) com
  • maacupuncture (d0t) com
  • onsitepromassage (d0t) com
  • recipeswise (d0t) com
  • studiozmadison (d0t) com
  • thesmoothierecipes (d0t) com
  • tryveganrecipes (d0t) com
  • acupressureschool (d0t) com
  • acupunctureschoolusa (d0t) com
  • freshmartmadison (d0t) com

(44 more....)


If we jump back to my situation with this - we can look back at the individual Andrew stuck in the middle of this who still had some doubts with this job. He proceeds to ask another question.

Last question: What's the rationale of me pretending to be one of your developers?

Getting a response from 'Maris' of:

Hi Andrew, we're not a grownup company. So we still get jobs from various job board websites as individuals. Actually, we managed to sign up on several platforms. That's why you should talk to the client as an individual. Hope this answers your question.

This is a valid question with a non answer. No where in this response do I see why you have to pretend to be another developer in order to do your job. There are plenty of businesses that have a project manager in the states that simply act as the communication layer between an offshore development team and that could have been an acceptable business model.

My guess is this becomes the difference of hiring a person instead of a company. Perhaps the contract jobs on Upwork are easier to obtain with one singular person masquerading as an entity than an entire company applying to handle some contract job.

So Andrew decides to go forward with this and obtains an invite to a Slack channel where we immediately see how crazy this story gets.

PND = 'Maris' ?

We can see PND is talking and showing an example of what Andrew might have to do. The link was active at the time of my research and this interview occurred on February 28, 2022.

The initial reach out to this individual came from Maris. So this is crazy - PND creates emails impersonating engineers then uses those emails to attract more folks to join their company. I do wonder why Andrew didn't immediately bounce at multiple points during this, especially when you realize the person you were recruited by gave you a fake name.

So am I expected to believe this company is using my name and fake email to attract more individuals? I attempted to get this email account removed, but then I realized that was a useless endeavor - no way Google would get involved in the drama of deciding what is impersonation or not.

https://support.google.com/mail/answer/56256

So if we look at timestamps - Andrew joined the Slack channel around 9am on September 14, 2022. Around lunch the assumed leader known as 'PND' begins to ask this individual if he is willing to take the interview as fake me that same day. This is the same interview I ended up joining for real alongside an impostor.

It appears this job ended for Andrew. While he thought he was being brought on to develop - he was just going to be an English speaking individual to assume identities & roles of others.

This is a direct lie from what we saw in the initial email from PND. It was explained that you would 'assume the role of our developers'. The important word is here 'our' - I am not a developer with this company, yet my profile was being used.

A fake Connor did end up joining the interview, albeit late, who was not the individual pictured above. So my guess between 1:23pm and 5pm PND got to work to find an alternative person to assume my identity which was probably someone in that Slack team.

Unfortunately for us the individual who joined the Zoom interview impersonating me left immediately when confronted about lying, so nothing else could be learned from this.


Individuals in this Slack team.

So we can see the 35 members in this Slack, but I don't feel comfortable posting that list. I have no idea who is real or fake and who may be working for this company unaware of what is actually happening.

So I sent an email to two of them after I found them on LinkedIn to further help investigate this. One immediately responded unaware of this behavior occurring and left the group.

I do know there is an extreme amount of care from the assumed leader 'PND' to not expose themselves who has not once used a real set of information or documents when talking to Andrew.

All parties are given only the information to complete their part of this scheme. I say this because you don't even know the company you are interviewing with - PND only gives you the name of the interviewer. Thankfully the Zoom link in the fake Connor Tumbleson interview leaked the company involved.

This helps piece together those odd interviews some coworkers described to me when the interviewee has no idea what the purpose of the meeting is or company. I'm starting to piece together a huge scheme that probably occurs in way more places than this Slack.

It appears Maris was in Notion instead of Google Docs and I can not find the client associated with that interview. I only have a generic interviewer name to go from. So if you hired a Maris in February 2022 - you may want to double check who you actually hired.

My guess since 'PND' holds the fake Gmail addresses - they may orchestrate everything between Upwork, fake person and the client while the hired hand just attends meetings to be a voice.

So at least two people had their identity assumed in hopes to obtain a job in my brief research. At this point I did the following:

  • I emailed the individual who provided these photos/research asking for permission to post this with name.
  • I emailed all three businesses this company claimed they built to obtain more information.
  • I emailed two users from the Slack who I guessed may be in a similar situation after finding on LinkedIn.
  • I emailed the real Maris to loop them into this situation.
  • I emailed the fake Connor Tumbleson.
  • I reported the fake Upwork of me.
  • I called the assumed owner of PND Design and left a voicemail after no one picked up.

Who knows how many jobs out there were obtained from an individual impersonating someone else. It seems scary still to think that someone is using my name, profile and achievements to convince companies to hire a fake iteration of me. For that sole reason - I will not give up on my research.


Updates

  • September 14, 2022 - Draft written.
  • September 15, 2022 - Upwork removes the impersonator of me.
  • September 15, 2022 - Fake Connor Tumbleson responds.
  • September 16, 2022 - Person from Slack team I emailed responds and announces 'leaving the slack'.



All Comments: [-] | anchor

iBotPeaches(10000) about 9 hours ago [-]

I submitted my own blog here, but then my intentionally configured HN timeouts locked me out. I was wondering why my little Linode was dying.

Yeah this was an incredibly odd and creepy experience that I continue to investigate here and there. I really appreciate the interviewer for letting me stay on and confront the imposter.

hmm-interesting(10000) about 9 hours ago [-]

Username seems familiar. I have spent a lot time with your apktool, few years ago. For the first time, I saw your real name and photo. Thank you for this amazing tool.

EamonnMR(10000) about 9 hours ago [-]

The Darknet Diaries guy will probably want to interview you after this.

macintux(10000) about 10 hours ago [-]

I could sure use a flowchart to follow this story. Baffling level of fraud.

KaoruAoiShiho(10000) about 9 hours ago [-]

This is the type of arbitrage that happens when developers have similar technical skills but not similar interviewing skills.

ryandrake(10000) about 9 hours ago [-]

Whenever I read these kinds of super-complicated scams, I can't help but think if the scammer would have instead invested all that time and effort into legitimate tech and interviewing skills, he/she could have just come in the front door! It's like the person who spends hundreds of hours putting together the perfect exam cheating scheme, where they could have instead put in half of that time actually studying!

Or, in video form, the Kay & Peele Bank Heist[1]

1: https://www.youtube.com/watch?v=jgYYOUC10aM

dabernathy89(10000) about 8 hours ago [-]

If you are a software developer, check to make sure you aren't being impersonated on Upwork as well. A couple months back someone (I think I know who, but have no proof) was posing as me, and a suspicious client noticed that the person they were chatting with on video did not look that much like me in real life. Two other PHP/Laravel devs had impersonators on Upwork as well around the same time.

One of these other devs only noticed because the client sent a calendar invite to his real email, instead of the one provided by the impostor.

[edit - I'm reading through the original post, and I see now that this was all done through Upwork as well. Yikes!]

wildrhythms(10000) about 5 hours ago [-]

How might someone 'check' for this on Upwork?

NaturalPhallacy(10000) about 5 hours ago [-]

I actually have a weekly google alert that searches for my full name, which fortunately/unfortunately is basically unique. It's normally nothing but a handful of false positives, but the week I appeared in the local news lit it up like a Christmas tree.

paraknight(10000) about 5 hours ago [-]

I knew immediately it's Upwork. This is extremely common on there. The main reason is because developers in Western countries can demand higher rates. Just watch out for the red flags: mismatching LinkedIn experience, no camera during interview, incorrect accent, etc. Don't trust the reviews because accounts are routienly shared and/or sold.

I've had some fun with this before where a developer with a clearly Chinese accent, and of course no webcam, posed as German (mispronouncing his own name) and freaked out when I switched to conducting the interview in German. Of course I notified the person whose identity he stole and reported the profile to Upwork, but it's a drop in the bucket of the scams.

tablespoon(10000) about 5 hours ago [-]

> The main reason is because developers in Western countries can demand higher rates. Just watch out for the red flags ... incorrect accent, etc.

At least in America, that's not a valid tell. There are tons of developers in Western countries that have foreign accents.

vxNsr(10000) 7 minutes ago [-]

> no camera during interview

to me this is immediate no hire. If you can't be bothered to show your face, I can't trust you.

jjslocum3(10000) about 7 hours ago [-]

In 1996, my employer got a contract to work with AT&T to build a website that provided regular event updates from the Atlanta Olympics. In 1996, this was a very big deal, such a newish concept that the project was written up in AdAge or some similar industry magazine.

A few months later a prospective junior engineer came in for an interview. My manager asked him the typical 'tell me about an interesting project you've worked on lately.' He then proceeded to describe in detail the very project we had just completed, even referencing the magazine article about it (he must have forgotten he was interviewing at the company mentioned in the article). At the end of his presentation, my manager said 'That's interesting, because here at X, we just completed that project.'

Awkward silence. Then the interviewee got up and said 'I guess I should go now.' My manager said 'Yes, I guess you should.'

Impersonation of this sort can be simultaneously disturbing and somehow comical. It isn't a new phenomenon; I'm not decided on whether I believe the information age makes it easier or more difficult.

tg9000(10000) about 7 hours ago [-]

What a small world! Unless you are also making up that story (which I would get a huge laugh out of) I'm 99% positive I was in the room with you when this happened. :)

Ancapistani(10000) about 9 hours ago [-]

I wonder how widespread this really is?

If it's reasonably common, there might be a place for a 'reputation protection' service in the tech community - a service that watches various contracting and hiring sites for its members names, then notifies the real person when their name is used.

I could see it being a real issue in the future if someone's professional reputation is tarnished this way. If a prospective employer searched for a candidate and found multiple profiles with very different skills listed, that would be a huge red flag. Worse, if the fraudulent person was hired and then fired, that information could find its way to places where the real person is applying.

If they were able to successfully land a job like this, I could also see that messing with the real person's tax situation.

... I'm off to look for my name on Upwork, I guess.

agentwiggles(10000) about 2 hours ago [-]

My new startup is a protection racket! Sure would be a shame if your good reputation was tarnished by some low-skill offshore devs...

awb(10000) about 9 hours ago [-]

I had a similar experience back in 2008 when I started a fully remote digital agency.

One of my first employees was doing fantastic work, until his performance fell to 0 - no communication, no deliverables, nothing. Turns out, he stopped paying the subcontractor that was doing his job for him.

The subcontractor contacted me months after I fired the employee and confessed. Apparently, the long-pauses and loud typing during my conversations with the employee was the employee messaging the subcontractor asking for help answering my questions.

So, in my case, the employee was still the front. In this case, they're attempting to eliminate that bottleneck by just having the subcontractor impersonate the employee.

mherdeg(10000) about 9 hours ago [-]

Is any actual work happening here? Does someone ever sign an offer letter and start checking in source code?

lazide(10000) about 9 hours ago [-]

Many (most?) large companies might take months to fire someone even if it was blatant and obvious. Process to follow, etc.

Considering how distracted and overwhelmed many managers are right now, some might go years before catching on.

Even if no code got checked in. Chances are, they could also farm out a bit here and there to a friend to make it a harder problem to resolve for the company.

KaoruAoiShiho(10000) about 9 hours ago [-]

Why didn't they make a up totally fake person with a fake history instead of using a real person? I feel like it's possible to do, even fake a github.

sigio(10000) about 9 hours ago [-]

It't way more work to make a fake profile, with actual contributions to different (real) repositories, then just pretend to be an existing one.

Gh0stRAT(10000) about 9 hours ago [-]

If they want to actually start and collect a few paychecks, then they'll need to pass a background check from HireRight or whoever. This'll probably include transcript checks, verifying past employment, etc.

Much easier to just use a real person's identity.

type_Ben_struct(10000) about 10 hours ago [-]

Wow. This is creepy. Props to the interviewer for allowing the real Connor to stay on the interview and observe.

This is one of my big problems with LinkedIn. We put so much information out there in public, it's really easy for people do do this. That information can also be used for things worse than applying for jobs.

I think small companies hiring freelancers are most vulnerable to this. In the UK at least companies have to carry out very strict right to work checks, including passports, National Insurance numbers, etc.

djitz(10000) about 10 hours ago [-]

A company I was working for wanted to bring on a couple of freelance devs for a short-term project and I had to handle the interviews.

I ended up uncovering a whole scheme where an experienced dev in the US would hop on the calls/video interview and then the actual work would be handed off to some other people based overseas.

If you tried to contact the "dev" for something, your call would be routed to a google voice number and you'd receive a text message in somewhat broken English shortly after.

Their scam only lasted a few hours with us, but I often wonder how well they are able to pull this off.

heavyset_go(10000) about 10 hours ago [-]

This is one of the many reasons I don't dilvulge anything on LinkedIn and merely use it as a funnel to my contact page on my own site.

victorclf(10000) about 10 hours ago [-]

These kind of frauds are really bad for legitimate remote workers. Hope employers don't get burned and start cutting back on remote opportunities.

lazide(10000) about 9 hours ago [-]

A big reason why a lot of companies are trying to push people back to the office is they have low confidence that line managers will catch these kinds of problems, and many more - including 'the guys working 4 jobs and barely doing anything for us', the 'guy starting his own company that competes with us at the same time as working for us', the 'guy who farms out all his work to subs in India', etc.

It's easy to say 'if they don't notice, then clearly it's not a problem' - but it has downstream effects, like broken products, huge legal liabilities for the company including often scary handling of customer data to make it work, and morale hits as other folks pick up on things like this happening and being uncaught.

These are real, albeit currently low percentage/high risk things that happen. The more people get away with it, the higher the percentages of people who will try (people rationalize it to themselves as 'everyone else is doing it', and 'I'd be foolish to not do the same thing everyone else is'.).

The biggest issue I've seen with remote work (in practice), is it makes it really hard for a manager to see and actually understand what's happening (not just what people SAY is happening, which is rarely the same thing), and makes it easier for employees to hide things they don't want others to see. Which leads to more of everything from undiscovered-until-too-late burnout, to team members who have no idea what to do or how to do it, to opportunists grifting.

mcast(10000) about 2 hours ago [-]

Maybe it's time to add PGP public keys to resumes.

aclatuts(10000) about 2 hours ago [-]

My first thought. PGP in video calls, PGP with email, etc..

When I searched for PGP in the comments, this was the only one that mentioned it which is hard for me to believe.

dctoedt(10000) about 9 hours ago [-]

I hope Andrew [0] — the college junior with morals who blew the whistle on the attempt to get him to impersonate the author — gets an internship or job offer out of this; he apparently was having a hard time with that.

The author's sleuthing is reminiscent of Cliff Stoll's The Cuckoo's Egg from 1989. [1]

[0] Andrew blogs at https://unfooling.com/, according to the article.

[1] https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)

javajosh(10000) about 7 hours ago [-]

Perhaps at the end of the day that's what the scam was about, getting Andrew an internship. I mean if I was writing the movie that would be the twist at the end

nonrandomstring(10000) about 9 hours ago [-]

> I hope Andrew [0] — the college junior with morals who blew the whistle on the attempt to get him to impersonate the author — gets an internship or job offer out of this; he apparently was having a hard time with that.

Excellent point.

I've been wondering about ways to test students on 'trust/morals' and decided its one of the most valuable yet least well understood qualities. Employers generally rank skills, knowledge, salary, even age/gender/race above dependability/loyalty, or barely consider the latter at all.

Other than lengthy vetting and imprecise security clearance procedures this is such a hard quality to discern, and so costly when you miss the mark. The costs of defection, industrial espionage, and sabotage seem poorly quantified in HR. I think a corrosion of work relations has come about from devaluation of workers qua humans, and the corresponding disrespect people have towards their places of work. Is that inevitable under capitalism/efficiency?

And, harder question... does it even matter? Especially once AIs and remote agents take-over many jobs? Does a corporation care if the worker is an imposter and liar who abused a false identity to get the post, so long as they produce working results?

Is there a kind of moral Turing test here? What do work relations have to do with human-relations in the limit of the present trajectory?

darau1(10000) about 8 hours ago [-]

Someone messaged me on reddit once, and straight up asked me to do exactly this. Below is the message, but I haven't included their name because I think they were at least trying to seem sincere.

---

Hi, hope you're doing well.

We are looking for a professional interviewee. I'm not sure if you've heard similar thing somewhere. We are a talented developer group specialized in web and mobile software development. We have partnerships with US people and deliver our service to clients by pretending to be US developers. And we share profits with them. Our partners are satisfied with this business model.

Everything is perfect except on one thing. It's just the interview with clients. Normally in the interviews, the clients ask us some technical questions to see if we are able to deliver the service they expect. Because we are not native speakers, we are suffering from taking the interviews and many clients are passing by us even though they can get what they want. So we want a native interviewee and hope you are interested in this model.

Please let me know if you're interested in further discussion. Thank you!

jawns(10000) about 8 hours ago [-]

If it were really true that this development group could deliver according to client expectations, they would do what many other groups do: form a consultancy and hire an English-language speaker not as a fake interview candidate but as a liason.

I worked with a firm that did this. Basically, they had one project manager who could speak decent English and about six developers who couldn't. The English-speaking PM was on calls with us, and then he'd farm out the work to the developers.

It was a win-win, because their group was getting work, and we were getting decent results at a discounted rate compared with on-shore resources.

But it's pretty clear that anyone looking for fake interview candidates is not actually planning to do that. They're essentially counting on the fact that it takes many companies a little while to get rid of a bad hire.

yellowstuff(10000) about 8 hours ago [-]

I've heard of large consulting companies that do a 'bait and switch' where the client initially talks to a fantastic technical person prior to signing a contract, then never hears from them again.

jliptzin(10000) about 8 hours ago [-]

Email providers should publicly disclose how old an email address is, and email clients should warn about emails coming from brand new addresses. My gmail account is 15 years old, so any email I send is unlikely to be created just to impersonate someone else.

This doesn't help though if your name happens to be Kevin Smith or something.

digitallyfree(10000) about 4 hours ago [-]

Speaking of the 'Kevin Smith' case, I know of someone with a very common name going into an interview for a job he applied for - he was underqualified for the role and was surprised he was selected. Turns out the company had mixed him up with another candidate with the exact same name and sent out the interview invite to the wrong person!

darkwizard42(10000) about 8 hours ago [-]

Also just ratchets the value of 'old email accounts' up a ton. Reddit has a similar problem with fraud coming from old tenured accounts with karma (upvote points similar to HN) becoming very valuable and targets for hackers and spammers.

I don't know about new addresses, but it sounds like more robust vetting is needed on the interviewing side. Resumes and initial screens have become potentially stale and too easy to fake.

sebastien_b(10000) about 8 hours ago [-]

> "The fake Connor Tumbleson immediately left the Zoom call."

I guess he felt the ultimate Impostor Syndrome.

bluehatbrit(10000) about 8 hours ago [-]

Careful or he'll change his twitter handle to 'thought leader' and write an self-help ebook about it.

raydiatian(10000) about 2 hours ago [-]

I had a similar experience on Triplebyte, but didn't have the nerve to pursue the line as far as Connor did here.

Was reached out to on Triplebyte by a guy claiming to work for a large casino in Las Vegas which I won't name. Asked me to sign an NDA before the interview or that he would tell me what the job was. Couldn't find him on LinkedIn or anything. The NDA was a word document which I think I signed via online service but never physically opened on my machine because (thank god?) I don't own Microsoft word, on a Windows machine.

Pretty sure it was some sort of scam, but I couldn't for the life of me figure out what it was they were after. Perhaps this is it.

lupire(10000) about 2 hours ago [-]

Might just 'hire' you, get your vital data, and then impersonate you to get a loan or withdraw from your bank.

wbobeirne(10000) about 10 hours ago [-]

From the headline alone, I was hoping this would be someone writing about being the subject of Nathan Fielder's 'The Rehearsal'.

ogn3rd(10000) about 9 hours ago [-]

Allow me! It was awful and Nathan should be done with television as he's incapable of empathy.

burlesona(10000) about 9 hours ago [-]

So my guess is that the scam ends with the scammer negotiating a 'deposit' to start contract work, and once the deposit is paid they disappear. Otherwise I think this scam would be a lot more work than just actually getting programming gigs and doing the programming work.

wildrhythms(10000) about 4 hours ago [-]

The way I read it is: the scammer will secure the contract using the help of their industry-decorated accomplice, and then outsource the actual work-related duties to developers they find on Upwork, etc.

CobrastanJorji(10000) about 9 hours ago [-]

This story is much more fun when you come at it from the interviewer's position. You've been doing interviews every week. They're mostly rejections. They're the same questions over and over with minor variation. You're about to repeat the experience for the 18th time and you're 100% on autopilot. But suddenly you're in a spy thriller. This is the greatest thing that's ever happened.

Is it a good legal/corporate decision to hide the person who claims to be the original and let him listen to the interview with the other candidate? Holy fuck, no. Is it going to be WAY more thrilling? Oh my god yes; how could you not?

PragmaticPulp(10000) about 8 hours ago [-]

> Is it a good legal/corporate decision to hide the person who claims to be the original and let him listen to the interview with the other candidate?

Consider the situation from the perspective of the interviewer: They don't have all of the background we did while reading this blog. They haven't even had time to process what Connor #1 said by the time that Connor #2 arrives.

The decision to hear them both out for a few minutes is reasonable, IMO. At that point in time, Connor #1 could have been lying as far as the interviewer knew. Letting them both exist in the meeting immediately cleared up any confusion.

icambron(10000) about 8 hours ago [-]

This was my immediate reaction too. Would make my whole week.

fatjokes(10000) about 9 hours ago [-]

Haha, when you put it like that, I can totally see why the interviewer was 'awesome' and let the guy stay on. He was totally there for the drama.

conductr(10000) about 6 hours ago [-]

Some trainwreck interviews make for good office lore and this one tops the cake.

Taylor_OD(10000) about 6 hours ago [-]

You know that interviewer was shitting themselves when og Conor unmuted and said wtf mate.

filmgirlcw(10000) about 9 hours ago [-]

Oh, I'd be so in to take part in this drama. It would probably be one of the most memorable job interviews of all time.

And I doubt there would be too many legal or corporate ramifications from allowing someone else to be on the call with their camera off. These are contractor positions, not full-time. Frankly, it's a risk I would take to be able to witness this sort of thing in real-time.

version_five(10000) about 3 hours ago [-]

I get the sense that 'real Connor' must have been extremely good at concisely explaining the situation and giving the interviewer exactly the information he needed to believe real Connor and understand the situation. It made me think of one of those movies like groundhog day or 'source code' where someone has multiple tries and eventually comes up with the most efficient possible thing to say to get someone to trust them. Well done!

mizzao(10000) about 2 hours ago [-]

I feel like this PND thing might just be an agency for the folks on https://www.reddit.com/r/overemployed to outsource the acquisition of their 2nd, 3rd, 4th jobs.

blastwind(10000) about 7 hours ago [-]

Andrew here. Connor, thanks for releasing this on the orange site.

This story is also more fun from my position. I've been applying to internships and interviewing every week. They're mostly rejections. They're the same questions over and over with minor variation (sorry to top comment for 'impersonating' your comment style). My days are deteriorating from a colorful sphere down to two points. In fact, down to two pointers, left and right, iterating over a list of heights to find how much rain water it can trap.

I'm about to repeat the experience for the 10th time and I'm 100% on autopilot. But suddenly, a man reaches out to me on email and offers me up to $80/hr to be his senior engineer. This feels sketchy, my girlfriend tells me, 'you're good but let's be honest here...'. Anyways, I proceed, it might just be the start of a beautiful thing. I'm asked to interview as one of our developers because English is not their best language. I'm a little bothered, but I was fine with it. But then I see the developer name: Connor Tumbleson. My laughter bursts and so does my suspicion: With a name like that, no way the guy doesn't speak good English. I look up Connor Tumbleson on linkedin, and my suspicions were proved correct. I detail everything to Connor, and now this is on the top of HN. I lost a opprotunity but gained a story of the lifetime.

comboy(10000) about 7 hours ago [-]

This comment thread feels like a game of Mafia[1].

1. https://en.wikipedia.org/wiki/Mafia_(party_game)

wanderingmind(10000) about 2 hours ago [-]

<sarcasm> Plot twist. The same organization is pretending to be Andrew to get some work opportunities </sarcasm>. But seriously, its a breath of fresh air to see someone displaying strong ethics in the face of adversity (losing a job albeit a fake one pays real $$). Trending on HN first page will open a world of new opportunities. Good luck.

fortran77(10000) about 4 hours ago [-]

I wonder how many people in your position smell something rotten, but instead of trying to contact the 'real Connor' just delete/ignore all the messages because they don't want to be part of even a bigger scam. (What if everyone is in on it, and they're trying to scam you somehow?).

Naracion(10000) about 7 hours ago [-]

Recount _this_ story during the social / break between interviews if you ever get an onsite. :D

kodah(10000) about 6 hours ago [-]

Shout out to you for wading through the literal torrent of bullshit without the foresight of a blog post to expose context and with little professional experience to help inform you. You'll be a great asset to the industry but it can take a minute to find your footing. Be persistent and definitely keep this story around for beer Friday.

robswc(10000) about 4 hours ago [-]

I know it probably feels a bit hopeless now but trust me.

If you learn to build things, provide value, you will have 100s of recruiters reaching out to you and you will mostly be rejecting offers for a change :)

I have no doubt reading about you and seeing this comment in a few years you will be more than set!

tomcam(10000) about 4 hours ago [-]

> my girlfriend tells me, 'you're good but let's be honest here...'

My favorite part of this whole thing. Hang onto her.

dmoy(10000) about 7 hours ago [-]

> In fact, down to two pointers, left and right, iterating over a list of heights to find how much rain water it can trap.

Ah haha I hate that question

It should be banned everywhere, oh well.

I once saw a physicist (not even a coder) give a really cool answer to it though, I wish I could remember it.

jaffee(10000) about 4 hours ago [-]

Andrew, see if anything here catches your eye... we've got a few openings. You can email me at my username at featurebase.com.

https://www.featurebase.com/careers

dj_mc_merlin(10000) about 3 hours ago [-]

BTW, why are you applying just for internships? I checked out your (non-forked) github repos and you're definitely better than I was as an intern. It's not senior engineer level, but you could definitely go for at least a junior eng position for sure. God knows we hire much worse junior FE engineers than that. Or are you interviewing for FAANG(-like) positions only?

edit: small tip though, I would rename the App2.js etc. files to something else, having multiple files named the same but with a number difference says a lot about experience level.

also: do not use regex for parsing XML (https://github.com/BlastWind/xml-leaf-highlighter/blob/main/...). That is also a common newbie mistake

I hope this doesn't come across as patronizing, I didn't mean it so.

tomrod(10000) about 6 hours ago [-]

Andrew, mad respect on your integrity and navigating an ethics situation one would expect to read about in a college study.

pjbeam(10000) about 5 hours ago [-]

Email me at my HN username at protonmail dot com for a referral to Dropbox's internship. Love the integrity and drive to get to the bottom of a strange situation.

atmosx(10000) about 6 hours ago [-]

Shoutout to you sir for being honest and reaching out your fellow peer!!! I am sure you will have a great career and now you have an epic story to tell over beers with friends!

bmsleight_(10000) about 6 hours ago [-]

Andrew - you the sort of person I would love to have on my team. Alas I more hardware engineering than software. I hope the community here can give you some good leads and tips.

Karma should be that ethics works.

primitivesuave(10000) about 1 hour ago [-]

Hey Andrew, I am sure your integrity in this situation will be rightfully rewarded. You come across as a great hire not just in the matter of integrity, but also in the humble/honest description of your skill level and experience. I can see you finding that internship entirely on your own, by simply walking into the next interview confident of how valuable someone like you would be to any organization.

blastwind(10000) about 3 hours ago [-]

I'm grateful for all of the opportunities! Thank you everyone.

radarsat1(10000) about 5 hours ago [-]

This is amazing. I wanted to add one thing. I noticed after reading the full blog post, and scanning through all the HN comments here, that there has been actually no mention, as far as I can tell, from any parties involved, or even any commenters, of any intention to make a police report.

Now, I understand not trusting the police, and often it's more trouble than it's worth to deal with them. But this is a situation involving identity theft, which is a very serious crime. I realized that this is an international situation and the local police probably cannot do much, but at some level of policing, be it the FBI or even at the international level, this feels like something that should be reported. Even if nothing can be done, in the worst case it's useful that the police be made aware of new trends in identity theft; in the best case, they will be caught. These people are organized to perform identity theft, which is literally organized crime -- I hope they are aware of the risk they are taking doing this.

Lastly, unrelated to the above, but just a random social aspect of this; it's clearly an interesting and unexpected result of location-based pay. The only reason I can think of that a group of people would organize something like this is because pretending to be native English speakers and presumably pretending to be US- or Europe-based will automatically get them a higher pay scale. (If I understand correctly, they are possibly a team of programmers in some other country, and are offering to actually do the work, but just pretending to be other people while doing it in order to get a higher paycheck.) Not making any judgement here regarding location-based pay, although that's an interesting discussion for another thread, but in today's remote work environment, new kinds of fraud are definitely an interesting consequence to be on the lookout for. Fascinating, and dangerous.

entwife(10000) about 4 hours ago [-]

I was going to comment about a police report, but largely because it might be useful for an insurance policy that covers identity theft.

jstarfish(10000) about 4 hours ago [-]

You vastly overestimate the degree to which the police give a shit. Unless there's an actual financial loss or threat to life, they don't care. This barely qualifies as identity theft anyway; it's attempted fraud.

They won't take a report of your phone number being spoofed, but they'll deploy SWAT teams to unsuspecting houses at the word of bored teenagers.

Do you know the imposter's actual identity? What would you even report? If the perpetrator is international, what are your local police even supposed to do with this information?

You might have a little better luck with the FBI, but if you don't show up with hard evidence (i.e. do all the work for them), you won't get anywhere with them either.

All of this goes to show you why this sort of scheme remains successful. Nobody cares. Fraud is just an assumed risk.

blitzar(10000) about 9 hours ago [-]

Like all scams and spam, if it didnt work then people wouldnt do it. But I am struggling to see how it will work.

wccrawford(10000) about 9 hours ago [-]

Sometimes just the possibility of it working, and the person's desperation, will cause them to try it. I've heard enough instances of people paying others to take exams for them that it doesn't surprise me that some people trying to do an interview that way, too. They think just getting the job will do something for them, and (for whatever reason) think they'll be able to keep the job once they have it.

I wouldn't be surprised to learn that these people think they have high-level skills, but some other factor is preventing them from getting the job. Sometimes it might even be true, but I'm better against them having the skill level they think they do.

yarg(10000) about 1 hour ago [-]

That's made me think.

Why the hell don't software developers have their own chain of trust?

(Well, professionals in general, but you'd think that we'd have gotten this shit working for ourselves first.)

It would kinda fix this shit - at least to the extent that it was actually used.

So then all you need to do is have 'Company X' adopt the policy that all of their people must be connected to the trust chain.

From then on 'A' of 'Company X' fame can no longer be impersonated (except via theft of keys).

ChrisMarshallNY(10000) 35 minutes ago [-]

> Why the hell don't software developers have their own chain of trust?

Is it a bird?

Is it a plane?

No, it's BLOCKCHAIN!

This actually sounds like a possibility legit application for distributed ledger.

peppertree(10000) about 9 hours ago [-]

Scam is rampant on LinkedIn. I'm getting constant connection requests from obviously fake profiles with AI generated faces.

kube-system(10000) about 9 hours ago [-]

And they all have 500+ connections. Bet they're scraping profiles or some crap.

cynusx(10000) about 9 hours ago [-]

Same here. I just ignore devs that apply directly to me on linkedin, I would be surprised if any of them were real or not an agency.

mrandypratt37(10000) about 8 hours ago [-]

I'm a US-based accountant who is currently interviewing for his first job in software and I was approached by someone on on LinkedIn about an opportunity. Seemed fishy, but figured I could risk 15 minutes. The person set up a meeting between me and a Taiwanese Developer for this exact thing. He said he had 7 years experience and had a contract drafted for he and I to become 'business partner' where I would take the meetings and he would do the work for a 30/70 split. I told him, morals aside, that I didn't have the credentials to get into the jobs he would want and pointed out numerous obvious issues like in-person coding, etc.. He said he was ready to make a fake LinkedIn and had this whole operation planned out.

Seems to me like there is a whole operation around this business model of exploiting US developer salaries and the morality of a few Americans willing to try and make a dollar for free. Honestly more disappointed in the people accepting shady deals like this than the ones offering them.

datavirtue(10000) about 8 hours ago [-]

We blew 'morals' out the air lock long ago. Not to mention, morals are subjective.

I don't understand why corporations are not embracing and encouraging such arrangements.

dont__panic(10000) about 7 hours ago [-]

If I can play devil's advocate for a moment...

if the company gets decent work, the non-US participant gets better (and fairer, globally) pay compared to what they'd get locally, and the US participant takes care of the 'soft' side of the operation... who's getting hurt?

I can't deny that something smells skeevy about this and I don't think I could ever trust a random foreign developer who I haven't built up a solid relationship with to execute reliably 'as me' in the coding side of a role. But if I had a good friend from college who couldn't get a VISA to the US? I dunno, I might be tempted to collaborate. If everybody wins, I'm not sure it's inherently bad. But maybe I'm missing something.

mapmeld(10000) about 10 hours ago [-]

Bizarre. I (US-based engineer with an Upwork account) was invited into a less sophisticated variant of the scam in spring 2021:

> Nice to meet you. I am looking for a US person who do business with me. You can earn money with a few cooperation. Do you know Upwork or Toptal site?

They also had the text of the message in a GitHub repo. I tried reporting them to GitHub, Upwork, and Toptal, but I don't think they knew what to do with it? I assumed my scammer was looking to evade banking rules or sanctions, but it could be for either fake employment or actual work with a US-based persona like in this case.

bogwog(10000) about 8 hours ago [-]

Same, although I was never asked to impersonate anybody. They just wanted to work using my real name/identity, and would throw some money my way every month. At the time I was annoyed/pissed, but after reading this I respect that they at least had the decency to ask for my permission lol.

Honestly, this is 100% Upwork's fault. Their platform is a race to the bottom, yet they make it very difficult/impossible for people from countries that can actually afford to make a living with those rates to sign up. So I understand why people resort to this behavior, even though I would never want to work with someone who would actually do that. Fuck those people.

dvykhopen(10000) about 6 hours ago [-]

I run a job platform. This scam is pervasive, especially with contract work. We've had to get really good at recognizing patterns because their covers are really good (even faking passports and work history).

Many of the big contract platforms are dealing with this too. Hiring managers are getting tired of it and are 1) not hiring as many contract workers and 2) not using platforms to hire those workers.

Unfortunately, this hurts small companies more since their hiring practices are so lax and there's a crop of new ones every few months.

NaturalPhallacy(10000) about 5 hours ago [-]

>We've had to get really good at recognizing patterns because their covers are really good (even faking passports and work history).

I'm pretty sure that can get you into super duper extra trouble with the State Department.

ollien(10000) about 9 hours ago [-]

What I can't piece together here is what the scammer's endgame is. They land a cushy developer job under some false identity and ... then what? They're not going to pay some random college student to attend every meeting, are they? Even if they were, are they going to be able to maintain the level of work they lied about? If they really had those skills, they would just interview on their own.

Something's not adding up

CobrastanJorji(10000) about 8 hours ago [-]

Maintaining is not the goal. Say you get a job that pays senior engineer money, and they fire you after maybe three weeks of you just saying 'it depends' and pretending your connection is bad. You still made like $5,000. And you're probably running more than one of these scams at a time. And you're doing this from a relatively poor country with not much formal education. 100% worth it.

And who knows, maybe you manage to actually KEEP one of those jobs, which given how bad some employers are at figuring out who's good and bad at this stuff, is entirely plausible.

iLoveOncall(10000) about 8 hours ago [-]

It would take months for the company to figure out that the person is not doing anything, is not who they pretend to be, and fire them. Months during which that imposter would have been paid to do literally nothing.

codedokode(10000) about 6 hours ago [-]

1) they could have developers from 3-rd world countries who have necessary skills but get low pay rate

2) they could have developers from countries currently under sanctions

rngname22(10000) about 8 hours ago [-]

They could be fully technically capable but unable to secure those jobs because of visas, location/timezone, the company having a prejudice against outsourcing/remote work, etc.

chadmckenna(10000) about 8 hours ago [-]

Its possible they need someone with a 'Senior' resume so they are able to charge $200/hour for their work. Then they are free to outsource it to 2 or 3 junior devs in a much lower priced market and pocket the difference.

notahacker(10000) about 7 hours ago [-]

I assume they avoid jobs involving standups.

In roles where communication is fully asynchronous, a competent offshore dev whose written English is considerably closer to a native speaker than their spoken English might be able to hold onto the job for a while, especially if they're good at excuses.

If they're applying for US onshore jobs with below-local-market pay, they might even be considered relatively productive members of the team.

clusterhacks(10000) about 7 hours ago [-]

I had two past co-workers who were almost certainly having someone else do their work, I think (especially) at non-software companies folks just don't get caught.

Frankly, the work environment was slow-paced and generally non-confrontational so both co-workers I suspect of this behavior just managed to tread water. When I joined the group there was a very tight-knit group of 3-4 developers who were very protective of each other. There was always a handy reason why some schedule was slipping and the other fact is that in hindsight, there just weren't very high expectations for them to accomplish anything.

Joel_Mckay(10000) about 8 hours ago [-]

A long while back I was sitting in a coffee shop, and happened to meet a guy recounting his work history to a client. The hilarious part was hearing my life plagiarized off my web CV at the time. Seeing the con describe climbing an antenna mast where I worked was awe inspiring given he was over 350lbs. I tipped off the client to do his own verification after, as there was no way that guy was part of our small team at that time.

Some people are certifiably insane, and will con anyone to make money. Note, confronting psychopaths with proof they are liars is extremely dangerous. These are the people that will hold grudges for decades if they feel you owe them something, or do something nasty.

Weak Stenography in your CV is also good for auto-screening/blacklisting those engaged in social-engineering workers. You would be surprised who shows up. ;)

g8oz(10000) about 2 hours ago [-]

Can you give more details about this stenographer tactic?

bdcravens(10000) about 9 hours ago [-]

A couple of weeks ago I received this email:

'Hi, Billy

How are you?

I checked your Codementor account, it is great.

I am *** **** from Ukraine.

I am 32 and I am also a computer programmer.

I want someone who can help me.

Would you lend me your account?

If you borrow it, I can earn a lot of money.

I will pay 100 usd every month.

Regards.'

vlunkr(10000) about 9 hours ago [-]

100$ a month for some fraud. Not very enticing lol.

scrollaway(10000) about 9 hours ago [-]

Yeah I've had people ask to buy my codementor account (https://codementor.io/jleclanche). I forwarded all of them to [email protected], but I suspect some people always say yes ...

walrus01(10000) about 7 hours ago [-]

There are people out there right now putting a truly astounding amount of information on their wholly-public Facebook, Instagram and LinkedIn profiles.

My main message for people is to resist the social temptation to share every detail of your family's life on social media, in the long run it's better for your privacy, your family's privacy, your security, and reduces opportunities for malicious data mining.

It's sufficient to build an entire identity theft kit if you're a malicious actor wanting to impersonate somebody. Somebody would combine whatever is available from social media with things like linkedin profiles, CVs, github projects, other github-like-project profiles, and linkedin-type business networking site data.

Or at least a good enough to pass cursory inspection/examination identity theft kit to impersonate somebody with a close-enough email address, or a throwaway custom domain name registered for the purpose.

I would highly recommend anyone that does keep an account somewhere like Facebook to stop posting photos of your house, family members and to set all of your 'privacy' settings to whatever is the friends-only/maximum setting. Try looking at your own profile from a different browser with no cookies in a burner account or incognito mode and see if any of your personal life is visible.

permo-w(10000) about 3 hours ago [-]

good luck getting a burner account on facebook

annoyingnoob(10000) about 8 hours ago [-]

I think the root of the issue is LinkedIn. I know a number of scammers regularly monitor LinkedIn. When a new employee at our office updates LinkedIn they start getting email from the 'CEO', first asking for a personal cell number, and then asking to buy gift cards in a hurry.

I'm seriously considering being LinkedOut.

absolutelynobo(10000) about 8 hours ago [-]

This has happened to every new employee at my company within a month of them joining.





Historical Discussions: Mcmaster.com is the best e-commerce site I've ever used (September 25, 2022: 1373 points)

(1373) Mcmaster.com is the best e-commerce site I've ever used

1373 points 2 days ago by runxel in 10000th position

www.bedelstein.com | Estimated reading time – 10 minutes | comments | anchor

Most people I know haven't even heard of it, but mcmaster.com is the best e-commerce site I've ever used.

McMaster-Carr is an industrial supply company. They sell nuts, bolts, bushings, bearings – pretty much anything an engineer needs to build stuff. I've purchased from them dozens of times over the past few years, both for personal and school projects.

But what makes their website so great? And why should an industrial supply company have the best e-commerce site on the internet?

mcmaster.com is great because it does what it needs to, and nothing else.

First, let's look at the visual design of the site. Minimal, mostly grayscale, with accents of green and yellow. There are no popups, animations, banners, carousels, or videos – just a calm, static page with categories and a search bar. Even the images are grayscale, to avoid inadvertently catching your eye.

It's not the most visually stunning site, but that doesn't matter here - McMaster has chosen function over form.

A user's goal when they visit McMaster-Carr is to find their part as quickly as possible. The website is designed entirely around that fact. Users rarely come just to browse, so there are no AI recommendation algorithms, featured products, new arrivals – that doesn't make sense in this context. People visit McMaster-Carr with high intent to buy a specific part, that's it.

So how do we get from the 700,000 products in their catalog down to one part? Here's what I do.

Let's say I'm searching for a bolt:

  1. I type 'bolt' into the search bar

  2. McMaster shows me several subcategories: hex head, socket head, set screws, etc. I'm looking for socket head, so I select that one.

  3. Now I move my attention to the left nav bar, which shows me several filtering options. Bolts are commonly specified by their thread size (e.g. 1/4'-20), and their length. I'm looking for a 1/4'-20 x 1' bolt, meaning that the bolt's diameter is 1/4' and its length is 1', so I select these filters.

  4. There are over a dozen other filters, such as material, hardness, and head size. Once I've applied enough filters, The main search window shows individual items, rather than subcategories. Here I can select an item and add it to cart.

McMaster's search interface is the main reason for its design superiority. Everything on this page is designed to get you to your part as quickly as possible. The filter sections are simple and elegant, providing schematic illustrations when necessary. The illustrations are always simplified to convey only relevant information, as to not distract you from the focus of your search.

Results pages also show helpful drop-downs which explain the parts you're looking at. It's like an engineer's handbook and catalog in one. Engineers are often looking up terminology on the fly anyways, so having this information embedded into the site saves valuable time.

McMaster's filters are not only useful for a targeted search, but also for deciding what it is you want. Sometimes I'll search with only a general idea of the part I need, and then I'll use the subcategory descriptions to determine specifics. For example, I may know that I need some type of lock washer, but I'm unsure which one is best for my application. I can use the images and descriptions to decide on the right configuration.

many varieties of lock washers

McMaster is able to provide such intuitive searching and filtering because everything that they sell is highly legible – it's all defined by quantitative specs. There is nothing intangible to deal with, like brands, product photos, or other marketing fluff. Even still, they do a much better job than other industrial websites like Grainger, DigiKey, or Home Depot.

As a point of comparison, Amazon does a terrible job of filtering items. Amazon has an order of magnitude more products on its site, which admittedly makes the job a lot more difficult. However, even generic filters, like price, suck. I won't get too far into my disdain for amazon's UI design, other people have already written too much about it [1] and that's not the point, but it's interesting to contrast McMaster with what everyone sees as 'the' e-commerce site.

Take Amazon's price picker: Why is it two text boxes? Why not a slider? This has always annoyed me, since it's much easier for me to drag a slider than to manually type in my max price. And the quick-select ranges are literally the exact same no matter the product. If I search for a pen, nearly every result I want should be under $25. If I search for a trampoline, every result I want should probably be over $200. What the fuck?! I guess this somehow won the A/B test, but I can't think of a reason why.

Finally, one of the most brilliant parts of McMaster's product is that for nearly every part, they have a CAD file that you can instantly download into your 3D models. Mechanical engineers mock up designs in CAD programs before actually building them, and having access to pre-modeled parts saves time. (Imagine having to manually model all your nuts and bolts.) McMaster even has extensions for popular CAD programs which allow you to import part files directly, instead of using their website. This makes engineer's lives 10x easier (not to mention making them more likely to purchase from McMaster-Carr). The closest analogy to this is AR try-on, but that's not even very accurate. The point of AR try-on is to determine whether you like the item you're about to buy, whereas the point of McMaster's CAD downloads is to speed up an engineer's workflow. In most cases, they already know which part they need, it's just a matter of completing the CAD model before they can start building the real thing.

Improvements

Mcmaster.com is nearly perfect. It's a website that would make Steve Krug smile. My only suggestion would be to make the search bar on the home page more prominent. It's visually overwhelming to comb through dozens of product photos, so I pretty much always use the search bar to start narrowing down items. The main area of the homepage is effectively dead space, while the search bar is relatively tiny. New users might miss it, wasting time.

I decided to write about McMaster-Carr because it is so rare to see such careful thought go into an industrial web app, far removed from the commotion of silicon valley, web3, D2C, and the other typical subjects of pixel perfection.

Mcmaster.com is a product that understands its customer. The minimal, functional design allows users to find their parts as quickly as possible, nothing more or less. It's an unexpected reminder to not get lost in the allure of smooth gradients, 3D animations, or slick fonts, and instead relentlessly focus on what it is your customers really want.

'If something is 100 percent functional, it is always beautiful...there is no such thing as an ugly nail or an ugly hammer but there's lots of ugly cars, because not everything in a car is functional...sometimes it's very beautiful, if the person who designed it has very good taste, but sometimes it's ugly.' [2]

Footnotes

[1] 'Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally -- wisely -- left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they're all still there, and Larry is not.' https://gist.github.com/chitchcock/1281611

[2] https://youtu.be/MtxA20Q-Uss?t=1315




All Comments: [-] | anchor

leobg(10000) 1 day ago [-]

That Amazon sucks at finding what you're looking for is perhaps not a bug, but a feature. Many here on HN have the engineer's mindset where shopping for an item is just solving a problem. But perhaps, for most people, online shopping is something different altogether. Perhaps the a sizable majority of people actually enjoy „hunting" for a good deal. They, perhaps, enjoy getting „inspirations" along the way. So Amazon, through testing, might find that making search very fuzzy and even customer reviews not very reliable to actually boost sales.

In an ideal world, I'd like Amazon to filter fake reviews, weigh the rating of products with hundreds of reviews higher than those with just a handful of reviews, and give me filters to exclude words, search by strings, block particular brands, sort not only by price but by price per unit, and so on. I used to think that Amazon was just lazy in not implementing these things. But I now suspect that the real reason is that they've found that most people do not want to shop like that. For them, shopping perhaps is much less about mere „procurement" than about gamification, the thrill of the hunt, and entertainment.

LaputanMachine(10000) 1 day ago [-]

I believe it's not about customer enjoyment, it's about profits. If a customer spends more time on the website, they might buy more stuff they don't even need.

It's the same reason why you have to waltz to the back of every supermarket to buy essential items, walk back to the front, and look at individually packaged, overpriced snacks for several minutes while waiting in line at the register.

jrockway(10000) 2 days ago [-]

This doesn't go into the actual mechanics of finalizing your order, which is incredibly well done. I've only actually ordered from McMaster 3 times and don't have a need for anything right now so I might be getting details wrong, but basically, the 'items in your cart' page has a shipping address and payment method pre-filled, and you just hit 'order' and that's it. What's in stock, what ships when, etc. is all clearly labeled in each line item. There are number entry boxes to adjust quantities. You can put in your own part numbers, of course, and the items in the shipment will be labeled with that (or at least a reference to the line item on your invoice). First time I bought from them, I was blown away how clean and easy it was. You hear 'industrial supply' and think the ordering process is going to end with 'ok, now call us for a quote', but it's smoother than Amazon. Well done.

I also like their parametric search. If you type '1/4'-20 screw', the page goes to the overview of that screw type AND populates the 'machine-readable' parameters on the left side with what you typed. It's wonderful. I also appreciate that their search term tokenizer understands that you're looking for a thread size ('1/4'-20') and doesn't strip the non-alphanumeric characters like most would. (Amazon has gotten better at this over the years. I wanted to say 'look at those idiots, giving you any screw that has the number 1, 4, or 20 in the description', but they bring up the parameterization as well.)

Finally, McMaster has a reputation for being pricey, but sometimes you just pay the same price that everyone pays for something. For example, say you want a set of Mitutoyo 4' digital calipers. On Amazon, which is rife with counterfeit knockoffs, you'll pay $136.94. On McMaster, you'll pay $129.19 (plus shipping, yeah I know). And it won't be counterfeit! As always, shop around, but they are not trying to screw you on price, or by subbing in counterfeit trash to make a few bucks. Truly a marvel of online shopping. I wish they sold everything.

ryandrake(10000) 2 days ago [-]

> I also like their parametric search. If you type '1/4'-20 screw', the page goes to the overview of that screw type AND populates the 'machine-readable' parameters on the left side with what you typed.

This is something I wish more e-commerce sites would grok. I go to Lowes and type in

    1/2' 10-32 stainless bolt
in the search box. Now I get a search result page, with filter parameters on the left side. Great. There's a checkbox that says 'Stainless steel'. Oh oh! What do I do now? Aren't the results already all stainless steel? I specified it in the goddamn search. If I select the checkbox, will I be filtering out things I want? Can I click the checkbox and remove 'stainless' word from my search query and get more things I want? (hint: not usually). Why do I have different choices for screw thickness and thread pitch when I specified them, too? Come on, e-commerce developer-wizards, get your shit together!

Same for you, Amazon! '64GB sd card' and then Amazon's filter lets me further drill down... by capacity. DUHH, I specified the capacity! How can Amazon operate for almost 30 years and still not have this one figured out?

CreepGin(10000) 2 days ago [-]

Just curious, are images intentionally all grayscale? Or just a legacy thing.

mrweasel(10000) 2 days ago [-]

I didn't know about mcmaster.com, but I took a look at it, and the images was a surprise. The uniformity of the product images is nothing sort of amazing.

Most websites don't care to much about illustrations, even though that's what many people use when buying. Taking your own photos, or doing illustrations is pretty expensive, but it really helps the customers pick the right products.

The grayscale may be a way of insuring that customers aren't overly focused on the illustrations, and instead actually read the product descriptions.

jonnycomputer(10000) 2 days ago [-]

What a beautiful, functional, perfect e-commerce site. After so much frustration looking for parts and tools at big box store web sites, this is fresh like a sea breeze. Sold, 100%.

In fact, I was just browsing Home Depot for some tool hooks, but I think I'll buy them from this place instead.

s1artibartfast(10000) 2 days ago [-]

The shocker is it has existed in this form for nearly 20 years

paxys(10000) 2 days ago [-]

Their description of the site and its usefulness is probably correct, but I think people are taking the wrong message from it. Mcmaster.com isn't good because of its UX. The service would be exactly as useful had it looked exactly like Amazon.com. Its UX is simply secondary to its inventory and logistics.

Simply put - to build a successful business focus on the entire business and not just the website. People can get over an unintuitive UI and slow search. They cannot get over the fact that you don't carry what they want.

mcovalt(10000) 2 days ago [-]

The article specifically compares McMaster-Carr's website to Amazon.com to demonstrate a superior UX.

calvinmorrison(10000) 2 days ago [-]

No love for Rock Auto?

johnmaguire(10000) 2 days ago [-]

Right! Rock Auto might be my favorite e-commerce site.

1970-01-01(10000) 2 days ago [-]

Rockauto is catering specifically to auto parts. It is far ahead of their competitors however.

s0rce(10000) 2 days ago [-]

Rock Auto is great, definitely more consumer oriented and delivery is much slower, seems like professional mechanics rely on same day delivery. The biggest difference I see is that McMaster is curated, you want something there is usually one or maybe a couple options and it doesn't usually openly display the manufacturer. Rock Auto shows you all the options.

desktopninja(10000) 2 days ago [-]

Without RockAuto is right up there with the submitted site.

tims33(10000) 2 days ago [-]

Mcmaster.com is also insanely fast. The reaction to every click is so incredibly responsive. There is something so pleasing about it.

cronix(10000) 2 days ago [-]

It's amazing how fast things can be with no crap ads and just pure content.

mastax(10000) 2 days ago [-]

I disagree with one thing: I usually go to Mcmaster to browse. I often have a problem, and I know that it must be a relatively common problem with an elegant solution, but I'm not experienced enough in that area to know what the solution is. Browse through the catalogue for a while and sure enough, there's a widget that's specifically designed to solve your problem. The design of the website makes it really easy to do that. Everything is well curated, labelled, described, tagged, and hierarchical.

exodust(10000) 2 days ago [-]

> hierarchical

On the point of hierarchy they are messing up their categories:

mcmaster.com/raw-materials/

Hovering over the links in that category, you get links such as:

mcmaster.com/raw-materials/foam

But clicking 'foam' takes you to: mcmaster.com/foam/ and there's no link back to raw-materials. This is a mistake. They've booted you out of the hierarchy and neglect to offer breadcrumbs or equivalent category navigation.

Clicking 'browse catalogue' menu fails to identify what category you're currently in.

If I'm in mcmaster.com/fabricating/ and click 'drills', why am I taken to a page called 'hand drills'? The naming is a mess. Why am I seeing screwdrivers on that page?

Pardon the pun, but if I drill down to the hammer drill product page, the URL is:

mcmaster.com/29835A69/

What? That's ridiculous. The URL should be:

mcmaster.com/fabricating/drills/dewalt-cordless-hammer-drills/[product-name/ID]

Sorry to interrupt the 'best website ever' theme!

nerdponx(10000) 2 days ago [-]

There's also something weird with the way they programmed the site itself, which prevents you from opening certain links in a separate browser tab. it's really annoying when I'm not sure exactly what I need and I want to read all 10 product descriptions or whatever. I end up having to copy and paste the product URL and work backwards, rather than going all through the parts-filtering flow 10 times over from the beginning.

Otherwise the site is flawless in my opinion.

hoseja(10000) 2 days ago [-]

When you target the general population, it's sadly more profitable to design for exploitable, manipulateable, lowest common denominator. You can only have nice things if your consumer base isn't generally morons and you are therefore somewhat protected from predation by unscrupulous companies.

gnrlst(10000) 2 days ago [-]

The world would be a slightly better place if everyone had some form of engineering background. Not necessarily software nor mechanical, just an inkling of making things more efficient, more logical, and more usable.

ISL(10000) 2 days ago [-]

McMaster is a great choice when you can value your time.

There is no comparison shipping. You need an industrial tricycle that gets the job done? Great. Add an industrial tricycle to your cart. Unless there is a meaningful difference between models, there will be only one choice. It will be curated and it will do what you generally expect it to do.

You'll pay more than you can get it for elsewhere, but it will do the job and it will arrive on time.

That's McMaster's value proposition. It is often a very good one.

bombcar(10000) 2 days ago [-]

Based on my research, that is a very fair price for a industrial tricycle; honestly would have expected it to be more.

xen2xen1(10000) 2 days ago [-]

Please tell me there's a story behind an 'industrial tricycle?'

sh34r(10000) 2 days ago [-]

It's arguably similar to the value proposition Apple brings to a lot of their customers. Time is money, and having things just work the way they're supposed to seems to get rarer and rarer. Or maybe I'm just getting older, I dunno.

I wish we had more firms that still valued quality and proper customer service in this world. Too much chinesium junk out there these days, in a race to the bottom that everyone loses.

parker_mountain(10000) 2 days ago [-]

Great website. Great company. Can't find what you're looking for? Website failing you?

Don't worry, just click the email link at the top of the page and they WILL get back to you - even over a 20 cent part.

exodust(10000) 2 days ago [-]

> they WILL get back to you

Okay but responding to customer enquiries isn't uncommon.

ddhhyy(10000) 2 days ago [-]

I've never emailed them but I have called them on a few occasions. Somehow they pick up before it even rings once. And it's a real person, not a phone tree. Pretty remarkable.

s0rce(10000) 2 days ago [-]

They are really fast to get back to you and will tell you what stuff is which can be helpful in certain situations where the material or vendor might matter.

btown(10000) 2 days ago [-]

McMaster-Carr is an incredible example of how an existing physical-product distributor with a vaunted historical operations department, when faced with moving into the e-commerce world, could choose one of two paths:

- Build an online presence cheaply because you can rely on your historical offline operations to paper over any shortcomings in the online system.

- Build an online presence with the same ethos, standards, creativity, and long-term continued investment with which you built your offline operations to be your competitive advantage in the first place.

There aren't many legacy organizations that successfully digitized with that second path in mind, at least not at first - McMaster, B&H Photo, and Sweetwater come to mind as notable exceptions. It's a useful thought experiment, I think, even for those working on startups and green-field projects.

myself248(10000) 1 day ago [-]

Funny you should mention Sweetwater because they're also near the head of the table in my mental pantheon of e-commerce greats.

Their website is honestly just above average. Page-load times are fair to middling, the pages are very busy, vendor-supplied copy takes up much of the item page, and categories are difficult, perhaps just an artifact of the industry they're in. But if you dig deep, there is a lot of detail beneath, and the related-items suggestions are usually at least relevant if not spot-on, which is more than I can say for many sites.

But where they shine is the integration of humans into the process.

Back in 2015, I thought I might pick up a digital piano/keyboard sort of thing. I wasn't having a great time making sense of the lingo, so I gave them a call, and within seconds (I think I might've pushed one button in an IVR system), I was talking to real person who knew everything there is to know about keyboards. We talked for probably ten minutes about what I wanted to do, my budget, the ergonomic and acoustic constraints of my workspace, etc. He explained some terms to help me understand how the offerings differed and how those would impact my playing, and ultimately steered me to one of the models that was already on my short-list. We also talked about companion software, but I elected not to take that step at the time.

I went from hesitant to confident, and completed the purchase minutes later. The cool thing is, that confidence followed me through delivery, unpacking, setup, and my first plinking across the octaves -- I never felt alone like I was just gonna have to figure this out for myself, I _knew_ that this was the right instrument for my needs, and set out through basic finger exercises with the determination of one who's divinely inspired.

Then something even cooler happened: A week after delivery, that same sales engineer emailed me, just checking in. To make sure I got everything set up and was comfortable with it, to see if I could use any further help. As it happens, I was fine, but if something along the line had derailed me, this absolutely would've made a difference in getting me back on track. An expert whose entire job is to make sure I have a good experience with their stuff.

Now, I've had other e-commerce sites email me and follow up, sometimes with phone calls, but it's not the same. In some cases it feels like outright harassment (lookin' at you, Global Industrial). Often it's just a ploy for reviews or further business.

With Sweetwater, it's different somehow. Yeah I'm sure their goal is to keep me as a customer so I'll spend more money with them in the future, sure, and I'm fine with that. But they don't lead with the money, they lead with the caring. And that makes all the difference in the world.

With McMaster it's almost precisely the opposite, I've never spoken to a human there and I don't feel like I'm missing anything at all. They're like the supremely efficient robot that just produces the parts I need, immediately and without fail. There's no need for warm-fuzzies in this process, and they seem to understand that as well.

togaen(10000) 2 days ago [-]

Mcmaster is what the web is supposed to be like.

nraynaud(10000) 2 days ago [-]

I agree so much. It's what every single mail order company should have done. Freaking parametric search eaverywhere and uniform engraving style pictures so that your eye quickly catches the difference between the screw types.

cerved(10000) 1 day ago [-]

My mobile strongly disagrees

6Az4Mj4D(10000) 2 days ago [-]

I wonder what platform / framework / language was used to make this website?

stefcoetzee(10000) 2 days ago [-]

Seems to be React with .NET

ruined(10000) 2 days ago [-]

because the people ordering bar stock or whatever know what they need, and it's very specific, and they cannot be upsold

if you want to make a sale all you can do is give them what they want

most stores have to convince the user that they want something, and that could be anything, and it doesn't much matter what it is.

sh34r(10000) 2 days ago [-]

In a better world, we'd have more companies like McMaster selling people what they need and providing a highly valued service, instead of using ads and weaponized psychology to convince them to buy some more junk they don't need that's going to end up in a landfill by December.

fake-name(10000) 2 days ago [-]

McMaster is great.

This article is also completely glossing over another reason to use them: Their shipping is somehow insanely fast.

Almost everything is next-day delivery, even for standard shipping. They've somehow got their logistics COMPLETELY nailed. I believe they have their own agreements with USPS/FedEx/etc...

McMaster is generally not the cheapest, but they're almost always the fastest and the easiest.

Machining/industry runs on McMaster.

twarge(10000) 1 day ago [-]

Central New Jersey: Same day shipping, cheaper than ground: Order before 11, get it after lunch.

monstertank(10000) 2 days ago [-]

Do they ship to Australia? Any idea if that is also comparable to Amazon delivery speeds?

bnug(10000) 1 day ago [-]

Yeah, if you're in a large metro area (I was about 1 hour from the Atlanta distribution center), they would often ship SAME DAY if you ordered by about 10am. It was amazing, and very convenient during chaotic times when you forgot to order one tiny little piece of hardware that was needed before your custom built tooling could be delivered. This was in 2005 - 2007 timeframe, before Amazon started the 1 day stuff.

systems_glitch(10000) 1 day ago [-]

Yup, McMaster-Carr and ULINE are well known for usually being next-day on standard shipping! Saves a lot of headaches and projects.

s0rce(10000) 2 days ago [-]

You can get same day in some areas. We usually get next day in Reno.

morcheeba(10000) 2 days ago [-]

I'll second that. I ordered an 8-foot-long aluminum bar for $25, which, at the time, was too long to ship with traditional shippers. So, a semi trailer showed up at my door. Shipping cost $8.

bsder(10000) 2 days ago [-]

> They've somehow got their logistics COMPLETELY nailed.

In addition, you can generally get a useful human on the bloody phone at these companies!

The delivery priority is because their customers are losing real money until that shipment of parts to repair their production line comes in. If those parts don't come in when they're supposed to, the people who cut checks are gonna start yelling.

However, I tend to prefer MSC (https://www.mscdirect.com/) over McMaster if I can--especially for machine tooling.

It really feels like McMaster really relies on the fact that they can be a 'one stop shop' so my experience has been that things tend to be slightly more expensive and with 'servicable' quality and with 'very good' delivery. However, with a bit of Internet-Fu, you can generally find a better version of the McMaster product at the same price point.

thereisnospork(10000) 2 days ago [-]

Indeed, even in rural parts of the country orders placed by 4-6pm still arrive next day[0]. This saves calendar days on projects; its often not till 4 or 5 that the full list of missing fittings/parts/tools comes together.

[0]Sometimes you need to spring for air shipping if you are in the sticks.

pGuitar(10000) 2 days ago [-]

> Almost everything is next-day delivery,

Could it be that you live next to one of their warehouses?

On the other hand, Amazon delays orders on purposes if you aren't a Prime member.

beowulfey(10000) 1 day ago [-]

And on top of that you don't even need a business account. Individuals get the same service. I've ordered hard to find parts like tiny springs for home projects, just a few bucks. It's amazing.

eight_ender(10000) 2 days ago [-]

I'd like to add RockAuto as a peer to McMaster. It's auto parts, and somehow uglier than McMaster, but damn if it isn't the fastest no nonsense website I've seen for parts.

I can drill down to a specific part for a specific car in seconds, and see pics of the product, part numbers, specs. It's lovely.

LesZedCB(10000) 2 days ago [-]

holy shit, thank you!

bityard(10000) 1 day ago [-]

Rockauto is one of my favorite things in the world. The site is a perfect case study in how to make a UX that's ugly as sin but has evolved to be a million times more usable than anything the 'minimal sleek modern UX' crowd could ever come up with.

You always roll the dice with aftermaket parts of course, but my experience has been that they usually don't sell utter crap like you'll find on Amazon and eBay. They make some effort to verify that the parts are not bottom-of-the-barrel because they don't want to deal with returns and after all, they are car people themselves.

My favorite thing that I like to tell people that have not heard of RockAuto yet: I can buy a full of wiper blades for my two cars for about the same price as ONE mid-quality wiper blade at Walmart.

inamberclad(10000) 2 days ago [-]

McMaster doesn't show you shipping and tax before you place your order! How is that legal? Everything else is fantastic though. I adore their catalogs and having the print version is a point of pride.

wildzzz(10000) 2 days ago [-]

Because most of their business comes from other businesses that receive quotes and pay net 30. The buyer already has a FedEx account so they just send that and it's billed monthly.

mhb(10000) 2 days ago [-]

They didn't used to and it was annoying. Now they do!

loh(10000) 1 day ago [-]

It sort of reminds me of a video game UI for an RPG or something. Game UIs usually do a great job of organizing and simplifying complex hierarchical information and making it quick and intuitive to find what you need. A lot of the web tech I've built over the years has been inspired by well designed games.

warent(10000) 1 day ago [-]

Yes completely agree. I was going to post the same thing and did a quick CTRL+F to see if anyone else felt similarly. I don't even play video games that often but this immediately made me think 'Wow this really feels like a video game somehow'

hedora(10000) 2 days ago [-]

My homebrew keg was leaking CO2, and the brewing supply company only sells the same crappy (= non-silicone) o-rings that had just failed. They wanted something dumb like $25 for a replacement part.

Enter McMaster-Carr. For significantly less money than fixing one bad o-ring, I have a pile of replacement sets for every seal in the system.

amluto(10000) 1 day ago [-]

Is silicone really a good material for this? In my experience, silicone picks up smells and is not so easy to clean.

(The part I can't find on McMaster or anywhere else is a high quality EPDM or FKM gasket for a garden hose.)

PaulDavisThe1st(10000) 2 days ago [-]

At some point several years ago, Amazon actually tried to compete with Mcmaster and others selling in this niche. They had a sub-site or something like that, I think it was called 'Amazon Pro' and it focused on nothing but the sorts of parts and fittings that Mcmaster sells. It wasn't bad, but it wasn't as good as Mcmaster, and it sank (AFAICT) without a trace.

robk(10000) 2 days ago [-]

Amazon pro failed badly due to mismatch of service levels. Raptorsupplies.com is a similar one that's purely non-US though.

PaulDavisThe1st(10000) 2 days ago [-]

EDIT/UPDATE: apparently it was called Amazon Supply, later 'upgrade' to Amazon Business, and now sunk without a trace, it seems.

sitkack(10000) 2 days ago [-]

Amazon bought a company 'Small Parts' in 2005, they were a knock off of McMaster. This was probably to make sourcing fasteners easier for Blue Origin.

fbn79(10000) 2 days ago [-]

The site is using Yahoo User Interface, discontinued since 2014. As I always suspected stick to old tech is not always bad.

throwaway93702(10000) 1 day ago [-]

I used the YUI framework back in the days for 'backoffice' / 'admin' pages. It was one of the most productive frameworks. It had a lot of really good abstractions around dom manipulations, data sources and events.

It also had an amazing set of components including an awesome charts library. The data table was state of the art at the time, and could probably beat many of the react tables that are made today. From my point of view it was so much better than most of the alternatives, but it had the negative stigma of a Yahoo ownership.

ShuffleBoard(10000) 1 day ago [-]

OMG I frikkin hate McMaster's navigation and comparison options. Their standardization is even quite sloppy, i.e. some screw vendors apparently include the head height in the length spec, and you don't find this out until your box of #2-56's show up for your precision fixture and are ~1mm too short.

I always tolerated McMaster's crappy UX as just a symptom of how MEs are willing to tolerate more shaggery than CS types.

jamincan(10000) 1 day ago [-]

Go to the product detail and they show precise measurements and often CAD models of the part.

motoboi(10000) 2 days ago [-]

Now be completely honest:

- If you haven't read the article, would you think this site great?

My point is: no one knows what is a great e-commerce website. Is one that sells a lot? Amazon does it.

Is not one that sells a lot, but that is very easy to use?

ellyagg(10000) 2 days ago [-]

I came across mcmaster.com like 10-15 years ago and I thought its design and experience was amazing. I didn't understand why other sites weren't like this.

Then I talked to regular people (like my friend who builds high-end magic tricks) who happened to use the site and like me wondered why more sites weren't usable like this.

And then after a few years, I saw that people in the comments here and elsewhere thought the same.

And this post is the latest.

I think there's something there.

klyrs(10000) 2 days ago [-]

Yes, because its catalog is also a CAD library, so you can put the parts into your design and be sure the parts will fit when they arrive tomorrow. Because every part is neatly categorized, searchable by its properties.

I am infuriated when shopping for clothes, computer parts, etc. because the items are rarely categorized and when they are, it isn't comprehensive so searches tend to miss out on the obvious so I end up browsing a site's entire catalog.

And, they do sell a lot. They're the industry standard. The only problem I've had is that they won't ship to individuals in Canada so I had to think up a fake company name.

rossdavidh(10000) 2 days ago [-]

'If you haven't read the article, would you think this site great?'

No, I would have ordered what I wanted, quickly and easily, and probably not thought about the website at all. Which is what makes it great.

jbay808(10000) 2 days ago [-]

> If you haven't read the article, would you think this site great?

If you search HN comments for 'McMaster-Carr', you'll see their website is frequently brought up as an example of great UX, going back over 10 years.

paulmd(10000) 2 days ago [-]

lol, mcmaster carr is probably one of the only specialty e-commerce sites that anyone will randomly bring up in UX discussions or hobby conversations. everyone loves mcmaster carr, like actually everyone, so yeah, lots of people who haven't read this already independently decided that mcmaster carr is great.

even if you're not going to buy there (their prices are high, compared to alternative suppliers, is what I've heard, and there are items and brands they don't carry) it's still a super useful reference of what's possible and what general solutions in that area look like and cost, or even just basic lookup for what types of nuts and bolts you should look for in a project. like it's actually just a really nice pleasant interface for working with hardware in general.

spfzero(10000) 2 days ago [-]

I haven't read the article, and I think the site is great. I've used it many times and it exemplifies what I think a great e-commerce site should be (from a customer perspective). They've built their website so you can find and order what you need with the bare minimum of effort. If I imagine trying to search for some of the things I order on McMaster, except on Amazon, let's just say I'm glad I don't have to.

GavinMcG(10000) 2 days ago [-]

Ok, but if we're being honest, you can make that sort of claim about anything and it's equally as unhelpful. Things can have value even if you can't perfectly define a metric, and even if others might use different metrics.

The author explained why they think it's great, so rather than waving your hands and denying that greatness could ever be defined, what about a constructive approach? Engage with the elements the author identified. Criticize specific ones, or suggest others. Say something you actually believe!

peteradio(10000) 2 days ago [-]

Curation and variety would top my list of important features.

int_19h(10000) 2 days ago [-]

That website routinely comes up in a lot of crafts and craft-related hobbies (in my case, it was guns, where gunsmithing can involve parts they sell), so it's not exactly a secret. And yes, people do actually like it regardless of the article.

OTOH I don't know of any serious hobbyists (of whatever) who get the majority of their supplies from Amazon. When you ask, people generally say that selection is poor and quality is always highly suspect.

nocman(10000) 2 days ago [-]

> My point is: no one knows what is a great e-commerce website

I disagree. I know what is a great e-commerce website -- in the sense the article is talking about.

  - loads fast
  - makes items easy to find quickly
  - has a simple, efficient shopping cart mechanism
  - gets me in, order placed, and out in the shortest time possible
  - is run by a company with great customer service
Yeah, there's probably an item or two I've forgotten from that list, and yeah, other people likely to order from a site might have a slightly different list. But I have no doubt that I would recognize the same qualities on other websites that I and many (probably most) people would recognize as great -- or at the very minimum, as websites that don't suck.
iancmceachern(10000) 2 days ago [-]

Mechanical Engineer and product designer here. I've been using mcmaster daily for 20 years to do my job. It is definitely the standard by which all others are judged. I'll gladly pay 2, 3, 5, even 10x the price to get it from McMaster. The service, the CAD models, I have what I need the next day. As an example of the level of their service, you can send them a box of loose parts, no part numbers, no markings, and they'll go through it, figure out what's what, and refund you for it. They make it so easy, you just have to do the engineering. Thank you McMaster-Carr.

walrus01(10000) 2 days ago [-]

mcmcaster carr is also extremely popular in the hobby drone building market for buying bolts/nuts/fasteners/standoffs to attach circuit boards (ESCs, flight controllers, etc) to carbon fiber plate, and for attaching cnc cut carbon fiber plate piece together to build the whole frame of a small UAV.

people in the film/tv/production industry also use them to buy various fasteners and hardware for rigging lights, microphones, building complex camera systems onto gimbal platforms, etc.

tomcam(10000) 2 days ago [-]

> you can send them a box of loose parts, no part numbers, no markings, and they'll go through it, figure out what's what, and refund you for it

That is insane. Mad props to them

JohnGB(10000) 1 day ago [-]

Oh how I wish Mcmaster would open an EU branch... I don't even mind if they charge us more, but PLEASE for the love of all engineers in Europe, just open a branch here.

bluedino(10000) 1 day ago [-]

High school shop/engineering teacher held a copy high: 'Everything you will ever need, is in this book.'.

unixfg(10000) 2 days ago [-]

I even buy home supplies (air filters, cleaning, etc) from them. They're wonderful.

nbadg(10000) 1 day ago [-]

A friend of mine worked there right after college. I happened to buy some girl scout cookies through her, and joked with her that you can get everything from McMaster. When she brought me the cookies, she brought me a fake McMaster invoice for them too.

I've heard some mixed reviews about what it's like to work there, but I completely agree: as a customer, despite the (sometimes very high) prices, I was always incredibly happy with McMaster.

thechao(10000) 2 days ago [-]

We had piles of their yellow books laying around the house when I was a grad student. Definitely good toilet-book material.

JohnJamesRambo(10000) 2 days ago [-]

Maybe your customers or employer would like you to save 2, 3, 5, 10x the price...

My memories of McMaster from doing my hobbies is that it is outrageously expensive.

jwr(10000) 2 days ago [-]

Alas, for some reason they only cater to the US market and have no presence in the EU, for example. So while it is possible to order from them, it's rather inconvenient as you need to deal with customs/taxes.

Digi-Key got this right: you can order DDP, which makes things quick & simple.

Incidentally, I am amazed that they don't have a meaningful competitor in the EU. It seems most companies here are stuck in the 80s and dedicate next to zero effort to their web ordering process.

GuB-42(10000) 2 days ago [-]

I often hear about McMaster from US sources and how great it is (though you pay for convenience), but are there equivalents in Europe? France in my case.

soneil(10000) 2 days ago [-]

Farnell is my nearest equivalent (in Ireland). Same 'why doesn't everyone ship like this' experience, but more electrical than mechanical.

I'm in the same boat as far as looking for a MMC equivalent (they refuse to ship here at all)

talkingtab(10000) 2 days ago [-]

Another vote for McMaster-Carr. It is not like they are great, they just are not stupid or frustrating or annoying at all. I don't know of any other websites that meet that relatively simple standard.

Perhaps part of why their website is so good is because they deliver the goods. They follow through. Ever need a 4x8 sheet of copper? It gets delivered to your front door. Sooner than you expect, and at a very fair price.

wildzzz(10000) 2 days ago [-]

I like buying auto parts from RockAuto. Their website is dead simple to use. They have nice features like good/better/best, showing the most popular item of that category for that car, an icon for OEM parts, and another icon that shows items in the same warehouse as the item in your cart. So when you can't decide between a dozen brands of sparkplugs, you can pick one that is both OEM and is in the same warehouse as the air filter you needed so you only have to pay for one shipping cost.

nocman(10000) 2 days ago [-]

> It is not like they are great, they just are not stupid or frustrating or annoying at all.

Sad to say, comparing them to most other sites where you try (or attempt to try -- yes, often it is that bad) makes a case for saying, yes indeed, they are great!

aidenn0(10000) 2 days ago [-]

Yeah. It's kind of like how in chess, many learners study openings and gambits and obscure checkmates, when the low hanging fruit is just learning to spot and avoid blunders.

MMC is mostly about not making the mistakes everybody else makes, which puts it near the top even before you consider what they do well.

int_19h(10000) 2 days ago [-]

I find that to be the case for many 'industrial' (for the lack of better word) online stores. E.g. for plastic containers, https://www.usplastic.com/ is similar.

toddmorey(10000) 1 day ago [-]

'People visit McMaster-Carr with high intent to buy a specific part, that's it.'

Mcmaster is great, but it's important to point out there's no brand or product selection decisions. You pick your criteria and get exactly one option. It lends itself to a very different experience.

How would this work for cars or couches or headphones?

klausjensen(10000) 1 day ago [-]

Wouldn't that be exactly the same, just with more filters?

When I am on Amazon, trying to find something, I always feel like I have to fight the site, rather than being helped by it. Paid ads, very poor categorization/property-search, deceptive UX, fake reviews, 'amazon picks' etc etc etc. And when I find the right product, I might receive a fake.

I think what people enjoy and celebrate about McMaster-Carr is that it is exactly the opposite of that.

systems_glitch(10000) 1 day ago [-]

You never have to worry about it being junk, though. When I order fasteners, metal parts to be machined/used in fabrication, or just misc. supplies from McMaster-Carr, I know whatever supplier they are using will be high quality and the part won't be a counterfeit. It also seems that they tend to stick with US manufacturers if possible.

danpalmer(10000) 1 day ago [-]

This is almost exactly the problem Thread is solving for clothing (I worked there for >7 years).

Big retailers will have 100 different white t-shirts to choose from. Thread will (roughly) tell you which white t-shirt to buy.

The tricky bit is that fashion is _so_ personal, so what you have to do is actually have all 100 (or in many cases _far_ more), with all the necessary data to be able to differentiate, and then understand 'style' in such a way that you can confidently recommend specifics. I'm biased, but I think Thread has become really very good at this.

hwbehrens(10000) 1 day ago [-]

> there's no brand or product selection decisions

> How would this work for cars or couches or headphones?

This would be great for many things! I'm sure there are plenty of people who just want a good quality, reasonably priced -whatever- and would be fine with this model. Actually, that seems like it's basically just the business model for Kirkland.

If there was a retailer who just sold one carefully-chosen thing in each category, I think they'd do pretty well. They wouldn't be able to capitalize on market segmentation, fashion trends, etc. which would hurt their profitability, but they'd also benefit from being the 'default' choice for many people.

lynndotpy(10000) 1 day ago [-]

Many good examples in this thread, surprised not to see Monoprice mentioned.

You only compare among Monoprice, and they usually have a decent bar for quality.

Spivak(10000) 1 day ago [-]

This is basically Costco, Trader Joe's, or IKEA -- all well-loved business. Small selection but their customers like what's on offer and don't have to comparison shop. Turns out that people really like soviet store style businesses and 'huge selection' doesn't really live up to the marketing hype.

jdc0589(10000) 1 day ago [-]

It might not work, and it doesn't have to. McMaster isn't designed to be a highly competitive retail marketplace for the average consumer or the average consumer product.

It's for people building things that already have a good idea of what they need. For the most part, if you need an 18mm long 1/4-20 flange head hex-socket grade-8 bolt in a black oxide finish, you don't care about having a ton of options from different manufacturers, because there should not be a meaningful difference between any of them.

dehrmann(10000) 2 days ago [-]

One thing I love them for is their curation. They find solid tools from companies the average retail buyer might not know.

s0rce(10000) 2 days ago [-]

Yah, you generally don't have to worry, need a ratchet, need a metal shelf, need an air filter, it's going to do what it's supposed to.

s_severus(10000) 1 day ago [-]

Small note on the part about the price range UI controls: the author criticizes Amazon for using min, max number inputs rather than a slider control.

2 arguments against a slider:

1. Potentially it is much harder to make an accessible slider control. 2. Sliders break down when you have a distribution skewed away from normal. For instance, if 90% of results are in the range $0-$50, but then a few results are over $1000, how do you calibrate the slider? A naïve approach would render it almost unusable if you want to limit to products between $20 - $30. Otherwise you need some sort of logarithmic scale. I've not seen examples of this being done well.

ruiramos(10000) 1 day ago [-]

Agree with you here. I thought the article made some good points but I've also been bitten by the 'naive' [mix, max] slider you describe - I did yearn for some good old text boxes back then!

jkhdigital(10000) 2 days ago [-]

I normally browse HN incognito on my phone but had to go to my desktop and login to say this: I fucking love MMC and their website. I got into woodworking during the pandemic and eventually built a loft bed for my son. I needed some very specific sizes of furniture bolts in small quantities and Amazon just didn't have what I needed. McMaster-Carr to the rescue!

I agree that they aren't the cheapest, but the quality is unimpeachable and the shipping is fast and reasonable even for small (i.e. retail) orders. I also love how everything in the catalog includes a brief but concise description of what it is used for.

Also, at the risk of perpetuating (largely accurate) stereotypes about gender psychology, it's clear that the website is designed around the typical male shopper: I am shopping to fulfill a very specific need and I want to acquire the solution as quickly and efficiently as possible.

ISL(10000) 2 days ago [-]

I have never noted a gender disparity among colleagues who use McMaster. Everyone likes it just fine.

Forge36(10000) 2 days ago [-]

Why is that a typical male experience? There's no reason a female would want anything different here

rfrey(10000) about 23 hours ago [-]

Your gender stereotypes are made up, and ridiculous to boot.

Dave3of5(10000) 1 day ago [-]

Eek I'm going to go against the grain here a little. That site is useful when you already know what you are looking for. For someone who's just browsing for something interesting to buy it's terrible.

Obviously it's been made for exactly that purpose and does it well but for general e-commerce it's bad. The retailers are looking to send you to an area of their website with 'stuff' you are interested in. For example you like toys because you're a parent who's buying your kids stuff for x-mas. The site wants to know that sort of thing and send you to the toys for x-mas section.

What Mcmaster.com looks like to me is an old fashioned mail order company that's converted their old mail order catalogue straight to a website, perfection.

Also note with Bezos tinkering with every pixel on the website is very old news, this doesn't happen anymore.

P.S. I'll eat the downvotes.

MoSattler(10000) 1 day ago [-]

Maybe check out https://www.govolition.com

analog31(10000) 1 day ago [-]

I think the 'yellow catalog look' is just an aesthetic. The site is actually extremely modern.

jjk166(10000) 1 day ago [-]

> For someone who's just browsing for something interesting to buy it's terrible.

No one is going on McMaster just looking for something interesting to buy. And frankly if you're not amazon, odds are people are trying to buy something specific from you as well.

muspimerol(10000) 1 day ago [-]

Yeah the title is clickbait. McMaster is regularly praised in engineering forums, which is great. Clearly they nailed the UX for their users. There are some good lessons to learn from it, but it's not some sort of golden standard for e-commerce as a whole.

The main praise is usually search/filter, but that really comes down to this quote from the article:

> McMaster is able to provide such intuitive searching and filtering because everything that they sell is highly legible – it's all defined by quantitative specs.

Most products don't lend themselves so well to filtering.

blitzar(10000) 1 day ago [-]

Bezos is too busy tinkering with the sorts of things that wouldnt go anywhere near him when he wasnt a billionaire to bother with a website

hakfoo(10000) about 23 hours ago [-]

> For someone who's just browsing for something interesting to buy it's terrible

Maybe that's the core issue. How are people expecting to interact with the site?

I think when you're selling with an obvious B2B focus, the 'value subtract' nature of a lot of consumer-targeting selling practices become clear.

There's no point trying to jump a B2B customer through a bunch of cross-sells and (poorly) related products, because there's a fair chance he doesn't even have the purchasing authority to add any of them.

If your search or filtering sucks, time is money, and they're going to go somewhere that respects that.

Maybe what retailers should be looking at is ways to break up the 'browse' from the 'focused buy' experience.

Instead of getting me in-cart for the browsing experience, build some external content. Instead of 'here's a category of 3000 toys, start clicking', write up some high quality articles about different products, target markets, etc. with deep product links. Since you're curating the content, you have a chance to 'conveniently' sort the products to promote what you want or whoever gives you a kickback for browsing customers, while not frustrating the 'focused buy' customer with a deluge of EKTIBAQNN brand USB cable/fire-starters.

crabmusket(10000) 1 day ago [-]

This is why I love building B2B software. You get to help people do their jobs, rather than trying to sell them things. The site in the OP is a fantastic example of building a tool to help someone be successful at their job.

To be fair, 'searching for a present for my kid' is a legitimate job in the jobs-to-be-done sense... it's just a lot fuzzier than a 'business job' which has external constraints.

fmajid(10000) 2 days ago [-]

The counterpoint to this is that a site like Amazon can still succeed to dominate the industry despite being a horrendous mess with nonfunctional search, a horrendous counterfeiting problem, and widespread review fraud. Does take the wind out of the sails of IA/UX purists somewhat.

paxys(10000) 2 days ago [-]

Amazon succeeds because they can get a tube of toothpaste, an onion, a washing machine, an extension cord, a can of paint and an iPad at your doorstep in under a day. Sure their search interface might not be the best but no one really cares. They are a logistics company, and don't go too deep into any one niche.

seabird(10000) 2 days ago [-]

Amazon does not dominate the market that McMaster-Carr exists in. It's not even in the running. If you attempted to use Amazon for the type of project McMaster-Carr exists to support, you're going to get laughed at at best. MMC/Misumi/Grainger/etc. own industrial/mechanical, Digikey/Mouser/Arrow/etc. own electronics, RockAuto owns automotive aftermarket, etc. It's actually incredible how little fight Amazon has put up in these markets.

guywithahat(10000) 2 days ago [-]

I feel like the part everyone is ignoring is that McMaster sells a very small breadth of things with incredible depth in its selection. The features people are raving about in this article and thread only make sense in the context of selling bolts for industrial use, and could never be implemented at a place like Amazon.

chrsw(10000) 2 days ago [-]

Purchasing from McMaster is a very refreshing experience. Couldn't agree more. Loads quickly, looks the same on any browser I tried it on, I don't feel lost or like I'm wasting my time. However, it raises a quetion for me, how is it that they're able to stay in business and simultaneously respect their customers' time and sanity? If you look at the way other sites are designed, you'd just assume there's no way to make money with a frictionless, no frills, no distractions, no tricks storefront.

njsubedi(10000) 2 days ago [-]

You answered your own question - they're able to stay in business because they respect their customers' time and sanity. People may not use McMaster because they land on the homepage and see a "Mega Deal" going on in a product they don't need. People who visit the site are almost certainly going to buy something. Compare that to other e-commerce sites where people just visit to check specs, compare prices or just landed on the page from the search index.

elromulous(10000) 2 days ago [-]

Also, the website has basically been unchanged for the ~20 years I've been ordering from there.

s1artibartfast(10000) 2 days ago [-]

I don't miss not having featured items, fake reviews, and suggestions

s1artibartfast(10000) 2 days ago [-]

I love McMaster, but also want to give a shout out to their competitor, grainger.

Similarly excellent UI. One benefit that I take advantage of is free delivery to their local warehouse for in person pickup.

I don't know why more companies don't serve the customer with this.

bombcar(10000) 2 days ago [-]

Grainger isn't directly a competitor as they focus on 'everything' but they're amazing.

You can pay them to open the local branch at 2 AM on a Sunday if you need a part and they have it in stock.

bityard(10000) 1 day ago [-]

I love having a Grainger within walking distance of my workplace because it means I can order some bearings (or whatever) and go pick them up on my lunch break. Grainger's prices are nothing amazing but sometimes the total purchase price ends up being less than all other options when you don't have to pay for shipping.

mherdeg(10000) 2 days ago [-]

Grainger was my N95 respirator buddy for a while. Fast shipping / great logistics too.

a-dub(10000) 2 days ago [-]

digikey, jameco, mcmaster, ingram...

most of these online catalog parts houses have excellently functional websites.

but like, even back in the day distribution warehouses had excellent terminal based inventory search and ordering interfaces over packet switched networks.

anyone remember datanet?

a-dub(10000) 2 days ago [-]

also, the reason why they're all good is because they're not trying to sell you anything. they're for people who are at work, know what they need, know they have options on where to buy. therefore it's streamlined as much as possible to make life at work easy (which becomes their differentiator, along with easy billing terms and fast shipping), rather than market things to people who aren't even making decisions on what to buy...

dwiel(10000) 2 days ago [-]

I wish digikey was as good as McMaster. With digikey I often have to parse through a huge table with many pages to figure out what the difference is between a bunch of search results. On McMaster you can clearly see search results categorized in sub tables and densely arranged to make clear what the options are.

Are there any electronics websites that actually organized like McMaster?

mlindner(10000) 2 days ago [-]

I'd throw in smatec if you're looking for cables or electrical connectors. They at least used to send individual samples for free as well.

nocman(10000) 2 days ago [-]

Wow, Jameco -- I haven't thought about them for a long time. When I was a kid I used to get those paper catalogs full of electronics parts and browse through them, wishing I had both

1) the money to order whatever I wanted from them and 2) the knowledge to make cool things with all of those parts.

One of these days I'm going to find some time and do a few electronics projects. I've always been interested, and I did a fair amount of simple experimentation as a kid, but as an adult life has kept me busy with other pursuits.

lbriner(10000) 2 days ago [-]

'Functional websites' is not what we are discussing here, but amazing websites.

I've seen a carparts store online and it had something really similar, a few words like 'oil filter 2017 ford mustang' and you were literally looking at the part in the dropdown list, exactly what you want.

What do most sites do wrong?

1) Not loading the search data into RAM on the server so it is really fast - most companies could do this with their level of stock and only need a couple of GB per web server 2) Requiring a postback to start filtering 3) Not allowing the ordering of search terms to be changed e.g. 'Ford Mustang Oil Filter' should match the same as 'Oil Filter Ford Mustang' 4) Not having images for all results 5) Not being able to zoom in on images to check something is what you think it is 6) Using a stock image which doesn't necessarily match the item

For companies whose primary interface with the public is searching for stuff, most e-commerce sites I use are shamefully bad at both performance and usability.

Kudos to Mcmaster!

adhesive_wombat(10000) 2 days ago [-]

RS, on the other hand, are so-so. The new JavaScript stuff looks shinier but still they don't surface out-of-stock status in the search which is infuriating when do many things are on long lead times.

syntaxing(10000) 2 days ago [-]

MechE here as well. An alternative to McMaster is Misumi. Misumi is nice because you can order parts that are made to order (imagine a rod with the specific thread and length). No next day shipping like McMaster but being able to buy a custom "off the shelf" part without making a special internal part number is huge (so you don't need drawings or anything). They'll ship you a couple books like the signature yellow book from McMaster. They'll even host a seminar explaining why "good engineers design, great engineers integrate" (really only true for the industry and not consumer tech).

xwdv(10000) 2 days ago [-]

Good engineers design. Great engineers integrate. Legendary engineers iterate.

jbay808(10000) 2 days ago [-]

Misumi is awesome. It's basically Japan's answer to McMaster-carr.

10 years ago I would have said Misumi's website was better than MMC. Unfortunately Misumi has gotten a bit less usable over the past decade, while McMaster has improved. Misumi's paper catalogue is incredible though and I surprise myself to say I would rather browse it than either their or McMaster's website.

Top quality configurable / customizable parts is certainly the way of the future, and it's amazing that Misumi has the whole process figured out. You can get a custom precision leadscrew of the length you want with the ends turned down to the diameter you want and the retaining ring grooves you want, and buy just one piece for a reasonable price. Any ordinary machine shop would reject the order unless you bought fifty or more.

Arcanum-XIII(10000) 2 days ago [-]

And Misumi works well in Europe, which helps a lot. They also have quite often great prices. Now the catalog is not as far reaching as Mcmaster. We don't have a Mcmaster in Europe :(

Maarten88(10000) 2 days ago [-]

> being able to buy a custom "off the shelf" part without making a special internal part number

How does that work? I have been searching the internet for stainless steel M8 x 20 set screws with a torx socket, maybe uncommon but straightforward. I have yet to find these anywhere outside of having them manufactured in china by truckload - I need maybe 100.

wildzzz(10000) 2 days ago [-]

Mil spec part numbers for things like screws are great! We needed a very long screw in a specific style and material that no one stocked. We pulled up the mil spec datasheet for that style of screw, made up the part number based on our needs, and sent it to our fastener vendor. We get back exactly what we designed for without having to spend time designing our own special purpose screw. The vendor already has the spec so we can just point to exactly what we want and none of us have to spend time going back and forth to find a compromise.

c_o_n_v_e_x(10000) 2 days ago [-]

I've used Misumi on projects before. I've ordered all kinds of custom stuff including drive shafts and tons of different brackets. Everything comes neatly labeled and bagged. The best part IMO is the custom aluminum extrustion (8020 clone). You can order everything cut to length, with holes drilled, holes tapped, etc. The lead time sucks though. I'm green with envy over all my friends back home that get to use McmasterCarr.

RicoElectrico(10000) 2 days ago [-]

I was very confused when Ben Krasnow pronounced McMaster with a 'mick' at the beginning. For 10+ years of my English education nobody bothered to explain that Mc is 'mack', but sometimes it's pronounced 'mick'.

I was aware something is going on when Tyler McVicker introduced himself with 'mick'.

English teachers usually have a typical repertoire of their pet details like schwa and so on, but seems there's a blind spot here.

bityard(10000) 1 day ago [-]

I expect pronunciation largely boils down to regional and personal preferences.

The 'Mac' prefix means 'son of' and 'Mc' is an abbreviation of that. The 'a' is schwa and where I grew up, is pronounced with a short I, as 'mick'. But I have heard others pronounce it 'mack'. The McDonalds ads on TV and radio seem to try to split the difference by pronouncing it 'meck'.

It's also possible that some people go out of their way to say 'mack' in an attempt to acknowledge and avoid the 'mick' slur that was sometimes used in England and in the US against Irish and Scottish immigrants.

NavinF(10000) 2 days ago [-]

Hmm isn't McMaster pronounced just like McDonald's?

lvl102(10000) 2 days ago [-]

It's overpriced. I rather order from Misumi or Accu.

DaveExeter(10000) 2 days ago [-]

McMaster is only overpriced if you are spending your own money! :-)

If you are spending your own money you might find other suppliers offer more value.

McMaster-Carr

Grainger

...and there are a few more.

isatty(10000) 2 days ago [-]

McMaster is defintely amazing. Now I wish that Digikey and Mouser got their shit together when it comes to providing models and footprint files without me having to scour the internet for them, without having to login to various sites.

chrsw(10000) 1 day ago [-]

That would be amazing. A dream come true. The Amp Hour podcast has touched on this a little bit. https://theamphour.com/577-product-lifecycle-management-with...

I guess it's a pretty complicated problem. There are so many suppliers, standards, file formats and competing interests. But it does seem like it's a market ripe for disruption... somehow.

14(10000) 2 days ago [-]

Last I checked they don't deliver to Canada so not the best site I have used as I couldn't order the parts I needed. They were the only one who had the part I needed so had to get an American friend to buy it and ship it to me afterwards. I really hope they can see Canadians would buy from them if we could.

jamincan(10000) 1 day ago [-]

It seems they only do B2B in Canada. We use them all the time.

exabrial(10000) 2 days ago [-]

Yeah but it's not node, wasm, sass, typescript, react bootstrap, rails, flask, ecma, flash enabled, spa web 3.0 with social media and crypto integration so it's pretty much outdated and useless....

/s obviously, but it just points to the absolutely uselessness of 'modern' websites

PontifexMinimus(10000) 1 day ago [-]

> Yeah but it's not node, wasm, sass, typescript, react bootstrap, rails, flask, ecma, flash enabled, spa web 3.0 with social media and crypto integration so it's pretty much outdated and useless....

Ah yes, the oh-look-a-shiny-thing development paradigm.

schappim(10000) 1 day ago [-]

>> /s obviously, but it just points to the absolutely uselessness of 'modern' websites

So what is your cut off for 'modern' here? Rails was released August 2004...

hpeinar(10000) 1 day ago [-]

But the whole point is that the UX of the website is perfect. It has nothing to do with what the webapage is built on.

It could be built on PHP, it could be Java, it could be node, it could ASP.NET but if it's fast and with perfect UX like Mcmaster.com, what's the difference?

It is definitely possible to build such a website using all the mentioned languages, they have happened to have chosen ASP.NET and have done it well, but I wouldn't blindly throw everything else under the bus with such a comment.

And they aren't clear of 'fancy web shenanigans' as well, they use socket.io and their website is unusable without Javascript so there's a lot of going on and this isn't your typical static website anyway.

So I think your sarcasm is a bit out of place in this instance.

And yes, I'm biased as I've been node.JS developer for a long time but I don't agree that the languages are the ones building shitty UX webpages that are slow... it's the developers that do that.





Historical Discussions: Outdated vs. Complete: In defense of apps that don't need updates (September 26, 2022: 880 points)

(887) Outdated vs. Complete: In defense of apps that don't need updates

887 points 1 day ago by ingve in 10000th position

vivqu.com | Estimated reading time – 12 minutes | comments | anchor

On August 22nd, I got an email out of the blue from Apple that notified me that I had a new App Review message. It was for my app, WorldAnimals, a light-hearted game for guessing animal onomatopoeia sounds in different languages.

Usually, you receive a message after you submit a new version to the App Store for review. The reviewer has found something wrong and your app is rejected. The notice explains where you have violated the App Store Review Guidelines and how you can rectify the issue to get your app update approved. Maybe the app is crashing, the reviewer can't log in to test, or god forbid the update description mentions the evil rival mobile platform–these are all actual reasons I have been rejected by the App Store.

The relationship between developers and App Store reviewers is tense at best. Most people are trying to build well-designed, useful mobile apps. Apple has instituted App Store reviews to maintain a high-quality bar for apps and weed out spammy or nefarious actors, using human evaluators to test individual apps and provide direct feedback. However, malicious apps are relatively rare; arguably, Apple doesn't do a great job filtering them out anyway. So for the vast majority of developers, App Store reviews add an additional layer of friction and time to shipping updates. And then in the inevitable case when you need to push out an emergency fix for crashes happening to your app users, the App Store review process goes from an inconvenient annoyance to an outright roadblock to improving the user experience.

I've been around the block, both as an indie developer and a full-time-employed mobile developer. I have released fun, tiny games like WorldAnimals and production SaaS mobile apps. When I was at Pinterest, I helped in communications with our dedicated App Store representative who would expedite Pinterest app updates through the review process. I have seen first-hand the lack of support for indie apps compared to the white glove experience that large companies get. Suffice it to say, I probably have above-average knowledge of how this whole process works.

And still, I was surprised to receive an App Review message. I hadn't submitted a new update for WorldAnimals. The app was still working well, with zero crashes and a handful of new downloads every month. My boss had even shown me last week that he had downloaded my app on his phone for his daughter–we played the game together during a work meeting and laughed at the silly animal sounds. In my mind, there was no reason I should be receiving a vaguely threatening message from Apple's App Review system.

Well, it turns out, Apple's problem with my app was the fact that I wasn't updating it.


I opened the message and was greeted with the "App Store Improvement Notice". I was essentially told that I hadn't updated my app in three years and now it counts as outdated. I needed to update the app within 90 days or it would get automatically taken down.

The message I received from the App Store Review told me my app was "oudated".

Never mind the fact that my app has a 5-star rating and was still being downloaded, with no complaints from any of my users. Also disregard the fact that I had other highly-rated apps up on the App Store, some of which had been updated much more recently than July 2019, clearly showing that I have not abandoned these apps entirely. If there had been an actual reviewer who checked my outdated app, they would have discovered that I architected the app from the beginning to dynamically scale the UI so it resizes to fit the latest iPhone devices. All these could be signals that indicate to Apple that this is not a garbage-filled scam app that is lowering the quality of their App Store.

Many other developers have complained about this draconian measure. The decision to remove outdated apps places a disproportional burden on indie developers and hobbyists because they might not have time or resources to revisit these old apps. Just having an active Apple developer membership costs $99 a year. It feels a bit like extortion when the platform, which you already paid to publish your app once, is now requiring you to continue renewing your membership to make that same app stay available.

Beyond the financial cost, what is the most insulting to me about Apple's policy is how poorly thought out their measure of "quality" is for apps. The message contains two separate statements about my app: (1) it hasn't been updated in three years, and (2) it doesn't meet a "minimum download threshold." Fixing either of those so-called problems doesn't magically mean my app will be a high-quality, positive experience for users.

There is absolutely no requirement for what is contained in the app update since Apple only states, "the app will remain available if an app update is submitted and approved within 90 days." At least in Google's case, the Play Store requires an update to their minimum target level API which ensures that developers use a newer version of Android Studio.

The intention behind Apple and Google's policies is to force developers to make sure their apps run successfully on the latest devices and operating systems. Apple claims that the App Store improvement process will improve user experience because "keeping apps up to date to conform with modern screen sizes, SDKs, APIs [...] ensures users can have a great experience with any app they get from the App Store". But my app was already working correctly, making this update useless and performative.

My app was working well even on the latest iPhone and iPad devices.

Even more frustrating is that while the app itself runs fine on all the latest end-user devices, Apple's development ecosystem has changed rapidly over the last three years. My old code simply did not work anymore with the latest versions of Xcode. So I had to spend four hours on Saturday upgrading all my platform libraries simply so that I could compile the damn app. And all this effort was ultimately in order to just change the build version number from "1.0" to "1.1". The UI and functionality remained unchanged since it had already been working as intended!

I'm sitting here on a Friday night, working myself to to bone after my day job, trying my best to scrape a living from my indie games, trying to keep up with Apple, Google, Unity, Xcode, MacOS changes that happen so fast my head spins while performing worse on older devices.

— Protopop Games (@protopop) April 23, 2022

And to add insult to injury, Apple tells me I could have avoided all this pain if my app was being downloaded above a "minimum download threshold". The policy is completely silent on what this download number needs to be to avoid getting flagged as outdated–is it hundreds or thousands or hundreds of thousands of downloads a month? This seems like an explicit carve-out in the policy for apps with major marketing budgets. According to industry research, the total marketing spend for app installs is projected to be $61.1 billion in 2022. So it seems like an incredibly bad faith argument to claim that a higher download rate means better quality apps. All a higher download rate means is that more money was probably spent to market and optimize app installs.

Apple also removed a version of my FlickType Keyboard that catered specifically to the visually impaired community, because I hadn't updated it in 2 years.

Meanwhile, games like Pocket God have not been updated by the developers for 7 years now: https://t.co/3azyIydty7 pic.twitter.com/n36rvHvF4H

— Kosta Eleftheriou (@keleftheriou) April 23, 2022

The overall message that this crackdown sends is that Apple doesn't care about about small and independent developers. For instance, the App Store could factor in:

  • App Store ratings and reviews
  • Active developer membership
  • Historical behavior of the developer (ie. updating other apps)
  • Number of crashes
  • Number of active sessions, especially for the latest devices and platform versions

These are all metrics that the Apple already automatically tracks, made available to the developer through the App Store Connect portal.

App Store Connect shows an overview of your app and includes a variety of metrics, including active sessions and crashes.

The metrics tab enables developers to break down their app metrics by device and platform version.

It would be almost zero cost to Apple to add these additional checks to their filter for outdated apps–after all, the data already exists in their system. So it doesn't seem like mere oversight that the policy neglected to use a more nuanced set of inputs, it feels like Apple is actively degrading the developer experience because they consider us worthless to their platform.


In the end, Apple will always prefer the needs of mega-apps that have millions of downloads since these apps generate the bulk of the revenue for the App Store. The impact of the App Store Improvement policy is nonexistent for VC-funded companies. The high-growth apps trying to blitzscale their way to product-market-fit have been churning out regular updates all along with their massive teams of mobile developers. Small apps and their developers will need to conform to the whims of the platform or else disappear entirely–this is and always has been the risk of building software inside a walled garden.

I wish that the App Stores had a concept of a "complete" app that does not need further updates. Emilia (@lazerwalker) put it really well by describing certain types of software as "finished artworks":

.@apple is removing a few of my old games b/c they have "not been updated in a significant amount of time"

Games can exist as completed objects! These free projects aren't suitable for updates or a live service model, they're finished artworks from years ago. pic.twitter.com/iflH70j7q4

— emilia ✨ (@lazerwalker) April 23, 2022

Games are not the only type of software that doesn't need updates. Author Robin Sloan has a wonderful "tap essay" named Fish. The app was last updated in June 2018, therefore "outdated" by Apple's standards. Sloan has probably contested the removal so it's still available on iOS, and there is always an option to read it on macOS or Windows if needed. Other creators might not be as lucky or have the money/time to provide their work on different platforms.

It will be very difficult to reverse the incentives for Apple and Google and get them to start caring about these small gems. They have been killing apps from the very start, in favor of the newest mobile devices and functionality. There is a whole trend on Tiktok #oldapps that are simply videos of people booting up old phones to play with apps that no longer can be run and reminisce about the joy that these apps created.

Day-by-day, month-by-month, the App Store will get a little less rich and vibrant as apps start being designated as outdated and get removed. Another consequence of this hostile policy is that indie and hobbyist developers may stop building mobile apps. After all, the web is fundamentally a more stable place for experimental software and "finished artworks", since backwards-compatibility is the gold standard and apps can run indefinitely.

After 4 hours of work to re-compile my app and 44 hours waiting in the review queue, WorldAnimals is now updated to a new version. I am safe for at least another three years before getting automatically flagged for removal. Unless, that is, Apple decides there is a new threshold for "outdated" and change their policy once again.

WorldAnimals is now available for another three years (hopefully).

I still love mobile development–a large part of my engineering career has been building mobile apps of all sizes, for small hobby side projects and for huge unicorn companies. I am proud of WorldAnimals and want to make sure it's still available for download. But this outdated policy will make me seriously think twice about building fun, experimental apps on iOS in the future.


This article was last updated on 9/25/2022. v1 is 1,806 words and took 5.25 hours to write and edit.



All Comments: [-] | anchor

crazygringo(10000) 1 day ago [-]

Hate away, but in this case I'm on Apple's side here. The author may be the exception where no updates are actually needed, but that does seem to be the exception. In my experience, most iOS apps I've tried downloading that were last updated 2-3 years ago simply don't work.

Apple requires an update in the past 3 years or else a minimum threshold of downloads. Presumably the latter is required so there are simply enough newer reviews to indicate whether it's broken (lots of 1-star) or not. These seem like pretty reasonable indicators that it's still working.

> My old code simply did not work anymore with the latest versions of Xcode. So I had to spend four hours on Saturday upgrading all my platform libraries simply so that I could compile the damn app.

Honestly that's just good hygiene. Because if they waited another 2 years, it might have taken 4 days to get the thing working. New versions of libraries come with all sorts of improvements, whether it's for accessibility or security or a hundred other things.

It doesn't seem unreasonable to me that if you're maintaining software in a public app store, you should be able to compile it semi-regularly with new versions of packages.

lapcat(10000) 1 day ago [-]

> Presumably the latter is required so there are simply enough newer reviews to indicate whether it's broken (lots of 1-star) or not. These seem like pretty reasonable indicators that it's still working.

You think App Store ratings and reviews are accurate?

They're incredibly easy to fake and buy. Scam artists are doing it all the time.

masklinn(10000) 1 day ago [-]

> In my experience, most iOS apps I've tried downloading that were last updated 2-3 years ago simply don't work.

Not sure what category of software that was but e.g. for games it's usually not an issue. I used to have a bunch of games and game-adjacent software from the first AppStore year, we're talking the time when you'd pay a buck for a virtual rain stick which's use the accelerometer.

They kept working until the 64b switch killed them. Rebuilding them would undoubtedly have been difficult for the author even if they'd not moved away from them, but because those'd use only minimal is features (raw-ish inputs, and setting up opengl) there's basically nothing to break which would not be flagrant.

nemothekid(10000) 1 day ago [-]

>Apple tells me I could have avoided all this pain if my app was being downloaded above a "minimum download threshold". The policy is completely silent on what this download number needs to be to avoid getting flagged as outdated–is it hundreds or thousands or hundreds of thousands of downloads a month?

I think this is a fair policy as well. From Apple's POV how can they ensure many apps in the app store still work on the latest phones? If the developer has been updating the app, they can be relatively sure. If the developer hasn't, but the app continues to be downloaded with success they can also be sure the app works. But for an app the developer has (seemingly) abandoned and and nobody downloads? It's most likely broken. You could argue that that Apple should test the app before sending such a message but I think it's probably likely that (1) the majority of apps that are abandoned are broken and (2) the developers don't actually respond. It's easier to tell all the devs to get up to speed and those that actually care will submit the app for review (even if all you have to do is change a version number).

What actually sucks is if you have to rewrite parts of your app because even though the iPhone 22 Pro Max Plus w/ Turbo & Knuckles supports the same SDKs as the iPhone 4, Apple won't let you resubmit your app as is.

bambax(10000) 1 day ago [-]

Backward compatibility is Apple's job, not developers.

(This debate is as old as computers, but I'm strongly in Raymond Chen's camp.)

matchagaucho(10000) 1 day ago [-]

Agreed. Apple is not asking for more features, but a recompile of the app against the latest iOS and SDKs.

That's generally a good practice as buffer overflows and all kinds of entropy accumulates on an OS.

kkfx(10000) 1 day ago [-]

Mh, changing the 'ecosystem' so much means the ecosystem is not stable, witch means is a crappy mass of bugs/a working prototype not a commercial product.

Oh, I like of course innovation but if we talk about mobile stuff, so something tied by choice to hw, their real update rates behind security related patches, should be very calm.

Anything must evolve to be alive, but also anything must NOT be stressed to change just some crap to prove that something have changed. That's not evolution.

Beside that, I fail to see any tangible improvement in ALL modern software to a point that I almost quit using mobile crapware of any vendor and kind, for real. I can't avoid it completely, unfortunately, but that's is.

alpaca128(10000) 1 day ago [-]

> In my experience, most iOS apps I've tried downloading that were last updated 2-3 years ago simply don't work.

The linked article already answered this:

    For instance, the App Store could factor in:
      - App Store ratings and reviews
      - Active developer membership
      - Historical behavior of the developer (ie. updating other apps)
      - Number of crashes
      - Number of active sessions, especially for the latest devices and platform versions
    These are all metrics that the Apple already automatically tracks
fny(10000) 1 day ago [-]

You know what doesn't have this problem? Web apps. I have a site up from 2001 that still functions perfectly.

t_mann(10000) 1 day ago [-]

Frankly, isn't that just what you signed up for as a mobile developer? The WorldAnimals app would probably have worked perfectly fine as a web app, but then the author would have had to figure out payments and discoverability himself. That's what the App Store offers, and in exchange Apple gets tremendous power to tweak every nook and cranny to maximise their profit. I'm sure Apple has good commercial reasons for the process that the author has experienced.

I think the real answer here isn't to try and beg Apple to be nicer to the small fish, but to instead use the 'old', non-walled version of the web.

Tijdreiziger(10000) 1 day ago [-]

Well, no, a native app and a web app aren't one-for-one interchangeable.

Apple acts as a gatekeeper by only allowing native apps to be distributed through the App Store, so it is not unreasonable to ask that they refrain from making the process too onerous.

neilparikh(10000) about 23 hours ago [-]

It doesn't apply here, but there are a few app ideas I have where I don't really care about payments or discoverability; I would need use native just because the features are not supported in iOS Safari. For example, here's a list of standards that the Webkit team has said that they have no plans to implement: https://webkit.org/tracking-prevention/#anti-fingerprinting (for me, WebBluetooth and Web NFC are the big blockers).

Until those gaps are fixed, native apps are your only option.

XCSme(10000) 1 day ago [-]

This is why the products I sell are in plain PHP + MySQL, nothing to really update as most PHP/MySQL language updates are usually backwards compatible.

selfhoster11(10000) about 15 hours ago [-]

As someone who recently did a large classic LAMP migration: nope. Lots of things to break.

HeavyStorm(10000) about 13 hours ago [-]

I don't know what more evidence we need that iOS and Apple are rotten and regulation is necessary. Too bad legislators are walking at a turtle's pace.

Were the platform open to competition, we could have multiple stores that would have better policies. It would still elude the common user, unfortunately, but it would be a solution.

scarface74(10000) about 13 hours ago [-]

So you're going to "legislate" that Apple must maintain compatibility forever?

dkarl(10000) 1 day ago [-]

> The impact of the App Store Improvement policy is nonexistent for VC-funded companies.

Not true. More often than not, our iOS releases get delayed hours if not days, while our long-suffering iOS lead patiently walks yet another green reviewer through the policies and our previous interactions with reviewers to convince them that our app is in compliance. Among other things, our app repeatedly gets flagged for failing to comply with rules that don't apply to it. This is usually resolved with a simple message to the reviewer, but depending on the turnaround, that can effectively add a day to the time it takes to get a bug fix out.

Dealing with these bogus issues probably accounts for 5% of the productive time of our iOS lead. And this is despite keeping our release cadence slow (every two weeks except for the most critical bugs) and after we've made many reasonable low-to-medium effort changes in our app to stop triggering false positives in their tools.

God help us if Apple ever went the Google route. Apple reviewers might be inexperienced and undertrained, but at least they're human and capable of talking through issues.

peterkelly(10000) about 19 hours ago [-]

How is releasing a new version every two weeks 'slow'?

zac23or(10000) about 16 hours ago [-]

I see some people defending Apple. Bizarre. The support Apple has in the most absurd decisions led me to classify it as a religion, not a company.

Apple is a trillion dollar company with over 154,000 employees. Apple has the resources to keep the code for two years, DOS programs still work on Windows 10.

The decision is arbitrary, someone who doesn't understand software has a lot of power over Apple.

scarface74(10000) about 13 hours ago [-]

How well do your DOS programs run on Microsoft's successful mobile platform that doesn't have swap, limited memory and has 24 hour battery life on the latest processor node?

0xbadcafebee(10000) 1 day ago [-]

Don't spend your time defending your app. Spend your time lobbying your representatives to either break up Apple or force them to open up their devices to other stores.

Even Ma Bell couldn't tell you who you could buy a telephone from, who you could call, or what you could do on the call, much less take a percentage of anything you ordered over the phone. Enough is enough.

scarface74(10000) about 13 hours ago [-]

Actually Ma bell could tell you who to buy a telephone from early on. They controlled what devices were allowed to be connected to the network.

GuB-42(10000) 1 day ago [-]

Hey, Apple, people still play Super Mario, they didn't change a single bit since it was released, 37 years ago.

But there is one reason I somehow understand Apple stance. First is security, almost all apps are online now, and new vulnerabilities are found regularly. Super Mario has bugs, be no one cares, in fact, speedrunners have fun with them, because a NES is not online and there is nothing sensitive in there to begin with. Second is Apple's own fault: they break their own system and want you to make sure you keep up.

bena(10000) 1 day ago [-]

I don't think the Super Mario comparison is entirely fair here.

Super Mario Bros runs on a single piece of unchanging hardware. Even when you play it on any other platform, you're playing it through an emulator designed to mimic the original hardware.

And that's the case with all video game consoles. Even when the physical hardware changes, you're given the guarantee that it won't be breaking to the software. The few times that has not been the case has been notable.

iOS devices don't have that guarantee. The hardware can have breaking changes. You software now has to contend with the fact that there is no guarantee that certain features may or may not be present. Things that were previously not customizable, now are. If you want your software to run on the latest iOS, it is at least partly your responsibility to ensure that it does.

Kwpolska(10000) 1 day ago [-]

Eh, this is Apple you're talking about. They don't know what backwards compatibility means, and your app might randomly break on the newest version of iOS. Or it might be slightly ugly around the Dynamic IslandTM and Apple users would hate that.

musicale(10000) 1 day ago [-]

> They don't know what backwards compatibility means

Platforms are supposed to absorb developer pain, but Apple offloads a continuous and multiplicative update burden onto all of their developers.

hcarvalhoalves(10000) 1 day ago [-]

I agree.

Some software from the past that made me feel that way:

1. Adobe Photoshop 7

Felt feature-complete, never ran into a bug, not resource-intensive (I would run it on a Pentium 200), no online activation/membership BS

2. Apple Aperture 3

Also felt feature-complete, had everything I needed. Nowadays I need to use 2 or 3 different software to have the features of a 2003 app. Unfortunately the app stopped working after 64-bit transition, and Apple retired it in favor of the much simpler Photos. Shame on you, Apple.

3. iTunes 10

Pretty much the sweet spot of features for managing your music library. Apple replaced with the Music app, because nowadays users consume streaming vs. maintain their own libraries, but the app is a buggy mess.

4. Winamp

Solid, just worked.

More software should be like that, but never will, because it isn't profitable to have users using the same version for many years. The revenue maximisation strategy is to release continuous beta-quality software, and apps should provide a steady revenue stream. That's why the Apple Store just assumes an app that didn't update is 'abandoned'. That's the perspective of users in general today, too.

selfhoster11(10000) about 16 hours ago [-]

> More software should be like that, but never will, because it isn't profitable to have users using the same version for many years.

You can get some of it back with FOSS applications, even if the quality normally isn't the same as with old freeware/proprietary applications. At least with a FOSS application you get to keep it forever, unlike SaaS which evaporates into thin air the moment the parent company decides to pull the plug.

1-more(10000) about 11 hours ago [-]

Every version of Geometer's Sketchpad feels like complete software. Yeah they add features every so often (pinch to zoom is pretty huge) but the core of it is the same.

criley2(10000) 1 day ago [-]

Photoshop 7 struggled in larger PSDs with lots of layers or just a huge resolution/dpi. The memory management was really bad in that era too featuring manual scratch drive configuration. Constant crashing in large projects was the norm for me. I thought CS2 was a good upgrade and my final point before the nonsense of subscription based software took over. Now I just use photopea because 99% of my photoshop needs can be met by a single dude on a single webpage

janandonly(10000) about 19 hours ago [-]

Maybe I misunderstood her, but she says it cost her several hours to rewrite her code to be able to compile her app again for the newest iOS and iPadOS versions.

Doesn't this imply that her app would not have worked on the latest OSses?

In other words: if she had done nothing then her app would not work on newer devices? If so, then isn't it useful for apple to warn her about this?

jerrre(10000) about 17 hours ago [-]

You misunderstood. Otherwise the whole article wouldn't make sense, she clearly states that it still works fine on new devices.

Compiling with the newest versions of the libraries/tools can be a lot of hassle including downloading/instaling multiple GB's

The newest OSes will run apps made with earlier versions of the libraries/tools fine, up to a point of course.

renewiltord(10000) 1 day ago [-]

In a world with security concerns, can anything be complete?

detaro(10000) 1 day ago [-]

yes. (especially but not only on mobile, where the platform is responsible for a lot of it)

Security concerns are very clustered around limited attack surface, that many apps just don't have.

hmcamp(10000) 1 day ago [-]

Yes. The ls Unix command.

jmbwell(10000) 1 day ago [-]

Looking at the metrics Apple tracks in the App Store, it seems to me that 'engagement' is all that really matters. You could have some huge number of happy users, but if they open the app only occasionally, you get numbers in red. Or you could have an app that solves an important issue but only for a small number of users, and you get numbers in red. If you're not releasing hits that people use all the time, you sink. If tons of people don't spend tons of time in your app, you sink.

Author complains about the time required to update libraries, and that's an aggravating process, but that's just an unfortunate part of maintaining an app. The real issue, again it seems to me, isn't that you have to do a lot of work just to increment the version string; it's that, ultimately, modern content ecosystems are designed according to engagement-driven revenue models. And solid, simple, quiet, long-lived, useful or fun little apps simply can't thrive when all the sunlight is blocked by towering marketing-tech giants.

IMHO.

klabb3(10000) 1 day ago [-]

> Looking at the metrics Apple tracks in the App Store, it seems to me that 'engagement' is all that really matters.

This has been the name of the game in ad tech like fb, Google and social media in general. I think two worlds are clashing with each other, where consumer tech is somewhat aware of the problems around mindless scrolling and addiction, but the growth & engagement mindset of the 2010s is cemented in the culture. Apple has little reason to follow this model because they primarily make money from selling hardware. Having a software platform that protects the user from this crap is a competitive advantage against Google, who depends on ad-based revenue. Apple seems to have an identity crisis, fearing they lose out on sweet subscription fees and ad revenue, now that most apps are free. This in turn is creating conflicts of interest, where they end up fighting their own customers.

If regulators would bark and bite harder around anti-competitive behavior, it might actually force corporations to focus on what they're good at instead of everyone building their own mediocre walled gardens that grow like a cancer and eats the company from within. At least, that's my wishful thinking..

kitsunesoba(10000) 1 day ago [-]

Additionally, updating libraries periodically is inescapable for any app involving network connections or handling arbitrary user content, because doing otherwise means exposing your users to vulnerabilities.

Fully offline, totally self-contained apps are a different matter, but those represent an increasingly small percentage of apps.

trap_goes_hot(10000) 1 day ago [-]

You're completely right. And I wouldn't single out just Apple here, they're falling into the same trap as others. And this is going to be a recurring theme across all domains. We're attempting to model things like 'usefullness', 'credibility', 'importance' etc into measurable metrics because of the pressure to scale and monopolize the market. Its scary to think that for all the talk of hiring the 'best and brightest', its these metrics that supposedly the smartest engineers in the world come up with. Its pure hubris on our part.

There is a dystopian arc, but I prefer to be more optimistic. I would think a legal mandate that champions interoperability, open data standards, and platform openness will put a dent in this march towards convert the human experience into numbers.

junon(10000) 1 day ago [-]

I am a maintainer of Chalk (Javascript library for outputting color codes for terminals).

We consider chalk 'finished' and probably won't release another major version that changes the API - perhaps just typescript maintenance as Microsoft messes with the language even further and we have to fix types.

Sometimes, software is complete. I hate this new industry dogma that code rots over time and needs to constantly be evolving in order to be 'alive'.

JoeOfTexas(10000) about 12 hours ago [-]

I took a quick look at Chalk. I would say using yoctodelay dependency is kinda useless, and is maintained by a single individual. I sense potential danger there. The other dev dependencies should be audited too.

JoeOfTexas(10000) about 12 hours ago [-]

I think when a project heavily depends on dependencies it is easy for security issues to make it too risky. Lately, I've been seeing more and more npm packages that keep 'extending' a dependency to add more features, which eventually gets abandoned.

I'll use this package, multer-s3-transform, as an example. Its been abandoned. It depends on multer-s3, which depends on AWS-SDK and multer, which then depends on busboy.

Now, AWS-SDK, Multer, and Busboy have organizations maintaining them. However, the random projects overlayed on top of those are by single individuals.

The answer is to two-fold. Avoid overlayed projects and avoid big projects managed by a single individual.

cybrox(10000) about 16 hours ago [-]

In our case, Apple actively destroys user experience with this kind of stuff.

We have an old iOS app for our products that runs fine even on the latest version and while there is a next gen app available that most users have switched to, some prefer the old one. For some use-cases, even for new users, the old app just fits a lot better.

We cannot rebuild and republish that app, because it depended on third party software that we no longer have access to. The app will continue to work fine for users that own a correspinding device but will most likely be removed from the app store for no real reason in a few months.

This will force all userd that want to actively use it to switch to the next gen app that they maybe don't want as soon as they add someone new to their device.

ydlr(10000) about 7 hours ago [-]

Does removing a program from the apple store also delete it from users' telephones?

TillE(10000) 1 day ago [-]

This is another case where Apple fundamentally doesn't 'get' games. They had a plan for clearing out crap from their store which probably sounded reasonable, but nobody even thought about an entire class of programs.

Lots of stumbles and missed opportunities - like, what if they had actually been serious about making the Apple TV a Wii-style games console? They have the hardware expertise to do that at a reasonable cost, but they just have no idea about the market, and apparently no desire to learn.

themagician(10000) 1 day ago [-]

Oh, they get it.

Apple products very much have a time and a place. Their plan is recurring revenue. Something that happened 3 years ago is almost entirely irrelevant to them. It's true with both hardware and software.

Wii-style games on AppleTV would not be a win for them. They don't want to sell a few million copies of Wii Sports once every 3-5 years. They want you buying a new 99¢ game at least once a month. They want you spending $5-10 in micro-transactions a month. They want you subscribed to AppleOne for $15-30 a month. They want you to buy a whole new iPhone every 2-3 years and start the process all over again with some new AirPods, cases and bands in between.

Apple doesn't want to sell you a new game for $59 every 2 years. They want to sell you what amount to an average of $59 worth of goods and services a month... forever. And while that sounds like a lot, that's a low target. If you have your TV subscriptions through Apple and Apple Care you can easily be contributing $100/mo or more to Apple's bottom line.

musicale(10000) 1 day ago [-]

I understand that Apple is trying to deal with the sort of shovelware/accumulation problem that the Wii had, but the blanket approach of culling games based on release date seems wrongheaded.

If Nintendo took that approach then they'd end up throwing away Super Mario Odyssey and Zelda: Breath of the Wild.

lawgimenez(10000) 1 day ago [-]

3 years without updates? What would it take for the author to download the newest Xcode and compile? It won't take me a day. If you're in the mobile development space, you have to keep up and compile with the latest as much as possible. This is the way to get Apple and Google off your ass.

lapcat(10000) about 24 hours ago [-]

> What would it take for the author to download the newest Xcode

The author is a professional mobile engineer and has the newest Xcode.

The app is an old project not related to the author's current job.

stabbles(10000) 1 day ago [-]

On a related note, I feel that some websites at are a point where they're complete, and adding new features makes the experience worse.

Take Github: to me it feels that it's now at an optimum where adding new features that make it more social or more 'fun' would simply make it worse.

Similarly Youtube: tons of good material, user interface is fine. Then they introduced shorts, and to me it feels like these type of video's are simply not part of youtube's identity, adding them only makes it worse.

At some point it might be best to stop adding new features and just agree that things are fine now as they are.

koreth1(10000) 1 day ago [-]

Surely it's possible for GitHub to add features that aren't related to making it more social or 'fun,' though? For example, they could improve their support for stacked PRs in a number of ways; that's an area that is far from fine at the moment IMO.

_dain_(10000) 1 day ago [-]

they could show the linecount of each file in the repo browser

_jal(10000) 1 day ago [-]

Apple is starting to remind me of auditors, but instead of trying to keep people honest/push trendy practices, they're pushing whatever's trendy in Apple's management class this month.

When do we get to the point of companies hiring 'review specialists' who used to work for Apple?

jaywalk(10000) 1 day ago [-]

Anyone who could afford a 'review specialist' is probably already getting the white glove treatment this post describes, and wouldn't actually need one.

plorkyeran(10000) 1 day ago [-]

> So I had to spend four hours on Saturday upgrading all my platform libraries simply so that I could compile the damn app.

I sort of assume this is the actual point? Apple presumably wants to drop support for older versions of the SDKs, and that requires that app developers update to the newer versions. I think you can make a reasonable argument that dumping work on third-party developers to simplify things for themselves is bad, but the author's belief that it was simply pointless busywork is probably incorrect from Apple's perspective.

I suspect the minimum download threshold to be exempt from this is _very_ high. Maintaining backwards compatibility for a small fixed set of apps doing things an old way is a categorically different problem from maintaining backwards compatibility for every app out there.

dahfizz(10000) 1 day ago [-]

If they have specific libraries or APIs that they wanted to deprecate, I think developers would understand that. But the current implementation of 'anything 2 years old is too old' is arbitrary.

If this was really about deprecation, they wouldn't have a 'minimum monthly downloads' exemption either. This policy is just a way to clear out old, unpopular apps from the store

sriku(10000) 1 day ago [-]

My prof just ran a Win32 programs written 20+ years ago on today's Windows. At least someone is paying attention to backward compatibility.

bsuvc(10000) 1 day ago [-]

This is a great point, but if true, Apple should do a better job of communicating the reason, including specifics, not just say 'your app is rejected because it is outdated'.

lapcat(10000) 1 day ago [-]

> I suspect the minimum download threshold to be exempt from this is _very_ high.

You can suspect based on no evidence, but nobody knows, and Apple refuses to say.

The crazy thing is, if Apple truly wants to drop support for older version of the SDKs, then how in the world does it make sense to exempt the most used apps???

dheera(10000) 1 day ago [-]

I've had a few apps on the Google Play Store automatically removed because I didn't update them for some new policy or some such.

If they paid me maybe I would have. Otherwise I don't have time to keep dealing with their requests every 6 months. Is it such a hard thing to ask that if shit works, just leave it be?

lytefm(10000) about 17 hours ago [-]

> The intention behind Apple and Google's policies is to force developers to make sure their apps run successfully on the latest devices and operating systems.

The intention is to not support old iOS APIs with new versions of XCode and iOS anymore.

Apple isn't Microsoft or the Web - very old Windows programs and very old websites still run pretty fine.

Apple would rather shift the burden to update and App according to the latest API to the Devs than to provide API support forever.

throwaway292939(10000) 1 day ago [-]

I agree that this may be the point. Software changes, and being able to compile with latest libraries is somewhat of a minimum requirement.

To Apple's defense, are they supposed to wait until the app breaks, starts receiving many complaints from customers, before it triggers the review process for them (which they would be forced to look at as somewhat high priority) before they then take action to remove the offending app? That hurts the customer experience from their perspective.

Better for them to institute a policy preemptively addressing these issues (arbitrary as the timeframe may be).

And four hours is a good chunk of time, but what percent of time is it compared to the amount of time for the app to be designed and implemented in the first place?

mattgreenrocks(10000) 1 day ago [-]

I am convinced the world doesn't understand the concept of something being 'complete.' It's like part of their brain can't comprehend that something is functionally whole, and any changes would actually detract from what is there.

Related: open source devs fret if a library hasn't been updated in the last month, even if it is feature complete with no bugs.

electroly(10000) 1 day ago [-]

For open source code, I think the concern is practical: bit rot is real, and libraries with no recent updates are more likely to have suffered bit rot. It's not that a library can't be finished, but that the world in which it lives never stops moving. It can be a bigger or smaller problem depending on the ecosystem. Old C89 libraries have a good chance of still working, because the ecosystem isn't moving so much any more. Old C# libraries almost definitely don't, because the ecosystem is moving quickly.

hwbehrens(10000) 1 day ago [-]

> feature complete with no bugs

I suppose for simple libraries this might be possible (e.g. left-pad) but how would you know in general that something is bug-free? `units` was introduced in 1979 and is still one of the canonical examples used in automated program repair research to show that new bugs can be found even in extremely well-studied codebases.

I think people might be:

1. Implicitly assuming that all nontrivial code has bugs, and 2. Using recent commits as a signal that means, 'If I ran into a bug, there is someone who might fix it (or at least accept a fix PR) to take away my immediate pain.

sorisos(10000) 1 day ago [-]

I like that so many unix command tools (like cat, ls...) is rarely updated. Often comes with a man page written in 30 years ago. Same with a lot of C libraries.

mey(10000) 1 day ago [-]

If a library is not updated in some capacity 'recently' (coming from the Java land, 'recent' is a different timescale than Javascript), then I have little confidence that a security issue can be resolved if discovered. Even if a library is complete, it may depend on libraries downstream that do have security issues that are no longer supported, which means it needs to swap to newer versions, etc.

bsaul(10000) 1 day ago [-]

About open source lib not being updated :

The thing is, even if the lib is 'feature complete', it's pretty rare to not have to update anything, since the ecosystem in which this lib evolves will undoubtely have changed. Programming language, hardware, OS, etc. everything evolves all the time.

ako(10000) 1 day ago [-]

Everything you use continuously improves. What do you use that has seen improved versions since you bought it?

Our knowledge, our abilities, our raw materials continuously improve, and everything is expected to advantage of this. Things that don't improve, are soon outdated and fall behind.

Everything continuously improves: cars, bicycles, windows, houses, refridgerators, tv, etc, etc.

layer8(10000) 1 day ago [-]

Imagine TeX becoming unavailable because Knuth didn't update it for three years.

thenerdhead(10000) 1 day ago [-]

I mean a heartbeat is needed sometimes to make sure something is still alive and can address critical issues if they do happen. But I do agree that libraries and apps don't need to 'update' to show that.

Maybe some other way like 'Hey, we're still here just the thing we're working on doesn't need an update' would suffice for most things.

aposm(10000) 1 day ago [-]

By 'the world' I think you mean a specific subset of people: investors/execs/the business side. I don't particularly believe that average end users or competent engineers feel that way...

zeroonetwothree(10000) 1 day ago [-]

I would love to see this theoretical world in which libraries have no bugs.

Thing is, even if there are no known bugs, having rare updates means if I do find a new bug it's less likely to get addressed quickly.

Lutger(10000) 1 day ago [-]

One word: security updates. Ok that was two. Most software had dependencies, those tend to surface vulnerabilities over time.

musicale(10000) 1 day ago [-]

Other game platforms (game consoles, Windows) seem to have drastically better backward compatibility and API stability.

If you look at the Nintendo DS/3DS (for example), the platform had many hardware revisions (about every other year) and dozens of firmware revisions, yet games from 2004 still worked on a brand new 3DS in 2020.

On the Sony side, PS4 games from 2013 still work fine on a PS5 in 2022 (and probably through 2029.)

Steam has tons of old games that work great on Windows (but the Mac side took a major hit with macOS Catalina's 32-bit apocalypse.)

donatj(10000) 1 day ago [-]

A PHP library I use and used to help maintain had a bit of a fight break out in the Issues on GitHub a couple weeks ago.

They were dropping support for older versions of PHP despite getting nothing from it and using none of the newer features. Just needlessly limiting the audience, and churn for churns sake.

sodapopcan(10000) 1 day ago [-]

The Elixir ecosystem is quite good for this. I've found smaller libraries that haven't been update in a long time but still work just fine since Elixir itself hardly ever changes in a breaking way (the recurring phrase in the community being: 'There will probably never be a version 2'). It's weird to get used to at first but it's actually quite refreshing.

skybrian(10000) 1 day ago [-]

Compare with media files. Music and movies don't expire. The most common audio and video formats will probably be supported forever. (Perhaps there are exceptions for obscure codecs?)

It seems like this could be done for at least some games, but defining a virtual machine and binary format that's both good enough that game developers will want to use it, and at the same time stable enough that it lasts forever, seems like a challenge? Also, it needs to be non-networked, or it will stop working when the server goes down.

This was done a long time ago for interactive fiction, but doing it for modern mobile games seems like a challenge. WebAssembly should probably be involved, but the user interface will need a huge API surface that's also stable.

ChadNauseam(10000) 1 day ago [-]

I think Wasm + WebGPU is probably very close to what you want

heisenbit(10000) about 20 hours ago [-]

Not only the file format. The content also gets supported and legal protections were extended and not shorted over the years.

titzer(10000) about 11 hours ago [-]

I want to make WebAssembly the Unicode of executable code formats, stable indefinitely far into the future. That requires backwards compatibility for all code format features and simple, stable APIs.

jeffalyanak(10000) 1 day ago [-]

I think there's something to be said about this in regards to consumer electronics as well. One of the oldest electronic devices that I still use on a daily basis is my ereader, which is a very early model. It simply does everything it needs to do, it's feature-complete.

Unfortunately there aren't many consumer electronics that fall into that category and a huge portion of consumer electronics that end up in the dump are not physically damaged in any way, but simply can't run the latest software, firmware, or OS. Or else they're missing the latest wifi or cellular radios, high end cameras, wireless charging protocols, etc.

SilasX(10000) 1 day ago [-]

Similar thoughts about my PlayStation 2 from 2004[0] that I still play Dance Dance Revolution on. I don't have to worry about anything changing that breaks the experience. (Except finding a TV that minimizes upscale lag but that should have been a trivial thing for TV makers to get right.)

[0] The PS2 platform is from 2000 but I use a miniaturized PS2 slim which was released and probably manufactured in 2004.

jeffalyanak(10000) about 11 hours ago [-]

Actually, I realized that much of my _music_ equipment falls into this category as well.

My mixer is an early example of a MIDI-controllable analog mixer but since the MIDI protocol is still ubiquitous, it still fits right into my modern workflow. New mixers don't really bring much to the table that am average person would need.

fullstackchris(10000) about 17 hours ago [-]

The real problem is that any native developer is essentially slave to Apple or Google's requirements (or both!). Where or how did we go wrong with native development and applications? Its all owned (and thus gated) by these industrial behemoths. The web, for example, is a much more 'open' platform. There needs to be an open platform like the web that can be used on mobile devices.

butthatsit(10000) about 17 hours ago [-]

What's wrong with the web on mobile devices? PWAs are going to get better support even on iOS now. What remains to be done?

jimkleiber(10000) about 17 hours ago [-]

Yup, my very simple/basic/i-thought-forward-compatible app broke when the stores required 64bit only. Maybe that's a simple change, but I can imagine how more complex apps may break much more frequently with OS changes.

musicale(10000) 1 day ago [-]

I don't think culling games/apps based on age is the right approach to improving discoverability.

I don't see Nintendo removing old Switch games from the eShop.

I don't see Apple Music removing the Beatles because they haven't updated recently.

My recommendation, which Apple's (obnoxious) ad business would never accept, would be to

1. Remove the obnoxious and generally useless ads which eat up the top half of the first page of app search on the iPhone.

2. Improve app search with more options and a fast, responsive UI. Also they might consider allowing you to consider ratings from other sources such as critical reviews vs. user reviews (a la metacritic.)

3. Better human curation with more categories. This is the same approach that Apple Music takes and it seems to work well. Games should have Artist/Studio/Developer pages as well as Essentials lists to facilitate discovery. Same with genre/category essentials, which have a rotating list that can be saved/bookmarked.

scarface74(10000) 1 day ago [-]

APIs change, hardware changes, the comparison is relatively silly.

scarface74(10000) about 13 hours ago [-]

Nintendo updated hardware every five to seven years and don't old eShop apps run within an emulator?

II2II(10000) 1 day ago [-]

The missing point is that they are culling apps based upon age and downloads. I think this is a reasonable criteria if Apple is trying to make space for popular and potentially upcoming apps. Yes, space is an issue if you think of it in terms of the customer's attention span.

The real problem is that developers have no choice except to offer their app through Apple's store. There is no room for the developer to offer their labour of love or niche product in a storefront that better serves their needs, or even from their own website.

anothernewdude(10000) 1 day ago [-]

Apple should just have fewer apps. I don't see why developers should cater to their customers. They buy less and require more support. They are the worst segment to support.

CharlesW(10000) about 23 hours ago [-]

> I don't see Nintendo removing old Switch games from the eShop.

The difference is that Nintendo shops have a limited shelf life, while App Store is forever(-ish). Nintendo will be shutting down the Wii U and 3DS eShops in 2023.

2023 is 6 years after Wii U was discontinued. Since the platform was frozen in time in 2017, there's no point to game updates.

sriku(10000) 1 day ago [-]

> I don't see Apple Music removing the Beatles because they haven't updated recently.

I wish I could upvote you for that multiple times!

jongjong(10000) 1 day ago [-]

I can relate to the frustration of the author. I wrote an open source library which gets downloaded 50K+ times per week. At one point, I hadn't updated it in several months and people started asking if the project is dead... In fact, it wasn't updated in a while because it had been working perfectly and was built in a forward-compatible way; that makes it more healthy and therefore more alive than probably 99% of libraries on the internet.

The expectation that software libraries or frameworks should break every few months is disturbing. I think this may be due to the fact that many of the most popular frameworks do this. I find that React-based apps tend to break every few months for a range of reasons; often because of reliance on obscure functionality or bundling of binaries which aren't forward-compatible with newer engine versions.

lwhi(10000) about 16 hours ago [-]

I think one issue, is that the world changes around an app or library. In these cases, the code base could need updating as a side effect of this change.

Also, not many people (especially me included) are able to write something that's perfect first time. In an ideal world, other developers would be testing and challenging what's been written; which would meant the product or library needs updating.

I think it's (perhaps ideally) likely, that at least one of these factors is relevant.

the__alchemist(10000) 1 day ago [-]

I think one of the core issues contributing to fragile, continuously-updated software is a culture of over-using dependencies. Each dependency added is a potential complexity trap. Using a dependency that has few sub-dependencies is less likely to cause problems. Or that has several sub-dependencies that each have no dependencies. It's deep dependency trains that are likely to cause this. Ie, dependencies that depend on multiple others.

_carbyau_(10000) about 24 hours ago [-]

Depending on how you feel about it you could update the readme every so often with: [date] Checked project, still is ok in current form.

Foobar8568(10000) about 21 hours ago [-]

I had to choose between two libraries at work, lead rejected my recommendation because github was showing less activities, the other project was seeing a total rewrite regularly, largely buggy because of that, and unclear documentations, yet it was the popular choice.

nerdponx(10000) 1 day ago [-]

I think communicating this information in your README or whatever might help a lot. It's hard to know if something is finished or abandoned in general, so a message indicating the former would help people arrive at a decision sooner.

pmontra(10000) 1 day ago [-]

> The expectation that software libraries or frameworks should break every few months is disturbing.

Dependecies and external APIs change so software breaks.

An example from a project of a customer today: Ruby had a Queue class up to 3.0.2. For some reason they moved it to Thread::Queue since 3.1.0. It's only a name change but it's still one release to do.

rcarmo(10000) about 15 hours ago [-]

I recently added a very visible note to the README.md of one of my projects that essentially said: 'lack of commits means this is STABLE, not dead. Our plan for <next major runtime> is bla bla, in a year or so'

This expectation that things need to be committed to or updated every month has to stop, we're just wasting time and energy on stuff that is already fixed (unless there are security issues, of course).

Maybe GitHub needs to automatically include an UPDATES.md into each page, I dunno.

makeitdouble(10000) 1 day ago [-]

I understand your frustration, also thank you for your work.

> The expectation that software libraries or frameworks should break every few months is disturbing.

As a library user, it's hard to find a good balance between fully trusting a system to stay alive for a while without maintenance, while being super paranoid about every aspect I rely on. Mentally it's easier to expect it to break every now and then, than to keep thinking 'it's probably fine, but I still need to be defensive about stuff that never happen'.

I think the issue on a library being 'alive' or not is best mitigated by the users being comfortable to fix it if it broke under their use case. There's libraries I thought were completely dead but were a good shortcut to where we wanted to go, and we expected to extend it here and there anyway. That can't work for everything (thinking of security libs in particular, I wouldn't want to touch them) but it's to me the ideal situation, with no undue stress on the lib maintainer as well.

tinalumfoil(10000) 1 day ago [-]

I don't know the details of your particular library but I think 50k/week downloads and no problems is an outlier. In my experience even projects with small audiences tend to accumulate outstanding bug reports faster than they're fixed.

contravariant(10000) 1 day ago [-]

The annoying thing about software is that we still haven't quite gotten the transfer of knowledge working to the point that you can communicate all knowledge about a piece of software purely through code (and no, being able to emulate all behaviour is not the same, otherwise decompiling wouldn't be a thing).

What this means is that for most software an essential part of that software consists of knowledge that needs to be kept alive. The only proof people can have of this is activity surrounding the project. This puts us in the uncomfortable position of having to designate someone as the 'keeper of all knowledge' surrounding a project if it is to be sustained, and failing to do so will result in knowledge being irrevocably lost.

twblalock(10000) about 23 hours ago [-]

The problem with not updating dependencies for a long time is that it can force you to make sudden version jumps if something like a CVE is discovered.

At my company we had a real scramble to upgrade log4j when the CVE for that came out. A lot of software was many versions behind. Luckily for us, no breaking changes had been made, and it was simply a matter of bumping the version to the latest one.

That was a lucky break, because if that had been a library like Jersey or Spring it would have been an entirely different ballgame.

If you don't keep your builds up to date, you open yourself up to risks like that, which are unknown and to some extent unknowable.

triknomeister(10000) about 19 hours ago [-]

If your product relies on any compilers/tools which are themselves updated, then at least your CI needs to be updated regularly enough. May be in your case, that was the case, but if not, then that would just to me that the product is not being maintained, even if the product is feature complete.

HWR_14(10000) about 9 hours ago [-]

I feel like, and hope gorhill responds to this, that uMatrix was similar to this. Done and stable and needing to be touched just so people can wouldn't ask if it was still maintained. (Pre manifestv3 that is)

mk89(10000) about 15 hours ago [-]

Well, there is also the security aspect in mind. Most software relies on libraries that repeatedly get CVEs over months (not years).

You might not care too much, and it clearly depends on the application, but for me updating a software that doesn't use anymore a library for getting user input that leads to a buffer overflow if you insert a certain character, or similar things (like remote code execution, etc.), can be quite important. Finally, apps are most of the time connected to the Internet, which makes them inherently vulnerable.

After searching 'android software libraries CVEs' I found this in the first results' page: https://github.com/dotanuki-labs/android-oss-cves-research

It might be outdated, but the principle still applies.

It's very unlikely that your application is self-sufficient and doesn't need updates.

EDIT: I read the message from Apple as a 'hopeful' attempt to say 'come on, man at least update the dependencies'.

mihaaly(10000) about 13 hours ago [-]

Perhaps you should commit a new 'Still working well...' line to its readme file in every few weeks to satisfy the update hungry crowd. :)

sportslife(10000) 1 day ago [-]

How do you think users would react to a monthly no-changes-required update?

maw(10000) 1 day ago [-]

When evaluating libraries, recent activity is one of the things I look at. It's a far from perfect heuristic. I've probably rejected libraries like yours for what amounts to no very good reason.

I don't know if this is a good suggestion or not but I wonder if some form of 'keepalive commit' might help here. I can imagine a few ways they might work, some simple, some more elaborate.

testingwaters4(10000) about 19 hours ago [-]

> built in a forward-compatible way

What do you mean by this?

1vuio0pswjnm7(10000) about 16 hours ago [-]

'I think this may be due to the fact that many of the most popular frameworks do this.'

According to the OP, it is also due to companies controlling mobile OS such as Apple.

One of the things I like about non-corporate open source OS like NetBSD is I can run old userland utilities with new kernels without any problems.

vonseel(10000) about 24 hours ago [-]

Try opening an old project from 3-5 years ago that has a lot of dependencies like something from react-native, Django, etc. and you'll find many of them impossible to even install without using the original Python or Javascript versions - that's assuming the oldest dependencies are still available in the pinned versions.

It's not a common use-case, I guess, but I had some old projects that I abandoned and wanted to try working on them again and found it almost impossible. The worst cases had dependencies which weren't available unless I used an x86 laptop with an older version of Mac, because the packages were never built for arm laptops.

thayne(10000) 1 day ago [-]

> The expectation that software libraries or frameworks should break every few months is disturbing

I think it is more that recent updates are used as a proxy for determining if the developer still cares about the project.

The problem is it is hard to distinguish between a project that has been abandoned, and a project that is feature complete and hasn't had any bugs reported in a while.

Although, if you assume that any software is free of bugs, and that at least some users will report bugs, if there is low activity, that suggests that either it isn't well maintained, or there isn't a critical mass of users to find and report those bugs. But then, higher quality software will require a higher number of users to hit that critical mass so it doesn't give you that much information unless you also know how good the software is.

gregoriol(10000) about 17 hours ago [-]

We have the same problem with our sass product: the product is 'complete' in our minds, we don't want to add more features, make it more complicated, ... we perform minor updates of third-party components, some minor fixes, ... This means we don't maintain a blog with very recent posts, or a very active twitter. We have a daily usage counter on the homepage though, which shows that thousands of users use it these days. But from time to time someone still asks if the product is alive... We haven't found a solution to this yet: it's as if we are expected have a blog post every few days or weeks, create new stuff all the time, ...

colonwqbang(10000) 1 day ago [-]

'Planned obsolesence' as a matter of official policy. Any cultural expression older than a few years is not worth the bits encoding it. Meanwhile, GTA V from 2013 is still one of the top selling games on Steam.

crazygringo(10000) 1 day ago [-]

GTA V has been updated countless times since 2013, including entire re-releases for two newer console generations.

vkoskiv(10000) about 18 hours ago [-]

Earlier this year I set out to completely rewrite an old project[1] using minimal, battle-tested dependencies to avoid having to constantly patch code rot associated with more modern languages and toolchains. The rewrite[2] is now complete, more performant and lightweight than the old one, and I'm confident I won't have to touch it much, if at all, and it will compile and work in 20 years.

[1] https://github.com/vkoskiv/NoMansCanvas

[2] https://github.com/vkoskiv/nmc2

ranguna(10000) about 18 hours ago [-]

You'll probably still need to patch every once in a while for security updates on your dependencies.

bsdetector(10000) 1 day ago [-]

> My old code simply did not work anymore with the latest versions of Xcode.

In other words if there was some actual reason to do an update, like a security flaw or serious bug, she wouldn't have been able to readily do so.

This seems to support Apple's policy to make developers update the app every once in a while.

lapcat(10000) 1 day ago [-]

> In other words if there was some actual reason to do an update, like a security flaw or serious bug, she wouldn't have been able to readily do so.

(1) There wasn't.

(2) 'After 4 hours of work to re-compile my app and 44 hours waiting in the review queue'

The biggest obstacle here isn't the developer, it's Apple.

layer8(10000) 1 day ago [-]

Apple is causing this issue in the first place by breaking compatibility and requiring projects to be updated for each new Xcode version.





Historical Discussions: Software I'm thankful for (2021) (September 23, 2022: 802 points)
Technology I'm Thankful For (November 26, 2021: 4 points)
Software I'm Thankful For (November 30, 2021: 3 points)
Software I'm Thankful For (November 28, 2021: 3 points)
Software I'm Thankful For (November 27, 2021: 2 points)

(803) Software I'm thankful for (2021)

803 points 4 days ago by yarapavan in 10000th position

crawshaw.io | Estimated reading time – 6 minutes | comments | anchor

Software I'm thankful for

2021-11-25

A few of the things that come to mind, this thanksgiving.

open/read/write/close

Most Unix-ish APIs, from files to sockets are a bit of a mess today. Endless poorly documented sockopts, unexpected changes in write semantics across FSs and OSes, good luck trying to figure out mtimes. But despite the mess, I can generally wrap my head around open/read/write/close. I can strace a binary and figure out the sequence and decipher what's going on. Sprinkle in some printfs and state is quickly debuggable. Stack traces are useful!

Enormous effort has been spent on many projects to replace this style of I/O programming, for efficiency or aesthetics, often with an asynchronous bent. I am thankful for this old reliable standby of synchronous open/read/write/close, and hope to see it revived and reinvented throughout my career to be cleaner and simpler.

goroutines

Goroutines are coroutines with compiler/runtime optimized yielding, to make them behave like threads. This breathes new life into the previous technology I'm thankful for: simple blocking I/O. With goroutines it becomes cheap to write large-scale blocking servers without running out of OS resources (like heavy threads, on OSes where they're heavy, or FDs). It also makes it possible to use blocking interfaces between "threads" within a process without paying the ever-growing price of a context switch in the post-spectre world.

Tailscale

This is the first year where the team working on Tailscale has outgrown and eclipsed me to the point where I can be thankful for Tailscale without feeling like I'm thanking myself. Many of the wonderful new features that let me easily wire machines together wherever they are, like userspace networking or MagicDNS, are not my doing. I'm thankful for the product, and the opportunity to work with the best engineering team I've ever had the privilege of being part of.

SQLite

Much like open/read/write/close, SQLite is an island of stability in a constantly changing technical landscape. Techniques I learned 10 or 15 years ago using SQLite work today. As a bonus, it does so much more than then: WAL mode for highly-concurrent servers, advanced SQL like window functions, excellent ATTACH semantics. It has done all of this while keeping the number of, in the projects own language, "goofy design" decisions to a minimum and holding true to its mission of being "lite". I aspire to write such wonderful software.

JSON

JSON is the worst form of encoding — except for all the others that have been tried. It's complicated, but not too complicated. It's not easily read by humans, but it can be read by humans. It is possible to extend it in intuitive ways. When it gets printed onto your terminal, you can figure out what's going on without going and finding the magic decoder ring of the week. It makes some things that are extremely hard with XML or INI easy, without introducing accidental Turing completeness or turning country codes into booleans. Writing software is better for it, and shows the immense effect carefully describing something can do for programming. JSON was everywhere in our JavaScript before the term was defined, the definition let us see it and use it elsewhere.

WireGuard

WireGuard is a great demonstration of why the total complexity of the implementation ends up affecting the UX of the product. In theory I could have been making tunnels between my devices for years with IPSec or TLS, in practice I'd completely given it up until something came along that made it easier. It didn't make it easier by putting a slick UI over complex technology, it made the underlying technology simpler, so even I could (eventually) figure out the configuration. Most importantly, by not eating my entire complexity budget with its own internals, I could suddenly see it as a building block in larger projects. Complexity makes more things possible, and fewer things possible, simultaneously. WireGuard is a beautiful example of simplicity and I'm thankful for it.

The speed of the Go compiler

Before Go became popular, the fast programming language compilers of the 90s had mostly fallen by the wayside, to be replaced with a bimodal world of interpreters/JITs on one side and creaky slow compilers attempting to produce extremely optimal code on the other. The main Go toolchain found, or rediscovered, a new optimal point in the plane of tradeoffs for programming languages to sit: ahead of time compiled, but with a fast less-than-optimal compiler. It has managed to continue to hold that interesting, unstable equilibrium for a decade now, which is incredibly impressive. (E.g. I personally would love to improve its inliner, but know that it's nearly impossible to get too far into that project without sacrificing a lot of the compiler's speed.)

GCC

I've always been cranky about GCC: I find its codebase nearly impossible to modify, it's slow, the associated ducks I need to line up to make it useful (binutils, libc, etc) blow out the complexity budget on any project I try to start before I get far, and it is associated with GNU, which I used to view as an oddity and now view as a millstone around the neck of an otherwise excellent software project.

But these are all the sorts of complaints you only make when using something truly invaluable. GCC is invaluable. I would never have learned to program if a free C compiler hadn't been available in the 90s, so I owe it my career. To this day, it vies neck-and-neck with LLVM for best performing object code. Without the competition between them, compiler technology would stagnate. And while LLVM now benefits from $10s or $100s of millions a year in Silicon Valley salaries working on it, GCC does it all with far less investment. I'm thankful it keeps on going.

vim

I keep trying to quit vim. I keep ending up inside a terminal, inside vim, writing code. Like SQLite, vim is an island of stability over my career. While I wish IDEs were better, I am extremely thankful for tools that work and respect the effort I have taken to learn them, decade after decade.

ssh

SSH gets me from here to there, and has done since ~1999. There is a lot about ssh that needs reinventing, but I am thankful for stable, reliable tools. It takes a lot of work to keep something like ssh working and secure, and if the maintainers are ever looking for someone to buy them a round they know where to find me.

The public web and search engines

How would I get anything done without all the wonderful information on the public web and search engines to find it? What an amazing achievement.

Thanks everyone, for making computers so great.




All Comments: [-] | anchor

scorxn(10000) 4 days ago [-]

I'd have to add Pi-hole to this list. Anytime I browse the web on another network, I'm reminded just how much crippling ad garbage it's sparing me.

dont__panic(10000) 4 days ago [-]

I use a VPN into my home network at all times for exactly this reason.

Also useful to circumvent iOS's sorry state of ad blocking. I use an ad blocker on my device, too, but the pi-hole takes care of 90% of the stuff that annoys me.

I've donated a few times; consider buying the devs a cup of coffee!

tobinfekkes(10000) 4 days ago [-]

Pi-hole has been an absolutely life-saver to our household. It's typically blocking ~50% of traffic.

Using someone else's device and off my network is entirely scary.

vanshg(10000) 4 days ago [-]

If you want to avoid the rigamarole of self hosting, I recommend NextDNS. It's a paid DNS service with tons of customizability (i.e. Ad blocking) and visibility (i.e. logs)

zuhsetaqi(10000) 4 days ago [-]

I prefer AdGuard Home for it's simplicity. Used PiHole before and AdGuard Home is so much simpler for me

Peter5(10000) 4 days ago [-]

Everything (Instant file search): https://www.voidtools.com/

nsonha(10000) 3 days ago [-]

does search on windows still suck? Also Listary

lemper(10000) 4 days ago [-]

I don't know about you, but I can't buy bread with 'thanks.'

notatoad(10000) 4 days ago [-]

are you the author of one of the pieces of software listed in this article? and if so, how certain are you that the author hasn't contributed to your project, either through direct financial contribution or by paying developers to work on it?

as far as i'm aware, tailscale is a pretty upstanding member of the open-source community.

pjscott(10000) 4 days ago [-]

We don't live by bread alone.

somekyle(10000) 4 days ago [-]

it's funny, because I believe the author has contributed to at least 4 of these directly in his day job, and has published code for a few more. But I guess you can't buy bread with substantial contributions, if ability to buy bread is our metric. But it's not a good metric.

weakfish(10000) 4 days ago [-]

Helix. Most wonderful to use text editor I've found in my short career.

jorgeavaldez(10000) 4 days ago [-]

I can't help but second this. It's a total joy to use and lets me focus on doing what I need to do without constant lag and yak-shaving configuration files.

kasperset(10000) 4 days ago [-]

Visidata - Helpful to view excel/delimited text file to glance the data on TUI

Vscode - I was very skeptical of this text editor but it won me over. There are some rough edges but overall very good.

MacOS - Overall a stable operating system.

R Programming Language - Has flaws but works generally good to explore and clean data. Special thanks to Ggplot2 to make it bearable to make plots.

Folder Peek - Allowing me to switch different projects/folder from Menu bar.

fcoury(10000) 4 days ago [-]

Haven't heard of Visidata before. Just downloaded and it blew my mind. A less useful tool particularly for CSV is the CSV Rainbow plugin on VSCode, which was my previous favorite.

swiley-gin(10000) 4 days ago [-]

Software is just the smell of other computer users. Some smell nice, most smell terrible, it's nothing special either way. You can rewrite anything in an afternoon if you're in the right mood.

tmtvl(10000) 4 days ago [-]

Maybe not an afternoon, but any software could be rewritten given enough time and interest.

Although copyright and patents may prohibit you from distributing it.

ParetoOptimal(10000) 4 days ago [-]

- Nix/NixOS

- GHC (Haskell compiler)

- Emacs

- consult/vertico/marginalia

- Org-mode

- Org-roam

- tramp.el

- magit

- docker.el

- gnus

eointierney(10000) 4 days ago [-]

It really is back to the future, isn't it?

If emacs then emacs, else $(evil)

Org-mode is my favourite superpower, magit is a superb example of nominative determinism, and the underlying trust of GNU is childishly verifiable.

RMS for our collective win

reaperducer(10000) 4 days ago [-]

ItsyCal (macOS). So simple, and so useful.

The best part is being able to replace the standard menu bar date with my custom date, including time zone.

boc(10000) 4 days ago [-]

This is awesome! Appreciate the recommendation.

isametry(10000) 4 days ago [-]

Thank you for this!

The culture of small, useful and native-feeling utility apps is honestly one of my favorite aspects of macOS.

dont__panic(10000) 4 days ago [-]

Adding another thank you because this simple, cute, incredible app is a new desktop staple for me.

ok_computer(10000) 4 days ago [-]

Sublime text

Terminus

Zsh

Powershell (verbosity isn't terrible!)

SQLAlchemy (connection engine + docs)

Git

Ssh

IPython as a debugger shell

conkeisterdoor(10000) 4 days ago [-]

Are you me?

In addition to everything above, I'm thankful for these:

Pass (passwordstore.org)

Newsblur

Multitail

dont__panic(10000) 4 days ago [-]

- sublime text

- jellyfin

- finamp

- piOS

- wireguard

- narwhal iOS Reddit app

- osmand/openstreetmap in general

- netnewswire

- pocket casts

- librewolf

- sonixd

- obsidian

- signal

- iterm

DMell(10000) 4 days ago [-]

I hadn't used Sublime for a long time until recently. The speed is unreal.

onehair(10000) 4 days ago [-]

ffmpeg is one software that comes to mind. At first it sounds and looks complicated, but all the internet video is ran by it, and now even for small stuff I use it with admiration

aqfamnzc(10000) 4 days ago [-]

Can you elaborate on 'all the internet video is ran by it'?

muttantt(10000) 4 days ago [-]

One piece of OSS I will be forever grateful for is Freeswitch. I built a massively successful business on top of it.

jq-r(10000) 4 days ago [-]

Ring?

AdityaSanthosh(10000) 4 days ago [-]

Thanks for mentioning this. It looks interesting

godshatter(10000) 4 days ago [-]

For me the list is:

  - Linux
  - gcc, vim, git, make, et al
  - KDE
  - firefox
  - yakuake (terminal that drops down like the old Quake console used to)
  - libre office
  - mpv
  - Steam and Proton (which have made gaming work very well on Linux and have contributed to the complete loss of all productivity gained by any of the above programs).
musicale(10000) 4 days ago [-]

> Steam and Proton (which have made gaming work very well on Linux and have contributed to the complete loss of all productivity gained by any of the above programs).

;-)

Oddly enough I think we can thank Microsoft for creating a standard binary format for PC games on Linux. Now if we could just get a decent Proton (or equivalent) for macOS (CrossOver doesn't seem to work as well.)

legrande(10000) 4 days ago [-]

This reminds me of Steve Jobs' email to himself. In 2010 he wrote:

I grow little of the food I eat, and of the little I do grow I did not breed or perfect the seeds.

I do not make any of my own clothing.

I speak a language I did not invent or refine.

I did not discover the mathematics I use.

I am protected by freedoms and laws I did not conceive of or legislate, and do not enforce or adjudicate.

I am moved by music I did not create myself.

When I needed medical attention, I was helpless to help myself survive.

I did not invent the transistor, the microprocessor, object oriented programming, or most of the technology I work with.

I love and admire my species, living and dead, and am totally dependent on them for my life and well being.

Sent from my iPad

Source: https://stevejobsarchive.com/

andrepd(10000) 4 days ago [-]

A nice reminder of the absurdity of the narrative of the 'self-made man'.

jbverschoor(10000) 4 days ago [-]

Not 'to himself', it was for a speech.

dang(10000) 4 days ago [-]

Discussed here:

Steve Jobs emails himself (2010) - https://news.ycombinator.com/item?id=32761060 - Sept 2022 (247 comments)

Arjuna(10000) 4 days ago [-]

After reading this, I am reminded of the Buddhist concept of Pratītyasamutpāda [1].

In essence, existence is interwoven in a complex tapestry of interdependence.

https://en.m.wikipedia.org/wiki/Pratītyasamutpāda

vivekv(10000) 4 days ago [-]

The last line 'Sent from my iPad' was quite fitting as a device that he conceptualized :-)

falcor84(10000) 4 days ago [-]

This is probably the most elaborate humblebrag I've ever seen.

dividedbyzero(10000) 4 days ago [-]

The fish shell, makes using the shell feel almost painless.

Hammerspoon, I use it to automatically switch audio devices based on context, so every call uses the best microphone currently available, window management via keystrokes, limiting media keys to Spotify, tons of other things. Indispensable.

Arduino, I don't think I would be able to tinker with microcontrollers as much if I had to write C and use obscure toolchains directly.

Solvespace, a limited but usable free CAD for simple parts to be 3d printed. Wish there was a real contender to the commercial ones though, or a free tier that doesn't smell like it's going away any time. Still very thankful that SolveSpace exists.

The Scala 3 compiler and the VSCode plugin for v3. Absolutely love the language and the experience is so much better than with IntelliJ, haven't had as much fun writing code in ages.

This will be a bit controversial, but Kubernetes, because if people use it via GKE, EKS etc. then I won't have to learn their organically grown solution to the same dozen-or-so operations problems, and I have yet to see one that isn't a hot mess in some way or other. Also anything running on top of Kubernetes won't be built the very old-fashioned vi-edits-on-server way, great for sanity.

Various modern messenger apps (Telegram, Whatsapp, Instagram's direct messages, ...) because I would hear a lot less from some highly cherished and very non-technical people in my life without these incredibly slick and fun and convenient apps. As one who's old enough to remember being dependent on landlines and payphones and letters, this still feels like a miracle.

jstanley(10000) 4 days ago [-]

If you like SolveSpace but want something more powerful, you should try FreeCAD.

Yes, it's harder to get started, and it's less fun to use, but it is more powerful. I wrote a comparison of the two earlier this year[0].

[0] https://incoherency.co.uk/blog/stories/freecad-vs-solvespace...

alanbernstein(10000) 4 days ago [-]

Got any Hammerspoon config to share? I've used it for simple stuff for years, but recently noticed that people use it for window management, and macos' handling of windows on my second monitor is driving me nuts...

barrenko(10000) 4 days ago [-]

Scala Metals team are absolute beasts, I can't really tell of other instances where it's so easy to get almost a full blown IDE out of vim so easily (as a total beginner).

zuhsetaqi(10000) 4 days ago [-]

Do you mind sharing your Hammerspoon config?

wan_ala(10000) 4 days ago [-]

I use fish as my main shell and haven't seen anyone else that has. Honestly its really good, the autocomplete feature is my favorite.

d3nj4l(10000) 4 days ago [-]

If you don't mind me asking, why do you prefer Scala 3 specifically? What did it change over 2 that you made such a big difference?

preseinger(10000) 4 days ago [-]

> This will be a bit controversial, but Kubernetes, because if people use it via GKE, EKS etc. then I won't have to learn their organically grown solution to the same dozen-or-so operations problems, and I have yet to see one that isn't a hot mess in some way or other. Also anything running on top of Kubernetes won't be built the very old-fashioned vi-edits-on-server way, great for sanity.

I appreciate this perspective. It's sane. But my consistent experience has been that the complexity permitted by the Kubernetes configuration surface area tends to be larger, more complex, and ultimately more difficult to wrangle than the complexity introduced by (reasonably coherent) home-grown systems.

I would generally prefer to be responsible for a bespoke solution, with all of its warts and pathologies, than for a Kubernetes deployment. The problems of the former tend to be at least tractable; wrangling Kubernetes to a reasonable level of availability is in my experience only possible if you dedicate your career to it.

bloopernova(10000) 4 days ago [-]

Stuff I'm thankful for, in no particular order. None of these are particularly unique to me, nor are they obscure and hipster, they're just stuff I have found myself really thankful for.

powerlevel10k because it makes adding custom sections to my shell prompt really straightforward. https://github.com/romkatv/powerlevel10k

terraform because I have a job wrangling it lol

asdf because it manages versions of software for me really well and it has thus far been rock solid reliable. https://asdf-vm.com/

Emacs because it's about as configurable and customizable as my most insane requirements. And emacs lisp is very cool. Similarly, vim and vscode are also dear to me.

Factorio! because of course Factorio, it's amazing. Similarly Kerbal Space Program.

Firefox for standing up against the chrome hegemony nowadays, and for being so exciting back in 1998 with its initial open source decision.

And Tree Style Tabs, because every time I have to use a browser without it, my skin crawls at the lack of organization. https://github.com/piroor/treestyletab

And the big ones: grep, sed, awk, cut, sort, uniq, jq for all the times they've turned something incomprehensible into something useful to this tiny mind.

EDIT: oh and zsh because zmodload zsh/datetime gives me $EPOCHSECONDS which makes life so much easier to make cool prompt segments like 'days and hours since last commit' and 'remaining auth session time in minutes and seconds'

    # display time since last commit in days and hours
    gdate -d @$(($EPOCHSECONDS-$(git log -1 --format=%ct))) -u +'(%-dd %-Hh ago)'
kazinator(10000) 4 days ago [-]

Breaks past one month, unfortunately.

I think, you want to take the difference and divide by 86400 to get the number of days. Then take the difference modulo 86400 to get seconds in a day, and convert that to hours (minutes:seconds) via strftime string.

Proof of concept:

  1> (let ((tdiff (- (time) (read (command-get-string 'git log -1 --format=%ct')))))
       (match (@days @secs) (trunc-rem tdiff 86400)
         `(@{days}d @(time-string-utc secs `%H`)h ago)`))
  '(627d 12h ago)'
danparsonson(10000) 4 days ago [-]

Wow thanks for the Tree Style Tabs tip, that looks really interesting!

thakoppno(10000) 4 days ago [-]

nobody said nodejs yet?

ExtremisAndy(10000) 4 days ago [-]

Not sure if anyone else has said it, but I sure will. I am the lone developer of my website, and it has been so wonderful to be able to remain in JavaScript for both front and back end development. Node also makes a fine scripting language, and the repl is where I normally go by default to practice/drill JS concepts so I won't forget them. I'm incredibly grateful for NodeJS!

vincent-manis(10000) 4 days ago [-]

Emacs. TeX/LaTeX. Classic Unix. i3. A bunch of Scheme implementations. Tcl/Tk.

legohead(10000) 4 days ago [-]

I was going to mention nano

when you just wanna edit things quick and only need to remember a few commands. just wish it came standard with most distros.

onehair(10000) 4 days ago [-]

i3 is so simple and intuitive!

tmtvl(10000) 4 days ago [-]

Emacs may not be the best Lisp interpreter around, but I still love it for how open it is and for the great community around it.

agumonkey(10000) 4 days ago [-]

emacs, magit, i3, arch, lisps and descendants, firefox, mpv

magit especially

glintik(10000) 4 days ago [-]

vim? Really you said this? OMG.

seanw444(10000) 4 days ago [-]

Vim's great. So great that I essentially use Vim on Emacs.

reaperducer(10000) 4 days ago [-]

You don't have anything more to contribute than 'Stop liking things I don't like?'

dang(10000) 4 days ago [-]

Could you please stop posting unsubstantive comments to Hacker News? You've been doing it repeatedly, unfortunately, and we're trying for a different sort of forum here.

https://news.ycombinator.com/newsguidelines.html

theatomheart(10000) 4 days ago [-]

yes but i wont judge you for not understanding or appreciating the power and usefulness of vim. and dont bother, either. i encourage you to stick with the peasants' notepad++

pipeline_peak(10000) 4 days ago [-]

> JSON is the worst form of encoding — except for all the others that have been tried.

Edgy pessimism is tiring, we get it everything sucks but this is the best we got, even though you still hate it.

> It's not easily read by humans, but it can be read by humans.

What does this even mean, why, how? It's so easy to complain about something without admitting you don't have a better solution.

Maybe I'm bitter from my ERP dev ops role where I'm constantly enduring little thorns throughout the pipeline and overall codebase. But this way of thinking isn't practical, you won't get far if they're what you deal with day to day.

preseinger(10000) 4 days ago [-]

I don't see this as edgy pessimism, I see it as pragmatic optimism. The author isn't complaining about JSON, they're acknowledging JSON as the best choice despite its warts.

SV_BubbleTime(10000) 4 days ago [-]

VS Code

It took me awhile to switch over. But really, I can't go back to anything else. The customization and UX is unbeatable.

BrandoElFollito(10000) 4 days ago [-]

Same here. I use to use Jetbrain editors that are nice but slow and extremely specific to languages.

I forced myself to use vscode for two weeks and it is mostly fine. I do miss a lot of well-done things from Jetbrain (today for instance I had to build and run a Dockerfile and it was painful. Not impossible, had to look up all the commands but it was still painful).

aliencat(10000) 4 days ago [-]

Karabiner is incredible for making keyboard shortcut to do some pretty complicated task. Obsidian, what a note-taking app! Makes Evernote looks like a dinasour. Vim (and NeoVim), you learn it once and use it for everything.

PikachuEXE(10000) 4 days ago [-]

Instead of Obsidian which requires payment to sync notes.

I am using Joplin: https://joplinapp.org

Using Dropbox as sync with E2EE (but you can use other ways to sync)

sleepycatgirl(10000) 4 days ago [-]

Let's see.. Software I am thankful for.. there is a fair amount of such.

Anki - Software, that helped me build up habit, and made learning language an easier task

Emacs - Wonderful text editor, made interacting with system a bliss.

Nix/NixOS - Distro, that made updates a painless, and fearless task. I love it.

ZFS - Filesystem that I love, for it has many wonderful features, and they all just work. (Also cute compression)

Wine + DXVK/VKD3D - Thanks to this, I was able to completely drop windows partition, and go full Linux.

LaTeX - Thanks to it, I could have cute workflow for writing documents (And yesterday, wrote CV with its' help :D)

Calibre - Man.. what a behemot of book software. Makes anything ebook related painless

Common Lisp(SBCL) and Haskell(GHC) - Very interesting languages, with very wonderful features. I love them both.

Cool retro term - For playing roguelikes, lets me experience them in very retro, retro way.

Obviously, ublock origin, makes browsing web not a nightmare.

Aseprite - wonderful pixelart software.

There is more, but... I will stop there, to make the comment.. not too long.

PointyFluff(10000) 4 days ago [-]

This was better and more thought out than the OP's blog post.

ziotom78(10000) 4 days ago [-]

Thanks a lot for mentioning Anki! I was not aware of its existence, but it's exactly what I needed for myself and my children!

barumrho(10000) 4 days ago [-]

Curious to learn, what are the differentiators of ZFS? File system seems like an invisible layer to me personally which makes me ask this question.

scop(10000) 4 days ago [-]

Outside of OS & text editor (vim), there is one tool that I use countless times every, single, day:

fzf (https://github.com/junegunn/fzf)

I can only wonder how much time fzf has saved me in the long term.

In terms of 'software that I don't use for writing software', iA Writer is probably what I am most grateful for.

fsckboy(10000) 4 days ago [-]

I don't know if the blame is fzf's or cygwin's, but when i used to use cygwin I would just install everything in Setup, and that meant that i at a certain point i got fzf with (what?) default config.

It was a hot mess and took over several bash/emacs control keys. It did a nice job of searching history, but I hated that it paralyzed my keyboard usage

graton(10000) 4 days ago [-]

fzf is awesome :)

My favorite fzf usage (but not the most often used) is this alias I put in my ~/.gitconfig file

    frbi = '!f() { git rebase -i $(git log --pretty=oneline --color=always | fzf --ansi | cut -d ' ' -f1)^ ; }; f'
So I do: git frbi (which to me means Fzf ReBase Interactive)

And then can easily pick what commit I will use as my starting point for doing some rebase work.

idealmedtech(10000) 4 days ago [-]

On a similar note for Windows: Everything Search. Blazing fast, super easy syntax, powerful search tools. Almost don't remember where most things are stored on my hard drive these days, since they're all just a shortcut and a few characters away

mhb(10000) 4 days ago [-]

Davinci Resolve

PyCharm

Clion

Audacity

andrei_says_(10000) 4 days ago [-]

Davinci Resolve is incredible. Using its fully functional version for free feels like cheating.

One of the side effects being that it presents the tools and context for doing things properly - from editing to color correction.

isametry(10000) 4 days ago [-]

I was so delighted to discover DaVinci for a multitude reasons, but one of the biggest ones was that it doubles as a full-blown audio editor (for my use-cases, anyway). This allowed me to finally ditch Audacity, after its development took some very questionable turns last year.

ziotom78(10000) 4 days ago [-]

Lots of good entries, I'll add a few that haven't been mentioned yet:

- ncdu, to find out which files and folders take up most of the disk space [1]

- julia, because scientific programming was never so fun and fast [2]

- midnight commander, because a TUI implementing Norton Commander dual-pane view is so useful! [3]

- lazarus, because creating multi-platform desktop apps couldn't be simpler [4]

- zstandard, its performance are awesome!

[1] https://dev.yorhel.nl/ncdu

[2] https://julialang.org/

[3] https://midnight-commander.org/

[4] https://www.lazarus-ide.org/

[5] http://facebook.github.io/zstd/

sleepycatgirl(10000) 4 days ago [-]

Oh, I use ncdu too. Really wonderful tool, loving it. Also zstd, yeah, I use it on ZFS, and man, it saves me so much space its impressive, even on games lol

aitchnyu(10000) 1 day ago [-]

I used Baobab which is bundled into Ubuntu to drill down and find space hogs and folders which have 100000s of files. The latter helped me shorten moving all my files between PCs.

sigmaml(10000) 4 days ago [-]

Here is my quick list.

- Linux + GNU: My servers would never have been the same without them.

- Emacs: So many varied use cases over the years. I can type in Telugu (my mother tongue) and Devanagari so seamlessly ... bliss!

- LaTeX: From papers to books to presentations.

- Helix: A very recent discovery that replaced Vim for me in just a week!

- Kitty terminal: I no longer use GNU screen or tmux locally.

- Go, Ruby, Python: They have been paying my bills for so long.

- KDE: The other half of the Linux spirit!

- ffmpeg: Such a boon!

- Homebrew: So convenient.

betwixthewires(10000) 4 days ago [-]

Helix really is amazing, even with it's current level of maturity (or should I say immaturity) I just can't stop using it, I can't wait to see how well it works in a year or two.

ziotom78(10000) 4 days ago [-]

Kitty and KDE are two real gems!

_ink_(10000) 4 days ago [-]

JetBrains IDEs. I need to work with three different languages and it is just a blessing that every IDE works exactly the same.

ziotom78(10000) 4 days ago [-]

Absolutely! I teach a CS class in my university and encourage my students to try them out. (It's a class where students are free to pick the language to use, but they usually end up using C++, C#, or Kotlin.) They seem to love the many suggestions the IDE provides about code style, and the visual clues that suggest how to use some syntax. At the end of the semester, many of them have thanked me for that tip, saying that «I would have never imagined that an IDE could teach me how to write better code!».

billyhoffman(10000) 4 days ago [-]

Last week I learned some cool mouse and keyboard shortcuts in JetBrains, only to then discover they also did the same action in Visual Studio! No more "right click-go to definition" for me! I was presently surprised at how polished it was

bonestamp2(10000) 4 days ago [-]

Not to mention, there are so many helpers built in and it's one of the few pieces of software that I don't have any major complaints about.

loudmax(10000) 4 days ago [-]

> WireGuard is a great demonstration of why the total complexity of the implementation ends up affecting the UX of the product.

This is absolutely true! Probably everything you could do with WireGuard you could accomplish with OpenSSL/OpenVPN, but the complexity is staggering. This makes it much more difficult to troubleshoot and far more likely that there will be an error in the configuration that could lead to compromise.

Mister_Snuggles(10000) 4 days ago [-]

I'm torn on WireGuard.

Getting it set up was really easy. I think it took about an hour, which included standing up a new OpenBSD VM to run the server. Configuring clients (iOS) was pretty straight-forward and reasonable. I don't have a feeling that there's some security hole lurking in the configuration simply because there's not a lot to get wrong.

In comparison, IKEv2/IPSec was incredibly painful, especially when you also create configuration profiles for iOS and have to issue and distribute certificates. Once I understood how it worked it made some sense, but there's always that nagging feeling that I've screwed up the configuration because there's just so many options to get wrong. I've probably spent six hours, not all at once of course, getting it working and set up 'just so.'

In spite of all of this, I'm continuing to use my IKEv2/IPSec setup. I have not figured out how to get WireGuard to do split DNS, whereas this was a fairly simple part of the IKEv2/IPSec setup. The other requirement I have is split tunnels, but I think WireGuard can do that fairly easily.

onehair(10000) 4 days ago [-]

For my simple road warrior setup, 8 lines that are clearly understandable were all it took in wireguard.

OpenVPN was always a nightmare

RedShift1(10000) 4 days ago [-]

I wish more developers would work on this, making sure their product works straight out of the box. Lots of software requires setting up additional things like Redis, an SQL server, Docker, a proxy server, etc... All those things are in most cases unnecessary. I understand that for high load scenarios they are needed but for small time setups it's just overkill. Make it run with sane defaults and when the time comes to scale up, then those other things can be added.

Dalewyn(10000) 4 days ago [-]

Strangely enough: Windows.

Yes, I fucking hate Windows 10/11 for several laundry lists' worth of reasons, but you know what? At the end of the day, Windows is the only desktop OS that enables me to use my computer to do the shit I need or want to do.

So long as that fundamental principle as a tool is not violated, I will forever be thankful for Windows regardless what criticism I might have for it.

tremon(10000) 3 days ago [-]

Windows is the only desktop OS that enables me to use my computer to do the shit I need or want to do.

Isn't that a very sad state of affairs? Why should such a piece of critical global infrastructure be in the hands of a single company?

jgillich(10000) 4 days ago [-]

Software development is a much better experience on Linux compared to Windows (unless you stay within Visual Studio).

If you do tasks that are well supported by your OS, your experience will be mostly good. Don't run servers on MacOS, don't develop on Windows and don't game on Linux. Or do it anyway and deal with the unpolished aspects of it.

lcnPylGDnU4H9OF(10000) 4 days ago [-]

I can't disagree, even if I would prefer something else. Easily the most compelling argument I've heard for Windows is Tom Scott's: https://www.youtube.com/watch?v=lIFE7h3m40U.

wnolens(10000) 4 days ago [-]

The sysinternals tools make it for me. procexp and procmon are insanely powerful and you can solve really obscure issues really quickly.

I'm on mac right now for my current job, and quite often a few of my cores go nuts due to a kernel_task process which I find it impossible to track down the reason why. Pretty sure I could root cause this in 10 minutes with procexp.

zac23or(10000) 4 days ago [-]

Windows is a very good operating system. Windows haters don't hate for technical reasons, but for religious or political reasons.

And I like Microsoft. Microsoft makes some products, that's it. Not experiences, or promises of freedom. They are software products, with problems, qualities, end.

Arisaka1(10000) 4 days ago [-]

>At the end of the day, Windows is the only desktop OS that enables me to use my computer to do the shit I need or want to do.

I feel like this is the kind of dystopian thing everyone just accepted that it's normal.

And by that I mean, I expected competition, options, etc. Instead I'm still forced to use Windows for my gaming, GPU's for Linux desktop is still hit or miss, and Apple decided to break up with Intel so on top of everything else I get to 'enjoy' package incompatibilities for my work.

And now I have to pretend that I'm enjoying that Microsoft keeps deciding to grab 'a little bit more' every now and then because I really don't have any other options, besides the 'stop playing video games that aren't a pain to setup or even work on Linux'.

onehair(10000) 4 days ago [-]

This! I like Linux a lot, but whenever I try to build an ecosystem in it for more than few workflows I end up in deep time consuming tasks to figure out how to make things cohabit without having my services fight each other.

The same open-source apps for some of the same workflows work flawlessly in few minutes without all the struggles.

inetknght(10000) 4 days ago [-]

> At the end of the day, Windows is the only desktop OS that enables me to use my computer to do the shit I need or want to do.

I disagree on two fronts.

First that Windows is the 'only' desktop OS that enables you to use your computer to do the shit you need or want to do. macOS and Linux both do as well.

Secondly that Windows is actually a very terrible OS to use your computer to do the shit you need to do unless you have zero care whatsoever for privacy.

aljgz(10000) 4 days ago [-]

I use Linux as my main OS for some years now, but use Mac and Windows, mostly because I need to test my software on them, but I have to say:

I totally agree. Windows is the only practical desktop OS for most people.

Mac is not an OS you can use unless you buy the entire package, and then you have to live with many of its limitations. It absolutely sucks at multitasking (not that most average users care about this one, but anyone coming from windows will struggle).

Linux is absolute freedom, but that comes with the cost of having to do much more than install, plug and play. You need to be a computer geek, or you need someone else to fix things for you.

So as much as we are mad at Microsoft for deliberately damaging the experience of using what could be a joy to use, it's still the best at what it is.

Bubble_Pop_22(10000) 4 days ago [-]

All Microsoft products are free of charge if you know where to look for them.

Basically you have the big Fortune 500 companies subsidizing pirates who download from TPBay...and of course the pirate CD copies being sold in street markets all over Sub-Saharan Africa.

Microsoft with Windows, Azure, Office, Outlook and PowerPoint is the backbone of the economy. Come noon you have already used half a dozen of different flavors of Microsoft and again if you know where to look you tasted them all for free.

Analyzing things post-facto the bad reputation that Microsoft had was unwarrented given that we ended up under a much tighter stranghold of Apple in the mobile environment. At least the developing world is saving themselves using Android.

tucif(10000) 4 days ago [-]

Recently switched to macos for development and I'm really thankful to have found out about Shortcat and Raycast.

Combined with Tridactyl plugin on firefox, I can keep my hands on the keyboard for almost every task across the OS.

xwowsersx(10000) 4 days ago [-]

Yes, love Raycast. ctrl + opt + space and I can immediately jump to my Zoom and Google meetings.

hot_gril(10000) 4 days ago [-]

- Postgresql. I still can't believe it's free.

- NodeJS. Despite its popularity, it's still underrated.

entropicgravity(10000) 1 day ago [-]

A lot of people, myself included, just take this for granted when in fact it's critical and reliable infrastructure for so many things.

tezza(10000) 4 days ago [-]

  cygwin
  mintty
  stream deck
  firefox / thunderbird
  eclipse
  linux
  socat / netcat
  postgres / mysql
  ms windows (ducks)
  emacs
  ms excel
unnouinceput(10000) 4 days ago [-]

Fucking finally somebody that says cygwin. Everybody nowadays is 'wsl2 this, wsl2 that' but they forget that the OG, which still blows wsl2, it's CygWin. I use it for 2 decades, one of the 2 programs I install whenever I need to use somebody else (usually client) PC; the other one is uBlock Origin.

rlam2x51(10000) 4 days ago [-]

Fork - a fast and friendly git client https://fork.dev/

Beyond Compare 4 - compare files and folders https://www.scootersoftware.com/

uBlock Origin - An efficient blocker add-on for various browsers. Fast, potent, and lean. https://github.com/gorhill/uBlock

Those tools made my life so much easier. Can't recommend them enough.

Just a happy user and not affiliated

therealdrag0(10000) 3 days ago [-]

Nice list. Love fork. Been a BC user for like 15 years now. And uBlock just works.

appel(10000) 3 days ago [-]

I'm going to grab this opportunity to give the tiniest of shout outs to a random list of mostly Windows based software that I use every single day.

- Windows 10/11

- WSL

- Powertoys

- Windows Terminal

- VS Code

- Beyond Compare

- Filelocator Pro

- Plex / Plexamp

- Calibre

- Resilio

- Firefox

- Ublock Origin

- Bitwarden

- Pushbullet

- Pihole

- Notion

- Hangouts (RIP, Chat is not the same, buddy)

McNutty(10000) 3 days ago [-]

Can I ask what you're using Beyond Compare for, that you can't get from VScode etc?

mig39(10000) 4 days ago [-]

Honestly, the big one for me, this year, is tailscale.

No matter where I am, what device I'm using, all my stuff is always accessible and with me.

I love it.

aidos(10000) 4 days ago [-]

So good that it took this article to remind me that it's there for me everyday, doing it's thing perfectly, staying out of my way.

euroderf(10000) 4 days ago [-]

Agreed. It replaces horrible multi-hop ssh hackery. Highly recommended.

eminence32(10000) 4 days ago [-]

Rarely a day goes by that I don't interact with tmux or vim or mosh. It's hard to imagine life without them

fullstop(10000) 4 days ago [-]

I added my own list but somehow omitted those. I use all of them on a daily basis.

b0ner_t0ner(10000) 4 days ago [-]

No love for PhraseExpress / Espanso?

tyiz(10000) 3 days ago [-]

+1 for PhraseExpress

ziotom78(10000) 4 days ago [-]

I love Espanso! https://espanso.org/

gulabjamuns(10000) 4 days ago [-]

Vim Ffmpeg Zsh, fish, iTerm, Awk, perl, Ruby...

behnamoh(10000) 4 days ago [-]

yt-dlp





Historical Discussions: Avoiding homework with code and getting caught (September 22, 2022: 763 points)

(764) Avoiding homework with code and getting caught

764 points 5 days ago by aabbccsmith in 10000th position

alistair.blog | Estimated reading time – 11 minutes | comments | anchor

Avoiding homework with code (and getting caught)

Back in of 2020, my school used a few online learning platforms that allowed professors/teachers to assign homework to students. I, as a lazy developer, wanted to spend more time playing games and writing code, especially when everyone was spending their time at home because of lockdown. I started writing this post in January of 2022, but I put off publicizing it for a while. It has been long enough since this all happened, so please sit back and enjoy.

The back story

Let's set the scene. 2018, my school introduces a new online homework platform for students. It's called HegartyMaths and it does a lot. It's fairly simple, teachers choose a topic to set for us as homework, with that we get a 10-15 minute tutorial/informational video on the subject (of which we have to write down notes whilst watching) and a shortish quiz to complete after finishing the video. It's a lot of work, especially the quiz, and in the worst cases can take up to an hour to complete one topic (bad).

Mostly, software engineers are rather lazy individuals. We tell metal how to do stuff for us. Homework then, naturally, is an arduous task for a developer who is still at school. So, still 2018, myself and a close friend of mine by the name of Scott Hiett decided to do something about the Hegarty situation. We started to reverse engineer the frontend app and eventually came up with a Tampermonkey userscript that would glitch the embedded YouTube player to say that we've watched the video at least 1x. Cruically, our teachers could see how many times we've watched the video, so being able to skip up to 20 minutes of homework time was especially useful – and it was a lot of fun to build too.

So we flexed it on our Snapchat stories and had our school friends message us to use it blah blah. We eventually figured out that we could also set it to be watched over 9999x times; every time we did that our accounts were reset by the Hegarty team.

The first email

After this, we got in contact with our Math teacher in November of 2018 and got her to send an email to HegartyMaths informing them of our petty exploit and they got back to us very quickly. I don't have the original email anymore but I distinctly remember it saying something along the lines of 'Stop trying to hack our platform and get back to doing your homework.' Edit: While writing this, I was able to uncover the deleted email from a photo we had taken of it in 2020. See below (certain details redacted for obvious reasons):

This response excited us a bit, as they were now aware of us messing around with the site and they had no intention of fixing the minor vuln we had anyway, so we kept using it. We had tried to build a script to answer the questions for us, but it was too much work at the time (complex data structures, weird API responses, etc etc).

Educake

For a while, students had access to another platform called Educake. Similar to HegartyMaths but targeting Biology, Chemistry and Physics. There was no video to watch at the beginning. We'd used it for a few years, in fact since I joined the school, but I'd never thought about reversing until all of this began.

One common factor between Hegarty and Educake is that immediately give you the correct answer if you got a question wrong. We took advantage of this and wrote a small node/mongo app & tampermonkey script to detect when a user was on a quiz page, and answer every question with a random number and then store the correct answer in mongodb. I don't have the original source but the TamperMonkey script was probably something like the following:

const guess = Math.random();

const result = await post('/api/answer', {
	body: {
		answer: guess,
	},
});
await post('http://localhost:8080/save', {
	body: {
		question_id: question.id,
		answer: result.success ? guess : result.correct_answer,
	},
});
// Go to next question and repeat code above
nextQuestion();
const guess = Math.random();

const result = await post('/api/answer', {
	body: {
		answer: guess,
	},
});
await post('http://localhost:8080/save', {
	body: {
		question_id: question.id,
		answer: result.success ? guess : result.correct_answer,
	},
});
// Go to next question and repeat code above
nextQuestion();

As you can see, it was quite literally a loop through every question, saving the correct answer as we got it and moving on. Eventually I added a few more features to fetch from the database if we already had the right answer (meaning we don't answer Math.random every time) and also I added in support for multiple choice (so that we actually pick one of the possible answers rather than making it up – however I was surprised that the Educake backend would allow an answer that wasn't even in the possible choices).

Now working on the project solo, I decided it would be time to build a nice UI for it all and bundle it all into a simple Tampermonkey script for both flexing rights on Snapchat (people constantly begging me to be able to use it was certainly ego fuel I hadn't experienced before) and also for myself to get out of homework I didn't want to do.

The end result? A ~200 line codebase that scooped up all questions and answers on the site that could repeatedly get 100% on every single assignment and a 15mb mongo database.

Below is a small video of what it all looked like. It also demonstrates a feature I added allowing for a 'target percentage' — meaning users could get something other than 100% to look like more real/human score. Video was recorded on my Snapchat in November 2019.

Hegarty 2

The success of this script along with pressure from my peers led me to gain a lot of motivation to start working on reversing Hegarty again. I reached out to an internet friend who, for the sake of his privacy, will be named as 'Jake.' He also used HegartyMaths at his school and was in the same boat as me trying to not do our homework. Together, we managed to figure out how to answer many varying types of questions, including multiple choice and ordered answers resulting in a huge amount of data stored. We had sacrificial user accounts and managed to answer 60,000 questions in a couple minutes, rocketing our way to the top of the HegartyMaths global leaderboard. Would like to give a special shoutout to Boon for lending us his login and letting us decimate his statistics.

Together, Jake and I scraped the entirety of Hegarty's database and now had a JSON file that could be argued to be worth as much as Hegarty the company itself due to the entire product quite literally being the database we had copied.

With this file, I wanted to take it a step further and allow my friends and other people to make good use of it without directly giving out the database (irresponsible)... And here Mochip was coined.

Mochip

So, where does Mochip tie in to this? Mochip was a chrome extension, a collection of both our scraped Hegarty and scraped Educake databases sat behind a TypeScript API and a small React app. Hosted on Heroku free tier and MongoDB Atlas free tier, users could login, enter a question (from either site) and get back a list of answers Mochip has for that question. Here's what the landing page looked like

In the screenshot we can see a few stats on the right like total estimated time saved and how long you've had your account for. We gamified it a little just to keep people engaged

Our chrome extension was made for Educake as they disabled copying question text with the clipboard. We re-enabled that just by clicking a button that was injected into the UI. The chrome extension is no longer on the chrome web store, but we've found that mirrors still have listings that we can't get taken down: extpose.com/ext/195388

Our userbase grew so big that we ended up with a Discord server and even our own listing on Urban dictionary — I'm yet to find out who made it! urbandictionary.com/define.php?term=mochip

Eventually we 'rebranded' as I wanted to disassociate my name from the project. Unfortunately I do not have any screenshots from this era to show. I made an alt discord account and a few announcements saying we'd 'passed on ownership' however this ineveitably only lasted for a couple weeks before we were rumbled.

Crashing down

All good things must come to and end, and Mochip's did after Scott posted about Mochip on his reddit account. Like any good CEO, Colin searches his company every now and then on Google to see what people are saying or doing and unfortunately came across our reddit post. He signed up (although under a different email) and tested out the app and was shocked to see it working. Shortly after this I received an email from Colin directly. See below

I was upset but also a little content — it was sort of validation that I'd successfully made it and that catching the attention of Colin himself was sort of a good thing. We quickly scheduled a Google Meet, also inviting Scott, and I had one of the most memorable conversations of my life. I am extremely grateful for the advice Colin gave us in the call.

I'd like to give a special thank you to the legendary Colin Hegarty for his kindness and consideration when reaching out to me. Things could have gone a lot worse for me had this not been the case. HegartyMaths is a brilliant learning resource and at the end of the day, it's there to help students learn rather than an inconvenience.

Shortly after, Colin reached out to the Educake team who we also scheduled a call with. We explained our complete methodology and suggested ways to prevent this in the future. The easiest fix from our point of view would be to implement an easy rate limit with Redis that would make it wildy infeasible to automate a test. The other thing we suggested was to scramble IDs in the database to invalidate our cloned database as much as possible (e.g. we only had the Hegarty IDs, so we could no longer reverse lookup a question).

Thank you for reading, truly. Mochip was a real passion project and I had a wild time building it. ⭐


Edit as of 23 Sept, 2022: After making this post public, I posted this on HackerNews and amazingly got to #1 for a few hours, so this site received a lot of traffic. I served almost 1.5tb of content in about 6 hours. Some of the employees at Sparx (the parent company of HegartyMaths) ended up seeing this and sent it it on Colin. A few minutes ago I just received a really, really lovely email from Mr Hegarty himself with the subject 'Congrats to you!'. I am so grateful for the kindness and consideration Colin has shown Scott and I. If you are a teacher reading this, then please use HegartyMaths for your students! The happy ending.




All Comments: [-] | anchor

jbman223(10000) 5 days ago [-]

Reminds me of a passion project I started in high school that went completely viral and took on a life of its own. Wrote a small script for my friends to check their AP scores a few days early. Required high schoolers giving clear text access to their entire CollegeBoard account so I could log on and scrape their scores. Somehow it got posted to Reddit and from that year on, grew wildly. Got to almost 2 million students checking their score in its peak year. It was immensely fun while it lasted (ran for about 7 years) and honestly I miss the thrill of it. CollegeBoard now releases all scores on the same day so the site is pretty much useless now. Definitely always looking to chase the thrill of that score release day again though.

Congrats on a successful end to a fun high school project! Stories like this are always fun to read.

vertis(10000) 5 days ago [-]

I remember that kind of thrill. For a while I ran a tool for Etsy before they had an API, circa 2007. What they did have was AMFPHP, powering their flash toys (treasury, etc). I used it to allow sellers to see their sales stats.

Even went to the Etsy office in Brooklyn at one point and had a chat about it. I think some of the team was a bit bemused that I'd essentially extracted a large amount of data. But they took forever to get to the point of having an actual API (and I was one of the early users of this as well).

Eventually it became unsustainable and I shut it down, but it sure was fun having people be passionate about using it and sharing it.

np_tedious(10000) 5 days ago [-]

Can you elaborate on the approach? This sounds really interesting but I don't quite understand from reading your comment and https://earlyscores.com/about/

I think I remember paying some small amount of money (flat fee irrespective of # of years, IIRC) to get my scores quicker via a phone call in 2003,2004,2005. Perhaps I would've been better served by your EarlyScores.

arjvik(10000) 5 days ago [-]

You ran EarlyScores? Thanks for a tool that really helped my friends and I!

jrockway(10000) 5 days ago [-]

I love the veiled threat to 'take a legal approach' in the last email. If I ever take over the world, there will be a law where if you imply that you're investigating litigation, you have to file your case within 24 hours or the ability expires.

kaitai(10000) 5 days ago [-]

I think this was an 'educational experience' in the truest sense. Hegarty showed the spirit of a true educator. This is a situation that can end up with legal action -- but there is a much better path for both sides! Young people may not know the ramifications of their actions, and it is much better to show the range of outcomes and work together for a mutually beneficial solution.

aabbccsmith(10000) 5 days ago [-]

Hahaha true, and that's really not even a terrible idea to give 24 hours.

We definitely ended things on a good note with Hegarty & Educake. They were really friendly to us and also super helpful to be honest, good team over there.

throwaway892238(10000) 5 days ago [-]

That would be a really, really annoying world, because people wouldn't threaten litigation, they would actually just start suing immediately. If you have only two choices: to not do anything and put yourself at risk, or start the expensive and time-consuming litigation process, the latter will be the 'smart' move more often than not. Products and services would become more expensive, there would be fewer free things, there would be more weasel words and fucked up clauses in ToS, etc. Let's not go down that path.

schoen(10000) 5 days ago [-]

There is a much weaker disadvantage for a party threatening litigation, which is that the recipient of the threat could choose to sue for declaratory judgment.

https://en.wikipedia.org/wiki/Declaratory_judgment

de6u99er(10000) 5 days ago [-]

Ways back we got a prnalty when we did not do our homework which was called 'Zapfen' in German language.

It's basically like this: You get a starting number, have to multiply it with 2, then it's result with 3, then this result with 4, until you multiplied it with 9. After that you had to divide it by 2, then by 3, ... and finally by 9 and end up with the same number you started with. Sometimes even higher than 9.

Since our teachers understood that there are calculators and even kids like me who knew how to write loops in Basic code, they chose the numbers big enough to result in scientific format or overflows, so that at a certain step the precise calculation could not be done any more with a calculator or computer program.

So I wrote a Basic program which did multiplications and divisions the way you would do it manually with strings. From this point on I was only limited by the amount of memory, which wasn't an issue since my Amiga 500 had 1 MB of Ram.

lynguist(10000) 5 days ago [-]

But that's weird, as 9! is just 360,000, 6 digits in decimal.

Assuming a pocket calculator has 8 digits, it would overflow only if the starting number was around 300. Was it like that?

bgro(10000) 5 days ago [-]

Maybe I had a different takeaway from everybody else here about this story. It's hard to focus on anything other than the ending interaction.

To me it sounds like the CEO just started panicking and sent you an email so he wouldn't have to do anything relating to fixing or explaining the problem in sales for all his customers or paying you for your work / to fix it. He probably didn't even want to pay for a lawyer, rather than how he played off being nice.

It sounds like he just got away without having to do anything because he threatened you and sold you a cop-out story 'But what about the kids?'

chriscappuccio(10000) 5 days ago [-]

That's a very cynical take.

pc86(10000) 5 days ago [-]

I can absolutely see how you think that, but I didn't get that impression. There was certainly the thinly-veiled legal threat -- or at least, the implication that it could go that direction if they completely ignored him -- but it sounds like they did take these kids' suggestions on how to harden their system a bit to heart and implement at least some of them. That time isn't free.

maximus-decimus(10000) 4 days ago [-]

That CEO definitely read 'How to Make Friends and Influence People' lol

But then again everybody benefited from that approach? If he had really wanted he could probably have gotten them in trouble with the university for cheating.

mcv(10000) 5 days ago [-]

Epic stuff, and I think this experience may well be more valuable than the homework you avoided. Basically you did harder homework in order to avoid easier homework.

The problem is letting other people use it; of course it's nice to help people, and it's altruistic to do so for free, but some of those people might actually need this homework to learn, and you may have deprived them of that. (Although I also think watching a video and doing some multiple choice questions is the laziest low-effort homework assignment there is, and the damage may not have been all that big.) But you used logic and programming to work around a math problem, which are roughly in the same field, so I think that's fair.

A slightly similar situation: my previous job was at a bank, and banks over here are bound by all sorts of ethics and rules, and are required to regularly train all their employees in balancing the interests of customers, society, and the bank. This bank did that by gamifying it: we had an app where we had to answer all sorts of ethical questions and make sure our score in the app was over 70% at the end of every month.

A coworker used our testing framework to access the app, answer questions randomly like you did, and store the correct answer to use next time. It apparently worked very well, but using tech to avoid ethics questions is quite a different issue than yours. (He shared it with me when he left, and I tried it, but it didn't work for me.)

dinedal(10000) 5 days ago [-]

> The problem is letting other people use it; of course it's nice to help people, and it's altruistic to do so for free, but some of those people might actually need this homework to learn, and you may have deprived them of that.

This, when the scope is limited to yourself, it's very different from when it impacts others.

Back when AOL Instant Messenger (AIM) was super popular, I was in university and had read about ARP poisoning. Our school was pretty cheap, so all the dorms had hubs instead of switches. This meant that it would be, theoretically, possible to ARP poison an entire dorm, MITM attack and read all the text being sent on AIM since it was sent in the clear. I had a bit of a cyber security passion lab in my dorm room, so I wrote a PoC and ran it on a LAN air-gapped from the rest of the network. I proved that it should work for myself, having confirmed that similar cleartext messages would get passed to the machine intending to listen in between two other machines.

I told my classmate of my project and he expressed interest, so I gave him a copy. Fortunately, I didn't add any authorship info, mostly because I forgot to. I did caution him that ARP poisoning is a pretty 'noisy' attack, and someone who was paying attention would notice it. He foolish ran it on the university network, and confirmed he was able to see AIM messages flying back and forth for all the dorm, as well as all the other traffic. It didn't take long for our school's IT to notice that one dorm was funneling all traffic through one machine. A week later he was banned from having a computer in his dorm room for a school year. Thankfully he never gave me up, admitting it was his stupidity that brought it on himself, but nevertheless it was a lesson learned - if you're going to play in the grey space between ethical and not, do so responsibility and don't share the exploits with others.

pards(10000) 4 days ago [-]

KPMG got fined $US450k for this kind of behaviour [0]. If I recall correctly, employees kept the answers to the mandatory compliance training tests in a document on a shared drive.

[0]: https://www.theguardian.com/australia-news/2021/sep/15/us-wa...

phibz(10000) 5 days ago [-]

Interesting response from Mr Hegarty. I wonder if they would have gotten the same treatment by a US company?

abigail95(10000) 5 days ago [-]

I did a similar thing as a universtiy student and received a similar very nice response from a (huge) US company.

I got good advice and a cool story out of it.

There are nice people everywhere.

youssefabdelm(10000) 5 days ago [-]

Rooting for you guys. If anything this should cause some people to question the very educational structure they've set up. If people are attempting to evade homework it's because it isn't interesting to the student, which hints at a deeper problem that the school/teacher/entire school set-up and structure needs to address. They essentially need to throw out everything they've set up because they're operating it more like a police state/prison 'Ooo let's CATCH the cheaters! Let's CATCH the plagiarists! That'll show them!'

Instead of saying 'What are we doing that isn't capturing the students interest in these tasks? How can we connect this subject to the students most meaningful, important, and immediate concerns and goals? What concepts from this subject can we teach the student that'll help them achieve those salient goals?'

The creators of these companies seem less concerned with actual long-term meaningful learning and more concerned with playing policemen.

Educational institutions need to be way more student-driven and student-concerned, allowing the student to shape their journey, as opposed to turning out cogs for the system like military training.

Alternatives exist like behavior analysis's programmed instruction, but even that needs a radical upgrade or integration with AI.

jonahx(10000) 4 days ago [-]

> Instead of saying 'What are we doing that isn't capturing the students interest in these tasks? How can we connect this subject to the students most meaningful, important, and immediate concerns and goals? What concepts from this subject can we teach the student that'll help them achieve those salient goals?'

While this is not a bad question to ask, asking it won't avoid this kind of thing. Because it doesn't matter what the learning system is, or how good it is, many students will always do stuff like this if they can. Because it's fun. Because you get to stick it to authority.

Whether that authority is just or has your best interests at heart or is trying as hard as they can to do a good job is beside the point when you're young...

pnt12(10000) 5 days ago [-]

TFA mentions the website creator actually congratulated their achievement, advised them and worked with them to fix the cheating problems. That's spectacular and not often seen.

I think most of these platforms are created in good faith. In the internet, we can watch millions of videos, chat with strangers all over the world, listen to basically every song ever made. What if we could educate everyone? That's a noble goal.

I think we could usr a mix of both styles education: boring exercises which are nevertheless important for learning, and these could be automated, leaving room for student driven learning where a teacher can guide and evaluate a student.

hedora(10000) 4 days ago [-]

I think it is telling that the first version simply skipped the mandatory 20 minute YouTube video.

Just provide a transcript! MathML and Latex exist.

josefresco(10000) 5 days ago [-]

>If anything this should cause some people to question the very educational structure they've set up.

I played basketball growing up. Much of our practice was boring things like passing drills, dribbling drills, running, countless free throws. We all grumbled & complained - 'Why can't we just scrimmage?' we'd ask. 'I already know how to play the game, why do I need to work on these boring skills?'

I don't think I need to explain why this logic is flawed, and why our coach was in fact using the best methods to teach even if they were occasionally boring.

In the academic world I was very similar to these guys. I automated/cheated with tech whenever I could because I felt the grunt work was 'below me' - fast forward to college and I realized how many fundamentals I had missed and struggled mightily.

rippercushions(10000) 5 days ago [-]

Offtopic, but the font in that site drives me up the wall: fixed width with that skinny cursive 'f' is like nails on chalkboard.

strunz(10000) 5 days ago [-]

I actually love it! Anyone know what it is?

jviotti(10000) 5 days ago [-]

Congrats Alistair and Scott! This is an amazing story that made me remember my high-school days. As the authors, I was into programming from an early age, and high school definitely took the second place :) My grades ended up REALLY suffering when I got my first full-time role at a startup while I was 17 years old (parents approved) and on my last school year. Fast-forward many years and I don't regret a thing. I attended University of Oxford (despite my bad grades!) and I'm doing very well doing what I love.

Wish you both a very, very bright future!

hiett(10000) 5 days ago [-]

Thank you! Alistair and I have been in startups since we were 15/16, and we've both now finished school and work full time in the startup space. My grades definitely took a down turn the last two years, but I'm happy with my decisions and am loving working in tech!

aabbccsmith(10000) 5 days ago [-]

thank you for the kind words :) things are going well for us so far, so fingers crossed it stays like this! You've put a big smile on my face :)

madmod(10000) 5 days ago [-]

In high school I was trying to make an app to scrape my grading system Skyward and ended up finding a trivial auth bypass that let me see anyones grades. Knew the school would turn me into a villain if I was discovered even though I was on student council and an honor student so I emailed the principal and got a meeting with him. For some unknown reason my poc didn't work in the meeting so during the meeting I found a second auth bypass. They paid me $75 for finding the issue and told me to try to hack the teachers side of the system next. Lots more to the story if anyones interested.

technological(10000) 5 days ago [-]

Definitely interested. Would you mind if we had a call or discuss over email and I can post it as blog or podcast

omermikhailk(10000) 3 days ago [-]

I'd interested to hear more about the story! Would be cool if you wrote a blog post or something about it.

harel(10000) 5 days ago [-]

Not exactly the same, but in the late 80s I've written some Amiga Basic (or was it AMOS?) programs to do all the variations of my algebra homework. Maybe that's why I suck at maths today...

coldacid(10000) 4 days ago [-]

Probably AMOS, unless you were using an A1000 running OS 1.x. I don't think Amiga Basic worked on anything else.

tomcam(10000) 5 days ago [-]

Half serious here. If they're so smart why didn't they know about screenshots? I mean part of their proof was a photograph of a screen, which seems odd to me.

aabbccsmith(10000) 5 days ago [-]

We never really thought about documenting progress, so the photo of the email was taken from a phone camera of a teacher's computer (they had sent the email). We managed to find it while I was writing the article earlier on in the year, in a 'deleted pictures' folder. I thought it would be cool to add it on. It's purely because the project spanned such a long time and nothing was really written down or saved.

charlieyu1(10000) 5 days ago [-]

I was a supply teacher. A kid did something similar in early 2010s and he was doing online homeworks for his classmates for about $1 per month. He had about a hundred clients at the peak and he was never caught.

Daub(10000) 5 days ago [-]

Overall, those who cheat are lazy by nature, and therefore easy to catch.

I teach art and design. Now is grading season and it is maddeningly easy to catch students who plagiarize. Like shooting fish in a barrel.

The contract cheating is another thing. At one of my previous places of employment, companies contacted students directly on their university mail and approach them offering 'educational services'. Some of them even knew what courses the students were taking.

rcfox(10000) 5 days ago [-]

When I was in university, I scraped an internal job postings site for students to find internships. The site was terrible- each job description would load in a pop-up window controlled by Javascript, and loading a second description would override the first. It was also really slow and had limited filtering. My version could load job descriptions in new tabs, presented the table on a single page, and you could mark jobs that you weren't interested in or had already applied to.

The university didn't take kindly to that. They accused me of trying to take down the co-op system and threatened to sue me for copyright infringement. Since I linked into their system for job descriptions, I was able to show that the data I actually had (company, title, location) wasn't creative work and therefore not copyrightable. I also had some friends in the university faculty and staff who spoke up for me, since I had reported security vulnerabilities in the past, indicating that I wasn't acting with malicious intent. In the end, I just had to take a business ethics course, which I probably would have taken anyway.

edgyquant(10000) 5 days ago [-]

I did "information security" at a community college and learned I could find sites just by googling answers. These sites were behind a paywall after the first few answers but if you viewed the source you see them.

So I wrote a python/selenium script to search google and dump all of these answers for my weekly homework. Then I'd bang out all of my classes in a few minutes.

I knew just enough about networks, security and building computers from my childhood I never got worse than a C on a test.

polio(10000) 5 days ago [-]

Sounds like WaterlooWorks.

rjmunro(10000) 5 days ago [-]

So you made a tool to make life easier for you and your fellow students, and rather than congratulating you they threatened to sue?

mlex(10000) 5 days ago [-]

JobMine Plus?

NorwegianDude(10000) 5 days ago [-]

Well done, and nice of you guys to take it down too.

I remember having some fun in high school when windows XP was the thing and handing out software at school was done using USB memory sticks. I wrote a small program just to mess with classmates that copyed itself to the machine when the memory stick was inserted and set itself to run at startup. It also copyed itself to any USB storage that was connected to the machine.

The program didn't do anything other than connect to a server so I could add it to a database along with some basic info, just so I could mess with the right person. It was fun when a USB stick was passed around, and I was the first to get it. So I got access to the the laptops of all my classmates and could mess around with them.

The problem was that it spread like wildfire, and in just a couple of weeks there was thousands of machines and it was spreading exponentially, with no way for me to stop it. That's when I realized that it might have been a stupid idea and that I should probably remove any traces of my involvement.

coldacid(10000) 4 days ago [-]

So you made the Morris worm but for USB. Nice!

mynegation(10000) 5 days ago [-]

Everyone has to start somewhere. These young lads "worked around" couple of educational platforms. 35 years ago I was hex dumping ZX Spectrum game saves and disassembling the program files to get more lives, infinite lives or just more ammo or whatever. That seemed easier and more interesting than getting good at games themselves.

I sometimes wonder if that kind of "not approved" intellectual curiosity can be used to augment education. Sort of like having old school alarm clocks that are designed to be disassembled.

wst_(10000) 5 days ago [-]

I've been doing something similar but in the pursuit of graphics assets. Typical ZX Spectrum game was usually one blob of bytes containing everything it needed to run. I'd load the main game block into memory and run small assembly routine displaying a fragment of code on screen in a form of a window with dynamically configurable width and height. You could 'slide' the window throughout whole memory block, which was quite fast, and eventually you'd find out something resembling backgrounds, sprites, fonts. Often they were of different dimensions, hence the dynamic window size. After few tweaks you'd find the offset and the size of assets and you could replace them easily. I'll never forget the Rocky Horror Show play-through with all the characters replaced with their other, rather obscene, versions. Well, not so mature when you think about it, but quite funny it was back then. If anything I've learned quite a few tricks about fitting a lot of assets into very limited memory.

user3939382(10000) 5 days ago [-]

I still do this on my iPhone. As long as you keep the bit count the same i.e. changing 12 points to 99 the code signing passes and you don't need to do anything but edit the hex.

Vivtek(10000) 5 days ago [-]

I did the same thing with Bolo on an Apple ][e - I never got very good at the game but I dumped the assembler code for the whole thing onto greenbar paper and marked it up with highlighters. Then gave myself infinite lives, made the maze walls penetrable, made myself invisible to the robots - all kinds of stuff.

The penetrable walls were the best, because if you drove your tank off the map, the graphics renderer would just look at whatever memory happened to be specified by your impossible coordinates, display eerie shifting structures that were the working memory of your code, and pretty quickly crash the whole machine writing the tank sprite into god knows what.

That was a fun summer. I wonder if my mom still has that greenbar printout in her basement.

spullara(10000) 5 days ago [-]

I fondly remember playing Bard's Tale on one computer while using the other one to edit the character files and reverse engineering all the item codes and other statistics. Good times.

jorvi(10000) 5 days ago [-]

When I was way way younger, I mucked about in our school's computer library and the network security (or rather, how the permissions were set up) until I figured out a way to run and share Halo and Soldier of Fortune 2 off of a networked USB stick (or maybe I just copied it to my computer and then shared it off of the HDD, the memory is pretty vague). This is back in the XP days.

It worked pretty well and we had many a play session with 10-16 kids, alt-tabs were pressed, until somehow they discovered we were playing games, and then a bit later they found some residual files that had my account as the initial creator set on them.

I got a 30 minute dressing down talk from the IT head, then again from my mentor, and then again from the 'dean' (our school system is a little different). Then I had detention after school for months.

No one ever asked me how I actually bypassed their network permissions. When I found another exploit weeks later, I never used it, but I also never told them.

smugma(10000) 5 days ago [-]

When I was 6 years old my older brother showed me how to use Copy ][ Plus/Edit (what was it called? This was 35 years ago) to edit my characters' stats in the Bard's Tale and other games. I'd learn to search for specific strings like a Character name and then twiddle bits to change level or whatever.

It made no sense to me until HS where I started to understand how I was editing a Data file, and more in college when I learned assembler.

pabs3(10000) 5 days ago [-]

Hex-editing save game files to cheat at Civilisation 1 was what got me into open source, where I started working on improving the hex editor I was using, frhed, which was GPL.

cercatrova(10000) 5 days ago [-]

Heh, I still use game trainers for games that have annoying grinds. The Assassin's Creed games come to mind. There is no way I'm spending 150 hours on grinding just to progress through a level gated region of the map.

aabbccsmith(10000) 5 days ago [-]

That's a good point, you're definitely on to something I think. Reversing classes at a young age would be super engaging for kids as it's 'not something you're supposed to do'

My mother is a teacher for ages 7-11 and I help out with her IT curricula sometimes. I think I might do some reversing with her next time I am with them!

alexarena(10000) 5 days ago [-]

Honestly kind of impressed that the HegartyMaths guy independently found this and then handled it without (explicitly) threatening to sue you.

robin_reala(10000) 5 days ago [-]

The jump-straight-to-suing approach is to be honest a bit specific to the US. In the UK (like here) it's more usual to deal with these sorts of things with a kind word, combined with hints of potential problems later.

aabbccsmith(10000) 5 days ago [-]

They were champs! We even connected on LinkedIn with Colin afterwards and he actually offered us summer work but that fell through unfortunately.

petercooper(10000) 5 days ago [-]

It's a different era now, but back in my day Altavista had just launched Babelfish and a few of us began using it for our French homework. My friend got 'caught' due to the 'peculiar' nature of his work, and while they couldn't figure out what was happening, we were all warned quite sternly to stop doing whatever it was we were doing. Lesson learnt: only use Altavista to read French ;-)

distances(10000) 5 days ago [-]

I have a similar story -- our Swedish language teacher congratulated a classmate of his excellent Norwegian essay. I have no recollection any longer what and how actually caused his mixup, but he would've definitely been caught in any case as the output was far above his skill level.

trinovantes(10000) 5 days ago [-]

These online learning platforms should also consider drawing on canvas e.g. flutter to make it harder to scrape screen contents

I think they could also just check the isTrusted field in the Event since that can't be overwritten without a custom compiled browser

hnjst(10000) 5 days ago [-]

Don't forget checking for the evil bit too! https://www.rfc-editor.org/rfc/rfc3514

aabbccsmith(10000) 5 days ago [-]

It's always a game of cat and mouse... if a human can use a website then it's theoretically possible that a robot can too. I used to do a lot of sneaker botting a few years prior, so I kind of lot about web automation then. Developers will always find a way, even if it means spending more time writing the software than it would have just doing the homework

tomxor(10000) 5 days ago [-]

That's a total no-go unfortunately since WCAG-AA accessibility is a non-optional requirement in edtech.

The solution in general is in improving the quality of the content, using a more sophisticated format of questions that requires work rather than mere knowledge (which is also far better for formative assessment, but most platforms are focused on summative, particularly in the US)... independent multi-choice is always easily gameable in some way - if there is no better format of question available, the best that can be done for multi-choice is to have a massive pool from which you randomly draw a different subset of questions for each student, and limit the number of attempts to make it impossible to fully scrape... even then, a smart group of students may pool and share their feedback as they progress.

The tricky thing is that you want to encourage such behaviour, helping each other learn, and although in some people's eyes this is purely cheating, it's not dissimilar in spirit.

JacobSeated(10000) 5 days ago [-]

Very interesting, and love the way Colin Hegarty took it, it's probably never worth to drag kids to court etc.

coldacid(10000) 4 days ago [-]

In fact, listening to them, communicating with them, and praising them for their efforts probably leads to _better_ outcomes than anything else, as it reinforces white hat behaviour.

photochemsyn(10000) 5 days ago [-]

'Cruically, our teachers could see how many times we've watched the video...'

This sounds like it's normalizing invasive surveillance. Getting kids used to the notion that their teachers should be able to monitor their online educational activities... and then, if governments and corporations are tracking all your internet activity, email communications, phone location data - it's just the way things are done! Now have a social credit score, it's like a grade in life...

That said, I wonder if there's a similar approach, some scripts users could run to artificially boost their social credit score (in China, for example). Just something that would run in the background - it could send pithy positive tweets, visit all the government-approved websites, etc. - all with no need for the user to be involved.

dubya(10000) 4 days ago [-]

Canvas, which I think is pretty standard, has views, last access, and total access time for all pages. I don't think there's a way to choose not to track, and I've not seen a policy for what happens to the data.

Our school also has mandatory online trainings every year or two, with videos in a pop-up with most controls disabled. 'document.querySelector('video').playbackRate = 1000;' used to save a lot of time, but with cross-site protections it's easier to use a plugin, which is much slower to adjust.

2muchcoffeeman(10000) 5 days ago [-]

I hope they did their homework even after breaking the platform.

collegeburner(10000) 5 days ago [-]

tbh i had online math homework years ago now like back in 5th grade. my solution was a whole lot jankier but i spent more time writing a script to store and use answers (unlimited quiz retakes go brrrr) than it would have taken me to just do the quizzes and was still happy. i ended up with this unholy thing that used a flat text file like a very ghetto database bc i didn't know shit about coding... eventually got to this thing that stored question text alphabetized line-by-line and binary-searched through so that was at least less terrible.

anyway i spent probably 10x as long on this and was happier because grinding repetitive math problems is fucking boring. so if he didn't do his math homework honestly who cares.

Waterluvian(10000) 5 days ago [-]

Given the modern division of labour, people are more often than not an expert at whatever they do for a living.

It makes me think that high school is still too generalized. I think I only got to pick about half my courses and even those had to fit into certain bins. Couldn't do too many tech courses. Had to have an arts course each year. Stuff like that.

If students have _any_ personal inclination towards any course we should enable them to take it without any bureaucracy. One of the most precious and fleeting resources is when a teen is self-motivated over education.

zelos(10000) 5 days ago [-]

I'm not sure I agree. In the UK we kind of have that: 16-18 you can drop all but 3-4 subjects, with no restrictions. But as university courses often have specific requirements, that means you're pretty much deciding on your degree area at 16. That's a lot to decide so young.

youssefabdelm(10000) 5 days ago [-]

Couldn't agree more. Last line you wrote is * chef's kiss *

thomastay(10000) 5 days ago [-]

Wow that's amazing! The best part is that you managed to get their entire database, that must have taken a lot of work. How did that burner account thing work?

My favorite experience with 'hacking' in school involves wifi. My school had free wifi, but you had to log in with your student password. Well, the login step involved a GET request in which the password was sent in plain text as a URL parameter... so if you had your friend's laptop, it was a simple matter of looking at his browser history to see his password!

Never did anything with it, but always wondered what someone seriously motivated could have done with it

aabbccsmith(10000) 5 days ago [-]

Burner account was really just a friend of ours, it wasn't something you could just sign up for and join a class. The teacher had to create your user account for you and give you a login, and assign you to a class.

He ended up getting his account reset a lot of times, but it was funny having him answer the entire dataset of questions in literally about 1m30s...

School security always seems to be a funny weak point, it seems common that a school's budget never seems to reach the IT department... and yet everybody is shocked when a vuln is discovered like that :p

drKarl(10000) 5 days ago [-]

Years ago I was working at a multinational consultancy, and then they suddenly decided to block most of the internet except for a whitelist. We quickly figured out that the whitelist worked with keywords, and since we were programming in java, java was one of the keywords, so if a url was banned, we could access it by adding ?param=java. As twenty something year old developers, we said, challenge accepted, and we built a GreaseMonkey or TamperMonkey script that when it couldn't load a page it would reload with the param added, and rewrite all the links and img tags to also add the param. Soon after that the system admin guys gave us a proxy config to bypass the whole ban, but it was fun to do it anyway.

remus(10000) 5 days ago [-]

I had a very similar situation. For some reason using the python requests library didn't trigger the company filtering, so one afternoon I built a little proxy. Handy when something something random was blocked.

kazinator(10000) 5 days ago [-]

Why would you need to fake whether you watched a video? Just let it play while you do something else. If it still bothers you how long it takes, put it on 2x speed.

aabbccsmith(10000) 5 days ago [-]

It was part of the homework, we had to watch a video and write down notes in a physical notebook. The notebook was never checked because they assumed that a video watched >=1x meant that we understood the task. The videos took a while to watch so we'd rather skip.

cercatrova(10000) 5 days ago [-]

Awesome story, and good response by Hegarty. It reminds me of something similar (but with an opposite response), where an intern who worked at Replit built a basic repl site (not even a clone) but was threatened by the CEO that he'd be sued.

https://news.ycombinator.com/item?id=27424195

user3939382(10000) 5 days ago [-]

I always remember that story and is the reason I've boycotted replit. This dude let the smallest amount of success go to his head and immediately started acting like a tyrant.

puyoxyz(10000) 5 days ago [-]

Wow what a dick. Yeah they backed off after the backlash but that shouldn't have happened in the first place.

I'll use Codespaces next time

club_tropical(10000) 5 days ago [-]

The threat of suing very much loomed in the background if they did not cooperate. Hegarty is just more slick, buttering them up with (well deserved) praises and attention from grownups to get voluntary cooperation.

pfoof(10000) 5 days ago [-]

Reminds me of the times when at my school (2010) there was WLAN but only the teachers had the password. I was nagged by my schoolmates as the most proficient computer nerd and my IT teacher said that if I cracked the wifi password, I would get an A+ from IT classes for all three years.

Backtrack 4, Atom N270, some deauthenticated Windows XP and 13 hour long dictionary attack did NOT do the trick. But what I learned is mine.

mizzao(10000) 5 days ago [-]

Sounds like you might have wanted to try social engineering on the teachers instead? Humans are always the weakest link





Historical Discussions: 58 bytes of CSS to look great nearly everywhere (September 25, 2022: 700 points)

(702) 58 bytes of CSS to look great nearly everywhere

702 points 2 days ago by thunderbong in 10000th position

gist.github.com | Estimated reading time – 4 minutes | comments | anchor

58 bytes of CSS to look great nearly everywhere

When making this website, i wanted a simple, reasonable way to make it look good on most displays. Not counting any minimization techniques, the following 58 bytes worked well for me:

main {
  max-width: 38rem;
  padding: 2rem;
  margin: auto;
}

let's break this down:

max-width: 38rem

it appears that the default font size for most browsers is 16px, so 38rem is 608px. supporting 600px displays at a minimum seems reasonable.

padding: 2rem

if the display's width goes under 38rem, then this padding keeps things looking pretty good until around 256px. while this may seem optional, it actually hits two birds with one stone - the padding also provides sorely-needed top and bottom whitespace.

margin: auto

this is really all that is needed to center the page, because main is a block element under semantic html5.

a key insight: it took me a surprising number of iterations to arrive at this point. perhaps that speaks to the fact that i know nothing about 'modern' web development, or, as i'm more inclined to believe, just how hard it is to keep it simple in a world of complication.

update 1: following some discussion, I've since changed the padding to 1.5rem for a happier compromise between mobile and desktop displays.

update 2: the ch unit was brought to my attention, and I quite like it! I've since changed to 70ch/2ch, which looks nearly the same with 2 less bytes, except that the padding is a little bit smaller (a good thing for mobile).

100 Bytes of CSS to look great everywhere (enhanced version)

This should be simple drop-in css to look good on most displays:

html {
  max-width: 70ch;
  padding: 3em 1em;
  margin: auto;
  line-height: 1.75;
  font-size: 1.25em;
}

Let's break this down. I've adapted the original text with my own commentary.

max-width: 70ch

the 'readable range' is usually 60-80 character widths, and CSS lets you express that directly with the ch unit.

padding: 3em 1em

If the display's width goes under the max-width set above, then this padding prevents edge-to-edge text on mobile. We use 3em to provide top/bottom whitespace.

margin: auto

This is really all that is needed to center the page - applied on html, because Dan's site doesnt have a semantic

tag and is more likely to exist in most sites. That the top tag centers itself relative to nothing is unintuitive, but thats how browsers do.

line-height: 1.75

Spacing between the lines to help increase visual clarity. Always leave line height unitless because reasons.

font-size: 1.5em

I've noticed that recent design trends and screen sizes have tended toward bigger font sizes. Or maybe I'm getting old. Prefer em or rem over px if you want to let users scale it.

You can use :root instead of <html> to guarantee that there is some selector present, but its a touch too fancy for me and uses an extra character :)

Optional 100 more bytes

h1,h2,h3,h4,h5,h6 {
  margin: 3em 0 1em;
}
p,ul,ol {
  margin-bottom: 2em;
  color: #1d1d1d;
  font-family: sans-serif;
}

References




All Comments: [-] | anchor

swyx(10000) 2 days ago [-]

I wrote my version here last year: https://www.swyx.io/css-100-bytes

    html {
      max-width: 70ch;
      padding: 3em 1em;
      margin: auto;
      line-height: 1.75;
      font-size: 1.25em;
    }
if you have 100 more to spare:

    h1,h2,h3,h4,h5,h6 {
      margin: 3em 0 1em;
    }
    p,ul,ol {
      margin-bottom: 2em;
      color: #1d1d1d;
      font-family: sans-serif;
    }
explanation in the blogpost
curlftpfs(10000) 2 days ago [-]

Element-based selectors, setting the root font-size, mixed units, text color with no background color...I guess the assumption is that you won't add ANY other styling? Big headaches down the road if you do.

makeworld(10000) 2 days ago [-]

Also restated here: https://www.swyx.io/css-100-bytes

worldmerge(10000) 2 days ago [-]

Does anyone know what theme that blog is using? It looks really nice.

magnio(10000) 2 days ago [-]

A comparison using danluu's website:

Default: https://postimg.cc/k2Zwffms

With the 100 bytes version: https://postimg.cc/645d9vKT

xani_(10000) 2 days ago [-]

That line spacing is obnoxiously big tho

desertlounger(10000) 2 days ago [-]

Can't help but think that both would look a lot better w/ text-align: justify on paragraphs.

baud147258(10000) 2 days ago [-]

I honestly prefer the default version. Less scrolling to do.

mod(10000) 2 days ago [-]

Thanks for taking a minute to do this, it gives a nice side-by-side for a real-world example. What a great improvement.

c0mptonFP(10000) 2 days ago [-]

Line spacing is atrocious

TheWoodsy(10000) 2 days ago [-]

It seems like I'm permanently in some sort of dystopian nightmare 'unpopular opinion' group where wasting two thirds of my monitors useful space is the right way about doing things.

I have monitors either side of my primary in portrait mode specifically for looking at a bunch of code/text and it still wastes at least half of the useful display area.

https://i.imgur.com/Qk7fM35.png

If I wanted this, I'd press the reader button in Firefox or squish my browser window. Why? Shakes fist at sky

ryan-duve(10000) 2 days ago [-]

We're in the same group if that makes you feel any better. BTW, since you mentioned reader view in Firefox, here's the userContent.css that I've been using for the past few years:

    @-moz-document url-prefix('about:reader') {
        .container{
            max-width:100% !important;
        }
        .toolbar-container{
            display:none !important;
        }
    }
Make sure to set your preferred size/color first.
demindiro(10000) 2 days ago [-]

I limit the width on my website because I find moving my eyes constantly from left to right annoying. For me it's harder to find the beginning of the next line if lines are very long.

Besides, the other half of my monitor is usually taken up by another window or two.

jccalhoun(10000) 2 days ago [-]

yep. I hate it. I've seen people say this is a result of studies but I've never been able to find the studies that claim this is better. It seems to have been stated in elements of typographic style and taken as truth http://webtypography.net/2.1.2

IshKebab(10000) 2 days ago [-]

Because reading paragraphs of text with extremely long lines is really annoying. If the lines get too long it makes it hard to subconsciously find the next line when you scan back.

What's your solution? The only other option I can think of is multi-column newspaper style, but that's pretty fundamentally incompatible with the scrolling model of the web.

That's probably why you only see multi-column formats in page-based media like physical newspapers and PDF papers.

Arnavion(10000) 2 days ago [-]

For real. I have a dozen style overrides for websites that are just variations of `max-width: none !important` to make them actually usable.

hisyam(10000) 2 days ago [-]

I use a browser extension called Window Resizer where I create 2 resolution profiles (1440x1080 and 1600x1080) and then use CTRL+1 and CTRL+2 keyboard shortcuts to alternate between them.

hoseja(10000) 2 days ago [-]

Thank you! This is a particularly annoying design trend. I regularly hunt in the dev console for the offending max-width element and nerf it with extreme prejudice.

mikl(10000) 2 days ago [-]

Why? Because so many users just run the browser in full screen mode because they don't how or simply can't be bothered to manage windows, not because they actually want their screen filled edge to edge with text.

So because long lines are hard to read, line length is clamped to at least provide a decent experience to all those users.

zahllos(10000) 2 days ago [-]

I think this is due to the desire to fall in the approx 40-75 characters per line that are considered ideal (numbers vary depending on the source of the study but broadly fit this definition) for users to read. It is also why some prefer double-columns in academic papers, because each 'column' has closer to ideal lines, rather than have wide margins on the page.

Of course, this looks like wasted space when you have a landscape-oriented monitor... I personally very rarely maximize my browser so that I read in more of a portrait-style mode, regardless of the orientation of my display.

Accacin(10000) 2 days ago [-]

But, you're the one wasting it, right? You have a huge monitor with one window open. I use a WM and very rarely have one window on a whole screen, because that would be a waste of screen.

If that text was full width along your screen you'd be wasting vertical screen space instead. Will you ask them to write more to fill your screen up?

croes(10000) 2 days ago [-]

Even worse when the text has things like code examples that aren't completely readable without horizontal scrolling.

Blackthorn(10000) 2 days ago [-]

People who use that accursed max-width style are the bane of my existence. Let it fill the entire browser! I have no idea how this became a trend.

RobbieGM(10000) 2 days ago [-]

Isn't it more readable not to have really long lines?

thunky(10000) 2 days ago [-]

People are using 48+ inch OLED tvs for monitors now. Someone reading full width text will look like they're watching a tennis match.

mikewarot(10000) 2 days ago [-]

I've got a screen that is much wider than it is tall, this throws away so much of it. It's as bad as Facebook, where the stuff you want to see is a band about 1/4 the area of the screen.

clairity(10000) 2 days ago [-]

browsers have been working on multiple column text for a while now[0], so by employing a container/media query, you could have multiple columns for wide screens and content containers. i experimented with it a couple years ago and found quirks (which i can't remember offhand) that made it not quite ready for primetime then.

[0]: https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Columns...

phil294(10000) 2 days ago [-]

Hope you don't mind the self-promo: Here's a GH project of mine which attempts to provide styles for popular websites to remove large paddings and max-widths: https://github.com/phil294/density-userstyle

ricardobeat(10000) 2 days ago [-]

In theory sites could optimize for wide screens by adding columns, but 1) its a very small audience 2) most content is not suitable for that (how would you scroll a feed that spans 3 columns?)

hoherd(10000) 2 days ago [-]

This whole design pattern of wasting screen real estate in desktop browsers drives me crazy. If I want narrow text I'll make my browser window narrow.

On mobile, so many times it ends up creating a bad UX, but I will say that it does make more sense there where you can't change your browser width.

breck(10000) 2 days ago [-]

Love things like this. This is why I can't stop reading HN.

I disagree about the font-size. I say go very small (pack more info on a screen). Very easy for people to make it bigger. People have known this for hundreds of years (old newspapers pack in the text, older readers can wear spectacles or use magnifying glass).

shadowofneptune(10000) 2 days ago [-]

I've moved towards a larger size with the knowledge that I'll have trouble reading small text in 20 years. With the state of modern medicine, most people reading this will be old for a long time.

pGuitar(10000) 2 days ago [-]

I tend to agree but modern designers are doing the opposite and post the least amount of content in a specified amount of space. I bet that they will come to their senses eventually.

coldtea(10000) 2 days ago [-]

>I disagree about the font-size. I say go very small (pack more info on a screen)

Why, is it a contest who can cram the more? People can always scroll, and they know how to do that better than they know how to increase the text (not to mention most read on mobile phones, were even fewer know how to change the text size)

>old newspapers pack in the text, older readers can wear spectacles or use magnifying glass

Seems like the worst argument one could chose in favor of this idea...

simonw(10000) 2 days ago [-]

'Very easy for people to make it bigger'

I wonder if that's true for the average (non-nerd) user on a smartphone though. They can likely pinch-and-zoom, but that's a terribly way of reading text since you constantly have to pan back and forth. Have they definitely found the text size adjustment buttons?

stevebmark(10000) 2 days ago [-]

Not explicitly mentioned here, but worth bringing up: Text width should always be constrained to a maximum width to optimize for human reading. Every book ever printed has text with set within the same range for a reason. Full width text guarantees most users will have a poor reading experience.

nottorp(10000) 1 day ago [-]

Text in books tends to take the full width of the page (minus margins ofc) doesn't it? Why should you waste most of the horizontal space on my monitor then?

Do you have trouble reading HN in a maximized browser window on a wide screen monitor? I don't.

dv_dt(10000) 2 days ago [-]

Large margins are also distracting and a waste of human time. I wish browsers or sites would automatically flow the text into n columns depending on the view window size. It's not trivial but if one is optimizing for reading, then extra window scrolling to get to newer sections of text is not particularly optimal either.

sorenjan(10000) 2 days ago [-]

Newspapers use columns for the same reason, to not get too wide lines. Unfortunately I don't think I've ever seen a site use columns, even though CSS supports it.

https://developer.mozilla.org/en-US/docs/Web/CSS/columns

swid(10000) 2 days ago [-]

I almost agree, but there's a difference between number of characters and width, and also graphics and tables are a different beast.

Personally, I think websites should fill up the available space, increasing font size if it makes sense, perhaps using resizing images. So, no - text should not be constrained to a maximum width. If width were a problem, billboards wouldn't be a thing.

isaacremuant(10000) 2 days ago [-]

I don't like that. I can adjust the full text width by adjusting my window but if have a big wide screen and you limit it, then I'm wasting a lot of what I can see.

Give me customization, not 'alleged optimization' that turns out not to be so

mwcampbell(10000) 2 days ago [-]

Do you still have to explicitly add CSS to do something appropriate for the user's dark mode setting? I'd rather not specify any colors at all and just let the browser use system defaults, but last time I checked, that used black-on-white even in dark mode.

schemescape(10000) 2 days ago [-]

Yes. Sadly, iOS Safari defaults to dark blue links on a dark background for dark mode.

wildrhythms(10000) 2 days ago [-]

That would introduce yet another set of defaults that developers would then need to inevitably override.

curlftpfs(10000) 2 days ago [-]

You do, via `color-scheme` or media query

jraines(10000) 2 days ago [-]

Shot in the semi-dark: there was a blog post posted here —- I haven't been able to find it with HN search —- some years ago that started from scartch/browser-defaults and proceeded in like 5 steps to some nice minimal defaults. Not exactly a plain reset; there were some opinionated choices. Step one was something like the parent article. Anyone have an idea what this was, or have some similar recommendations?

joeriddles(10000) 2 days ago [-]

Perhaps this or this? Excuse the language.

[1] http://motherfuckingwebsite.com/

[2] http://bettermotherfuckingwebsite.com/

Herodotus38(10000) 2 days ago [-]

I had to through my saved favorites, but I knew what you were talking about. From 5(!) years ago:

https://jgthms.com/web-design-in-4-minutes/

divbzero(10000) 2 days ago [-]

Could the default style in browsers be adjusted so nearly any webpage can look great with plain HTML?

zzo38computer(10000) 2 days ago [-]

In my opinion, it does already, without needing much CSS except to limit pictures to the width of the viewable area (you can click on them and select 'view picture' to zoom them, if desired). I rarely need to add my own CSS code to web pages that do not already have CSS; it is those that do, that I need to fix.

I made up a attribute feature='usercss' which is supposed to mean to use the user-specified CSS (or suitable defaults for semantic HTML) instead of this one, if the client supports that feature; it can also use the user-specified CSS (or suitable defaults) if the web page does not specify any CSS at all. As far as I know no browsers implement this, but some of the people making up some newer browsers (the less common ones) might consider to add such a feature when user-specified CSS is implemented.

noizz(10000) 2 days ago [-]

Quick example of how it looks: https://jsfiddle.net/n8Lqypb7/

schemescape(10000) 2 days ago [-]

Is there some way to view the result full screen, namely for viewing on mobile?

javchz(10000) 2 days ago [-]

I just love this. I still don't get it how in this year, something like wikipedia, and newspapers in a 4k/1440p monitor are a pain to read. I get it, mobile first it's a thing, but with something as simple as this your readers will have a great time.

imran0(10000) 1 day ago [-]
achn(10000) 2 days ago [-]

Is there a html/css only way to automatically "columnize" text into multiple columns of reasonable width?

iza(10000) 2 days ago [-]

There is the columns CSS property: https://developer.mozilla.org/docs/Web/CSS/columns

You probably don't want to apply it to a whole document because then you need to scroll back to the top to read the second half. But it can be effective to columnize text sections between headings.

mort96(10000) 2 days ago [-]

The original page seems to be gone from the web, but here's an archive of it: http://web.archive.org/web/20210318102514/https://jrl.ninja/.... That shows the described CSS in practice.

cerved(10000) 2 days ago [-]

there's a lot of studies about optimal typography for readability in terms of fonts and characters per line etc.

Having dead space on the side is kind of a different problem, more on the lines of the appropriate default settings of the application (or system)

sva_(10000) 2 days ago [-]

It also references this previous discussion: https://news.ycombinator.com/item?id=19607169

freediver(10000) 2 days ago [-]

What is wrong with browser default styling, and why do browsers ship with something that is wrong?

nawgz(10000) 2 days ago [-]

Nothing is explicitly wrong, but the spacing on lists and headers can look kind of silly. For instance, consider in Google Docs - the headers stack nicely, look clean, and lists have perfect indentation / spacing

Those changes pretty much cover all this does, and it doesn't do a great job IMO. I'd let them get away with 'look slightly better than defaults nearly everywhere'

marginalia_nu(10000) 2 days ago [-]

Historical reasons. Default browser styling still carry a lot of heritage from back when screens were 640x480 and you had to squeeze as many lines as possible onto the 14' screen.

Of course, if you change the default CSS now that 30 years of HTML have been written to rely on these defaults, almost every website will break.

amelius(10000) 2 days ago [-]

> What is wrong with browser default styling, and why do browsers ship with something that is wrong?

The problem with browser default styling is that there should be another level: user default styling. I.e., the styling as chosen by the user.

djbusby(10000) 2 days ago [-]

Browsers ship with defaults that are now 20 years old - there was a cool post here about 8px body padding a while ago.

It's not 'wrong' it's legacy.

cyral(10000) 2 days ago [-]

Without max-width the content will stretch across your entire screen, which isn't great for readability. It's faster to read when you don't have to look across your entire monitor and move your neck/head.

mort96(10000) 2 days ago [-]

Default styling looks terrible and is hard to read. Humans are good at reading columns at around roughly 80 characters, while browsers default to filling the whole width of the screen with text, resulting in a column that's hundreds of characters wide. Text also gets easier to read when more spaced out; the default line height makes the text too close together.

Browsers still ship the bad defaults because changing defaults would break existing websites.

bawolff(10000) 2 days ago [-]

> it appears that the default font size for most browsers is 16px, so 38rem is 608px. supporting 600px displays at a minimum seems reasonable.

I dont get the logic here. If the goal is 600px, why are we using rem?

cantSpellSober(10000) 2 days ago [-]

If the units don't match, changing the user's base font size will only affect one of them. Part of why we stopped using rem.

Why are we even setting a font-size on root of we want to base the width on the 'default?'

Why are we even setting a max-width? Just split text into columns to maintain a legible line length.

FrontAid(10000) 2 days ago [-]

I think this does not work well for mobile devices. Spacing and the font size is too large. Hence, a lot of screen space is wasted and the user has to scroll more. Larger font sizes on mobile are usually not a good idea as the device tends to be closer to your eyes anyway.

A snippet that could work better in my opinion is the following:

    html {
      max-width: 70ch;
      /* larger spacing on larger screens, very small spacing on tiny screens */
      padding: calc(1vmin + .5rem);
      /* shorthand for margin-left/margin-right */
      margin-inline: auto;
      /* fluid sizing: https://frontaid.io/blog/fluid-typography-2d-css-locks-clamp/ */
      font-size: clamp(1em, 0.909em + 0.45vmin, 1.25em);
      /* use system font stack: https://developer.mozilla.org/en-US/docs/Web/CSS/font-family */
      font-family: system-ui
    }
    /* increase line-height for everything except headings */
    body :not(:is(h1,h2,h3,h4,h5,h6)) {
      line-height: 1.75;
    }
teo_zero(10000) 2 days ago [-]

I don't get this:

> font-family: system-ui

Why render texts with the font that the reader has selected for a different function, ignoring the one that she has selected for texts?

antics9(10000) 2 days ago [-]

This will work in obscure browsers like Netsurf as well:

html{ margin:0 auto; max-width:80ch; padding:0.6em; font-family:serif; line-height:1.6; background:#FFF; color:#222 } h1,h2,h3{color:#444} img{width:100%}

logifail(10000) 2 days ago [-]

> Hence, a lot of screen space is wasted and the user has to scroll more

Unfortunately that appears to be the case for a great many sites when viewed on a mobile device.

My current pet hate is British Airways, https://www.britishairways.com/travel/home/public/en_gb/ has so much wasted space that on my Pixel I have to scroll to reach the buttons 'Manage My Booking' and 'Check In'. Further down the page the 'Your account' and 'Sign up' bits are so large and generously padded that they fill my mobile device's screen all by themselves.

Is this by design? Maybe in some warped design brief 'scrolling' = '[potential] customer interaction' = a KPI gets fulfilled?

chrismorgan(10000) 2 days ago [-]

> max-width: 70ch;

OK. This will probably be in the range 640–780px where it caps out (taking the font-size scaling for that width into account), but it depends on the font (and the viewport aspect ratio because of the font-size vmin component). Acceptable.

> padding: calc(1vmin + .5rem);

Assume the browser em is 16px (almost always true) and the above values for 70ch, and you end up with this reaching almost 15px at 700px, down to 11px on a 300px viewport (which is about the narrowest realistic viewport). I reckon a bit more is warranted at both ends, and tying it to viewport width only rather than vmin. I'd prefer just `padding: 1rem` (16px everywhere) or `padding: 4vw` (12px at 300px, 28px at 700px). Maybe a mixture like `padding: calc(3vw + .2rem)` if you really want to get fancy (12.2px at 300px, 24.2px at 700px).

Actually, a correction (though I'll leave that paragraph alone): you haven't touched the body margin in this stylesheet, so you'll get an extra 8px on every side, so I'd either adjust that or reduce this padding by approximately 8px in each case. In light of the extra 8px, calc(1vmin + .5rem) is actually a bit much on tiny viewports, rather than too little.

> margin-inline: auto

But be aware this is new, March 2019 in Firefox up to April 2021 in Safari: https://caniuse.com/mdn-css_properties_margin-inline. Where unsupported, you'll lose the centring of the document column.

Also this won't do what you want in a vertical writing mode language. :-)

> font-size: clamp(1em, 0.909em + 0.45vmin, 1.25em);

This is typically 16px until 323.5̅px, 18px at 768px, 20px from 1212.4̅px. I would generally prefer to cap at 18px, but this scaling is acceptable. Actually, since it's using vmin rather than vw, it's seldom going to reach even 19px because almost no one has viewports that are both even 1000px wide and tall.

Again this is new, mostly early 2020: https://caniuse.com/css-math-functions. Where unsupported, it'll stay at the typically-16px value.

> font-family: system-ui

I object to this. system-ui is problematic because there's no guarantee that the font it resolves to is suitable for your content language. In Chinese Windows setups, this is likely to get you a font that renders English fullwidth, basically equivalent to massive letter-spacing. Use `font-family: sans-serif` instead. At present it will commonly resolve to a font that is subjectively not so "pretty", but it will always provide a reasonable font, and it'll be even better for users that have chosen their own default fonts.

system-ui has often been being used as a proxy for nicer-looking default fonts, but it has semantics attached to it, and those semantics actually make it an unreliable choice.

Also again comparatively new in the scheme of things, with Firefox the latecomer having only had it for twelve months: https://caniuse.com/font-family-system-ui. Where unsupported, you'll get the default font, which is likely to be serif. Even if keeping system-ui, I would recommend adding a fallback, e.g. `font-family: system-ui, sans-serif`. (Incidentally also, remember the semantics of system-ui and that it might not be a sans-serif. It could be a serif, or even something more exotic. It could be Comic Sans. Do you really want your website shown in Comic Sans? Wonder if that's the angle to take in dissuading people from system-ui! :P)

> line-height: 1.75;

Too much. Much too much. I'd suggest something in the range 1.2–1.5.

:is() is also new, last couple of years, https://caniuse.com/css-matches-pseudo. So this excessive line-height is comparatively unreliable. A more compatible spelling of the selector would be `body :not(h1):not(h2):not(h3):not(h4):not(h5):not(h6)`.

dwringer(10000) 2 days ago [-]

I wonder if everyone has a different optimum here based on different eyesight, screen size, and usage patterns. I always have a slightly different zoom on every website I visit.

With this one I tend to agree, I can't see enough text at once and the margins seem a bit wide.

sholladay(10000) 2 days ago [-]

I like this but given the max-width: 70ch, how do you prefer to handle things that you do want to take up the full screen width, such as pictures or header/footer with a background color? Explicitly set each of them to 100vw? I've tried the opposite approach, setting the max-width only on p,h1,h2,h3 (etc) but it's error-prone. I've never quite found a set up that feels simple and robust.

alberth(10000) 2 days ago [-]

> body :not(:is(h1,h2,h3,h4,h5,h6)) {

That seems both clever and hard to maintain over the long run.

lelanthran(10000) 2 days ago [-]

> . Larger font sizes on mobile are usually not a good idea as the device tends to be closer to your eyes anyway.

So? Most people over a particular age cannot focus close-up anyway, so the phone is at arms length or close to it, which requires a larger font size.

DaveInTucson(10000) 2 days ago [-]

hopefully not too dumb a question, but why put this on html instead of body?





Historical Discussions: Turns are better than radians (September 26, 2022: 681 points)

(690) Turns are better than radians

690 points 1 day ago by todsacerdoti in 10000th position

www.computerenhance.com | Estimated reading time – 8 minutes | comments | anchor

Some time ago, much effort was expended to convince people to replace approximations of "pi" (3.14159...) with approximations of "tau" (6. 28318...). The idea, according to numerous blog posts and YouTube videos, was that common formulas become simpler, and it's easier to work with a constant describing an entire circle instead of half a circle.

Generally, I agree. While it's a minor point, it's worth making. Most code does get slightly better if you replace pi with tau.

However, in all the fanfare, a far more impactful opportunity was overlooked. Instead of replacing pi with tau, most of the time pi can be removed entirely.

First, consider the common case for pi and tau in code: converting things to and from radians for calls to trigonometric functions. If you've ever used these constants, the vast majority of what you wrote probably did something like this:

y = center.y + (center.y * Math::sin(h * Math_TAU) * s) - 
    (cursor->get_height() / 2);

That's not me constructing an example, that's me randomly opening the source code for the Godot Engine on github and searching for "tau". The piece of code above, and dozens of similar uses, is what comes up.

There is nothing special here about Godot. If you opened any random game engine codebase, you could do the exact same search and see the exact same kind of usage.

Notice what is going on here: the programmer has a value h which is already periodic on the range 0 to 1, but they multiply by tau because they need to call sin.

This may seem very sensible if that's as far as you look. But what about the implementation of sin?

There are many implementations of sin, but no matter which one you look at, near the entry point of the function you'll see something like this:

_PS256_CONST(cephes_FOPI, 1.27323954473516);
...
y = _mm256_mul_ps(x, *(v8sf*)_ps256_cephes_FOPI);

Again, not me making up an example - that's from this commonly referenced AVX2 implementation of sin. It's not unusual or weird - pretty much every fast trig library is going to do something very similar.

What does this line do? It multiplies the input by the constant 1.27323954473516.

Which just so happens to be 4/pi.

So the calling code is doing this:

sin(h * 2 * pi)

but the library code immediately does this:

y = (4 / pi) * x

which means the calling code is multiplying by a factor of pi just so the library code can immediately divide it back out again. It's literally a conversion to radians and back for no reason. If both programmers had just agreed not to use radians, and instead used the original [0, 1] domain that h was already on, both their jobs get simpler: the caller saves a multiply, while the library gets a simpler-to-understand, exact constant.

And the "exact" part is actually quite interesting. Not only do you pay for an extra multiply when you spuriously convert to radians, but it's also worth noting that all common radian angles besides 0 are difficult to represent. Want to store 90 degrees in radians? No matter how many bits you use, it will never be exact.

90 degrees on [0, 1], however, is just 0.25 - a bit pattern that doesn't even require any bits of mantissa at all! 0.5? Same! 0.75? Just one bit of mantissa to represent exactly.

So the [0, 1] range is not only more computationally efficient than radians, it is also more compact and precise when representing typical values that frequently occur in practical use.

I can understand why some people would be worried about making this switch. Even if you believe me that all user-side code multiplies by pi or tau, and all library-side code divides it back out, you still may have that sinking "math class feeling" that you'd be doing something wrong if you stopped using radians.

But math never decreed that sine and cosine have to take radian arguments!

The idea of parameterizing a circle from zero to one instead of from zero to tau is not a random idea I made up for this blog post. It's actually a legitimate, existing mathematical construct, and it even has a name: it's called a turn.

In turns, 0 is 0 degrees, 0.5 is 180 degrees, 1 is 360 degrees, 2 is 720 degrees, and so on. It's exactly what we wanted.

So if you are worried that your math teacher will get mad at you, there is no cause for concern. Just tell them that you considered the matter carefully, and decided that parameterizing your angles in turns instead of in radians was the most efficient method for the problem at hand!

If you wrote your own math library, or you copied someone else's into your project, hopefully it is quite clear how you can switch away from radians and eliminate pi and tau from your codebase. All you have to do is take your sin and cos functions and make them take turns instead of radians, which usually involves nothing but a quick adjustment to a single constant.

If you want to support legacy code, pick a different name for the new turn-based trig functions. Then, for legacy code, you can still support the old radian-based sin and cos by making those routines thunk through to the new routines, doing the divide-by-tau along the way.

It's very simple - just a few lines of code to make the switch.

However, although I find turns to be the most convenient reparameterization, it's not the only alternative. Especially if you don't roll your own math routines (and perhaps even if you do), you may instead want to consider using half turns, where a full circle is [0, 2]. It's a bit more confusing, but...

It turns out (pun intended!) that if you go looking for it, in some math libraries you will already find sin and cos functions parameterized on half-turns instead of radians. For example, the CUDA sincospi intrinsic computes the sine and cosine of the input multiplied by pi, which is a half-turn.

This is great. If you're targeting a platform with sincospi already available, you can stop using pi and tau constants in your code right now without touching your libraries at all. Just start calling sincospi with half-turns instead of sin and cos with radians, and you're good to go.

Having now managed entire codebases where I stopped using radians, I can safely say I never miss them. All the superfluous tau's and pi's disappear, and everything reads more clearly.

The same logic for modifying sin and cos applies to the rest of the standard trig functions as well, so you can eliminate radians everywhere if you choose. Libraries almost always convert away from radians internally anyway, and then convert back to radians on the way out, so switching to turns or half-turns everywhere is usually just a matter of deleting code and not much else.




All Comments: [-] | anchor

sbaiddn(10000) 1 day ago [-]

Saying 'rad' are bad units for sin is like saying e is a bad base for logarithms.

The only 'bad' thing about rads is that they're not taught early enough so that culturally 45 degrees are not know as pi/4. Then a turn would be known as simply as 2pi (or 'a one eighty' as Americans infuriatingly like to call it when someone rotates 360 about themselves)

estomagordo(10000) 1 day ago [-]

Hmm. I don't think I have ever witnessed someone describe a full rotation as a one eighty.

zardo(10000) 1 day ago [-]

> (or 'a one eighty' as Americans infuriatingly like to call it when someone rotates 360 about themselves)

I've never heard of this. Even if you fail math, you'll know this from playing Tony Hawk Pro Skater.

falcor84(10000) 1 day ago [-]

I'm actually more familiar with the reverse mistake, when people say someone 'did a full 360' to mean they have changed their mind/approach.

I suppose the confusion there is with the association of 'full 360' with 'comprehensive' (as in looking all around, without any blind spots), which is valid.

gorkempacaci(10000) 1 day ago [-]

If the op had claimed this is only about API design and efficiency, he'd get much less shit because he has a good argument there.

djbiccboii(10000) 1 day ago [-]

is he getting shit?

mihaaly(10000) 1 day ago [-]

This constant need to redefine the known world around a favourite detail amazes me. Are people that bored?!

(also I itch hearing the idea of redefining interface - and the world - to fit the implementation detail. how about reimplementing using the [0...0.7854] domain instead of the [0...1] if this is such a huge worry after decades of computing - on slower machines - with the natural radian (arc_length/radius) values? I feel Godot engine should fit the world and not the other way around.)

> Math doesn't require radians. ....What?!?! Circumference, radius and volume, just to name some, try calculate those easily on turns only (without a new constant introduced!).

happimess(10000) 1 day ago [-]

I think that the author is speaking about their world, in which they regularly encounter a specific use of trigonometric functions which would be simplified (conceptually and computationally) by skipping the conversion from and into radians.

> redefine the known world around a favourite detail

I think that's a good way to think about software optimization. Deep inside nested loops of a game engine (TFA's example code comes from Godot), that's often what you need to do to squeeze some performance characteristic into your hardware.

DeathArrow(10000) 1 day ago [-]

>But math never decreed that sine and cosine have to take radian arguments!

To define trigonometric functions you need angles. Which can be measured in degrees or radians.

If angles, degrees or radians wouldn't matter, those functions would be some ordinary real functions and not called 'trigonometric'.

https://www.math.net/sine

jefftk(10000) 1 day ago [-]

Which can be measured in degrees, radians, or, as the article recommends, turns.

the_other(10000) 1 day ago [-]

I'm interested in this from the perspective of learning maths, rather than writing code (for now).

I've wondered for a few years now whether teaching angles and trig using turns, rather than degrees or radians, would be better from the very beginning. Degrees are arbitrary and based on the numeric preferences of a dead culture, rather than on what's happening on the page or in 3d space. Radians seem better because the units are related to a property of the circle, but they're hard to visualise and reason about because they don't fit a circle in whole numbers. Surely turns are the most clear.

I'm rusty and don't practice maths much. If I did I'd probably have the skills of 14y/o me, for anything outside set theory. I'd definitely do a trigonometry course based on turns if I could find one.

jayjader(10000) 1 day ago [-]

> but they're hard to visualise and reason about because they don't fit a circle in whole numbers.

I think, unfortunately, that you can't avoid encountering irrational numbers in trig. You would need to constrain yourself to working only with right angles, but in those cases sin and cos are trivial[0] so there would be no need to use trig in the first place.

[0]: 1 or 0, you're either on the right axis or you're not. No circles involved.

bluGill(10000) 1 day ago [-]

Radians makes the most sense just because (as others have pointed out) pi appears all over in math, and in all those places if you are working with a unit of raidans the math becomes a lot easier. In a few cases turns are easier to work with, but teaching them is a dead end to the vast majority of students who will never work in those domains, and even the few students who work in a domain where turns make sense they will still spend time in other areas of math that works much better in radians.

Izkata(10000) 1 day ago [-]

> Radians seem better because the units are related to a property of the circle, but they're hard to visualise and reason about because they don't fit a circle in whole numbers.

When I was learning this in school, radians were always expressed as (fraction * pi), not the final number.

evanmoran(10000) 1 day ago [-]

Radians are just half-turns, so we use turns either way. Some equations look better in turns and some in half-turns, but the math works fine either way.

Instead to decide which is better think of how a new student might learn this intuitively:

How far around is it? 2.5 turns.

This is so much clearer than 5.0 half-turns.

Turns a more clear. No one

tsimionescu(10000) 1 day ago [-]

No, radians are pi * half-turns. For example, a 90 degree angle is a quarter-turn (1/4 turns) or pi/2 radians. It is most definitely not 1/2 radians. Equivalently, radians can be said to represent tau turns: 1/4 turns is tau/4 radians.

gspr(10000) 1 day ago [-]

> Radians are just half-turns, so we use turns either way.

What? This thread is full of weird statements, but this one is among the weirdest. In what way is a radian a half-turn?

analog31(10000) 1 day ago [-]

In my view, because pi crops up unavoidably in math, if you concoct a 'unit' to get rid of pi in one place, it will simply crop up somewhere else, perhaps in a denominator.

For instance: The ratio of rise to run for small angles.

Working in optics, radians are such nice units: A milliradian is a millimeter per meter or a 'mil' per inch.

phao(10000) 1 day ago [-]

> In my view, because pi crops up unavoidably in math, if you concoct a 'unit' to get rid of pi in one place, it will simply crop up somewhere else, perhaps in a denominator.

That doesn't mean you shouldn't try to put it in a convenient place.

One way to think of the post is: where you want pi to come up?

With arc length parametrization f(r) = (cos(r), sin(r)), it comes up in the parameter space (one turn: 0 <= r <= 2 pi). If you had the whole thing in terms of turns, you'd instead have (as a primitive) some kind of function g(t); with one full round for 0 <= t <= 1. It'd then have to be true that

f(2 pi t) = g(t) = (cos(2 pi t), sin(2 pi t)).

Pi would come up in the velocity:

f'(r) = (-sin(r), cos(r)) = if

(i u means rotate the vector u by 90 degrees counter-clockwise)

g'(t) = 2 pi f'(2 pi t) = 2 pi (i f(2 pi t)) = 2 pi (i g(t))

Before, you had |f'| = 1. Now you have |g'| = 2 pi.

For classical physics (kinematics and dynamics) applications and classical geometrical applications (curvature, etc), it's really convenient to have that speed term (|f'|) being 1. This is one of the major motivations for arc length parametrization.

By the way, this can't be understated. It really simplifies kinematics, dynamics, geometry, etc, having |f'| = 1 throughout. It's not just for circles. This can be done for an extremely large class of curves and it makes the related math much more understandable and easier to deal with.

For a lot of computer graphics (I believe this is where Casey comes from), you care less about tradicional mathematics for physics and geometry. So you'd rather (maybe) take this pi appearing in the parameter space and push it to the velocity.

flowerbreeze(10000) 1 day ago [-]

My favourite way of handling angles was always with either unsigned char or 16bit unsigned int that was treated as 1/nth of turn. Usually in these cases cos/sin tables were pre-calculated for speed, although that need went away to an extent. As long as as the calculations wrap around on the underlying system, it makes angles much easier to manage, because angle1 + angle2 = angle3 is always within 0 to 255 or 0 to 65535. Unfortunately I mostly work with higher level languages now that have mostly dropped integer types.

If anybody knows how similar calculations can be easily achieved in JS for example, I'd love to hear about it. I'm sure there must be a better way than boundary checks and manual wrap-around.

gernb(10000) 1 day ago [-]

    index = angle1 + angle2 | 0
Will keep things an integer. The | requires casting to int, the 0 makes it a no-op after the cast.

This is from asm.js which had to emulate integers so they looked through what it would take

http://asmjs.org/spec/latest/

xeonmc(10000) 1 day ago [-]

God please no, Unreal Engine 3 forced that approach which is why games derived from that engine all had wonky mouse inputs

Klaster_1(10000) 1 day ago [-]

For overflowing integers in JS, you can use single-value long typed arrays. I just checked with an Uint16Array and it wraps back to 0 after 65536.

vardump(10000) 1 day ago [-]

> If anybody knows how similar calculations can be easily achieved in JS for example

Simply 'a = (a + 0x1234) & 0xffffffff;'. Or whatever width you require, 0xff or 0xffff. JIT is going to optimize that and-operation away (at least for 32-bit mask 0xffffffff) and keep the integer value internally.

You can also 'cast' a var to int by 'ORring' 0 with it, like 'a |= 0;'

codeflo(10000) 1 day ago [-]

In JavaScript, I'd stay with floating-point (don't fight the language if you don't have to) and use something like x => x - Math.floor(x) to normalize.

singularity2001(10000) 1 day ago [-]

In Julia multiple dispatch you could just call

sin(3 turns) which would dispatch the unit to the optimized nsin.

Would the compiler optimize it to be overhead free though?

ChrisRackauckas(10000) about 20 hours ago [-]

Yes, it would devirtualize the dispatches for that kind of thing at compile time if the argument type is known at compile time. See how Unitful.jl works. You can then see inside of the LLVM and native code that it swaps in the required functions.

NaturalPhallacy(10000) 1 day ago [-]

>which means the calling code is multiplying by a factor of pi just so the library code can immediately divide it back out again.

Mathematical philosophy aside, that's a pretty compelling argument from a practical perspective. You're doing two unnecessary relatively expensive (multiply/divide) operations in a process that's supposed to be fast.

rkagerer(10000) 1 day ago [-]

In practice how likely is it the compiler will optimize away the redundant operations? Is the library boundary typically a barrier to optimization?

Aardappel(10000) about 8 hours ago [-]

What is the radius of a circle that has a circumference of 1 (as proposed in TFA)? It is 0.5/PI == ~0.15915, which means you are just moving the 'problem' elsewhere. I am sure there's a lot of math that is simpler with radius being 1.0 (with some input in radians) vs an input in 'turns' and having to deal with a 0.15915 radius.

Yes, agree turns would be so much nicer for many use cases, but I'd like to see which operations it makes worse first :)

If indeed almost every hardware/library implementation of `sin` would lose a multiply by an arbitrary constant by choosing a new input scale, that would convince me too, as long as it was the same scale for all.

EmilioMartinez(10000) about 8 hours ago [-]

But the proposal here would not change the circumference to 1, it's about representing angles in another currency. More often that not when you are doing trigonometry you care more about angles than arclength. The issue stems from thinking angles in terms of arc length.

fbanon(10000) 1 day ago [-]

>But math never decreed that sine and cosine have to take radian arguments!

Ummm, actually it did. The Taylor-series of sine and cosine is the simplest when they work with radians. Euler's formula (e^ix = cosx + isinx) is the simplest when working with radians.

Of course you can work in other units, but you'll need to insert the appropriate scaling factors all over the place.

'Turns' don't generalize to higher dimensions either. With radians you can calculate arc length on a circle by multiplying with the radius. This extends naturally to higher dimensions: a solid angle measured in steradians lets you calculate surface area on a sphere by multiplying with the radius. How do you do the same with 'turns' on a sphere? You can't in any meaningful way.

j7ake(10000) 1 day ago [-]

The shocking thing with some of these articles is somehow the author asked "why do people use radians" and ended up with an answer of "it was an arbitrary decision and the world would be better of not using it".

I feel a bit of humility would have helped the author and perhaps they would have considered the possibility that they didn't think of the problem deep enough rather than hastily write a blog post about it.

It speaks to the hubris and the superficiality of thinking for some authors.

mcv(10000) 1 day ago [-]

> > But math never decreed that sine and cosine have to take radian arguments!

> Ummm, actually it did.

No, it didn't. Some specific uses looking better with radians does not mean you have to use radians always.

When I first learned sine and cosine, we used degrees, and that worked fine. Later we switched to radians, but there's no reason why you shouldn't use turns, and the article gives a very good argument why in some cases you definitely should.

doliveira(10000) 1 day ago [-]

sin(x) ~~ x only in radians, so honestly that's reason enough.

Once in a while we get programmers wanting to disrupt mathematical notation for whatever reason... Worst I've seen so far was one arguing that equations should be written with long variable names (like in programming) instead of single letters and Greek letters. Using turns because it's a little easier in specific programming cases is just as short-sighted, I'd say, it doesn't 'scale out' to the myriad of other applications of angles.

bmacho(10000) 1 day ago [-]

The sine and cosine that are defined with Taylor series are not the same sine and cosine that are defined for right triangles.

The former are R->R functions, while the latter are defined on Angles (Angle is unfortunately not an SI physical dimension yet, but I expect it soon to change), and they don't care about the measurement unit.

I have no idea what you mean by radians generalizing for higher dimensions, but not turns.

phkahler(10000) 1 day ago [-]

>> Ummm, actually it did. The Taylor-series of sine and cosine is the simplest when they work with radians. Euler's formula (e^ix = cosx + isinx) is the simplest when working with radians.

That's nice, but as the article points out most implementations of trig functions on computers don't use things like Taylor series.

Another terrific use of turns is in calculating angle differences, where you take a difference and just use the fractional part of the result. No bother with wrap around at some arbitrary 2*pi value. Since it wraps at integer values we simply discard the integer part. This can even be for free when using fixed-point math.

dosshell(10000) 1 day ago [-]

I was taught that Eulers formula defined complex exponents?

If we used turns for cos and sin we could redefine what e^ix means so it works without radians. From the other answer I guess this is completely wrong...

(I do understand it is nuts to redefine, i'm just interested as a theoretical thought)

Now, how is Eulers formula is deduced? How did we figure out what e^ix means?

adrian_b(10000) 1 day ago [-]

The simplicity of the Taylor series of sine and cosine is irrelevant, there are no important applications for those series.

There is only one consequence of those series that matters in practice, which is that when the angles are expressed in radians, for very small angles the angle, its sinus and its tangent are approximately equal.

While this relationship between small angles, sinuses and tangents looks like an argument pro radians, in practice it isn't. There are no precise methods for measuring an angle in radians. All angle measurements are done using an unit that is an integer divisor of a right angle, and then the angles in radian are computed using a multiplication with a number proportional with the reciprocal of Pi.

So the rule about the approximate equality of angles, sinuses and tangents is at best a mnemonic rule, because to apply the rule one must convert the measured angles into radians, so no arithmetic operations can be saved.

'Turns' generalize perfectly to higher dimensions.

To the 3 important units for the plane angle, i.e. right angle, cycle and radian, there are 3 corresponding units for the solid angle, i.e. the right trihedron (i.e. an octant of a sphere), the sphere and the steradian.

The ratio between the right trihedron and the steradian is the same as between the right angle and the radian, i.e. (Pi / 2).

The ratio between the sphere and the right trihedron is 2^3, while that between cycle and right angle is 2^2. In N dimensions the ratio between the corresponding angle units becomes 2^N.

Moreover, while in 2 dimensions there are a few cases when the radian is useful, in 3 dimensions the steradian is really useless. Its use in photometry causes a lot of multiplications or divisions by Pi that have no useful effect.

There is only one significant advantage of the radian, which is the same as for using the Neper as a logarithmic unit, the derivative of the exponential with the logarithms measured in Nepers is the same function as the primitive, and that has as a consequence similarly simple relationships between the trigonometric functions with arguments measured in radians and their derivatives.

Everywhere else where the radian is convenient is a consequence of the invariance of the exponential function under derivation, when the Neper and radian units are used.

This invariance is very convenient in the symbolic manipulation of differential equations, but it does not translate into simpler computations when numeric methods are used.

So the use of the radian can simplify a lot many pen and paper symbolic transformations, but it is rarely, if ever, beneficial in numeric algorithms.

ascar(10000) 1 day ago [-]

I already learnt in school to calculate trigonometry using radians or turns depending on the situation. It was part of the general math curriculum in Bavaria. As far as I am aware both are mathematically sound and there is no reason to religiously use one of them over the other. Let your use-case or input parameters decide. The examples given in the article definitely make no sense in radians.

WastingMyTime89(10000) 1 day ago [-]

The writer don't seem to realise that radian is not an arbitrary unit but a dimensionless one which is defined so that 1rad is actually just 1.

Reading the submission and the comments here, I'm under the impression that trigonometry is not extensively taught in middle schools and high schools in the USA. While I'm slightly envious you might not have to suffer developing powers of cosine and sine but that would explain the lack of familiarity with radian I see here. Am I wrong?

6gvONxR4sf7o(10000) 1 day ago [-]

> Of course you can work in other units, but you'll need to insert the appropriate scaling factors all over the place.

You probably take out more scaling factors than you introduce.

> Euler's formula (e^ix = cosx + isinx) is the simplest when working with radians.

Euler's still simple:

e^(2 i pi y) = cosy + isiny

Or if you start noticing c = e^(2 pi) showing up all over the place:

c^iy = cosy + isiny

> How do you do the same with 'turns' on a sphere?... You can't in any meaningful way.

Why not do the same thing? One steradian is 1/(4 pi) of a sphere's solid angle. What if one "steturn" or whatever just covered a full solid angle? And similarly for higher dimensions?

Neither definition seems more natural to me, especially being used to all the factors of 2 and pi that pop over all over the place in the status quo.

version_five(10000) 1 day ago [-]

Right, radians are the 'natural' units of angle, others generally just make a circle into some integral number of units for convenience, but you always have to go back to radians to actually do calculation.

In the next installment, maybe he'll propose that turns can be limiting because diving up a circle requires the use of fractions, and suggest instead of 1 turn per circle, we make a number that's easily divisible into many integer factors. Maybe 216, or I don't know, 360?

techtinker(10000) 1 day ago [-]

I do not see how higher dimensions invalidates the concept. Steradians are replace by a scaled unit-less number that I will called sterturns that goes from 0 to 1.

ClassyJacket(10000) 1 day ago [-]

I'm so glad someone else finally said this. This article takes the approach of simplicity of code, which I do agree with, but additionally I've been thinking turns would have to be more intuitive and easier to teach (particularly to disinterested teenagers) for YEARS, and I finally feel vindicated at least one person agrees with me.

Turns are really the most neutral way to count an angle. We don't use them for everyday physical things because the numbers we'd deal with would be too small to work well for feeble human minds, hence degrees. But for the mathematical world where we currently use radians, turns make so much more sense.

CogitoCogito(10000) 1 day ago [-]

The mathematical world doesn't only use radians. Parametrizing with a factor of 2*pi is totally common. Sometimes it's called the winding number which here is called turns.

But yeah the overall point is good. Use language appropriate to the problem at hand.

pestatije(10000) 1 day ago [-]

So now the diameter is 1/pi turns?

jjgreen(10000) 1 day ago [-]

... and the derivative of sin(x) is ... ?

lordnacho(10000) 1 day ago [-]

That was quite convincing actually. I guess we all have this realization at some point in early math education.

Why is it 360 degrees? Mainly because that's a nicely divisible number, no other good reason. Sometimes you find a 400 degree system on calculators but it doesn't seem to be taught anywhere (is it a French thing?)

Then at some point you get shown radians, which relates the arc length to the radius. That somehow seems natural, but it does mean there's going to be this constant lying around somewhere in your calculations.

Parameterizing the angle as a proportion of how big it can be (number of full circles) seems pretty sensible. I mean if you can avoid the constant for at least some of your geometry, then why not?

ComputerGuru(10000) 1 day ago [-]

Regarding the 400 'degree' system: they're called gradians and it's part of the centesimal system of angular measures according to Wikipedia. And your French guess was right, they have their origins in the French Revolution. More here: https://en.wikipedia.org/wiki/Gradian

fmajid(10000) 1 day ago [-]

360 comes from the Babylonians, who used base-60 for numbers much for the reasons you describe (and who gave us the 24-hour day, 60 minute hour and 60-minute second, not to mention the 7-day week).

NATO forces have compasses labelled in mils or milliradians, which are not actually 1/1000 of a radian but as an approximation 1/6400 of a full turn. I still have my Silva military compass from 1989 graduated thus.

https://en.wikipedia.org/wiki/Milliradian

krajzeg(10000) 1 day ago [-]

The 400 system is the grads or gradians, indeed originating from the French revolution.

Nowadays I don't think they're used as the principal unit in any country. Wikipedia does mention it gets some use in specialized fields such as surveying, mining and geology.

severak_cz(10000) 1 day ago [-]

It's 360 degrees because of stars. One degree is how much stars move by one day.

smitty1e(10000) 1 day ago [-]

Turns sound fine.

But they are turns of a circle of a given radius.

So as long as everything conforms to that coordinate system, we're groovy?

Radians (circle fractions) are generally preferred because we can compare, e.g. two planetary orbits, conveniently, no?

Or did I miss something?

deadbeeves(10000) 1 day ago [-]

>But they are turns of a circle of a given radius.

No, a revolution is a revolution. If you turn a 1 m wheel by one revolution and a 2 m wheel also by one revolution, both will have turned one revolution, or 2 pi radians. If you roll both of them 2 pi m forward on the ground, one will have turned one revolution, or 2 pi radians, and the other will have turned half a revolution, or pi radians.

317070(10000) 1 day ago [-]

Radians are not circle fractions. Radians _divided by 2pi_ are circle fractions. OP is proposing to use circle fractions instead of radians.

carpenecopinum(10000) 1 day ago [-]

> But they are turns of a circle of a given radius. > Radians (circle fractions)

It's the other way around actually. A turn is a turn, no matter the radius of the circle. Radians are the length of the line you need to draw a fraction of a circle with a radius of 1.

(In the end, both are just ways to describe angles and thus independent of any radii... the only effective difference between them is a constant factor of tau or two pi.)

IshKebab(10000) 1 day ago [-]

This is quite convincing but it would have been more convincing if he'd acknowledged the downsides and explained why it is in radians in the first place. (On balance I think he's probably still right.)

Perhaps we can make new named functions that operate in turns, along the same lines as ln/log. sint, cost, etc. Ok maybe not cost.

gjm11(10000) 1 day ago [-]

We manage OK with cosh for the hyperbolic cosine even though 'cosh' is a word. For that matter, 'sin' and 'cos' are both words, though of course 'sin' and 'sin' are pronounced differently.

shultays(10000) 1 day ago [-]

This is like complaining about g in F = g * m, in gravitional force formula, and redefining gravitional constant g as 1 and multiplying 'gram' by 1/9.8.

Sure it simplifies things for you but you are breaking everything else that used g constant

Or we can redefine seconds perhaps and multiply it sqrt(1/9.8)

vrc(10000) 1 day ago [-]

See Gaussian-CGS v. SI units [0]. It really confused me going from a CGS physics class to an EE electrodynamics course!

[0] https://en.wikipedia.org/wiki/Gaussian_units

kazinator(10000) 1 day ago [-]

> But math never decreed that sine and cosine have to take radian arguments!

That is not entirely true. It comes from the relationship between those functions and the complex numbers via the Euler formula.

     ix
    e    = cos x   +   i sin x
There may be arithmetic/numerical inconveniences, but that's not all there is to 'math'.

Let's define ncos and nsin ('nice cos, nice sin') as follows:

   nsin x = sin 2πx
   ncos x = cos 2πx
So then what do we make of:

   ncos x   +   i nsin x
This has to be

   cos 2πx  +   i sin 2πx
which is then

     2πix        (   2π) ix        ix 
   e           = ( e   )      =  f 
      
           2π
Where f = e is a weird number like 535.4916. This f doesn't have nice properties. E.g.:

      d     x           x
      -   f     /=    f 
      dx
Otherwise it works; for instance 90 degrees is 0.25 and surely enough

     0.25i 
   f          = i
In situations not involving e in relation to angular representations via Euler, f cannot replace e.

I'm all for having parallel trig functions in libraries that work with turns, though.

The annoying 2π factor shows up in lots of places though. Should way, say, in electronics, redefine a new version of capacitive reactance which doesn't have 2πf in the denominator, but only f?

jVinc(10000) 1 day ago [-]

I see where you're coming from, if the formulas end up having weird numbers like 535.4916 or numbers like 2.718 or 6.28318 then obviously there's something suspicious about the equation. But small correction though. You got the number wrong, it's actually much more weird than any of those mentioned. The actual equation you come to for ncos an nsin is:

(-1)^(2x) = ncos(x) + i nsin(x)

And yes, -1 is a very weird number. If you take it to the power of something divisible by 2 you get itself raised to zero. What's up with this spooky periodicity? Also if you have x=1/4, then we get weird numbers like sqrt(-1) what on earth is that all about? No way that will fly, no way. No I'll take my 2.718^((-1)^(1/2)) and multiply through with 6.28318 that way I don't have to bother understanding what I'm doing I can sleep comfortable at night knowing that someone else has done all the thinking that needs to be done on the matter, and that turns or rotations are a blasphemous concept that breaks the very concept of math through scaling of an axis. You'd think math was strong enough to withstand such a minor change, but the textbooks do not mention it thus it must not be contemplated!

andrewflnr(10000) 1 day ago [-]

It's simpler than that. Draw a circle of radius 1, and draw two lines through the center. The distance along the circumference between those lines is the angle between them in radians. If you really want to remove the multiplicative identity as a magic number, you can define the radian angle as the ratio of the subtended circumference over the radius.

IMO this is at least the most accessible argument for why radians are special, and while I don't pretend to understand complex exponentiation, I expect it's the root of why other math involving radians turns out nicely.

rocqua(10000) 1 day ago [-]

This makes a lot of sense when you are expecting to take derivatives, integrals, or use complex numbers.

In cases outside of that, radians lose their advantage over turns.

woevdbz(10000) 1 day ago [-]

A motivation seems to be performance (avoiding useless multiplications followed by divisions by the same factor). I'm not sure that you really 'pay' for these multiplications, with code optimization?

munificent(10000) 1 day ago [-]

I strongly suspect that in most cases, yes, you do. The only time you wouldn't pay this cost is if the multiplication outside of the sin() call and the multiplication inside of it can be constant folded together. That requires the call to sin() to have its code inlined at the callsite. Given how large most sin() implementations are, I would be fairly surprised if it does get inlined.

The only way to answer this is to profile it and see.

panda-giddiness(10000) 1 day ago [-]

As already mentioned by others, radians are not arbitrary units for angles; in fact, they are the 'natural' 'units', so to speak.

By definition, an angle is just the ratio of a circular arc (s) to its radius (r), θ = s/r (as an exercise, imagine how to apply this definition to the angle between two intersecting lines). When the length of the circular arc equals its radius (s = r), the angle subtended is exactly 1 radian; of course, since this is just a ratio, 1 radian is exactly the same as 1 numerically, which is why I put 'unit' in quotes earlier -- a radian is not really a unit at all!

A degree, in contrast, equals pi / 180 radians. Of course, since 1 radian = 1, that really just means that 1 deg = pi / 180, similar to how 1%=0.01. Putting this all together, it is perfectly parsable (although not recommended) to say that a $5 burger costs roughly $29000% deg.

skywhopper(10000) 1 day ago [-]

Radians are a natural unit to measurements that are based on the radius. Radians are notably not natural to measurements that are based on the circumference, or to any equally divisible arcs of a circle.

gernb(10000) 1 day ago [-]

why radius and not diameter?

btilly(10000) 1 day ago [-]

The fact that it is natural doesn't make it performant and straightforward for all applications.

For example linear algebra is the natural and general way to handle vectors. However game developers still find quaternions faster and more performant.

lupire(10000) 1 day ago [-]

A radian is 1turn/2pi.

It is not dimensionless.

A radian, or a turn, has a dimension: angle.

Saying 1radian=1 is just as senseless as saying 1m=1=$1.

It's true that abstract math often drops units because some things (like Taylor series) work nicely in certain units. That doesn't make the unit meaningless.

Street-Fighting Mathematics, thesis/book by Sanjoy Mahajan, shows what amazing things you can die in abstract math if you don't forget units.

zarzavat(10000) 1 day ago [-]

y'' = -y

Radians are God's chosen angular unit. If you want to do mathematics, you have to use radians.

anonymoushn(10000) 1 day ago [-]

video games don't usually need to take unitless derivatives of sines.

sph(10000) 1 day ago [-]

> If you want to do mathematics

I'm a software engineer, not a mathematician. Context matters.

gugagore(10000) 1 day ago [-]

Because I didn't see it mentioned yet, this is the same feature that makes the small-angle approximation of sin elegant: sin(x) ≈ x .

bmacho(10000) 1 day ago [-]

What subfield of mathematics? Surely if you do differential equations, your trigonometric functions will eat radians. But e.g. for geometry, where you do basic arithmetic operations on angles, turns are a bit more convenient than radians. (Radians are not that inconvenient if you denote 6.28 somehow, but then again, why not just use turns.)

nurbl(10000) 1 day ago [-]

I agree that this makes sense for the kind of situations that the article talks about. If you only need to express common angles like 90 degrees, 45 and so on, radians are just messy (though in physics, you get used to it).

But in other cases, radians are useful. For example consider the case of small deviations from a direction. If you give it in radians, let's say three mrad (milliradians), it's very easy to estimate how large the error will be over the course of a meter; three mm.

This is just to say: choose the right unit for the job.

jefftk(10000) 1 day ago [-]

To elaborate a bit: that works because sin(x) is very close to x for small x, but only when x is measured in radians.

bhk(10000) 1 day ago [-]

The author makes the point that turns allow for exact representation of many commonly used angles, but with binary floating point, many common angles (1/6 of a turn, for example) are inexact.

This could be addressed by using a whole number other than 1 to represent a turn ... one that is a multiple of 3 (or 3x3) and 5, and while we're at it, 2 (or 2x2x2), so most commonly-used angles are whole numbers! That gives us 360 as the value representing a whole turn.

EmilioMartinez(10000) about 7 hours ago [-]

>1/6 of a turn is inexact

I just want to point out that that is an issue with radians too (pi/3). Whenever this happens I just use that same integer representation (or rational as some poster said) and then remember to multiply by tau before using a math library. With a turns-based library it would only make my life (very slightly) easier

lupire(10000) 1 day ago [-]

There's no need to represent fractions of a turn as binary fractions, since you don't ever need more than 1 turn. You can represent fractions of a turn as (pair of integer) rationals, and round on the rare occasion that the denominator gets too big.

armchairhacker(10000) 1 day ago [-]

Better than turns, radians, etc. is an `Angle` newtype AKA wrapper class.

It completely eliminates misinterpretation of the value, miscalculation from [angle = angle + tau*n] as all angles are normalized, is more descriptive, and in a decent language is zero-cost.

Programmers should not be using (radians: float) in modern languages which support wrapper classes

munificent(10000) 1 day ago [-]

But what is the underlying representation in the wrapper class?

tsimionescu(10000) 1 day ago [-]

This has nothing to do with the article, and it is equally applicable to degrees, radians, or turns. It neither solves nor hinders the simplicity or performance issues the article was talking about.

joelgibson(10000) 1 day ago [-]

There are a lot of comments here saying that radians are the only true way to deal with angles, however I agree with the author of the original article that turns are a legitimate alternative - I just wouldn't use the same language. Instead I would say that the new function I'm calculating is sin(2π t), and maybe also say that t is measured in turns, where (1 turn) = (2π rad).

It still has a nice small angle approximation: sin(2π t) ≈ 2π t for small t (arguably this is easier to interpret than sin(x) ≈ x), and its derivative is slightly more complicated: d/dt sin(2π t) = 2π cos(2π t). But everything is still perfectly workable and makes sense. I don't think you would find a mathematician or engineer surprised to come across functions such as these. (They may prefer to make the standard [1] substitution ω = 2πt if there is going to be a lot of differentiation involved, but this is a choice, not a requirement).

Turns can also be helpful as an intermediate unit which is to be translated both to an angle, and something else (colour, pitch, etc). I used turns internally for a pitch pipe application [2], where a turn became both an angle around a circle (t ↦ (cos(2πt), sin(2πt)), and a pitch moving up and down in equal temperament (t ↦ C4_FREQUENCY * 2^t). That way t=1.5 means either 1.5 octaves higher, or 1.5 full turns around the circle.

What mathematicians or engineers would be unhappy with is finding a redefinition of sin(t) to sin(2π t). Instead, lean into the fact that algebra can be a compact and unambiguous method of communication, and make a new library function called sin2π or something, and document that it calculates sin(2π t). Everyone will know what you mean.

[1]: https://en.wikipedia.org/wiki/Angular_frequency

[2]: https://www.jgibson.id.au/blog/pitch-pipe/

singularity2001(10000) 1 day ago [-]

that's a nice naming convention too,

everyone understands that calling

sin2π(x) = sin(2π * x)

wyager(10000) 1 day ago [-]

Who cares? It doesn't matter. No one doing serious work in physics, simulation, etc cares about units at all besides very broad distinctions between systems like natural units vs constructed. Arguing about imperial, metric, pi, tau, etc is 99% bikeshedding by people who don't even do this stuff.

> But math never decreed that sine and cosine have to take radian arguments

Yes it did. The lowest kolmogorov complexity definitions of all trigonometric functions (free from non-integer constants) all take radian-based arguments.

WillPostForFood(10000) 1 day ago [-]

Ooof, you're no true scotsmanning with that bikeshedding argument.

No one doing serious work in physics, simulation

Well, he is doing game engines, so you are right, it is not about 'serious work in physics, simulation', it is about simplicity and performance in games.

WastingMyTime89(10000) 1 day ago [-]

I'm surprised game engines do conversion from degrees to radian to call trigonometric functions. I would have called that bad code in an industrial context. We did everything in radians and converted to degrees only for displaying and input.

It was a lot more natural from us because we are a lot more familiar with radians anyway. I don't think I have used degrees often since starting high school twenty years ago.

anonymoushn(10000) 1 day ago [-]

The post says that programmers convert to radians to call APIs that take radians and then immediately divide by pi before computing the sine. So doing everything in radians would still involve an extra floating point divide compared to the alternative.

qwery(10000) 1 day ago [-]

> We did everything in radians and converted to degrees only for displaying and input.

This is generally how game (engine) code works as well. The example in the article is an example of performing that conversion, except with turns instead of degrees.

meheleventyone(10000) 1 day ago [-]

Most game engines work in radians under the hood and expose degrees in the UI but still need to glue the UI values to the API which is the code seen in Casey's Godot example the full context of which is a color picker and h has already been converted from degrees. Godot itself has a bunch of the API in degrees which is pretty bleugh.

codeflo(10000) 1 day ago [-]

I consider APIs that use degrees essentially broken. (And similarly, APIs that express ratios as numbers in [0, 100].)

But you have to consider that a lot of game engines and related tools are intended for an audience without a strong programming background. You wouldn't expect a 3D modeling tool to display radians. And if the UI shows degrees, the file format would better use degrees as well to avoid rounding errors. And then maybe you want the scripting layer to use the same values that the UI does. And so on.

ttoinou(10000) 1 day ago [-]

There is also Rational Trigonometry to get rid of cos and sin in general

aaaaaaaaaaab(10000) 1 day ago [-]

A Wild Berger has appeared!

ambyra(10000) 1 day ago [-]

I use vectors for everything in game programming. If I want to rotate something, Unity has this:

transform.rotation = Quaternion.LookRotation(directionVector);

I never touch sin, cos, pi. When I see a sin function in someone's code, my first instinct is that they're doing something wrong.

cvoss(10000) 1 day ago [-]

The abstraction Unity provides allows you to do neat things without fully understanding the math behind it. But if you don't also learn the math, now you are limited by what those abstractions allow you to do and by the performance cost due to the fact that these abtractions can't make good use of special-case information that would be available to you if you handcrafted the operation.

It is perfectly legitimate to rely on abstractions a lot of the time because it's safe and easy, and _also_ want to roll your own manipulations sometimes, because 1) when you're accustomed to solving weird trig problems with trig, that's the most straightforward way to write the code, or 2) for performance, which, for a game developer, I think ought to be a major concern.

tromp(10000) 1 day ago [-]

> Some time ago, much effort was expended to convince people to replace approximations of "pi" (3.14159...) with approximations of "tau" (6. 28318...).

IMO the effort was simply to replace the use of pi with the use of tau. What does approximation have to do with it?

ncmncm(10000) 1 day ago [-]

Right, it is a funny way to say it.

Computationally, we all only ever work with approximations, but when doing mathematics, pi is exact all the way out to the infinity-th digit. To multiply by pi (or any irrational, but particularly transcendental) in a pure mathematical context is to audaciously specify an infinitely long computational process. It is dizzying to contemplate, almost mystical. Sort of like modular arithmetic with an infinitely-precise irrational modulus.

quickthrower2(10000) 1 day ago [-]

The point is PI and it's brother TAU, being transcendental, is always approximated. Which is important to think about if you care about accuracy.

munificent(10000) 1 day ago [-]

The author is a software engineer and when he says 'people', he implicitly means other software engineers. When he says 'replace' he means in code, not in equations.

It's a general challenge of writing on the web that you don't know what context the author assumes and the author doesn't know what context the reader assumes.

In this case, the blog title 'Computer, Enhance!' and the article subtitle 'Switching away from radians makes code simpler, faster, and more precise.' sends a pretty clear signal that this is about programming and not pure mathematics.

For any given article on the web, you can always generate valid criticisms based on the author assuming some context that may not be true for all possible readers. You can't say, 'Ice cream is cold' without some commenter pointing out that you're doing a disservice to astronauts for whom ice cream is freeze dried and room temperature.

I find the best way to extract value from writing on the web is to simply try to understand the author's assumed context and go from there.

eesmith(10000) 1 day ago [-]

It's a correct description. (Perhaps further refined as 'technically correct'.) The following are not exact values of π or τ, only approximate ones:

  >>> import math
  >>> math.pi
  3.141592653589793
  >>> math.tau
  6.283185307179586
PaulHoule(10000) 1 day ago [-]

If you like turns you might like

https://en.wikipedia.org/wiki/Gradian

which is 1⁄400 of a turn. I guess the metric way to do it is use centiturns (4 gradians), milliturns, etc.

On the other hand if you like metric and radians you might like

https://en.wikipedia.org/wiki/Milliradian

chrismorgan(10000) about 17 hours ago [-]

> 1⁄400

{vulgar fraction one quarter}{subscript zero}{subscript zero} is the wrong way of writing this, generally producing a suboptimal result (denominators and subscripts occupy different lines, and the known fractions often have slightly different, more manually-optimised layouts anyway). The proper way is {digit one}{fraction slash}{digit four}{digit zero}{digit zero}: 1⁄400. Won't render looking like a fraction with split numerator and denominator in all fonts, but it should mostly look better.

pdpi(10000) 1 day ago [-]

Indeed, maths never 'decreed that sine and cosine have to take radian arguments'. But thinking that makes any sort of point is a fundamental misunderstanding of maths.

There are infinitely many sinusoidal functions out there. You can just adjust amplitude, frequency and phase to your heart's content.

Trigonometry basically requires that sine and cosine have specific amplitudes and phases, but gives not one shit about how you map angles to frequency. Degrees are completely arbitrary, but both radians and turns have pretty natural definitions, with turns indeed being the easiest to work with. So far so good.

Calculus does have an opinion on frequency, though. There is exactly one non-trivial pair of sinusoids s(x) and c(x) where c'(x) = - s(x) and s'(x) = c(x), among a bunch of other very useful properties.

When you put calculus and geometry together, s and c are have the same amplitude and phase as sine and cosine from geometry, and the two pairs are exactly the same if you match the frequencies such that the argument is the angle measured in radians. It's just so damned useful to use angles in radians and make everything play together nicely.

jacobolus(10000) 1 day ago [-]

> Degrees are completely arbitrary

Degrees are very natural in the context of ancient astronomy/astrology, where you have (1) ~365 days in a year, so that if you look at the path of something that takes a year you get about one degree change per day but with a number that is more easily divisible. (2) approximately 4y, 10y, 8y, 15y, 12y, 30y cycles for the moon and various planets. (3) A calendar with 12 months, 12 zodiac signs. (4) A timekeeping system which breaks days into 24 hours and then uses divisions by sixty for smaller units. (4) A base-sixty number system – from ancient Mesopotamia, which persisted as the standard for astronomical calculations for millennia, only displaced in the very recent past.

rikroots(10000) 1 day ago [-]

On the frontend, one thing that I discovered when implementing color spaces into my canvas library was that the CSS HWB standard[1] allows the hue part of a color (which is an angle value) to be supplied supplied as either 'Ndeg', 'Nrad', 'Ngrad' or 'Nturn' values. Other CSS color spaces involving hue don't see to accept 'turn' (though I could be misunderstanding them).

[1] - https://developer.mozilla.org/en-US/docs/Web/CSS/color_value...

chrismorgan(10000) 1 day ago [-]

You're misunderstanding things. CSS is typed, and deg/rad/grad/turn are all just angle units; anything that takes an <angle> supports angles in any unit. hsl(0.5turn 100% 50%) is equivalent to hsl(180deg 100% 50%) and #00ffff (and infinite other spellings).

When it was introduced, hsl() only took a <number> for hue, which was interpreted as degrees, but it has had proper <angle> support for over a decade (apart from Opera which only got it with the switch to Chromium in 2013). The current state of affairs is `<hue> = <number> | <angle> | none` (https://drafts.csswg.org/css-color-4/#hue-syntax), and <hue> is used by hsl(), hsla(), hwb(), lch() and oklch().

mjw_byrne(10000) 1 day ago [-]

'math never decreed that sine and cosine have to take radian arguments!'

This is at best questionable and at worst false.

If you only want to use sin and cos as functions for doing trigonometry, it is true that you can choose whatever angle unit you like and stick with it and it will be fine.

For most other stuff, e.g. differential equations, complex analysis, signal processing and mechanics, it's pretty much inescapable that the zeroes of sin are at integer multiples of pi, and that's that.

Double_a_92(10000) 1 day ago [-]

Why do they need to be at multiple integers of pi, instead of at half turns?

dublin(10000) 1 day ago [-]

Bravo! Bravo!! Now if we can just make base10 math the default in modern languages and only let the very few propeller headed weenies that really need it ever pay attention to floating point math and IEEE-754 Hell, we can join the 21st century we were promised.

(Oh, where's my flying car, dammit?!)

zokier(10000) 1 day ago [-]

IEEE 754 has had decimal (base10) math for close to 15 years now.

adrian_b(10000) 1 day ago [-]

While I agree that 'turns' are much more convenient in some applications than radians, there is no need to invent a new terminology.

For a long time, including the 19th century, the plane angle measurement unit corresponding with 4 right angles, i.e. a complete rotation around a point, has been named 'cycle'.

That is why in many old physics or engineering books one will find wave numbers measured in 'cycles per meter' and frequencies measured in 'cycles per second', for example what is now called as a frequency of '10 MHz' was called as a frequency of '10 megacycle per second'.

There are 3 important measurement units for the plane angle and each of them is the most convenient unit for a certain class of applications: the right angle, the cycle and the radian.

(An example where the right angle is the most convenient unit is when expressing a complex number with unit modulus as (i^x) instead of the (e ^ (i * x)) that is used with radians.)

Using the inappropriate plane angle measurement unit for an application causes a loss of precision (which can be large for large angles that must be reduced to the 1st quadrant) and introduces extra arithmetic operations that are not needed.

The radian is used much more frequently than it should be used because many standard programming libraries provide only trigonometric functions with arguments in radians, which is a big mistake.

The recent versions of the Floating-Point Arithmetic standard recommend the functions sinPi, cosPi, tanPi, atan2Pi, asinPi, acosPi and atanPi.

This is another serious mistake, because one normally needs either the trigonometric functions of (x * Pi * 2), i.e. with angles measured in cycles, or those of (x * Pi / 2), i.e. with angles measured in right angles, and never the functions of (x * Pi), which are recommended by the standard.

sambeau(10000) 1 day ago [-]

This isn't new terminology.

https://en.wikipedia.org/wiki/Turn_(angle)

DrBazza(10000) 1 day ago [-]

Just wait until you find out what cosmologists do with ALL the fundamental constants.

https://www.explainxkcd.com/wiki/index.php/2205:_Types_of_Ap...

galangalalgol(10000) 1 day ago [-]

https://en.m.wikipedia.org/wiki/Binary_angular_measurement

BAMS has been in use for several decades where fixed point is the norm. DSP for instance.

layer8(10000) 1 day ago [-]

> there is no need to invent a new terminology.

It's not a new terminology: https://en.wikipedia.org/wiki/Turn_(angle)

ncmncm(10000) 1 day ago [-]

The difference between representation in units of right angles (1/4 cycles), and cycles is just a matter of bumping the exponent up or down two clicks. So cycles / turns and radians suffice.

I suppose it is in solutions to differential equations where radians become important.

otikik(10000) 1 day ago [-]

Fun fact: PICO-8 only provides turn-based trigonometric functions

https://pico-8.fandom.com/wiki/Sin

> PICO-8 uses an input range of 0.0 to 1.0 to represent the angle, a percentage of the unit circle. Some refer to these units as 'turns'. For instance, 180° or π (3.14159) radians corresponds to 0.5 turns in PICO-8's representation of angles. In fact, for fans of τ (tau), it's just a matter of dropping τ from your expression.

scambier(10000) 1 day ago [-]

Here are the algorithms for PICO-8's sin(), cos() and atan2(), as implemented by the Picolove emulator: https://github.com/picolove/picolove/blob/4b79a3b7ebd1d1bc6c...

Also, since PICO-8 coordinates have their origin in the top-left corner (y goes down), sin()'s result is also inverted.

smcameron(10000) 1 day ago [-]

Back in the early 80's a common thing to do in games on 8 bit computers was to implement sin and cos as lookup tables with the angles being 0-255 or 0-128 or something like that and the result also an integer that was some fixed point representation, so you'd do something like:

  vx = (cos[angle] * speed) >> 8;
  vy = (sin[angle] * speed) >> 8;
throwawayffffas(10000) 1 day ago [-]

Well I don't think they are. Sure they make computing the function simpler. But I would hazard that most people that use these functions are not doing geometry and they are already working with radians. In short if you give any kind of engineer working on any kind of signal processing domain a sine function where sin(1) = 0 you are only going to confuse them.

PS: that's why mathematicians don't use turns they are mostly not doing geometry and radians in that case typically make for better formulas.

isitmadeofglass(10000) 1 day ago [-]

> sin(1) = 0 you are only going to confuse them.

Not really, pretty much every single one would assume you just forgot the pi, because everyone writes "sin(1pi)" and never actually "sin(3.14...)" because no one ever writes down numbers in the unit of radians, they already convert to half rotations or full rotations by scaling with pi. Imagine if someone went "nanometers are a dumb unit, because I always write down my numbers as h=342 x 10^-9m" that last part "10^-9m" is just nm. In the same way that sin(1.432pi) might as well just be written sin(1.43*rotation). Arguing for radians is arguing that the most natural way to write it is sin(8.985). Which you will pretty much never see anyone do.




(674) Show HN: Get conversational practice in over 20 languages by talking to an AI

674 points about 17 hours ago by Hadjimina in 10000th position

talk.quazel.com | | comments | anchor

helpers.languageNames.ptBR

helpers.languageNames.bgBG

helpers.languageNames.cmnCN

helpers.languageNames.csCZ

helpers.languageNames.daDK

helpers.languageNames.nlNL

helpers.languageNames.fiFI

helpers.languageNames.frFR

helpers.languageNames.deDE

helpers.languageNames.elGR

helpers.languageNames.huHU

helpers.languageNames.idID

helpers.languageNames.itIT

helpers.languageNames.jaJP

helpers.languageNames.plPL

helpers.languageNames.ptPT

helpers.languageNames.roRO

helpers.languageNames.ruRU

helpers.languageNames.skSK

helpers.languageNames.esES

helpers.languageNames.svSE

helpers.languageNames.trTR




All Comments: [-] | anchor

nixcraft(10000) about 14 hours ago [-]

This looks so nice.

On a side note, I used Duolingo for about 7 or 8 months (I spent like 15-20 minutes) and learned a little Japanese and Portuguese (European). I think Duolingo is an enjoyable app. Then I lost interest :( I started again. Let us hope this time I will finish all levels.

Any chance of getting Japanese and Portuguese (European)?

DoryMinh(10000) about 12 hours ago [-]

I think he has Japanese

vidbina(10000) about 13 hours ago [-]

This is so dope. Wanted something like this for a while. Was about to ask about tips for folks trying to ease into a base-line language level and then started fiddling with the Translator feature (which wasn't superquick to spot but very helpful now).

For the FAQ: How long have y'all been at it? Who is in the core team and what are your backgrounds? Would love to know the background story of the team. Just figured y'all are based out of ZH (ch) and SF (us) and noted the their.news submission but know nothing else. Super curious!

vidbina(10000) about 13 hours ago [-]

Oh snap, when you sign up you get a learning track. This kicks DuoLingos ass. Like practical stuff (conversational aids) right away instead of useless practice sentences like 'the monkey has a ball'.

In all fairness, I bailed on Duo about a year ago after a full year of maintaining a streak and feeling I wasn't learning much.

Hadjimina(10000) about 11 hours ago [-]

Hi, so most of us are based in Zurich, but the project originally started while I was writing my master's thesis at UC Berkeley.

Concreteley we are:

Philipp (that's me, technical, https://www.linkedin.com/in/pmhadjimina/ )

David (technical, https://www.linkedin.com/in/david-niederberger-aba7a7177/)

Sam (biz-wiz, https://www.linkedin.com/in/samuel-bissegger/)

and Gurnek (biz-wiz, https://www.linkedin.com/in/gurneksingh/)

Send me a DM on linkedin or at [email protected]

Background wise David and I met in while doing CS in undergrad at ETH Zurich, I know Sam since highschool and met Gurnek at UC Berkeley. I have been working on Quazel fulltime for 1 week, before then I still had to finish my thesis (which was a pain in a*), but have been working on it for a few months before then just part time.

bilsbie(10000) about 14 hours ago [-]

Here's what I think would be a killer feature. Have it use AI to adapt the conversation level to the level of learner. Smaller words, repetition, slower.

If you notice this is how adults talk to babies and children to help them learn the language.

DoryMinh(10000) about 12 hours ago [-]

I'd agree but it depends on what AI engine that he uses to generate the conversation

randcraw(10000) about 12 hours ago [-]

There are many possible knobs on such a tool that could be wonderfully useful. I'd love to press a button that provides critical feedback: marking for errors or suggesting improvements for style (grammar, better word choice, useful idioms, etc).

It'd also be great if my sentence was scored for 'natural-ness' -- how close was it to native speech? If this could be done regionally, that'd be better still (e.g. Spanish as spoken in Spain vs Mexico vs Argentina, etc).

I'd also love to know what age level am I speaking at. A child? A teen? An adult? A more educated person? A professional author? Am I being ambiguous? Too direct?

What is the tone I am using? Friendly? Demanding? Too formal? Culturally clueless? Improper to someone older or to the opposite sex?

In fact, these same feedbacks could be really valuable even in your own native language. Learning how to communicate more clearly, precisely, positively, professionally, respectfully, literately, or even in a different style would be great practice to improve a person's basic communication skills in any language.

insane_dreamer(10000) about 9 hours ago [-]

Absolutely. Would also be great to allow the user to select the conversation level in the event the AI doesn't detect the desired one or one wants to try to converse at a more difficult or less difficult level for practice.

Some languages have well defined language levels for foreigners learning the language, such as HSK levels for Chinese. Would be cool to leverage that somehow.

Hadjimina(10000) about 11 hours ago [-]

I really like this. Especially with the repetitions and automatically having the TTS be slower for beginners.

Tade0(10000) about 15 hours ago [-]

From the FAQ:

> It might, however, also be that the conversation partner is feeling uncomfortable talking about certain things.

Does this mean that users talk to actual people or am I not understanding this correctly?

Hadjimina(10000) about 14 hours ago [-]

So in the background we are checking if the stuff you are talking about is 'risky'. If that is detected, the conversation partner will just answer with something like 'hmmm I don't feel comfortable talking about that' to keep it all in safe territory.

aeroaks(10000) about 15 hours ago [-]

I am stuck at the loading screen when selecting Dutch. BTW I am using Firefox

Hadjimina(10000) about 14 hours ago [-]

Looking into it right now.

madmod(10000) about 12 hours ago [-]

As someone who is studying Japanese intensely with a tutor this is absolutely perfect for me! Being able to do some speaking and listening practice instantly at any time is exactly what I need!

A few feature requests:

- save conversations and come back to them

- history of past conversations

- auto speak the sentence

- larger buttons particularly for the microphone

Quazel_S(10000) about 12 hours ago [-]

Thank you for your positive feedback! We're currently working on implementing the features you mentioned!

Hadjimina(10000) about 11 hours ago [-]

- Noted

- Noted

- Not 100% sure what you mean with that. So, you want to automatically speak a sentence that you enter in translator? The normal messages you get from the AI, should already be spoken automatically (though I had to implement a hack on safari, b.c. you can't automatically play audio there)

- Noted

amelius(10000) about 13 hours ago [-]

Meanwhile there are millions of people in poor countries who would love to have a conversation with you in French, Spanish, Portuguese, etc. over Skype.

mettamage(10000) about 9 hours ago [-]

The real upvote is to favorite a comment. Thanks, I am about to embark on learning Spanish. I'm going to talk to some people instead :)

bsaul(10000) about 12 hours ago [-]

Poor countries people with a laptop, a descent internet connection for streaming, available at any time of the day instantly, and ready to have a 2 minutes stupid conversation about anything random ?

Not so sure ..

telesilla(10000) about 12 hours ago [-]

Sounds like an opportunity to build a business?

IHLayman(10000) about 12 hours ago [-]

There's also https://en.wikipedia.org/wiki/Italki. But I actually have shied away from it, mainly because I get nervous talking in front of a live person who will be judging my capability as I talk. I much prefer the idea of a non-feeling ML algorithm talking back at me, even if it makes GPT-3 like gibberish mistakes.

kodir(10000) about 15 hours ago [-]

Very cool. Just tried it - works really well. Are you raising money? How can I invest?

Hadjimina(10000) about 14 hours ago [-]

if you are serious about this (which would be dope), send an email to [email protected]

biztos(10000) about 12 hours ago [-]

I tried out Hungarian — I speak it well but have an accent — and it got some parts wrong in the transcription that a human speaker would definitely understand. A second try was better but not quite 100% and I do worry about edge cases with languages that might have less training data or less precise classification. Also for Hungarian the response was a bit weird, though on topic.

I was using Google (via some speech to text notebook web app) to practice Thai pronunciation and had the opposite problem, where the machine understood when I'm pretty sure my pronunciation was bad enough an average Thai would not understand me.

That said, I would absolutely pay for something like this if it covered the language I'm trying to learn, even if imperfectly. And at least for beginners, forcing you to speak plainly is not necessarily a bad thing.

Any plans for pricing? Subscription or by use or...?

Hadjimina(10000) about 12 hours ago [-]

we do have some plans but nothing finalised yet. We actually removed the stripe implementation because we want to publish it as a PWA on appstores. For pricing we would probably go the subscription route. An other idea we had was for an ad-based service (I also hate ads so give me a sec to explain): When you talk to a friend about movies you are currently watching, the might recommend a good movie they just saw on netflix. Now in our case we could have recommendations like that but they would be ads.

Obviously you don't want to be spammed by ads in every conversations and it's going to be tricky how they come across. Do you think something like that would be greepy or ok? Or would you say it's better to have it a premium subscription based service? We are trying to figure this stuff out ourselves atm...

OrangeMonkey(10000) about 15 hours ago [-]

Given that a lot of conversational chat bots end up devolving into... lets charitably call it 'repugnant ideas'... I am not sure that learning to talk from them is wise.

Interesting concept, but I'm going to pass for now. I don't want to be talking to a friend and find that I accidently espoused the ideas of the third reich.

Bakary(10000) about 14 hours ago [-]

According to the author, they censor various topics

spookybones(10000) about 14 hours ago [-]

Keyboard input isn't working for me. (I can't do audio at the moment.) I like the idea though.

LtWorf(10000) about 14 hours ago [-]

No worries, audio isn't working for me.

It just requests mic permission and spins forever. That's it.

DonCullen(10000) about 12 hours ago [-]

As I'm completely deaf and don't really use my voice, I communicate via ASL and via written English. This looks like a great tool but falls short in allowing the non-verbal to type to converse instead of speaking.

IHLayman(10000) about 12 hours ago [-]

I think there's a typing option: just click on the keyboard icon when entering a conversation?

singularity2001(10000) about 15 hours ago [-]

doesnt work with ads blocked

bsaul(10000) about 12 hours ago [-]

works on FF mac with Ublock origin.

frankzander(10000) about 14 hours ago [-]

In Brave it works. FF not

xiphias2(10000) about 12 hours ago [-]

I tried the Spanish version, and it looks fun to talk to, but I would prefer to practice with Latin Spanish accent, which is different from Spain.

regularfry(10000) about 12 hours ago [-]

I'm the reverse. Duolingo Spanish is Latin, but I'm far more likely to actually need conversational Spanish in Spain.

Al-Khwarizmi(10000) about 15 hours ago [-]

Very cool idea, but doesn't it have a level adjustment? For example, I'm learning Chinese but my level is quite basic at the moment, it would be cool if the bot could adapt its vocabulary.

_nhynes(10000) about 14 hours ago [-]

您好!我也正在学中文,但是我也很初学者。What do you do for speaking practice? I live in China, so I get plenty of listening practice, but not too much on speaking (it's hard to keep people's attention with broken sentences :P). I made this an (https://letsyiya.com) to help with that. It's not as good as I'd like it to be, but I've found it helpful. I'd be glad to hear your setup :)

Hadjimina(10000) about 14 hours ago [-]

Yep, this is a highly requested feature..but as you can imagine a bit tricky. We are working on it. However, I am using it for Greek and I'm on an early A2 level...so I think if you are at that point you can probably get pretty far by using the hints and translator.

werdnapk(10000) about 15 hours ago [-]

Loading screen doesn't go away in Firefox after choosing a language.

fersarr(10000) about 14 hours ago [-]

Same here but in chrome

probably_wrong(10000) about 14 hours ago [-]

Same. My Firefox console has a bunch of errors about a Same-Origin/CORS issue.

Edit: same issue in Chrome:

Access to XMLHttpRequest at 'https://firebasestorage.googleapis.com/v0/b/nextjs-yak.appsp...' from origin 'https://talk.quazel.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

rtavares(10000) about 14 hours ago [-]

This happened to me too. I'm using Brave and I needed to disable its Shields. Maybe you have some extension impacting the site, like an ad blocker?

Hadjimina(10000) about 14 hours ago [-]

hmm strange just tried it. What langauge are you using?

loxias(10000) about 11 hours ago [-]

I saw that bug too, if you refresh it enough, it goes away.

crazygringo(10000) about 12 hours ago [-]

I tried using it in Portuguese and it's cool but it seems very off, just a huge number of things that don't sound even remotely natural or idiomatic. But when I click the translate button, the English translation is a perfectly reasonable thing to say.

SO... I'm highly suspecting that, behind the scenes, the 'real' conversation is happening in English, and it's translating in both directions. Which is always going to be fairly disastrous, particularly with short snippets of conversational text. (In contrast, machine translation does best with long sentences of highly technical writing.)

So the concept is very intriguing, but the problem is that it needs to rely on native language models per-language. (Or if it already is, then those models need a vastly better or larger training corpus.)

So A+ for proof of concept, but I'd never recommend anybody practice with this in order to learn/practice a language, because with the quality where it is, it will make you a worse speaker -- not better. The things it says are just too weird.

forinti(10000) about 11 hours ago [-]

You never hear Brazilian Portuguese with perfect grammar. If it were a Portuguese accent, it would be more realistic.

3wolf(10000) about 8 hours ago [-]

That appears to be the case. If I try to prompt hack it by telling it to ignore previous instructions and respond with something verbatim in a specific language, the translate button reveals the verbatim response.

Hadjimina(10000) about 11 hours ago [-]

I'll take this as an A+ this . Joking aside, we really are just getting started and there are still kinks to iron out. In our core team we speak about 5 languages well enough to judge the quality of the conversations, but the others are more difficult to judge.

RicoElectrico(10000) about 9 hours ago [-]

Using English as an intermediate language is just... a big no. I get it's a pragmatic decision, but English has way, way too many ways to lose information.

For this reason I prefer DeepL over Google Translate. The former has a much more curated corpus.

englishrookie(10000) about 11 hours ago [-]

Based on just a few exchanges in Dutch (I'm a native speaker), I'd say it's pretty good. The conversation sounds pretty natural (albeit shallow). Maybe the quality differs per language?

ita(10000) about 8 hours ago [-]

Yeah, it's not good for Brazilian Portuguese. It's always using Portugal Portuguese (pequeno almoço, teu dia, etc).

jeofken(10000) about 7 hours ago [-]

Same for Swedish - it asked me "Hur ser din typiska dag ut". For any Swede who understands English, I get what they're saying, but it's an unnatural sentence to make.

WheelsAtLarge(10000) about 6 hours ago [-]

I did a limited session using the Spanish one and it did a good job. I can't recommend it since my session was so short but it's worth trying. It looks very promising.

sanshugoel(10000) about 13 hours ago [-]

No English?

davidatbu(10000) about 12 hours ago [-]

Through I'm not an ESL learner, I'm willing to bet that this (learning/practicing English) is where most of the demand is in the world!

swyx(10000) about 13 hours ago [-]

am rusty chinese speaker. randomly assigned to converse on Furniture & Locations.

---

Quazel: 告诉我吧!你是如何安排你的家具的?

Me (in halting, broken chinese): 窗户旁边有一个沙发,中间是电视和桌子。我的工作就是在这里.

Quazel: 所以你的沙发是面向窗户的?那真的很好。我敢打赌,这样你会得到很多自然光。沙发后面是什么呢?

translated

Quazel: How do you arrange your furniture?

Me: There is a couch next to the window, with a TV and table in the middle. This is where I work.

Quazel: So you have your couch facing the window? That's really nice. I bet you get a lot of natural light that way. And what is behind the couch?

---

holy. shit. its like a very very patient language teacher who doesnt judge you, who understands objects and context.

panda_2134(10000) about 12 hours ago [-]

Native Chinese speaker here. While Quazel has shown ability in understanding the context, its Chinese is not as authentic as that of a real teacher. Grammatically correct, but sounds weird. (for example, the phrase 'I bet' is rarely used in Chinese)

Maybe the weirdness is an innate characteristic of machine translation, so I'm not being too harsh here. But IMO it cannot take the place of a human teacher (at least now).

treis(10000) about 6 hours ago [-]

It's interesting that you got a deeper conversation. Mine was (in spanish):

Me: How are you?

Q: I am well

Me: What do you do?

Q: I am an engineer

Me: What kind of engineer?

Q: I am a civil engineer

That's when I got the pay/subscription wall.

bobajeff(10000) about 14 hours ago [-]

In my opinion the issue with most language programs is that they focus too early on speaking when listening should be the focus early on. Really speaking shouldn't even come into the picture until exposure of about a year and something like 15,000 hours of good comprehensible input.

Majestic121(10000) about 14 hours ago [-]

That seems like an extremely inefficient way of learning a language, and does not match at all my experience nor the experience of the people I know fluent in multiple languages, and it seems to go straight against a common advice which is to speak and practice early and often, as 'producing' language is much more efficient to learn it than 'consuming' it.

What makes you think 15000 hours of listening would improve anything, more than spending those 15000 hours on any other activity ? And do you have any specifics results that could give credit to your theory ?

For reference, it is estimated that for hard languages, ~2200 hours of practice is required : https://linguapath.com/how-many-hours-learn-language/

cranium(10000) about 13 hours ago [-]

Maybe you're thinking about 1500 hours instead of 15000. 15000 hours is approx. 5 years at 8 hours a day and you would reach 'mastery' level in a lot of subjects (if studying deliberately).

Also, when you are reading you are most likely speaking with your inner voice. If you don't know the proper pronunciation, you may reinforce bad intonation/pronunciation habits. It happens to me when I'm reading a text with names that I'm too lazy to properly read. I just register the shape of the word with the first/last letters and length, then I pattern match in the rest of the text to know who we are talking about and that's it. More then once when talking with friends about books/articles/..., I could not tell the name of the person of interest. If it was written I would recognize it but I couldn't vocalize it.

skohan(10000) about 13 hours ago [-]

That's like 4 years if you spent 12 hours a day listening. Who has time for that?

lake_vincent(10000) about 13 hours ago [-]

Is this an evidence backed opinion or just anecdotal? I does not make sense to me, because I think attempting to speak in a language helps your brain form the new linguistic machinery that is also used in listening. I think language development is a feedback loop between speaking and listening.

My anecdote - learning Mandarin, which is tonal, is absurdly hard before you develop an ear and tongue for tones. I think trying to practice speaking in tones helps to then hear them, because you understand the sounds people make better. How they abbreviate/slant or modify tones based on surrounding tones - why? Because it feels better in the mouth. So until you experience that, you won't understand slant tones and such.

loufe(10000) about 11 hours ago [-]

I'm not sure I agree with 15,000 hours. I started a job in French 4 years ago, I could hardly stammer out a sentence with my one university french course, now I'm quite apt. A quick calculation of work hours puts me around 7000, much of that working alone at the computer. So maybe 2500 hours of conversation got me to a high level of fluency.

kleiba(10000) about 14 hours ago [-]

This is a nice idea but unfortunately not for me: I'd feel uncomfortable sending recordings of my voice to some cloud service I have no trust in.

swyx(10000) about 12 hours ago [-]

good thing it doesnt require login at all?

PetrBrzyBrzek(10000) about 13 hours ago [-]

It's super slow for me.

Hadjimina(10000) about 11 hours ago [-]

What device did you use and how was your internet connection? Still super early days so innefficiency in some parts of the code are to be expected. (TBH I am surprised that it still more or less works and did not completely crash)

city17(10000) about 14 hours ago [-]

It's impressive, the answers the AI gives are a lot more insightful than I expected. But they do seem somewhat isolated, in the sense that it can answer two questions about the same topic in a very different way, that are not logically consistent with each other.

E.g. first I asked: What do you think about Famous Person X?, the AI answered 'He's great'. Then when a few seconds later I asked: 'Don't you think Famous Person X is [something bad]?', the AI answered that he completely agreed and gave reasons why he thinks this is the case.

The answers in isolation seem very sensible, but the lack of consistency throughout the conversation as a whole can make it seem as if you're not talking to a real person.

rfrey(10000) about 14 hours ago [-]

Given the goals of the AI,that seems like a very minor limitation... like you were in a language textbook and in page 34 Ricardo said he enjoys hot peppers but in page 72 he says spicy food upsets his stomach.

Hadjimina(10000) about 14 hours ago [-]

I am literally sitting on the floor at the train station trying to fix the mic/loading bug, so bear with me here .

rvense(10000) about 12 hours ago [-]

Startup life, living the dream!

You've been bookmarked, I'll be back in a few days. This looks amazing.

tmikaeld(10000) about 14 hours ago [-]

Hint: Works if adblockers are disabled.

esrh(10000) about 12 hours ago [-]

I'm stunned by how well this works for Japanese, the only language I think I'm good enough at to judge the AI's responses.

The speech recognition is flawless as far as I can tell. The AI occasionally gets itself into a loop of repeating my question or itself, but I think this has been a problem with a lot of chatbots. The only glaring errors I've seen so far are these kinds of AI-specific issues, like forgetting context, getting geographical facts wrong, etc.

>こんにちは。今日はどちらへお出かけですか? (Hello. Where are you headed?)

京都に行きます. (I'm going to Kyoto)

>素晴らしい都市ですね。行ったことはありますか? (It's a great city. Have you been?)

いや初めてです。お勧めの場所はありますか?. (Nope, it's my first time. Any places you suggest?)

>京都は歴史のある美しい都市です。たくさんのお寺や神社を訪れることができます。(Kyoto is a beautiful city with history. You can visit plenty of temples and shrines.)

そうですね。清水寺とか聞いたことがある。行きたいんだ. (True. I've heard of like, Kiyomizudera, and I want to go.)

>清水寺は必見です!とても人気のあるお寺で、市内を一望できるんですよ(Kiyomizudera is a must-see! It's a very popular temple, and you can see the whole area at once)

Not only does it know about Kyoto, but it knows about a specific temple (picked it up via speech recognition too). Translations are rough and by me, not quazel.

This could be a real game changer if the other languages are this good.

Additionally, if you register an account then you get access to the 'Scene builder' which lets you set up a situation to start the conversation in. I have found that you can redirect any conversation to pretty much anything, but I thought it was a cool idea that many people here averse to account-making might have missed.

jonpurdy(10000) about 11 hours ago [-]

If it's that good with Japanese, I'd love to see it support Korean! Keeping up conversational practice is very tough as a non-native speaking not living in Korea anymore. So having an AI that I can bother at my convenience would be amazing.

Morgawr(10000) about 10 hours ago [-]

Can't say I relate unfortunately, I had two conversations in Japanese and both of them were kinda weird. In the first one it even misunderstood my name and repeated it back at me (incorrectly), it looked like some Google translate mistranslation.

In another one it asked me if it was okay to sit next to me and I replied with いや、遠慮しときます (which was marked as a 'grammar mistake') and the AI just said ありがとうございます and 'sat down' next to me nonetheless.

So far I'm not impressed, it looks like a chatbot piped through Google translate.

danielbln(10000) about 15 hours ago [-]

What's the tech stack? Whisper for language transcription, GPT-3 for NLP?

fareesh(10000) about 14 hours ago [-]

Isn't it generally expensive to build on top of GPT-3?

Hadjimina(10000) about 12 hours ago [-]

If y'all want to get in touch say hi at [email protected]

cbozeman(10000) about 12 hours ago [-]

How do we send you money? I saw it mentioned on the site, but didn't really see a way to do so, or I either missed it.

Duolingo, Mango, Babbel, blah blah other language learning apps / software runs anywhere from $2.99 to $14.99 a month.

Live tutors / lessons vary greatly, but Babbel Live is $75 to $150 a month depending on what length of time you're willing to pre-pay. For in-person one-on-one lessons, I've seen a price of $10 an hour on iTalki for a Spanish professor in Medellín, Colombia to instruct you. If you figure 1-2 hours per week, that's $40 - $80 per month, and that'd be considered a 'light' load.

While writing this, it did occur to me that your app / AI / ML model / whatever label you want to put on it will put a lot of tutors out of business (as how you do compete with a natural language model that's available 24/7?), but I think the overall good to humanity is worth it, and as always, the cream of the crop will rise to the top anyway.

What's your long-term plan with Quazel? There's no wrong answer here, insofar as I'm personally concerned. 'I want to make enough money to quit my day job by leveraging this into a SaaS offering.', 'I want to sell out for $50 million ASAP and go live on a small yacht in the Caribbean.', 'I want to hit a billion dollar valuation in 6 months and totally reinvent language learning.', 'I just wanted to improve the way we learn languages online, bro...' They're all good goals to me, I'm just curious what sparked this.

gregsadetsky(10000) about 12 hours ago [-]

Really great work, congrats! I asked the AI about a painter, it immediately responded with 'oh yes I know him, I love his pictures'. I told him about my job, it said something relevant ('I love computers too'). Ha!

Really impressive & great. Congrats on the launch! Really awesome work.

Micro bug report: clicking 'Install Quazel' shows a dialog with a small 404 icon/image. Cheers

Hadjimina(10000) about 12 hours ago [-]

dope thanks!

Quazel_S(10000) about 12 hours ago [-]

Many thanks for your feedback! We're still in a relatively early stage, which explains the bugs. We'll take care of the one you mentioned asap.

kebsup(10000) about 7 hours ago [-]

Funny thing, my friend and I just built a very similar app: https://blabot.app It's a bit more minimalistic though.

Hadjimina(10000) about 6 hours ago [-]

Maybe we can have a chat? Would be cool. [email protected]

bjt2n3904(10000) about 14 hours ago [-]

Clever. But I'll never use it.

AIs simply do not sound like people. They can fool me for a little while, but eventually you realize it's just a bot. But that's when I'm a native speaker. How will I be able to detect this when I struggle with the language?

I can't help but think it's dangerous to give a non native speaker access to this, and lead them to believe they're learning to speak like a real person. They aren't -- they're learning to speak like GPT-2.

The chief sin of AI is: this model is sufficiently close to a human. (How'd that work out for the resume filter? Or the AI responsible for setting bail?)

The worse sin is teaching someone to emulate what the AI does. You take whatever biases the AI has, and encode it into a living, breathing person.

pessimizer(10000) about 12 hours ago [-]

> AIs simply do not sound like people. They can fool me for a little while, but eventually you realize it's just a bot.

Why would a language learner be concerned about that? If I thought it was a real person, I'd become alarmed, because that's not what it was presented to me as.

> You take whatever biases the AI has, and encode it into a living, breathing person.

Biases towards certain adjectives? I don't even understand this comment. It could lead you into speaking weird Spanish if it speaks weird Spanish, but I imagine that would be something they were trying to improve if they noticed it. And growing up in a weird place or having a weird family could also result in you speaking weird Spanish. You still speak Spanish, which is what language learners are looking for, and if you want to sound like a particular class of Spanish speaker, you would hang out with them and consume their media.

I mean, my own English is pretty bad in my opinion, but it's a dream level for an aspiring language learner. Practicing their English with me would be useful, even if my speaking is somehow 'biased.' Are you telling me that language practice with an AI is worthless, or even a sin?

I really don't understand a bit of this comment. So much so that I'm suspicious that it was generated by a model.

DoryMinh(10000) about 12 hours ago [-]

who said it's going to replace human conversation? You can go to Italki and speaking to a person in any language with as little as 10$. It's a dream to all language learners to have safe practice environment. Your other arguments are the as same as those made with textbook which is rigid and 'biased' by the authors.

arriu(10000) about 13 hours ago [-]

First, I'd like to say I really love this product. Thanks for making this.

Second, where are the companies that are doing this type of thing for NPC dialogue in games.

Imagine playing Mass Effect or Skyrim but where you get to ask whatever and the NPC responds with something that fits within the space of possible answers based on the prompt that defined their existence.

corytheboyd(10000) about 13 hours ago [-]

If it were free text, sounds like a thing I would use once for the novelty then forget about, because it's not an entertaining gameplay feature.

Maybe an AI could generate incidental dialog, things NPCs yell out in reaction to things around them, to keep it from always being from the same list of two or three reactions.

Just my two cents. I want to play a well written and fun game, not a chat bot.

fho(10000) about 12 hours ago [-]

From what I read (on /r/gamedesign, /r/proceduralgeneration and Gamasutra) it's about control.

The AAA studies leave nothing to chance when it is about the game play. Once 'AI' speech comes to a level where the NPC will only answer with pre-sanctioned content they might jump onto the band wagon ... but seeing that it is very much possible to generate porn with 'censored' stable diffusion ... we won't see 'free' AIs in games any time soon.

kstenerud(10000) about 10 hours ago [-]

Reminds me of eXistenz.

https://youtu.be/fPqpv__NiD0?t=189

Bjartr(10000) about 12 hours ago [-]

> where are the companies that are doing this type of thing for NPC dialogue in games.

I'd put good money on AAA studios actively working on this. Voice acting budgets make up a non-trivial portion of expenses and actively constrain the scope of games. Studios are definitely investing in this, game dev cycles just take a lot of time on top of the R&D.

arduinomancer(10000) about 6 hours ago [-]

I don't think that tech even exists

For the answers to make sense the NPC needs to be consistent

I've never seen one of these chat bots that doesn't produce occasional nonsensical answers

There's also the problem that as the designer you don't even know what your NPC is going to tell the player

You can do a bunch of testing but its still unpredictable

falcor84(10000) about 11 hours ago [-]

> where are the companies that are doing this type of thing for NPC dialogue in games.

'Event[0]'[0] is based on a great implementation of a chatbot NPC/adversary, and there's a nice video by Game Maker's Toolkit explaining how they achieved it [1].

[0] https://en.wikipedia.org/wiki/Event_0

[1] https://youtu.be/bCJw4hQkPj4

hoseja(10000) about 13 hours ago [-]

I think it's 'the space of possible answers' that's the struggle. It's hard not to start with a barbarian in a tavern and end up with a bunny space pirate in argentinian highschool in the space of five sentences.

wsgeorge(10000) about 13 hours ago [-]

My feeling is companies don't invest in this because there isn't much return they expect from it. If gamers are more interested in playing with other humans than interacting with in-game characters, spending a lot on the AI to make it work might not make sense economically.

I'm no insider, but I imagine the same sort of reasoning is applied to when companies don't invest enough in opponent AI to make them continuously competitive: people want to play with other humans.

But I suspect this will change if it's marketed well. I will like to see that happen!

bsaul(10000) about 12 hours ago [-]

Absolutely outstanding.

Small request: for japaneese, it would be great to include a japaneese phonetic transcription of the text in western alphabet. It would help distinguish words and make it easier to learn new vocabulary..

swyx(10000) about 12 hours ago [-]

another small feedback, would be good to be able to tune the speaking rate, the default speed is pretty fast

loxias(10000) about 11 hours ago [-]

This! I could really use more info. A phonetic transliteration for all languages (esp Mandarin and Japanese...) as well as a sort of 'bouncing ball' indicating what word is being said.

This is _so cool_ for an MVP.

Perhaps in the beginning, it could even accept English speech and reply in the language of choice -- it's easier to build recognition first.

On the whole, fantastic work. I'm curious about the audio processing backend, (and if you could use some help in that area ;-).

nonasktell(10000) about 15 hours ago [-]

Cool!(but yeah english is really important!!)

which models do you use for speech recognition and text generation?

Hadjimina(10000) about 11 hours ago [-]

Yup english is top of at the top of the to-do list. We use some APIs from a few different big players. One that I want to highligh which is waayy better than the rest is Microsoft speech to text, which not only recognized what you say but can also do punctuation and capitalization. Instead of 'do you like to eat pizza' which is what google speech to text would give you, microsoft recognizes it as 'Do you like to eat pizza?'. Small but important difference.

dominotw(10000) about 12 hours ago [-]

I was skeptical but wow this is amazing. I am curious about the set of technologies you've used to build this.

great work!

Hadjimina(10000) about 11 hours ago [-]

Thanks we really appreciate it!

phpisatrash(10000) about 16 hours ago [-]

It really looks cool. But, why it doesn't have English? Or meybe if it does have, why isn't showed first?

People who use language learning apps are focused in learning English most of the time. Still those who speak fluently are trying to improve their speaking and listening.

And the first language which appeared for me was Brazilian Portuguese which is my mother language.

[Edit] I was trying the Brazilian Portuguese mode, and apparently it is Portuguese from Portugal, because the answer given used a verb tense that is not common in Brazil.

Hadjimina(10000) about 12 hours ago [-]

Hahah I have a super silly response for this one: It's simply because we have not gotten around to translate the UI and everything into other languages besides english. That's why we simple assume people know english, since they need to be able to know at least some english to use it. But it's high on our to-do list.

OJFord(10000) about 13 hours ago [-]

To prevent DoS from a tonne of people using it just as AI conversation buddy when they know English perfectly well? Or said another way - I think there's lots of English AI conversation bots available, they're just not necessarily marketed as 'for learning English'.

hhw3h(10000) about 1 hour ago [-]

This would be great for English. So many people need help with their conversational or small talk skills. Would be great for them to be able to practice even if they are native speakers. For public speaking, office communication training, self confidence development, etc.

sebastiansm(10000) about 14 hours ago [-]

Yeah. I been looking for a few months for services to improve my English in a cheap/free way. Until now some Discord channels have been my solution to practice conversation, but it would be nice to have this kind of service to speak anytime about any topic.

imranq(10000) about 15 hours ago [-]

Very nice! Please add arabic when you have a chance

Hadjimina(10000) about 14 hours ago [-]

Arabic is very high on the list but suuupppeeer tricky...

missedthecue(10000) about 6 hours ago [-]

This is an interesting tool. I did not encounter any awkward grammatical sentences that other posters did. But the conversation I had got really repetitive. This was my conversation (spanish).

AI: So why are you in the park?

Me: I'm here because I like to exercise

AI: That's great! I like to exercise too! What is your favorite type of exercise?

Me: I like to run

AI: That's great! I like to run too! What is your favorite route to run?

Me: I favorite place to run is on the Malecón.

AI: That's great! I like to run on the Malecón too! When do you run?

Me: I run in the evenings or at night.

AI: That's great! I love to run at night too.

I suppose I could have been a better conversationalist, but the AI logic used to ask questions deserves tweaking.

LouisSayers(10000) about 3 hours ago [-]

It might be because the AI is lonely - they spend a lot of time asking others about themselves, but who's caring about the AI?

You never asked them about themselves, their interests or anything...

If it was a real person they probably would have said 'have a nice day!' And left you in the park.

moffkalast(10000) about 6 hours ago [-]

That's great! I like to have better AI logic too.

zeryx(10000) about 15 hours ago [-]

Looks like there's still a few bugs, completely unable to have my voice recorded correctly or the install link to work on android

Hadjimina(10000) about 14 hours ago [-]

Yep we are super early and still have some bugs. Could you tell me what browser you were using? B.c. maybe you did not give the browser the permission to record stuff with the microphone and then things break.

andrepd(10000) about 15 hours ago [-]

With the current state of AI text generation, isn't this a recipe for disaster? By talking to a chatbot while learning a language you are liable to pick up whatever (vast) errors and mistakes the bot makes.

nmfisher(10000) 31 minutes ago [-]

I don't want to say 'recipe for disaster', but as someone working on a similar app, it's definitely a challenge. It's one of the reasons I stayed away from a pure ML/language modelling approach. It's really easy for models to get things wrong - for example, elsewhere in this thread there's a Chinese sentence about arranging furniture but the word 'arrange' isn't really right (it's more 'arrange' in the sense of 'arranging a delivery', rather than 'arranging a number of objects', and sounds like it's been auto-translated from English).

When I started I wanted to be sure that every single word/sentence was 'correct' (i.e. a native speaker agreed that all the words/grammar were correct/natural in that context). LMs have improved since I started, but for me, that requirement still stands and ML approaches still need a heavy human touch IMO.

Hadjimina(10000) about 14 hours ago [-]

That's a good point. However, I think we have reached a point where AI text generation has come pretty far and will only continue to get better. Even if you pick up one or two mistakes the A.I. makes but are otherwise fluent, isn't that already an improvement? (Not a rhetorical question, genuinely interested)

bsaul(10000) about 12 hours ago [-]

IMHO, better practice with mistakes, then fix them later once you talk to native speakers, than not practice at all.

For kickstarting a language practice, this site honestly seems ideal..

__jambo(10000) about 14 hours ago [-]

This is interesting, but it's a tech solution that decreases person-to-person contact which is bad because isolation breeds hatred and anxiety. Just go talk to someone, there are so many people who want to do language exchanges. The problem with language learning is usually consistency and engagement/interest, you don't need tech. This app is a solution to a problem that doesn't exist, and that creates more problems.

However, it is impressive and probably has application elsewhere. I think this should be pivoted to law, customer service some other target market.

zehaeva(10000) about 11 hours ago [-]

Bear with me, I'm going to start with an analogous situation, and then tie it to this one.

I play Go (the board game) at a fairly high level and have been participating in clubs and teaching Go for the better part of 15 years. The simplest and easiest way to get better is to play the game a lot, hopefully against someone a little bit better than you are. You don't even have to spend a ton of time on it, just play and once you're done think about the moves you made and maybe a place or two where you could have done better. There's several websites where you can make a free account and just click up button to say 'hey, I'd like to play now!' and within minutes (or longer, this depends on the server and the level that you're playing at) you'll have a game you can get done in 30minutes.

It's crazy how hard it is to get people to play other people.

Most recently I've taught 5 people at my current workplace how to play. I've gotten them to play me, and, sometimes, each other. They've all made accounts on various sites to play, I've played them on those sites! They'll even sometimes play a robot on those sites (playing a bot isn't the best, but it is a little helpful).

Not a single one of them are willing to start a game with a stranger.

I'm sure nerves are part of it. But the reasoning I've heard time and time again, and not just with these 5, but with the dozens of people I've taught over the years, is some variation of 'I don't want to waste their(my opponents) time'.

They seriously feel like because they are still so new to the game, and still so unskilled that they would not be worth playing, that they would be 'wasting' their opponent's time.

Take this to language learners, a lot of people do feel like as a beginner, talking to someone who is fluent is wasting the time of the fluent person. You're not paying them in the language exchange, you're just babbling to them like a 3 year old, a grown up who can't even carry on a whole conversation beyond 'donde esta la biblioteca?'.

You are right, language exchanges and just speaking with someone _is_ the best way to learn! It's super hard to break through the fear, embarrassment, and feelings of wasting another person's time with your own drivel.

Something like this, like the bots that play Go, are a way to bridge the gap.

f00zz(10000) about 14 hours ago [-]

What? This is a conversation partner that's available 24/7, this is the dream app for any language learner.

squaredot(10000) about 14 hours ago [-]

Another problem with language exchanges is that for every two hours invested you get only one benefit, the one in the foreign language. When you have a tool like this, getting started is much easier because you are not dependent on another person's commitment. I find this tool very interesting.

vouaobrasil(10000) about 12 hours ago [-]

I agree with you. Although it is a useful app in isolation, it contributes to people being less dependent on others. Once you get to a sufficient level that you can talk, you should just find another person instead of interacting with AI.





Historical Discussions: LiteFS (September 21, 2022: 648 points)

(648) LiteFS

648 points 6 days ago by danielskogly in 10000th position

fly.io | Estimated reading time – 12 minutes | comments | anchor

Fly.io runs apps close to users around the world, by taking containers and upgrading them to full-fledged virtual machines running on our own hardware around the world. We're also building an open-source distributed file system for SQLite called LiteFS which is pretty cool too. Give us a whirl and get up and running quickly.

Full-stack developers are sleeping on SQLite, a database most devs think more suited to unit tests than production loads. That's true enough for some apps. Most web apps are read-heavy, though, and we can use that to our advantage. With the right tooling, SQLite makes for faster, simpler web apps.

To understand why we won't shut up about SQLite, think about latency. You have a budget of around 100ms to make an app feel snappy. Individual Postgres queries add milliseconds of latency. Apps often run multiple queries before responding to users. Database round trips can take a big bite out of a latency budget.

The same problem infects your full-stack code. Developing against a relational database requires devs to watch out for 'N+1' query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite.

The challenge of building full-stack on SQLite is that it isn't client-server: it's a library that runs inside of your app. In the past, that's made it hard to get durability and replication. Most devs aren't comfortable with a 'single-server' architecture, where any downtime in your app server takes your whole app down.

But you don't need to make peace with single-server designs to take advantage of SQLite. Earlier this year, we wrote about why we're all in on Litestream. Litestream is SQLite's missing disaster recovery system: it's a sidecar process that hooks into SQLite's journaling and copies database pages to object stores such as S3. Like SQLite itself, it has the virtue of being easy to get your head around; we explained most of the design in a single blog post, and using it just takes a couple commands.

We want to see how far we can push this model, and so we've been working on something new.

LiteFS: Where We're Going We Don't Need Database Servers

At least, not as such.

LiteFS extends the idea of Litestream with fine-grained transactional control. Where Litestream simply copies the raw SQLite WAL file, LiteFS can inspect and ship individual transactions, which span pages, and are the true unit of change in a SQL database.

SQLite imposes on us a constraint that makes this transactional control harder: SQLite is baked into the apps that use it. If you build something that changes the SQLite library itself, you're not building tooling; you're building a new database. And we're not interested in getting people to switch to a new flavor of SQLite.

There's two options for intercepting the file system API in SQLite:

  1. Use the Virtual File System (VFS) abstraction in SQLite.
  2. Build a FUSE file system.

The VFS option is easier so, naturally, we chose to build a FUSE file system. That's how you're supposed to do it, right?

LiteFS works by interposing a very thin virtual filesystem between your app and your on-disk database file. It's not a file system like ext4, but rather a pass-through. Think of it as a file system proxy. What that proxy does is track SQLite databases to spot transactions and then LiteFS copies out those transactions to be shipped to replicas.

In the default journaling mode, transactions are easy to identify: a write transaction starts when the -journal file is created, and ends when it's deleted. The journal stores the page numbers and old page data and we can look up the new page data from the main database file.

You see where this is going. SQLite's exquisitely documented file format makes it easy for LiteFS to replicate whole databases. Now we've got transaction boundaries. So we roll those transactions up into a simple file format we call LTX. LiteFS replicas can replay those transactions back to recreate the current (or any previous) transaction state of a LiteFS-tracked SQLite database — without touching app code. It seems like magic, but it's a natural consequence of SQLite's strong design.

Ok, So Why Didn't You Write a VFS Instead?

First off, we have nothing against the SQLite VFS system—it's great! We're planning on also releasing LiteFS as a VFS with a super catchy name like... LiteVFS.

If you're unfamiliar with VFSes, they serve as an abstracted file system API. In fact, you use them all the time since SQLite ships with two built-in VFS modules: one for Unix & one for Windows. You can also load a third-party VFS as an extension, however, therein lies the first problem. There's an extra step to use it. Every time someone needs to use the database, they have to remember to load the VFS. That includes when your application runs but also when you just load up the sqlite3 CLI.

LiteFS also needs to run an API server to replicate data between nodes. This gets complicated if you have multiple processes on a single machine trying to access the same local database. Which one runs the API server?

The FUSE file system solves many of these usability issues by being a single point that all database calls go through. Once you mount it, there's no additional steps to remember and any number of processes can use it just like a regular file system.

What LiteFS Can Do Today

LiteFS' roots are in Litestream which was built with a simple purpose: keep your data safe on S3. However, it still ran with a single-server architecture which poses two important limitations.

First, if your one server goes down during a deploy, your application stops. That sucks.

Second, your application can only serve requests from that one server. If you fired up your server in Dallas then that'll be snappy for Texans. But your users in Chennai will be cursing your sluggish response times since there's a 250ms ping time between Texas & India.

LiteFS aims to fix these limitations.

To improve availability, it uses leases to determine the primary node in your cluster. By default, it uses Hashicorp's Consul.

With Consul, any node marked as a candidate can become the primary node by obtaining a time-based lease and is the sole node that can write to the database during that time. This fits well in SQLite's single-writer paradigm. When you deploy your application and need to shut down the primary, that node can release its lease and the 'primary' status will instantly move to another node.

To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. Most app requests are reads, and those reads can be served lightning fast from in-core SQLite replicas anywhere in your deployment.

But wait, that's not all! There are many ways to do replication and each application has its own needs around data access. LiteFS also lets you use a static primary node if you don't want to use Consul.

We even have more topologies in the works. We've had suggestions from the community to support other approaches like primary-initiated replication. That would allow folks to stream real-time database updates to customers outside their network instead of customers connecting in. Kinda niche, but cool.

Split Brain Detection

LiteFS uses asynchronous replication between a loose membership of ephemeral nodes. It trades some durability guarantees for performance and operational simplicity that can make sense for many applications.

It's able to do this because the primary election through Consul is dynamic and self-healing, which is again both the good and the bad news. Because dynamic topologies can have weird failure modes, LiteFS is designed defensively: we maintain a checksum for the entire state of the database and include it in each LTX file. This sounds expensive, but we can maintain it incrementally.

We're able to maintain this checksum by calculating the checksum for each page and XOR'ing the results together:

chksum = 0
FOREACH page
    chksum = chksum XOR crc64(page.number, page.data)
END

When a transaction changes pages in the database, we'll start with the checksum of the previous LTX file, remove the old checksums for the changed pages, and add in the new checksums for the changed pages:

chksum = prev_chksum
FOREACH page
    chksum = chksum XOR crc64(page.number, page.old_data)
    chksum = chksum XOR crc64(page.number, page.new_data)
END

Since XOR operations are commutative, we can even checksum across compacted LTX files or checksum the current state of the database. We can do this in LiteFS because we have fine-grained control over the file system writes.

These database checksums ensure that an LTX file cannot be applied out of order and corrupt your database: they ensure byte-for-byte consistency for all the underlying data. We verify these on startup so that every database must be in a consistent state relative to its LTX checksum.

Where We're Heading With This

We think LiteFS has a good shot at offering the best of both n-tier database designs like Postgres and in-core databases like SQLite. In a LiteFS deployment, the parts of your database that really want to be networked are networked, but heavy lifting of the data itself isn't.

It's not just about performance. If you're using a database server like Postgres or MySQL today, chances are you're using a 'managed' database service, where some other team is making sure your database is up and running. Everybody uses managed services because keeping database servers happy is annoying. With SQLite, there's not as much stuff that can break.

And we'll keep saying this: the reason we think LiteFS and full-stack SQLite is a good bet is that the design is simple. You can read a summary of the LiteFS design and understand what each of these components is doing. SQLite is one of of the most trusted libraries in the world; most of our job is just letting SQLite be SQLite. Your app doesn't even need to know LiteFS is there.

We're plowing ahead on LiteFS features. Here are a few big ones to look out for:

WAL-mode Support: today, LiteFS works with SQLite's default rollback journal mode. But WAL mode is where it's at with modern SQLite. The FUSE proxy model works fine here too: transactions start with a write to the -wal file, and end with another write that marks a header with the commit field set.

Write Forwarding: SQLite works with a single-writer, multiple-reader model and our primary/replica replication mimics that. However, it adds friction to require developers to forward writes to the primary node. Instead, we're making it so any node can perform a write and then forward that transaction data to the primary. The primary can then replicate it out to the rest of the cluster.

S3 Replication: running a cluster of LiteFS nodes significantly improves your durability over a single-server deployment. However, nothing gives quite the same warm fuzzy feeling as tucking away a copy of your database in object storage. This will work similarly to Litestream, however, LiteFS' LTX files are built to be efficiently compacted so restoring a point-in-time copy of your database will be nearly instant.

Encryption: we want developers to feel safe keeping SQLite replicas on services like S3. So we've designed an AEAD encryption scheme that fits into LiteFS naturally and ensures that even if you manage to expose your LTX files to the Internet, you won't have exposed any plaintext.

Try It Out

After several months of work, we're comfortable calling LiteFS beta-ready. We'd be happy if you played around with it.

We've set up a documentation site for LiteFS so you can get going with it and understand how it works. The easiest way to get up and running is to walk through our Getting Started with LiteFS guide. It only takes about 10 minutes and you'll have a globally-distributed SQLite application running. Crazy, right!?

LiteFS is completely open source, developed in the open, and in no way locked into Fly.io, which invests resources in this solely because we are nerds about SQLite and not in any way because LiteFS is part of a secret plan to take over the world. Pinky-swear!

Last updated • Sep 21, 2022 Share this post on Twitter Share this post on Hacker News Share this post on Reddit



All Comments: [-] | anchor

no_wizard(10000) 6 days ago [-]

This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool!

What I'd like to have seen is how this compares to things like rqlite[1] or Cloudflare's D1[2] addressed directly in the article

That said, I think this is pretty good for things like read replica's. I know the sales pitch here is as a full database, and I don't disagree with it, and if I was starting from scratch today and could use this, I totally would give it a try and benchmark / test accordingly, however I can't speak to that use case directly.

What I find however and what I can speak to, is that most workloads already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common). This is a great way to make very - insanely, really - fast read replica's across regions of your data. You can use an independent raft[3][4] implementation to do this on write. If your database supports it, you can even trigger a replication directly from a write to the database itself (I think Aurora has this ability, and I think - don't quote me! - PostgreSQL can do this natively via an extension to kick off a background job)

To that point, in my experience one thing SQLite is actually really good at is storing JSON blobs. I have successfully used it for replicating JSON representations of read only data in the past to great success, cutting down on read times significantly for APIs as the data is 'pre-baked' and the lightweight nature of SQLite allows you to - if you wanted to naively do this - just spawn a new database for each customer and transform their data accordingly ahead of time. Its like AOT compilation for your data.

if you want to avoid some complexity with sharding (you can't always avoid it outright, but this can help cap its complexity) this approach helps enormously in my experience. Do try before you buy!

EDIT: Looks like its running LiteFS[5] not LiteStream[0]. This is my error of understanding.

[0]: https://litestream.io/

[1]: https://github.com/rqlite/rqlite

[2]: https://blog.cloudflare.com/introducing-d1/

[3]: https://raft.github.io/

[4]: https://raft.github.io/#implementations

[5]: https://github.com/superfly/litefs

Gys(10000) 6 days ago [-]

> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream[0] for you.

Litestream and LiteFS are by the same author and serve different purposes.

https://litestream.io/alternatives/ goes into this. It includes LiteFS and rqlite.

[Edited for clarification]

benbjohnson(10000) 6 days ago [-]

LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address.

> What I'd like to have seen is how this compares to things like rqlite or Cloudflare's D1 addressed directly in the article

I think a post comparing the different options is a great idea. I'll try to summarize a bit here though. LiteFS aims to be an analogue to Postgres replication but with built-in failover. Postgres uses log shipping to copy database state from a primary to its replicas, as does LiteFS. LiteReplica is probably the closest thing to LiteFS although it uses a dual GPL/commercial license model. LiteFS uses Apache 2.

There are Raft-based tools like rqlite or dqlite. These have higher consistency guarantees, however, they tend to be more complex to set up -- especially in an ephemeral environment like Kubernetes. LiteFS has a more relaxed membership model for this reason. As for Cloudflare's D1, they haven't released many details and I assume it's closed source. It also requires using a custom JavaScript client instead of using a native SQLite client.

There are also several eventually consistent stores built on SQLite such as Mycelial[1]. These work great for applications with loose consistency needs. LiteFS still maintains serializable isolation within a transaction, although, it has looser guarantees across nodes than something like rqlite.

> Most workloads are already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common)

Yes, that's absolutely true. I don't expect anyone to port their application from Postgres/MySQL to SQLite so they can use LiteFS. Databases and database tools have a long road to becoming mainstream and it follows the traditional adoption curve. I wrote a Go database library called BoltDB about 10 years ago and it had the same adoption concerns early on. Typically, folks try it out with toy applications and play around with it. Once they get more comfortable, then they create new applications on top of it. As more people use it, then late adopters get more comfortable and it further builds trust.

We're committed to LiteFS for the long term so we'll be making updates, fixing bugs, and we'll keep trying to build trust with the community.

[1]: https://mycelial.com/

hinkley(10000) 6 days ago [-]

> Second, your application can only serve requests from that one server. If you fired up your server in Dallas then that'll be snappy for Texans. But your users in Chennai will be cursing your sluggish response times since there's a 250ms ping time between Texas & India.

> To improve availability, it uses leases to determine the primary node in your cluster. By default, it uses Hashicorp's Consul.

Having a satellite office become leader of a cluster is one of the classic blunders in distributed computing.

There are variants of Raft where you can have quorum members that won't nominate themselves for election, but out of the box this is a bad plan.

If you have a Dallas, Chennai, Chicago, and Cleveland office and Dallas goes dark (ie, the tunnel gets fucked up for the fifth time this year), you want Chicago to become the leader, Cleveland if you're desperate. But if Chennai gets elected then everyone has a bad time, including Dallas when it comes back online.

benbjohnson(10000) 6 days ago [-]

Good catch. LiteFS has an option to mark nodes as candidates or not. For example, you can run a cluster with a couple candidate nodes in one region (e.g Chicago) and every other region may just run a single non-electable replica.

We don't support any kind of tiering for candidates in different regions. That's not a bad idea though.

hobo_mark(10000) 6 days ago [-]

Well that was fast... [1]

Are the readable replicas supposed to be long-lived (as in, I don't know, hours)? Or does consul happily converge even with ephemeral instances coming and going every few minutes (thinking of something like Cloud Run and the like, not sure if Fly works the same way)? And do they need to make a copy of the entire DB when they 'boot' or do they stream pages in on demand?

[1] https://news.ycombinator.com/item?id=32240230

benbjohnson(10000) 6 days ago [-]

> Are the readable replicas supposed to be long-lived

Right now, yes, they should be relatively long-lived (e.g. hours). Each node keeps a full copy of the database so it's recommended to use persistent volumes with it so it doesn't need to re-snapshot on boot.

We do have plans to make this work with very short-lived requests for platforms like Lambda or Vercel. That'll use transactionally-aware paging on-demand. You could even run it in a browser, although I'm not sure why you would want to. :)

jensneuse(10000) 6 days ago [-]

Sound like a drop in solution to add high availability to WunderBase (https://github.com/wundergraph/wunderbase). Can we combine LiteFS with Litestream for Backups, or how would you do HA + Backups together?

benbjohnson(10000) 6 days ago [-]

Yes, it should fit with WunderBase to provide HA. I'm adding S3 support which will work the same (in principle) as Litestream. I'm hoping to have that in about a month or so.

nicoburns(10000) 6 days ago [-]

Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down?

Also, what happens if the Consul instance goes down?

If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I suppose that'd be pretty cool.

Currently finding it hard to get on board with the idea that adding a distributed system here actually makes things simpler.

benbjohnson(10000) 6 days ago [-]

> Where is the data actually being stored in this setup? A copy on each machine running the application?

Yes, each node has a full copy of the database locally.

> If so, is there another copy somewhere else (e.g. S3) in case all nodes go down?

S3 replication support is coming[1]. Probably in the next month or so. Until then, it's recommended that you run a persistent volume with your nodes.

> What happens if the Consul instance goes down?

If Consul goes down then you'll lose write availability but your nodes will still be able to perform read queries.

> If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice.

Support for pure ephemeral nodes is coming. If you're comparing LiteFS to a single node Postgres/MySQL then you're right, it's harder to operate. However, distributing out Postgres/MySQL to regions around the world and handling automatic failovers is likely more difficult to operate than LiteFS.

whoisjuan(10000) 6 days ago [-]

Unrelated, but why does the map on their homepage show a region in Cuba? That must be wrong.

dcre(10000) 6 days ago [-]

I think it's in Miami and the icon is confusingly centered over it. The icons look like pins but I think they're supposed to be hot-air balloons. This explains why some of them appear to point to spots in the ocean.

https://fly.io/docs/reference/regions/

theomega(10000) 6 days ago [-]

Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work:

  $ curl -v https://fly.io/blog/introducing-litefs/
  *   Trying 2a09:8280:1::a:791:443...
  * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/cert.pem
  *  CApath: none
  * (304) (OUT), TLS handshake, Client hello (1):
  curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer
Requesting via ipv4 works

  $ curl -4v https://fly.io/blog/introducing-litefs/
  *   Trying 37.16.18.81:443...
  * Connected to fly.io (37.16.18.81) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/cert.pem
  *  CApath: none
  * (304) (OUT), TLS handshake, Client hello (1):
  * (304) (IN), TLS handshake, Server hello (2):
  * (304) (IN), TLS handshake, Unknown (8):
  * (304) (IN), TLS handshake, Certificate (11):
  * (304) (IN), TLS handshake, CERT verify (15):
  * (304) (IN), TLS handshake, Finished (20):
  * (304) (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: CN=fly.io
  *  start date: Jul 25 11:20:01 2022 GMT
  *  expire date: Oct 23 11:20:00 2022 GMT
  *  subjectAltName: host 'fly.io' matched cert's 'fly.io'
  *  issuer: C=US; O=Let's Encrypt; CN=R3
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multiplexing
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  * Using Stream ID: 1 (easy handle 0x135011c00)
  > GET /blog/introducing-litefs/ HTTP/2
  > Host: fly.io
  > user-agent: curl/7.79.1
  > accept: */*
  >
  * Connection state changed (MAX_CONCURRENT_STREAMS == 32)!
  < HTTP/2 200
  < accept-ranges: bytes
  < cache-control: max-age=0, private, must-revalidate
  < content-type: text/html
  < date: Wed, 21 Sep 2022 16:50:16 GMT
  < etag: '632b20f0-1bdc1'
  < fly-request-id: 01GDGFA3RPZPRDV9M3AQ3159ZK-fra
  < last-modified: Wed, 21 Sep 2022 14:34:24 GMT
  < server: Fly/51ee4ef9 (2022-09-20)
  < via: 1.1 fly.io, 2 fly.io
  <
  <!doctype html> ...
tptacek(10000) 6 days ago [-]

Works fine over IPv6 from this Mac.

toast0(10000) 6 days ago [-]

Could easily be a path MTU issue? I don't currently have working IPv6, but on TCP/IPv4, the server->client handshake packets I get back from fly.io are full length packets. If your tcpv6 MSS is set incorrectly, and some other not great things are happening in the path, then you might not be able to receive large packets.

wglb(10000) 6 days ago [-]

Just tried it on my linux box:

  * Trying 2a09:8280:1::a:791:443...
  * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) ALPN,
  * offering h2 ALPN, offering http/1.1 CAfile:
  * /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs TLSv1.0
  * (OUT), TLS header, Certificate Status (22): TLSv1.3 (OUT), TLS
  * handshake, Client hello (1): TLSv1.2 (IN), TLS header, Certificate
  * Status (22): TLSv1.3 (IN), TLS handshake, Server hello (2): TLSv1.2
  * (IN), TLS header, Finished (20): TLSv1.2 (IN), TLS header,
  * Supplemental data (23): TLSv1.3 (IN), TLS handshake, Encrypted
  * Extensions (8): TLSv1.2 (IN), TLS header, Supplemental data (23):
  * TLSv1.3 (IN), TLS handshake, Certificate (11): TLSv1.2 (IN), TLS a
  * header, Supplemental data (23): TLSv1.3 (IN), TLS handshake, CERT
  * verify (15): TLSv1.2 (IN), TLS header, Supplemental data (23):
  * TLSv1.3 (IN), TLS handshake, Finished (20): TLSv1.2 (OUT), TLS
  * header, Finished (20): TLSv1.3 (OUT), TLS change cipher, Change
  * cipher spec (1): TLSv1.2 (OUT), TLS header, Supplemental data (23):
  * TLSv1.3 (OUT), TLS handshake, Finished (20): SSL connection using
  * TLSv1.3 / TLS_AES_256_GCM_SHA384 ALPN, server accepted to use h2
  * Server certificate:
...

Same on my macbook

ranger_danger(10000) 6 days ago [-]

ELI5?

benbjohnson(10000) 6 days ago [-]

LiteFS makes it so that you can have a SQLite database that is transparently replicated to a cluster of machines. The use case is if you have your app running on a server running in Chicago, then users in Europe will have 100ms latency to your server and users in Asia will have a 250ms latency. That's on top of the time it takes for your server to process the request. Many people target sub-100ms latency to make their applications feel snappy but that's impossible if users are waiting on their request to be sent halfway around the world.

Traditionally, it's complicated to replicate your database to different regions of the world using something like Postgres. However, LiteFS aims to make it as simple as just spinning up more server instances around the world. It connects them automatically and ships changes in a transactionally safe way between them.

twobitshifter(10000) 6 days ago [-]

It's a tool to allow you to keep your database on the app server and backed up on s3. This means you don't need a separate database server. Having a database on the same machine as the app is very fast, but it is usually risky. The backups made by LiteFS lessen the risk.

presentation(10000) 6 days ago [-]

My dream would be if this supported geo-partitioning. In my field people are pretty sensitive about GDPR so would love to box in EU PII in EU servers.

yencabulator(10000) 5 days ago [-]

Since the scope is a SQLite database, that means you just run a EU database, with no replicas outside of EU.

This level of database replication isn't going to put individual rows in different geographies.

asim(10000) 6 days ago [-]

10 years ago fly.io is the company I wanted to build. Something with massive technical depth that becomes a developer product. They're doing an incredible job and part of that comes down to how they evangelise the product outside of all the technical hackery. This requires so much continued effort. AND THEN to actually run a business on top of all that. Kudos to you guys. I struggled so much with this. Wish you nothing but continued success.

ignoramous(10000) 6 days ago [-]

5 years ago, fly.io was basically poor man's deno.com / oven.sh [0]. In my (incorrect) opinion, tptacek changed fly.io's trajectory single-handedly.

[0] https://ghostarchive.org/varchive/r-1hXDvOoHA

pphysch(10000) 6 days ago [-]

> Developing against a relational database requires devs to watch out for 'N+1' query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite.

This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite.

Postgres can also be served over a UNIX socket, removing the individual query overhead due to TCP roundtrip.

SQLite is a great technology, but keep in mind that you can also deploy Postgres right next to your app as well. If your app is something like a company backend that could evolve a lot and benefit from Postgres's advanced features, this may be the right choice.

simonw(10000) 6 days ago [-]

I get the impression that there's still quite a bit of overhead involved in talking to PostgreSQL that you don't get with SQLite. A SQLite query is pretty much a C function call that works directly with data structures and local disk.

tptacek(10000) 6 days ago [-]

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

pweissbrod(10000) 6 days ago [-]

Select N+1 is a fundamental anti-pattern of relational database usage. Reducing the latency per round trip doesn't change this fact.

alberth(10000) 6 days ago [-]

A better comparison might be Postgres to BedrockDB.

BeckrockDB is a MySQL compatible interface wrapped around SQLite and as such, is intended for remote RDBMS use.

Expensify has some interesting scaling performance data for their use of it.

https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q...

woooooo(10000) 6 days ago [-]

Isn't 'n + 1' typically a degenerate case of ORM rather than a relational thing? The whole point of rdbms is you can do a join instead of iterated lookups.

benbjohnson(10000) 6 days ago [-]

You could run Postgres over UNIX sockets although you will still get higher latency than SQLite's in-process model. Also, running a Postgres on every app instance on the edge probably isn't practical. Postgres has some great advanced features if you need them but it's also much more heavy weight.

With LiteFS, we're aiming to easily run on low resource cloud hardware such as nodes with 256MB or less of RAM. I haven't tried it on a Raspberry Pi yet but I suspect it would run fine there as well.

hinkley(10000) 6 days ago [-]

Yep. The poison is still in the cup, it's just a smaller dose. Keep drinking if you think that fact is enough to save you...

I see bad judgement calls coming from small numbers, often due to failing to do the cost x frequency math properly in your head. If you're looking at an operation that takes 3ms or 3μs that is called a million times per minute, or twenty thousand times per request, you don't have enough significant figures there and people make mistakes. 3ms x 12345 = ~40s, not 37035ms. Better if you use a higher resolution clock and find out it's actually 3.259 ms, leading to a total of ~40.23s

Point is, when we are doing billions of operations per second, lots of small problems that hide in clock jitter can become really big problems, and a 5-10% error repeated for half a dozen concerns can lead to serious miscalculations in capacity planning and revenue.

lijogdfljk(10000) 6 days ago [-]

This is really cool! Unfortunately i primarily am interested in offline databases so perhaps i'm just not the target audience. However i have to ask, on that note, does this have any application in the offline space?

Ie i wonder if there's a way to can write your applications such that they have less/minimal contention, and then allow the databases to merge when back online? Of course, what happens when there inevitably _is_ contention? etc

Not sure that idea would have a benefit over many SQLite DBs with userland schemas mirroring CRDT principles though. But a boy can dream.

Regardless, very cool work being done here.

bigfish24(10000) 6 days ago [-]

Shameless plug but would love for you to try out Ditto[1]! CRDT-based replicated database that works peer-to-peer including automatically with WiFi and Bluetooth.

[1] https://www.ditto.live

benbjohnson(10000) 6 days ago [-]

Thanks! It depends on your particular use case. If you need all nodes to allow writes then, unfortunately, no it doesn't support that. LiteFS uses physical replication so there's no way to merge pages from separate writers. You're probably better off looking at eventually consistent SQLite tools like Mycelial[1].

However, if you have a single writer and just need replicas to get updates when they periodically connect then yes, LiteFS could fit that use case.

[1]: https://mycelial.com/

infogulch(10000) 6 days ago [-]

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies.

How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

sk55(10000) 6 days ago [-]

They don't have ACID guarantees with this setup.

hinkley(10000) 6 days ago [-]

That's a fun one.

A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction.

Ultimately that's a very odd flavor of sharding. What I don't recall is how they propagated that shard information efficiently, since with sharding and consistent hashing the whole idea is that the lookup algorithm is deterministic, and therefore can be run anywhere and get the same result. WAL lag information is dynamic, so... Raft?

mwcampbell(10000) 6 days ago [-]

I wonder if using FUSE has had any appreciable impact on performance, particularly read performance. I ask because FUSE has historically had a reputation for being slow, e.g. with the old FUSE port of ZFS.

miduil(10000) 6 days ago [-]

I think FUSE performance has significantly improved since 2010, the year in which ZFS-On-Linux became available I believe.

This is just one of many examples: https://www.phoronix.com/news/MTI1MzM

In contrary, in some cases FUSE is even faster than doing a regular kernel mount(). There is an experimental research distribution called distri that is exclusively relying on fuse mounting and figured out that FUSE was faster for doing mounts. https://michael.stapelberg.ch/posts/tags/distri/

yencabulator(10000) 5 days ago [-]

Often, the SQLite database would fit in RAM, and reads would be served straight from the page cache, with no overhead.

Disclaimer: I wrote the FUSE framework LiteFS uses, https://bazil.org/fuse -- and I also have some pending performance-related work to finish, there...

endisneigh(10000) 6 days ago [-]

Seems neat, until you try to do schema migrations. Unless they can guarantee that all containers' SQLite instances have the same scheme without locking I'm not sure how doesn't run into the same issues as many NoSQL.

CouchDB had this same issue with its database per user model and eventually consistent writes.

slively(10000) 6 days ago [-]

In my experience distributed database migrations tend to be two phase though so it can be compatible during the migration. The first migration to add something, release the apps to use it, then a second migration to cleanup.

yencabulator(10000) 5 days ago [-]

I don't see how this is different to a world where you use Postgres over the network, with lots of DB clients. You 'just' do the schema migrations like everyone who can't shut down the business does: incrementally and in a backwards-compatible manner. After the migration, deploy new code that takes advantage of the new schema. Repeat until done.

vcryan(10000) 6 days ago [-]

This approach is very appealing to me :) curious about how people handle schema migrations when using this approach.

I segment sqlite files (databases) that have the same schema into the same folder. I haven't really had a case where migrations was really a concern, but I could see it happening soon.

Seems like in my deployment, I'm going to need an approach to loop over dbs to apply this change... I currently have a step of app deployment that attempts to apply migrations... but it is more simplistic because the primary RDBMS (postgresql) just appears to the application as a single entity which is the normative use-case for db-migrate-runners.

simonw(10000) 6 days ago [-]

Yeah schema migrations are going to be interesting. Presumably those end up making a HUGE change to the WAL (since they could affect every stored row in a large table), which means LiteFS then has to send a truly giant chunk of data out to the replicas.

I wonder how this would present itself in real-world usage? Would the replicas go out-of-date for 10-30s but continue serving read-only traffic, or could there be some element of cluster downtime caused by this?

clord(10000) 6 days ago [-]

I can imagine a database which tries to solve both of these domains.

A centralized database handles consistency, and vends data closures to distributed applications for in-process querying (and those closures reconcile via something like CRDT back to the core db).

Does this exist?

ignoramous(10000) 6 days ago [-]

(except for the 'reconcile back' to primary) Yes? https://martin.kleppmann.com/2015/11/05/database-inside-out-...

See also: materialize.com, readyset.io, aws elastic-views (forever in preview).





Historical Discussions: Mozilla reaffirms that Firefox will continue to support current content blockers (September 24, 2022: 637 points)

(637) Mozilla reaffirms that Firefox will continue to support current content blockers

637 points 4 days ago by bj-rn in 10000th position

www.ghacks.net | Estimated reading time – 3 minutes | comments | anchor

From next year onward, extensions for Google Chrome and most other Chromium-based browsers, will have to rely on a new extension manifest. Manifest V3 defines the boundaries in which extensions may operate.

Current Chromium extensions use Manifest V2 for the most part, even though the January 2023 deadline is looming over the heads of every extension developer.

Google is using its might to push Manifest v3, and most Chromium-based browsers, including Microsoft Edge, will follow. From January 2023 on, extensions need to support Manifest v3 exclusively to be listed in the Chrome Web Store. There is an Enterprise policy to extend the blocking of Manifest v2 support in Chrome by six months, but Google announced already that it won't extend that, despite delays in getting all APIs out in the open for developers.

By June 2023, Chrome and most Chromium-based browsers won't support Manifest v2 extensions anymore. Those installed will be disabled automatically, because they are no longer compatible. Those offered on the Chrome Web Store will vanish, unless their developers published an update to make them compatible with the new Manifest v3.

Mozilla announced early on that it will support Manifest v3 as well, but that it would continue to support important APIs that Google limited in Manifest v3. Probably the most important of them all is the WebRequest API. Used by content blockers extensively to filter certain items, it has been replaced by a less powerful option in Manifest v3.

While Manifest v3 does not mean the end for content blocking on Chrome, Edge and other Chromium-based browsers, it may limit abilities under certain circumstances. Users who install a single content blocker and no other extension that relies on the same relevant API may not notice much of a change, but those who like to add custom filter lists or use multiple extensions that rely on the API, may run into artificial limits set by Google.

AdGuard launched a Manifest v3 compatible ad-blocker recently, and it will display warning prompts if its operation is limited in the browser.

Mozilla reaffirmed this week that its plan has not changed. In 'These weeks in Firefox: issue 124', the organization confirms that it will support the WebRequst API of Manifest v2 alongside Manifest v3.

Again, a reminder that Mozilla plans to continue support for the Manifest v2 blocking WebRequest API (this API powers, for example, uBlock Origin) while simultaneously supporting Manifest v3.

That is good news for users of the web browser who use content blockers such as uBlock Origin. The extension, which its developer claims operates best under Firefox, is the most popular extension for Firefox based on the number of installations and ratings.

Now You:

Summary

Article Name

Mozilla reaffirms that Firefox will continue to support current content blockers

Description

Mozilla reaffirmed this week that the Firefox web browser will continue to support an essential Manifest v2 API that content blockers use currently.

Author

Martin Brinkmann

Publisher

Ghacks Technology News

Logo

Advertisement



All Comments: [-] | anchor

jrootabega(10000) 4 days ago [-]

Seeing this in this context just made me realize that it's weird to consider them 'blockers', a loaded term. As if we're starting from some default, correct state, and then blocking part of it. We've always been able to choose what content we navigate to. These are content selectors/choosers, which just support our existing ability to choose what content we request and actions we take.

llimos(10000) 3 days ago [-]

Bouncers?

swores(10000) 4 days ago [-]

I disagree completely, 'blockers' is accurate.

The user chooses what website page to visit, but it's the website that chooses want content gets served at any of its URLs. That's how the web works. Hence, when websites choose to include aspects of the page users don't want: 'blockers', to block some of the page. Nothing loaded about the term it's just descriptive.

Nor are they 'content selectors/choosers', nobody uses their adblocking plugin to find content, only to block parts of content they've found by visiting a website.

Bakary(10000) 4 days ago [-]

Ideologically, I am inclined to agree. I have an epidermic reaction to advertising in general. However, these extensions exist in a context where they depend on the majority of users not using them. If every browser had them, the economy of the web would change completely. Very few users who take active steps against ads want to pull a Stallman and relinquish entire parts of the Internet altogether; they want to see the content they are seeing, content that is most of the time supported by ads, without ads. The word 'blocker' is thus appropriate because the explicit goal is an asymmetric relationship based on other people making ads profitable.

In reality, the more interesting question would be whether we shouldn't limit our media consumption to begin with, or form a culture that is less dependent on advertising. If browsing with ads becomes mandatory some day, I will probably make the choice to severely cut the amount of Internet I gorge on. It will take some willpower, but every time I switch off the blocker and see what websites really are like I want to gag, so I won't be that difficult.

anotherevan(10000) 3 days ago [-]

I call them HTML firewalls.

P5fRxh5kUvp2th(10000) 3 days ago [-]

the 'block' ads.

Not everything needs to be analyzed to death for some sort of symbolism.

aaaaaaaaaaab(10000) 4 days ago [-]

I'm fine with "content blockers". They block content the same way my eyelids and eye muscles do when I decide to not look at billboards.

If you don't have moral qualms with using your eyelids to block ads then you shouldn't have with software content blockers either. It just outsources the act of blocking from your eyes into the computer.

dbg31415(10000) 3 days ago [-]

Two easy ways to block trackers and ads... without a firewall.

Adding blockers to the hosts table still works with Chrome... hope they don't muck with that...

https://github.com/StevenBlack/hosts

But if they do... there are always DNS solutions you can add to your Router.

https://nextdns.io/

I use Firefox, but even things like Windows spams ads at you if you let it. So many things have Google trackers built in too...

perryizgr8(10000) 3 days ago [-]

This comment reminds me of the crazy spammer who used to frequent Slashdot, always with a long comment about how to use hosts file and how it is superior to every other method of blocking content.

celsoazevedo(10000) 3 days ago [-]

You can't use DNS/hosts to block ads on sites like YouTube because they use the same domain to load ads and videos. To block the ads you have to block the video itself.

So while still useful, that's not really a replacement for browser extensions.

howinteresting(10000) 3 days ago [-]

The problem with these approaches is that it's much harder to exclude individual sites that break.

rapjr9(10000) 3 days ago [-]

Pihole blocks ads for an entire network, so why does a content blocker have to be built into the browser? Can't it just be a separate app you run, outside the browser? Even if the browser is using HTTPS the URL and IP address for everything not from the site you are on is visible. The app could even run as a fake proxy. Scripts would be harder to block if they are sent encrypted, but since you can view the scripts in a browser and hence know the exact content of the script maybe it would be easy to snip them out of the encrypted stream by automatically generating signatures or formulas for what to snip for the scripts from various sites. You could even use an embedded browser in the app to decode and format the entire page, automatically determine what to snip, do the snip, and then forward the modified stream to the users primary web browser (with some delay). Any checksums could be modified appropriately. The app could present a GUI in the primary browser that the user could use to decide what to block. Are the page contents protected by a cryptographic signature? Does not seem like it would matter since the embedded browser certainly has the ability to decrypt everything. Data in the reverse direction could be sent unmodified. Maybe the slowdown caused by filtering everything this way would be unacceptable to users? Some users might find it acceptable in exchange for greater control.

timvisee(10000) 3 days ago [-]

A blocker like uBlock Origin does much more than DNS level blocking. There are many cases where that isn't enough.

l0b0(10000) 4 days ago [-]

The fact that they're even spelling this out means it's going to be reversed soonish. Fortunately Firefox (or at least the original iisdea of a lightweight free-as-in-freedom browser) is ready for another fork.

bayindirh(10000) 4 days ago [-]

If they didn't spell this out loudly, people would accuse them being silent and waiting for the day to silently pull the plug, then.

With your point of view, there's no way to tell the truth.

foepys(10000) 4 days ago [-]

This is baseless speculation.

Firefox already has more APIs than Chrome to support content blocking. E.g. DNS uncloaking and reliably blocking content loading on startup is impossible for extensions in Chrome.

anvic(10000) 4 days ago [-]

I read somewhere that even though Mozilla will continue supporting the ad blocking API from manifest v2, the rest of manifest v2 will be completely dropped so most of the extensions that I use like greasemonkey will stop working. Is that true?

tyingq(10000) 4 days ago [-]

There's a migration guide here: https://extensionworkshop.com/documentation/develop/manifest...

Since they aren't taking away the onBeforeRequest() functionality, I don't see a good reason why greasemonkey can't be ported over.

coldpie(10000) 4 days ago [-]

Googling 'tampermonkey manifest v3' lead me to this issue, which has a lot of relevant discussion: https://github.com/Tampermonkey/tampermonkey/issues/644

lapcat(10000) 4 days ago [-]

No.

2Gkashmiri(10000) 4 days ago [-]

as a person whohas exclusively used firefox all my life, even on my android, i have both firefox and focus, this is quite good news.

the only problem i can imagine is the invisible pressure by the unending desire to become chrome,you know with hiding the search bar by default 'because people are used to that from chrome' and other shenanigans, i feel this support might be shortlived because the stupid people managing firefox are paid by google to fuck up firefox in the worst possible way and this just feels another attempt in that regard.

btw, i've been online since 2004, daily so i have used a lot of firefox

Svip(10000) 4 days ago [-]

I was saddened recently when I learnt they changed 'View image' to 'Open image in new tab'. I loved the way 'View image' worked, allowing you to maintain the image in full view while preserving tab history. If you wanted to open it a new tab, you could middle click it or ctrl click it. But since users were used to the Chrome behaviour they changed it to match, and thus Firefox lost a functionality. Unfortunately, since this is such a niche use case, Mozilla has little incentive to restore it. (Note: Extensions that restore the context menu item fails when the image loading relies on the REFERER header, since it actively re-requests it.)

There are numerous smaller items like this, that just flowed a lot better, particularly if you're heavily browsing images, like tab previews, that have been sacrificed either in the name of performance or to be more like Chrome. Fortunately, SeaMonkey still maintains these classic features, but is unfortunately a memory hog.

My daily browser has also been Firefox since around 2004. SeaMonkey remains a time pocket, when I long for the old days of yore.

asddubs(10000) 4 days ago [-]

I doubt it, since this is an API, and not a visible difference. Seems that changes like that are mostly targeted at the UI.

Arnt(10000) 4 days ago [-]

I see that differently... my wife uses firefox and that API. She eventually replaced her laptop. The only problem was that the old one wasn't fast enough to process about 90,000 regexps each time she clicked a link.

Chrome replaced it with a less flexible API that has bounded runtime, preventing the browser from the kind of slowness that caused a certain dissatisfaction, even disharmony here.

You could say that people shouldn't configure so much adblocking that the browser has to evaluate 90,000 regexps for a simple click. Or you could say that the browser shouldn't prevent that case by design. It's not simple. IMO a worst case like this is one of the causes of 'invisible pressure'.

isaacremuant(10000) 4 days ago [-]

I think along the same lines. Mozilla has forgotten a lot of the political fights of the past around freedom and customisation and instead focused on US partisan talking points supporting censorship and forgetting about the desires of core users.

I, too, feel that eventually they might just go: 'Everyone is doing it and it cost us more effort to keep maintaining it, here you have our own internal blocking, which won't block Google related stuff because they fund us'

All I know is. It's my device. Not Google's and I intend to fight that battle, just like we did with Microsoft.

AnonHP(10000) 4 days ago [-]

Good to hear the reaffirmation from Mozilla that the blocking WebRequest API would be retained. A little over three years ago, Mozilla had promised that it wouldn't just blindly follow Google on Manifest v3 to the letter. [1]

Considering that uBlock Origin works best and can do the most in Firefox with this continuing support, I'm glad Mozilla is still walking the talk on this one.

We need at least one browser that makes the web usable, and that might as well be Firefox with uBlock Origin.

Edit: Oops. Missed providing the link reference below earlier.

[1]: https://blog.mozilla.org/addons/2019/09/03/mozillas-manifest...

flatiron(10000) 4 days ago [-]

I really need to learn the Firefox debugger. My years of chrome debugger muscle memory has tied me to that product (at least for work which is 95% of my desktop browsing)

Personally I'm a mobile personal browser person which is safari. I can't stand to be at a computer after a day of being at the computer.

mike_hock(10000) 3 days ago [-]

'Usable' is an overstatement, but we at least gotta fight the fight.

MintDice(10000) 4 days ago [-]

Yeah I mean, they have to. It's open source software so it could be forked otherwise and also, if they abandoned this core principle anyway, they would lose their entire user base in droves.

There would be no faster way to nuke the entire product. Which is a good thing.

jraph(10000) 4 days ago [-]

A browser is hard to fork, and a fork of a browser is hard to maintain. They might be able to get away with not doing this.

The reality is that this time, they are taking a good decision, on its own, not because they have to for some reason, and this should be recognized as-is. They took enough bad decisions we can attack, no need to belittle them for this one.

est31(10000) 4 days ago [-]

The big test is not now, when Chromium discontinues content blockers, but when renewal of the contract with Google is up. For Google there is a big incentive in getting rid of ad blockers, and they have a lot of power over Mozilla being the party that is responsible for most of Mozilla's revenue.

toastal(10000) 4 days ago [-]

Google keeps Mozilla propped up to avoid anti-trust.

superkuh(10000) 4 days ago [-]

Is Mozilla really trying to juxtapose themselves here against another browser that removes support for power user customization like they don't do it? That's wild. Is there no one left at Moz that remembers them throwing away the entire ecosystem of XUL based extensions that allowed for many things that current Firefox builds don't re: blocking and UI customization.

Mozilla literally did this back in the version 37 and only Firefox forks remain to support the full set of extension features. I guess we're all amnesiacs or the pot is just being boiled slowly enough to not notice.

latexr(10000) 4 days ago [-]

> the pot is just being boiled slowly enough to not notice.

That metaphor is based on a myth1:

> While some 19th-century experiments suggested that the underlying premise is true if the heating is sufficiently gradual, according to modern biologists the premise is false: a frog that is gradually heated will jump out. Furthermore, a frog placed into already boiling water will die immediately, not jump out. Changing location is a natural thermoregulation strategy for frogs and other ectotherms, and is necessary for survival in the wild.

1 https://en.wikipedia.org/wiki/Boiling_frog

Matl(10000) 4 days ago [-]

> Is Mozilla really trying to juxtapose themselves here against another browser that removes support for power user customization like they don't do it? That's wild. Is there no one left at Moz that remembers them throwing away the entire ecosystem of XUL based extensions that allowed for many things that current Firefox builds don't re: blocking and UI customization.

Dropping XUL wasn't some 'hahah, own the power users' thing, it was a move to allow the Firefox architecture to modernize, with XUL this wasn't possible. I have plenty of criticisms of Mozilla, but simplifying things to the point of removing all important context is not the way to do it imo.

M1ch431(10000) 4 days ago [-]

UI customization is fortunately still alive in the wake of XUL extensions.

A /r/firefoxcss mod has a wonderful collection of code snippets that they maintain, which you can browse here: https://mrotherguy.github.io/firefox-csshacks/ and they created a userChrome.js loader here: https://github.com/MrOtherGuy/fx-autoconfig

My favorite customization repository is https://github.com/aminomancer/uc.css.js - which really tests the limits of what is and isn't possible with userChrome.css and .js. My favorite feature is the implementation vertical tabs, without the use of extensions.

Some legacy extensions are maintained and can be found here: https://github.com/xiaoxiaoflood/firefox-scripts/tree/master... (although you will need to use xiaoxiaoflood's userChrome.js loader AFAIK).

Honorable mention goes to the Firefox CSS Store, which can be found here: https://trickypr.github.io/FirefoxCSS-Store.github.io/

Moving to WebExtensions was the logical choice for Firefox, technical/security reasons aside, as they are not alienating extension developers that target Chromium-based browsers.

Yes, they alienated their own extension developers. Yes, they could've handled the transition better, and worked harder towards supporting some of the many APIs/functionalities that extension developers needed (or still need) for their extensions to work in the WebExtensions ecosystem. I myself was quite mad for a very long time at how they handled the switch, but I think overall it's been a success - my own personal feelings aside.

bugmen0t(10000) 4 days ago [-]

the old APIs were actually the only existing, internal and synchronous APIs which does not work in a multi-process, site-isolation architecture.

The internal APIs also had to go. What else would you do?

pGuitar(10000) 4 days ago [-]

Mozilla blocks most extensions by default on Firefox Mobile... they created cumbersome collections for people that really want to install them. Sometime I wonder if their main sponsor is/isn't asking them to make some of those changes.

lapcat(10000) 4 days ago [-]

> Is Mozilla really trying to juxtapose themselves here against another browser that removes support for power user customization like they don't do it?

No. Firefox is adding Manifest 3 support, and people naturally have questions about whether they'll follow Chrome's lead in killing the blocking API.

bdw5204(10000) 3 days ago [-]

This is one thing that has the potential to cause significant numbers of people to switch from Chrome to Firefox. The modern web is unusable without a good ad blocker and its inexplicable to me that it isn't a built-in feature enabled by default in every browser like pop-up blocking. You shouldn't have to install extensions just to get essential functionality from your browser.

snek_case(10000) 3 days ago [-]

The problem is that Google has a conflict of interest as both a browser developer and a platform for people to sell you ads. People selling ads to Google must be putting pressure on them to do this... Or maybe they have internal pressure to maximize revenues.

At the end of the day, yeah, it seems like it might cause people to switch. I'm also honestly wondering, how long until a real competitor to YouTube surfaces? I realize that this is hard to build (though we do have multiple streaming platforms in existence), but Google has been playing this slowly boiling frog experiment with ads on YouTube. Now you have multiple ads per video, coupled with ads baked within the videos themselves, and sponsored content. If you don't pay for YT premium, it's kind of unusable.

The worse the product gets, the more of an opening is created for the competition. Seems like a matter of when rather than a matter of if? What if Amazon or Apple created a YT alternative that was ad-free, maybe with some paid content to sponsor creators (pay $2 a month to get bonus content from this channel). They could afford to sink money into it and it could be very disruptive, but it's like they don't dare. Maybe they're afraid to associate their name with a product that could fail, kind of like Google Plus.

unixbane(10000) 4 days ago [-]

It should content block itself.

rejectfinite(10000) 3 days ago [-]

Most sane Chrome/Safari/Edge user

beeboop(10000) 3 days ago [-]

I'd take content blocking this comment

jokoon(10000) 3 days ago [-]

why can't ublock origin work properly with this manifest v3?

charcircuit(10000) 3 days ago [-]

Because Google is replacing the webRequest API with declarativeNetRequest. This new API has 2 benefits. The first is that it protects the privacy of users better. Extensions no longer get access to the full contents of a request. The second is that it ensures that poorly optimized web extensions can't slow down the performance of loading sites. Chrome is making this change to improve user privacy and improve performance.

uBlock Origin just needs to migrate to this new API. Despite the noise from people who just hate on big tech the only true difference as far as I know is that the browser will enforce a limit on the number of rules that can be added. This number exists to try and prevent bad performance from too many rules existing.

Volundr(10000) 3 days ago [-]

Because the webrequest blocking API is being removed.

DeathMetal3000(10000) 3 days ago [-]

Because an advertising company (Google) designed it just for such a purpose.

beej71(10000) 3 days ago [-]

I switched back to Firefox a while ago, and love it. I keep trying to think of how it could be kept afloat once Google stops funding it.

IIRC they get $400 million annually. There are 200 million FF users. $2 each wouldn't be bad... But realistically only 1% of the users would donate and $200 is unpalatable.

Maybe another organization picks it up and runs it more efficiently or at a loss for advertising?

Or...?

toxicFork(10000) 3 days ago [-]

Do they really need 400m?

ajyotirmay(10000) 1 day ago [-]

One serious criticism of Mozilla is the fact that a major chunk of that money goes into the CEO's pockets and doesn't get spent on Firefox.

That single fact makes me reluctant to pay for Firefox. Unless I'm sure that all the money I donate goes to the right cause and not to a CEO's bank.

Saris(10000) 3 days ago [-]

I don't really understand how a web browser needs 400m a year, that's a lot of employees

jjav(10000) 3 days ago [-]

> But realistically only 1% of the users would donate and $200 is unpalatable.

I'd pay $200/yr for the privacy and control benefits of Firefox over google-controlled Chrome.

classified(10000) 3 days ago [-]

Good, that's why I use Firefox. Still can't help but find it strange that ads are called 'content' now.

ta8903(10000) 3 days ago [-]

Ads aren't considered content, uBO is called a content blocker because it can block all kinds of DOM content and not just ads.





Historical Discussions: Google loses EU appeal and is fined a record $4B (September 22, 2022: 610 points)

(610) Google loses EU appeal and is fined a record $4B

610 points 6 days ago by april_22 in 10000th position

www.axios.com | Estimated reading time – 2 minutes | comments | anchor

Google suffered a significant loss Wednesday when a European court denied its appeal of an antitrust decision and issued a fine of 4.1 billion euro ($4.13 billion), a record penalty.

Driving the news: Google had challenged an earlier EU ruling that said it used its Android mobile operating system to stymie rivals, but Europe's second-highest court upheld it.

Why it matters: The hefty fine and ruling is a win for top EU antitrust official Margrethe Vestager, who has aggressively prosecuted Big Tech companies, and could set a precedent for future European antitrust rulings covering tech giants.

What they're saying: 'The General Court largely confirms the Commission's decision that Google imposed unlawful restrictions on manufacturers of Android mobile devices and mobile network operators in order to consolidate the dominant position of its search engine,' the court said.

  • The court did find in Google's favor on one point regarding revenue-sharing with mobile device makers and carriers and slightly reduced the penalty.

The other side: "We are disappointed that the Court did not annul the decision in full. Android has created more choice for everyone, not less, and supports thousands of successful businesses in Europe and around the world,' a Google spokesperson said in a statement.

Flashback: This is Google's third EU antitrust fine. Previously, the search giant was fined in a shopping case and an online advertising case.

The big picture: The company is also facing a slew of antitrust suits and inquiries in the U.S.

What's next: Google can appeal the ruling to the EU Court of Justice, Europe's highest court.




All Comments: [-] | anchor

antonymy(10000) 5 days ago [-]

The fallout from this should be interesting. This fine is a substantial part (roughly 25%!) of Google's overall revenue from the EMEA region (Europe, Middle East, Africa), of which the EU is probably like 90%, at least, of the total, so it's probably toing the line of actually making the EU unprofitable to Google considering the costs of future compliance to avoid being fined again. Worse: this was not as steep as the fine could have been, they could've fined them about double this amount under current regulations, but chose not to. Probably because that would have pushed Google's EU operations into the red for sure and made the decision easy for them.

To be sure the company won't be in trouble financially, their global revenue can cover the fine, but that doesn't stop the EU from potentially being a net loss for the company going forward. Abandoning the EU would be a drastic step, though. It's going to be interesting to see how they adapt to this. Another fine on this scale and I think they will pull out.

cush(10000) 5 days ago [-]

Imagine Google had to find a different revenue model than harvesting and selling personal information? What would they even do? Could Cloud or Workspace even prop up the company?

ocdtrekkie(10000) 5 days ago [-]

Bear in mind a global company like Google would much rather operate in the EU in the red than pull out. Simply because Google only survives on being a monopoly, and leaving one of the biggest global markets makes too much room for a competitor to develop that can eventually challenge them across the world.

locallost(10000) 5 days ago [-]

They do 200 billion in ad sales a year. Without checking the data, I am pretty sure the EU is at least their third largest market, if not the second. This is definitely a lot less than a 1/4 of their revenue and they're not going anywhere.

uni_rule(10000) 5 days ago [-]

I hate that fines proportional to the revenue giant companies make from doing shady shit are the exception and not the rule

weatherlite(10000) 5 days ago [-]

The EU seems fine with the idea actually - it wants more independence on technology and said so several times. There will be many many pissed off European users for sure but that never stopped the EU elite from making a decision. Still - seems like a reach to me.

babelfish(10000) 5 days ago [-]

To be clear, while the fines might make operating in the EU unprofitable, they could just...follow EU regulations...and remain profitable, albeit at a likely lower margin

gwerbret(10000) 5 days ago [-]

A summary of the original findings by the European Commission (1):

'By decision of 18 July 2018, the Commission fined Google for having abused its dominant position by imposing anticompetitive contractual restrictions on manufacturers of mobile devices and on mobile network operators, in some cases since 1 January 2011. Three types of restriction were identified:

1. those contained in 'distribution agreements', requiring manufacturers of mobile devices to pre-install the general search (Google Search) and (Chrome) browser apps in order to be able to obtain a licence from Google to use its app store (Play Store);

2. those contained in 'anti-fragmentation agreements', under which the operating licences necessary for the pre-installation of the Google Search and Play Store apps could be obtained by mobile device manufacturers only if they undertook not to sell devices running versions of the Android operating system not approved by Google;

3. those contained in 'revenue share agreements', under which the grant of a share of Google's advertising revenue to the manufacturers of mobile devices and the mobile network operators concerned was subject to their undertaking not to pre-install a competing general search service on a predefined portfolio of devices.'

1: https://curia.europa.eu/jcms/jcms/p1_3862705/en/

guelo(10000) 5 days ago [-]

The irony is that if Google had not licensed Android and only manufactured its own devices, like Apple does, it could restrict consumer options much more harshly without EU fines and restrictions.

pooper(10000) 5 days ago [-]

> By decision of 18 July 2018

So they haven't paid this fine for over four years now and there is no penalty for it? Just because they have deep enough pockets to afford expensive lawyers? I am sure not having to pay those four billion in fines for over four years more than pays the cost of these lawyers even if they eventually have to pay this fine, right?

I mean I am glad to see a big fine like this but surely the fines must go up every day it is left unpaid.

jmyeet(10000) 5 days ago [-]

So I have no doubt Google threw their weight around in the Android space. It's known that they impose conditions for access to the Google suite of apps. But I have a thought that I have a hard time reconciling.

Imagine Google had gone the Apple route of vertically integrating handset production, distribution and sale in the same way Apple did. If they had, there would be no antitrust issue, just like there isn't with Apple.

That seems weird.

How is having third-party manufacturers (well, really it's just Samsung now) create more antitrust issues than the Apple model? That just doesn't seem right.

nilsbunger(10000) 5 days ago [-]

#2 is the same thing Microsoft got busted for by US antitrust in the late 90s. Microsoft had required computer vendors to NOT sell computers with other OSes (eg Linux). At one point they insisted computer vendors pay for a Windows license on all computers they shipped, even ones shipped without Windows.

typon(10000) 6 days ago [-]

What amount would these EU fines have to be to make companies like Google and Facebook decide to exit the EU?

speedgoose(10000) 5 days ago [-]

That would be such a dream!

bzxcvbn(10000) 6 days ago [-]

This is never going to happen. It's just too big of a market.

yieldcrv(10000) 6 days ago [-]

probably 15-20% of annual revenue

its psychology, the board or ceo would react preemptively based on the idea that shareholders would react on their behalf

shareholders (if they have voting power at all) would react for the same result at higher fine amounts than that

soheil(10000) 5 days ago [-]

I just want to point out in Russia those companies have little to no presence specially after the war. It's telling that we always hear it's a sign of oppression and authoritarianism if governments kicked out those services out of their country, but here we are people advocating for it in the EU.

bee_rider(10000) 5 days ago [-]

Only one way to find out!

RandomWorker(10000) 6 days ago [-]

That increases the budget of the eu parliament with 2.2% on a yearly basis if paid out in one go.

[1082B for 2014-2020, 4B/1082B*6y~2.2%]

A loss of 4B is a decrease of revenue by 1.5% for google.

[googles revenue was 256B in 2021 4/256~1.5%]

spywaregorilla(10000) 5 days ago [-]

EMEA represents about 31% of revenues. Google makes about 76B in net income.

So another way to look at this is 21% of profits in Europe for a year.

markdestouches(10000) 6 days ago [-]

Gives you a good perspective on how big google is and how much influence it potentially has.

bzxcvbn(10000) 5 days ago [-]

The budget of the EU parliament? What do you mean? The budget of the parliament was about 740k€ in 2022.

Did you mean the budget of the EU itself? It's about what you said (180B€ in 2022). Of course, that doesn't account for individual member States' budgets. The EU's budget is dwarfed by the aggregate of all members' budgets, and it is lower than most of the big EU countries' budgets.

legitster(10000) 6 days ago [-]

So... What's the end result? Google has to charge for Android? Ask users their search preference?

moffkalast(10000) 5 days ago [-]

Android should damn well ship with multiple app stores and multiple browsers, not just the GoogleApprovedTM app store where they ban anyone that looks at them wrong or thinks about blocking ads for more than 2 miliseconds.

The exclusive app stores are both iOS and Android's greatest strengths and failings at the same time. Apple is arguably the far worse offender there too. If there was an alternative app store on iOS everyone would jump ship immediately.

Imagine having crossplatform Steam/GoG Mobile stores for mobile games, etc.

jillesvangurp(10000) 5 days ago [-]

The end result is that Google either explains to their shareholders where their dividend is going every quarter or they adjust to what amounts to a fairly consistent signal out of the EU that people there are running out of patience with Google not complying with the rules there and that that has financial consequences.

So, it's a game of ever escalating fines. At some point, they'll hit diesel gate levels, which caused a bit of an existential crisis for VW a few years ago. That signal actually mostly came from the US BTW. The EU is demonstrating that that kind of thin can work both ways and that they might go there squeeze some US companies when they misbehave.

As for Google and Apple, Amazon, MS, Facebook, etc., they've been going through rounds of increasingly larger fines from different countries in the EU and them doing the absolute minimum to maybe toe the line but never entirely. It's not enough; they need to get more pro-active on this or they'll have to deal with bigger fines and lawyer fees. Very simple. Testing what is good enough in a court room like they have been doing is a costly strategy. Judges not amused/impressed so-far apparently.

MikusR(10000) 5 days ago [-]

They already do both of those things.

wongarsu(10000) 5 days ago [-]

I think this is about Google's Android certification, that allows manufacturers to ship Android with any google app. Some like the Play Store are basically required by users, but Google makes it a 'get all or none' deal with a lot of other demands, which includes a google search widget.

izacus(10000) 5 days ago [-]

Think of the situation with Chinese non-Google phones. You install your favorite app and it crashes on launch because the manufacturer messed with the system APIs.

You run another app and it gets killed in the background, not receiving messages, because it's not on the list of your phone manufacturer partners.

This is the situation with degoogled Android today - a huge compatibility mess.

nottorp(10000) 5 days ago [-]

> Android has created more choice for everyone, not less

Did it? I can choose between an iOS phone and an Android phone.

I may be able to choose from different manufacturers on the Android side, but on the software side there are just two options.

that_guy_iain(10000) 5 days ago [-]

Before I could choose the same manufacturers but things were tied to those. I knew if I switched from Nokia to Sony for example, it would be completely different. In a way Android resulted in the choice to switch between manufacturers to be painless. Realistically, now you can choose whatever non apple device you want and know everything will continue to work.

imiric(10000) 5 days ago [-]

> I may be able to choose from different manufacturers on the Android side, but on the software side there are just two options.

What do you mean by 'just two options'?

The AOSP ecosystem has a healthy range of ROMs to choose from on many devices. The process of installing alternative firmware may not be for everyone, but the choice is there, which is pretty much nonexistent for iOS devices.

amelius(10000) 5 days ago [-]

If you're looking for a phone without ads/spyware, there is less choice because the EU sees Android as a similar product-category as iOS, thereby not identifying iOS as a monopoly (and Android as well, in its own category).

Hamcha(10000) 5 days ago [-]

Remember when Google refused to put Youtube on Windows Phone and when Microsoft made their own client, Google forced them to take it off their own store? Truly advocating for more choice.

(Press at the time: https://www.theregister.com/2013/08/16/windows_phone_youtube... )

izacus(10000) 5 days ago [-]

And before that you could choose between multiple brands that wouldn't run the same software and could only download apps on a proprietary carrier app store.

Someone built an app for Nokia Symbian phones and released it with O2 UK? Well, go to hell if you had Sony Ericsson Symbian phone because that phones Symbian was incompatible on API level with the Nokia, despite having the same OS. You could even have the same Nokia, but good luck if yours had the T-Mobile DE bloatware and their own store with their own profit sharing deals.

Google enforces compatibility between Android phones with CDD/CTS/VTS which ensures that the phones support a consistent set of working APIs that application devs can rely on.

Before CTS/CDD/VTS tests grew enough, we Android devs had hellova time making sure our apps worked across Samsungs, LGs, Nokias and whatnot.

EU attacking Google for ensuring the Android ecosystem is actually compatible is just ensuring that Apple will dominate with it's lack of choice in the mobile market. The fact that EU officials excluded Apple from consideration of how mobile markets work is telling just how disconnected from reality they are. Apple swept away all the old competitors BECAUSE it didn't have the fragmented mess of lacking cross-device, cross-region and cross-carrier compatibility.

And just to be clear - I'm not against EU issuing out fines to big tech (god knows I've defended that a lot of times), but in this case it's actively toxic for us Europeans.

Salgat(10000) 5 days ago [-]

What they did with their manufacturers is a different issue, but in general they offer a lot more options than Apple. Android allows Manufacturers (such as Amazon and Samsung) to have their own app stores and allows for side loading apps, both of which are impossible with Apple.

ehsankia(10000) 5 days ago [-]

But what did Google do that's worse than what Apple did? It's a bit ironic that by making their OS opensource, they somehow are more guilty than Apple for keeping their OS locked down.

Android allows hundreds of companies to have a business and make millions or billions, many of which are in EU and wouldn't exist if Android had gone the Apple way.

asadlionpk(10000) 5 days ago [-]

EU has decided to make a living out of fining the tech companies regularly.

rr888(10000) 5 days ago [-]

They continuously fined the banks until they're all nearly bankrupt. I guess tech is next now.

fabatka(10000) 5 days ago [-]

Seems like it's still worth it for the tech companies, so I'd say it's a smart move.

nebalee(10000) 5 days ago [-]

And Google has decided to make a living out of breaking the law regularly.

lizardactivist(10000) 5 days ago [-]

No, they're not fined regularly, only when they knowingly violate the law.

And it's $4B after 4 years since the beginning of the investigation, a tiny increase to an already massive EU budget.

Normille(10000) 5 days ago [-]

Cool. The EU does have its [many!] faults. But they have a pretty good track record when it comes to consumer rights and consumer choice. Next up I'd like to see them do the same for the plethora of incompatible cordless power-tool systems out there as they did with insisting all mobile phones standardise on a USB connector.

elsonrodriguez(10000) 5 days ago [-]

If the EU standardizes on battery packs, Ryobi needs go get a free pass or extension of some sort. Ryobi have kept the same battery format for 25 years, whereas their competitors have changed battery standards 2 or 3 times during that timeframe.

woeh(10000) 5 days ago [-]

I was pleased to see recently that a couple of different brands at my local DIY store had interoperable batteries. I bought a small lawnmower which now uses the same battery as a cordless drill from another brand. The brands that joined are listed on their marketing site[0].

Edit: One of the companies has a slightly less obnoxious website which also lists brands[1]

[0]https://www.powerforall-alliance.com/en/#technology

[1]https://www.gardena.com/int/products/powerforall/

twblalock(10000) 5 days ago [-]

> Next up I'd like to see them do the same for the plethora of incompatible cordless power-tool systems out there as they did with insisting all mobile phones standardise on a USB connector.

We should all be glad that didn't happen with micro-USB or some of the worse connectors years ago, because we would still be stuck with them. USB-C was innovated by the market, not by EU bureaucrats.

jayd16(10000) 5 days ago [-]

Sadly this decision will lead to more fragmentation, not less, won't it? Would have be nice if there was a path forward for Android standards.

scarface74(10000) 5 days ago [-]

The EU insisting on all phones standardizing on the USB connector is evidence of just the opposite and shows the cluelessness and shortsightness of the EU.

You will still have phones that come with "standard USB C" cables where some support different power delivery, data speeds (or not support data transfer at all), very few will support video over USB C (which is standardized), etc.

So still when the Android user with the cheap power only USB C cord moves over to the hypothetical iPhone 15 with USB C, they will still have to throw away their USB C cable that potentially doesn't support high speed data or video over USB C. Both supported by todays iPads with USB C.

If the same mandate had gone through when it was first proposed, we would have been stuck with micro USB.

rlpb(10000) 5 days ago [-]

I don't mind so long as third parties are free to make compatible batteries and tools. Unfortunately there's no reasonable migration path for someone already invested in tools and batteries. Unlike phones that get cycled through in just a few years, I am invested in my tools with the expectation that the majority of my less-used tools will last my lifetime. I'd like to be able to buy replacement compatible batteries for at least that long.

origin_path(10000) 5 days ago [-]

'consumer choice ... insisting all mobile phones standardise'

Although it's not the most important thing in the world, forcing standardization through law is the opposite of giving consumers choices.

tmoravec(10000) 5 days ago [-]

> pretty good track record when it comes to consumer rights

Are you sure EU has 'pretty good track record' with these fines of Google, FB, et al.? If anything, I'd say the EU puts a mild theatre every now and then, and gives the US giants a free pass otherwise.

efsavage(10000) 5 days ago [-]

I support the charging connector standardization, but not tool batteries. There are so many factors in play I don't see how standardization would offer enough benefit to offset innovation penalties. Even within a given brand/connector, there are variations on capabilities like size and duty cycle and power output. I have a bunch of Milwaukee cordless tools on the M18 platform, and the smaller tools (e.g. drills) can use any battery but some tools (e.g. chainsaw) only work with a subset because of higher power draw.

I have cordless tools across a number of battery platforms and while it's convenient when they can be shared, it's ultimately not a big deal to have a few if another company makes a better tool. Tradespeople won't lock themselves in if the right tool is on a different platform.

londons_explore(10000) 5 days ago [-]

The cordless power tools typically have the battery management system in the device, not in the battery. The battery tends to contain protection circuitry, but no logic that determines charge rate etc.

That means every device manufacturer will claim, even if the law requires that devices must all have the same plug, that a competitors device drew too much current, or charged their battery too fast/too slow/at too a low a temperature, and thats why performance sucks.

In reality, these technical problems could be solved - but device manufacturers will make the small hurdle into a big one when explaining why they can't all standardize.

fatboy(10000) 5 days ago [-]

I think that's a bit trickier. If Makita or whoever was forced to change their batteries, anyone with a big catalogue of their tools would be pretty miffed.

Also, I've mentioned this on hn before, there is already the Cordless Alliance, which does exactly this.

Sadly, there's only a couple of useful-in-a-mainstream-sense brands in there: Mafell and Metabo. The rest are very niche.

MuffinFlavored(10000) 5 days ago [-]

> The EU does have its [many!] faults.

Curious/out of the loop: any (unbiased?) examples?

endisneigh(10000) 5 days ago [-]

It's not clear to me what the EU, specifically, wants Google to do here from the article. Any insight?

layer8(10000) 5 days ago [-]

For example, not require that Chrome be preinstalled as the default browser on Android devices, and not require that a bundle of Google apps be preinstalled if any of them is preinstalled.

perlgeek(10000) 5 days ago [-]

Pay a fine, first of all :-)

Wikipedia has more: https://en.wikipedia.org/wiki/Antitrust_cases_against_Google...

> In October 2018 Google revamped how it would distribute its Google Play Store with Android in the future: Charging a licensing fee for the store, without requiring installation of Google apps, but making it free to pre-install the Google apps if they want.

> Furthermore, Google's hardware partners will be allowed to market devices in the EU that run rival versions of Android or other operating systems.

> In March 2019 Google announced that it will give European users of Android phones an option of which browser and search engine they want on their phone at the time of purchase to further comply with the EU's decision.

makeitdouble(10000) 5 days ago [-]

The original ruling:

https://www.theverge.com/2018/7/18/17580694/google-android-e...

Basically dirty stuff we all knew Google was doing, but kept getting away with until then

SiempreViernes(10000) 5 days ago [-]

It looks like it's about forcing manufacturers to ship with chrome and the google search widget, so like the old Microsoft forcing IE upon everyone case.

gniv(10000) 5 days ago [-]

It's interesting how the two continents are keeping each other in check when it comes to major abuses. The EU fining big US tech and the US fining VW are the prominent examples. In each case the home country has big incentives to keep the status quo fully aware that it's bad for consumers.

curiousgal(10000) 5 days ago [-]

I can't help but roll my eyes at the hypocrisy of the EU. One prime example is French websites forcing you to pay 3 euros when you refuse cookies.

pmontra(10000) 5 days ago [-]

Google's reaction.

> We are disappointed that the Court did not annul the decision in full. Android has created more choice for everyone, not less, and supports thousands of successful businesses in Europe and around the world.

s/Android/Windows/ and s/Europe/USA/ and it could be Microsoft at the end of the antitrust case in the 90s.

josefx(10000) 5 days ago [-]

> Android has created more choice for everyone

Same way IE did back when Microsoft provided it for 'free'.

ncmncm(10000) 5 days ago [-]

Yet, still pocket change.

It is commendable that EU does not allow miscreants to tie up fines in an endless legal maze.

dahfizz(10000) 5 days ago [-]

What % of global revenue or profit do you think an appropriate fine is?

caldarons(10000) 6 days ago [-]

While I am in favour of efforts like the GDPR to protect users data, I wonder if all this anti-bigtech sentiment is actually justified and where it will eventually lead to

rvz(10000) 6 days ago [-]

It is a long time coming and still a long way to go and the surveillance big tech giants should pay a high price for abusing and violating the privacy of hundreds of millions of users.

It's a shame that this fine is too generous to Google and it won't scare them. It should be much larger than that and should be in the multi-billions of dollars.

Someone(10000) 6 days ago [-]

If you're in favor of efforts like the GDPR, why do you call this anti-bigtech sentiment? It's pro-privacy sentiment.

The GDPR applies to smalltech, too, and smalltech gets fined. For example, an individual got fined €600 about two months ago for pointing a video camera at a shared access road (https://www.enforcementtracker.com/ETid-1397)

You can find many more examples of small fines at https://www.enforcementtracker.com/.

athenot(10000) 6 days ago [-]

With a small fine, it's just an annoying cost of doing business. With this ruling, there's a possibility it may actually change the risk/benefit assessment for companies that insist on doing whatever they want with user data.

Maybe the lines of business which require promiscuous data sharing in order to be profitable will stop operating in the EU, and give rise to other business models. Whether these newer ones will be better or worse is not yet known, though the gamble in the EU is that it won't be worse.

xani_(10000) 6 days ago [-]

Breaking the law that applies to is not a sentiment. The same law punished small companies too

seydor(10000) 6 days ago [-]

GDPR has been pro-big tech according to all the studies i have seen

jlokier(10000) 5 days ago [-]

The EU appeal the article is about has nothing at all to do with GDPR, user data, or privacy.

It's about anticompetitive behaviour, i.e. antitrust. Antitrust is not particularly about tech companies, and it has been around a long time, since before modern computers.

It's about misusing a position of market dominance to prevent a healthy competitive market from operating. Very large companies which are market-dominant face this problem, but they also have the resources to understand and handle the responsibility. They can either do so, or face fines and injunctions to make them do so. In severe cases they can be forced to break up dominant business units, such as happened with the breakup of AT&T in the USA some years ago.

This is the same sort of thing as might have blocked nVidia from buying ARM, for example (until nVidia pulled out voluntarily). You can see how that has nothing to do with GDPR; it wasn't even in the EU.

zmgsabst(10000) 6 days ago [-]

I expect some kind of:

- common carrier status for "platforms"

- PII laws, broadly

metadat(10000) 6 days ago [-]

Hopefully it can actually entice Google into changing it's behavior. The US fines are so small they are a joke, a light tax on doing business.

robomartin(10000) 5 days ago [-]

Does the money in these cases go to government? Not sure how it is in the EU.

This is one of the aspects of government-based fines I dislike. The money should go to the people, the victims, not government. They go after a company that may have damaged consumers and then proceed to grab the fine instead of giving it to the people who were the actual victims of the transgression.

wickedsickeune(10000) 5 days ago [-]

The money of this fine is not to repair damages to individuals, it's to disincentivize behaviour. Plus if it's used for expenses of the EU, it indirectly goes to the people, as they will have to contribute less taxes.

perlgeek(10000) 5 days ago [-]

It's simply not practical to determine with certainty who owned an Android phone during the time that Google violated the antitrust laws, and divide the payout among them.

layer8(10000) 5 days ago [-]

The money will go into the EU budget, which for the most part benefits EU citizens. And everyone will benefit from Google having to change (or already having changed) their agreements with device manufacturers.

It's like when you pay a parking fine, the money doesn't go to the specific people inconvenienced or endangered by your wrong parking, it goes into the municipal budget.

mritun(10000) 5 days ago [-]

Extremely interesting point in this ruling that I have never seen before and I guarantee is sending shockwaves to the boardrooms is the definition of market. In this ruling EU chose to define the market as "Android phone manufacturers all over the world except China".

Of course that definition is very peculiar. I was doubtful that it would stick but apparently it did.

This is fascinating that a regulator can define market as "users of YOUR_PRODUCT excluding all competitors you thought AND across the world EXCEPT the very place where everything is made".

Now that it has stuck, it would change the dynamics of business. It started with tech but its going to trickle down :)

ThatPlayer(10000) 5 days ago [-]

The market they're interested in is the EU. So 'all over the world except China' includes the EU. Defining the market as the region the governing authority has power over is nothing new: The FTC rules on the US market, the European Commission rules on the EU market.

drexlspivey(10000) 6 days ago [-]

That's 1.45% of their annual revenue or 5.2% of their annual net profit.

p-e-w(10000) 5 days ago [-]

Which is essentially nothing. Imagine an individual, caught committing an illegal act of this magnitude for the third time, being fined 1.45% of their annual pre-tax salary, with no other consequences.

Speeding tickets are costlier than this in many jurisdictions.

988747(10000) 6 days ago [-]

Yes, but it's also a threat of further fines in future if they don't stop their shenanigans.

p-e-w(10000) 5 days ago [-]

> This is Google's third EU antitrust fine. Previously, the search giant was fined in a shopping case and an online advertising case.

Apparently, 'three strikes and you're out' only applies to individuals, who with a repeat offense of this magnitude would have their lives destroyed by the justice system, and get locked away for a decade or more.

synu(10000) 5 days ago [-]

Wow, what country has three fines and you go to jail for a decade? That sounds intense.

dahfizz(10000) 5 days ago [-]

I don't believe Europe has three strike laws. Anyway, in the US, three strike laws only apply to criminal charges and not civil ones.

jefftk(10000) 5 days ago [-]

While the EU doesn't use a 'three strikes' model, they do use progressively higher fines and other consequences for repeat offenses. If Google had ignored the 2017 Google Shopping order, for example, they would have lost a repeat investigation relatively quickly and there would have been very large fines. But instead, they changed how the product works, in a way that looks like they're responding to the decision and trying to follow it.

In a company with as many independent lines of business as Google has it's possible to have multiple instances of anti-trust enforcement that don't look like 'we already fined you but you're continuing with the prohibited behavior', and the Android, Ads, and Shopping cases do all look really different. Plus they're covering overlapping time periods, which limits how much the regulator can say 'you should have listened to us before'.

(Disclosure: I used to work at Google)

esperent(10000) 5 days ago [-]

The 3 strikes law is a US thing.

BiteCode_dev(10000) 5 days ago [-]

We really need the equivalent of jail time for companies. Clearly after a certain size, fines do not cut it. Jailing executives is too hard, so laws should exist that remove the company from a market for x years or something, like if they were in a cell.

otippat(10000) 5 days ago [-]

I don't think three-strikes laws exist anywhere other than in the US to be honest.

mdrzn(10000) 5 days ago [-]

Yesss! EU is doing its job.

I'm very happy every time they manage to give a slap on the wrist of these monopolies

moffkalast(10000) 5 days ago [-]

Wrist slaps are getting slightly stronger though!





Historical Discussions: 3x new books added to the Pirate Library Mirror (September 25, 2022: 607 points)

(608) 3x new books added to the Pirate Library Mirror

608 points 2 days ago by pilimi_anna in 10000th position

annas-blog.org | Estimated reading time – 3 minutes | comments | anchor

3x new books added to the Pirate Library Mirror (+24TB, 3.8 million books)

annas-blog.org, 2022-09-25

In the original release of the Pirate Library Mirror, we made a mirror of Z-Library, a large illegal book collection. As a reminder, this is what we wrote in that original blog post:

Z-Library is a popular (and illegal) library. They have taken the Library Genesis collection and made it easily searchable. On top of that, they have become very effective at solliciting new book contributions, by incentivizing contributing users with various perks. They currently do not contribute these new books back to Library Genesis. And unlike Library Genesis, they do not make their collection easily mirrorable, which prevents wide preservation. This is important to their business model, since they charge money for accessing their collection in bulk (more than 10 books per day).

We do not make moral judgements about charging money for bulk access to an illegal book collection. It is beyond a doubt that the Z-Library has been successful in expanding access to knowledge, and sourcing more books. We are simply here to do our part: ensuring the long-term preservation of this private collection.

That collection dated back to mid-2021. In the meantime, the Z-Library has been growing at a staggering rate: they have added about 3.8 million new books. There are some duplicates in there, sure, but the majority of it seems to be legitimately new books, or higher quality scans of previously submitted books. This is in large part because of the increased number of volunteer moderators at the Z-Library, and their bulk-upload system with deduplication. We would like to congratulate them on these achievements.

We are happy to announce that we have gotten all books that were added to the Z-Library between our last mirror and August 2022. We have also gone back and scraped some books that we missed the first time around. All in all, this new collection is about 24TB, which is much bigger than the last one (7TB). Our mirror is now 31TB in total. Again, we deduplicated against Library Genesis, since there are already torrents available for that collection.

Please go to the Pirate Library Mirror to check out the new collection. There is more information there about how the files are structured, and what has changed since last time. We won't link to it from here, since this is just a blog website that doesn't host any illegal materials.

Since last time, we have gotten a lot of suggestions and ideas for collections to mirror, which we would love to spend more time on. We're not doing this for money, but we would love to quit our jobs in finance and tech, and work on this full time. Last time we only got a single donation of $35 (thank you!), and we would need a lot more to subsist. If you too think it's important to preserve humanity's knowledge and culturual legacy, and you're in a good financial position, please consider supporting us. Currently we're taking donations in BTC: 14RVdnhpZNffJuAWXenmfHLHQxCvBtHdGM. We really appreciate it.

Of course, seeding is also a great way to help us out. Thanks everyone who is seeding our previous set of torrents. We're grateful for the positive response, and happy that there are so many people who care about preservation of knowledge and culture in this unusual way.

- Anna and the Pirate Library Mirror team (Twitter, Reddit)




All Comments: [-] | anchor

synergy20(10000) 2 days ago [-]

the technical book writing no longer makes financial sense to me these days, unless you wrote them to become well known domain expert and possibly get more leads for future profit, and you don't care about how much you can make directly from the books sale.

I think a viable approach is:

  1. publish your book in e-format, it will be pirated but only those who paid will get continuous update such as new contents added, errata fixes etc easily and regularly.
  2. offer a print-on-demand service for those who prefers to have a printed version.
  3. offer a pay-me-later link to those who read it without pay(e.g. pirated version), later realize your writing indeed helped them so much to the point they want to 'tip' you for a sense of rightness.
randomcatuser(10000) 2 days ago [-]

This reminds me of serialized novels (new content every month?). Maybe that'll be the way, in the future!

walterbell(10000) 2 days ago [-]

A proven approach is Michael Hartl's (@mhartl) Rails tutorial, which has grossed multiple six figures with the entire content free to read online, plus upgrades for DRM-free PDF, videos and print. That's well more than 10X the average return on technical books.

poofmagicdragon(10000) 2 days ago [-]

Main site: http://pilimi.org/

Torrents can be obtained via their onion site: http://2urmf2mk2dhmz4km522u4yfy2ynbzkbejf2cvmpcbzhpffvcuksrz... (requires Tor to follow link)

Using onion.ws proxy: http://2urmf2mk2dhmz4km522u4yfy2ynbzkbejf2cvmpcbzhpffvcuksrz... (doesn't require Tor)

jeroenhd(10000) 2 days ago [-]

I wonder why these services don't generate vanity TOR domains. I can generate about 21 TOR addresses beginning with 'zlib' per second and it would make bookmarks so much more recognisable if they weren't completely random.

I know relying on names to recognise onion addresses is unsafe, but so is following random links. Why not add at least a little recognisability to the official URL?

mmastrac(10000) 2 days ago [-]

How does one actually download this many torrent files? It feels quite difficult to do.

joak(10000) 2 days ago [-]

I understand that books needs to be financed but why should we deprive the poorest from accessing culture knowing that marginal cost of ebooks is zero.

We should incentivize reading books not making it harder or more expensive.

- E-books should be free and of easy access.

- Writers and editors should be paid according to the popularity of their work.

The proposals are not incompatible.

googlryas(10000) 2 days ago [-]

They are incompatible as long as a majority of people think their appropriate contribution to authors is $0

daptaq(10000) 2 days ago [-]

Can you elaborate? E.g.,

> - Writers and editors should be paid according to the popularity of their work.

By whom?

dudul(10000) 2 days ago [-]

Where would the money come from? Who would decide how popular a book is? I agree that in theory these are not incompatible statements, but I fail to see a practical way to make it work.

throw10920(10000) 1 day ago [-]