Hacker News with comments/articles inlined for offline reading

Authors ranked on leaderboard
Last updated:
Reload to view new stories

October 19, 2019 18:36



Front Page/ShowHN stories over 4 points from last 7 days
If internet connection drops, you can still read the stories
If there were any historical discussions on the story, links to all the previous stories on Hacker News will appear just above the comments.

Historical Discussions: TurboTax's 20-Year Fight to Stop Americans from Filing Taxes for Free (October 17, 2019: 1498 points)

(1498) TurboTax's 20-Year Fight to Stop Americans from Filing Taxes for Free

1498 points 2 days ago by danso in 4th position

www.propublica.org | Estimated reading time – 39 minutes | comments | anchor

Richard Borge, special to ProPublica

ProPublica is a nonprofit newsroom that investigates abuses of power. Sign up for ProPublica's Big Story newsletter to receive stories like this one in your inbox as soon as they are published.

Last fall, Intuit's longtime CEO Brad Smith embarked on a farewell tour of the company's offices around the world. Smith had presided over 11 years of explosive growth, a period when Intuit had secured its place in the Silicon Valley pantheon, and the tour was like a long party.

In Ontario, employees wore T-shirts with Smith's quasi-spiritual sayings: "Do whatever makes your heart beat fastest" and "Repetition doesn't ruin the prayer." In Bangalore, India, workers put on Smith face masks as they posed for selfies with the man himself. Fittingly, the tour culminated in San Diego, the home of TurboTax, the software that transformed the company's fortunes. There, Smith arrived at his party in a DeLorean, and as he walked a red carpet, cheering employees waved "Brad is Rad" signs. To Smith's delight, his favorite rock star, Gene Simmons of Kiss, emerged. The two posed for pictures, Simmons clad in black and the beaming CEO flashing the "rock on" hand sign.

Intuit began in the 1980s as an accounting software company focused on helping people with their bookkeeping. Over time, the company, like the other giants of Big Tech, cultivated an image of being not just good at what it did, but good, period. In a recent Super Bowl ad, Intuit portrayed itself as a gentle robot that liberates small-business owners from paperwork. The company stresses values above all, urging employees to "deliver awesome" and pursue "integrity without compromise."

Get Our Top Investigations Subscribe to the Big Story newsletter.

Intuit's QuickBooks accounting product remains a steady moneymaker, but in the past two decades TurboTax, its tax preparation product, has driven the company's steadily growing profits and made it a Wall Street phenom. When Smith took over in 2008, TurboTax was a market leader, but only a small portion of Americans filed their taxes online. By 2019, nearly 40% of U.S. taxpayers filed online and some 40 million of them did so with TurboTax, far more than with any other product.

But the success of TurboTax rests on a shaky foundation, one that could collapse overnight if the U.S. government did what most wealthy countries did long ago and made tax filing simple and free for most citizens.

For more than 20 years, Intuit has waged a sophisticated, sometimes covert war to prevent the government from doing just that, according to internal company and IRS documents and interviews with insiders. The company unleashed a battalion of lobbyists and hired top officials from the agency that regulates it. From the beginning, Intuit recognized that its success depended on two parallel missions: stoking innovation in Silicon Valley while stifling it in Washington. Indeed, employees ruefully joke that the company's motto should actually be "compromise without integrity."

Internal presentations lay out company tactics for fighting "encroachment," Intuit's catchall term for any government initiative to make filing taxes easier — such as creating a free government filing system or pre-filling people's returns with payroll or other data the IRS already has. "For a decade proposals have sought to create IRS tax software or a ReturnFree Tax System; All were stopped," reads a confidential 2007 PowerPoint presentation from an Intuit board of directors meeting. The company's 2014-15 plan included manufacturing "3rd-party grass roots" support. "Buy ads for op-eds/editorials/stories in African American and Latino media," one internal PowerPoint slide states.

The centerpiece of Intuit's anti-encroachment strategy has been the Free File program, hatched 17 years ago in a moment of crisis for the company. Under the terms of an agreement with the federal government, Intuit and other commercial tax prep companies promised to provide free online filing to tens of millions of lower-income taxpayers. In exchange, the IRS pledged not to create a government-run system.

Since Free File's launch, Intuit has done everything it could to limit the program's reach while making sure the government stuck to its end of the deal. As ProPublica has reported, Intuit added code to the Free File landing page of TurboTax that hid it from search engines like Google, making it harder for would-be users to find.

Twelve years ago, Intuit launched its own "free" product: the similarly named "Free Edition" of TurboTax. But unlike the government program, this one comes with traps that can push customers lured with the promise of "free" into paying, some more than $200. Free Edition was a smash hit for Intuit and its pitch for "free" prep remains core to the company's growth. Recently, it launched a "free, free free free" ad campaign for the Free Edition, including a crossword puzzle in The New York Times in which the answer to every clue was "f-r-e-e."

Intuit knows it's deceiving its customers, internal company documents obtained by ProPublica show. "The website lists Free, Free, Free and the customers are assuming their return will be free," said a company PowerPoint presentation that reported the results of an analysis of customer calls this year. "Customers are getting upset."

An internal Intuit analysis of customer calls this year shows widespread customer confusion about ads for "free" TurboTax. (Highlights added by ProPublica.)

Intuit also continues to use "dark patterns" — design tricks to get users of its website to do things they don't necessarily mean to do — to ensure that as many customers as possible pay, former employees say. A marketing concept frequently invoked at Intuit, which goes by the acronym "FUD," seeks to tap into Americans' fear, uncertainty and doubt about the tax filing process.

An Intuit spokesman declined to answer ProPublica's detailed questions about its efforts to fend off a government filing system, but he provided a statement.

"We empower our customers to take control of their financial lives, which includes being in charge of their own tax preparation," he said, adding that a "government-run pre-filled tax preparation system that makes the tax collector (who is also the investigator, auditor and enforcer) the tax preparer is fraught with conflicts of interest."

The IRS is seemingly the biggest threat to Intuit and other commercial tax prep businesses, but it has more frequently acted as the industry's ally, defending the Free File program even in the face of critical internal reviews. The IRS declined to comment for this article.

The consequences of Intuit's efforts affect a huge proportion of the taxpaying public. Americans spend an estimated 1.7 billion hours and $31 billion doing their taxes each year. Just 2.8 million participated in the Free File program this year, down from 5.1 million at the program's peak in 2005.

Intuit's success has made the men who run the company rich. Smith, the CEO who stepped down last year and is now executive board chair, had a stake worth $20 million when he became chief executive. It ballooned to $220 million by last year. Co-founder Scott Cook is now among the country's wealthiest people, his fortune soaring to $3.3 billion.

Intuit CEO Brad Smith flashes the "rock on" hand sign next to Kiss' Gene Simmons during Smith's 2018 farewell tour at TurboTax in San Diego. (Rachael Marie Photography)

This year, Intuit was close to realizing a long-held goal: enshrining the Free File program in law, effectively closing the door on the IRS ever creating a free tax filing system. But an outcry followed ProPublica's reporting on the matter and Intuit's treatment of its customers, prompting the provision to be dropped and state and federal investigations into Intuit's practices.

Yet even after this setback, the company remained steadfastly confident that its clout in Washington would win the day.

"What we're not gonna do is fight this publicly because that is exactly what they want us to do," said Sasan Goodarzi, the new CEO, in a video released to staff this May and obtained by ProPublica. "We are actually working with the IRS and members of the Congress to ensure that the facts are very clear."


Intuit has dominated the tax software market since 1993, when for $225 million, it bought Chipsoft, the San Diego-based company that had created TurboTax. Even then, TurboTax was the most popular option, but Intuit pursued a plan of aggressive growth. The product necessarily came on a disk, and by the end of the 1990s TurboTax boxes were nearly ubiquitous, on shelves in office supply stores across America.

As internet speeds increased and dot-com mania took hold, it became apparent that Intuit's future was not in a box on a shelf. It was online.

The prospect of TurboTax's growth was vast for another reason. As late as 2001, around 45 million Americans still filled out their tax forms on paper. For Intuit, those were all potential customers.

But Intuit wasn't alone in seeing possibilities in the spread of high-speed internet. In Washington, lawmakers began pushing the IRS to modernize and get more taxpayers to file electronically. It was a no-brainer: Filing taxes online would be easier, and the IRS would save staff costs on processing paper returns.

The danger to Intuit's growing business was obvious. If the government succeeded in creating a system that allowed the vast majority of taxpayers to file online for free, TurboTax profits would plummet. Intuit recognized that the notion of "return-free filing" was not going away on its own.

And so in 1998, the company hired Bernie McKay, a onetime Carter administration aide and a senior lobbyist at AT&T, to be its vice president for corporate affairs. Intuit executives like to talk about having a "customer obsession" in developing their products. McKay's obsession is stopping government encroachment. Known to physically bang the table to drive home a point, McKay's style is "aggressive to the point of offense," said one fellow tax prep lobbyist. An Intuit spokesman said, "This mischaracterization of Mr. McKay is pure fiction."

McKay, for his part, when asked at a recent tax industry conference which Star Wars character he is, responded, "Darth Vader."

The year McKay was hired, Congress passed a major overhaul of the IRS. The bill, reflecting Intuit's lobbying, said that the IRS "should cooperate with and encourage the private sector" to increase electronic filing.

While McKay came through in his first big test, in 2002, the company found itself up against an unexpected foe, the George W. Bush administration. The threat came from a broad administration initiative to upgrade government technology. One of the proposals called for the IRS to develop "an easy, no-cost option for taxpayers to file their tax return online."

Without such an option, taxpayers were stuck either filing on paper or, to file electronically, paying a tax professional or software company like TurboTax. Providing an alternative would be an obvious improvement, said Mark Forman, an official at the Office of Management and Budget who led the "e-government" program. The technology wasn't all that complicated, and creating a free, automated filing system would help tens of millions of Americans. "This was seen as a low-cost, high-payoff initiative," Forman recalled in a recent interview with ProPublica. Standing in the way, he said, was an industry "that lives off the complexity of the tax code."

Intuit revved its new lobbying machine. Even before the OMB report was publicly released, a group of Republican lawmakers, led by TurboTax's hometown congressman, wrote to the agency arguing that there was no reason for the government to "compete" with the "well-established" private tax prep companies. Intuit's lobbyists also went above the OMB and pressed their case directly to the White House, Forman recalled.

At the IRS, "all hell broke loose," remembered Terry Lutes, who was then the head of electronic filing at the agency. Intuit's clout on the Hill meant that lawmakers were soon accusing the IRS of making "secret plans to undercut the industry," Lutes said. The agency ran the risk of seeing its funding cut if it were to pursue the Bush plan.

The IRS commissioner at the time, Charles Rossotti, also opposed the idea. The IRS' customer service staff, already too thin to respond adequately to Americans' questions about the tax code, would have to grow substantially to handle millions of software queries. Congress "will never give you sufficient funding," Rossotti told ProPublica.

So the IRS felt caught in the middle. The question became, Lutes said, "Is there some way to come out of this with something for taxpayers that addresses the administration's objective and at the same time is acceptable to industry?"

Intuit, it turned out, did have a way. Since 1999, as part of the company's strategy to head off encroachment, TurboTax had been offering free tax prep to the poorest filers. It was a program that served to bolster the company's arguments that government intervention was unnecessary.

This became the basis for a deal. The industry would offer free tax prep to a larger portion of taxpayers. In exchange, the IRS would promise not to develop its own system.

CEO Says Intuit Is Taking Its Case Directly to the IRS and Congress

In an internal video, CEO Sasan Goodarzi told Intuit staffers: "What we're not gonna do is fight this publicly because that is exactly what they want us to do."

Intuit organized a coalition of tax prep companies under the name the Free File Alliance, and after negotiations with the IRS, the group agreed to provide free federal filing to 60% of taxpayers, or about 78 million people at the time. Government officials touted the solution as a marvel of public and private cooperation. Americans would get free tax prep, and it would cost the government almost nothing.

For Intuit, it was the culmination of years of lobbying. The IRS had signed a contract that said it "will not compete with the [Free File Alliance] in providing free, online tax return preparation and filing services to taxpayers."

What's more, "free" wasn't as unprofitable as it sounded. The alliance, guided by a lawyer who was also an Intuit lobbyist, won a series of concessions that made the program palatable to industry. Free File only required the companies to offer free federal returns. They could charge for other products. The state return was the most common, but they could also pitch loans, "audit defense" or even products that had nothing to do with taxes.

Free File had another bright side: The companies could tailor their Free File offers so that they didn't cut into their base of paying customers. The agreement said the industry had to offer free federal services to at least 60% of taxpayers, but each company individually only had to cover 10% of taxpayers. Intuit and the others were free to limit their offers of free tax prep by age, income or state.

There was little incentive for the companies to publicize a free alternative to their paid products, and the IRS agreed that the Free File offers need only be listed on a special page of the agency's website.

For Intuit, it was a major victory in the war against encroachment. The company could now focus on turning whatever new customers it acquired through the program into paying users, both that year and in the future.


The first year of Free File was 2003, and for Intuit, things went well. On paper, the Free File Alliance was a collection of 17 companies, all of them vying to serve the American taxpayer. But in reality, it was a group made up of two giants and a bunch of gnats. Intuit's only significant competitor was H&R Block, and even it was a distant second. The rest of the alliance consisted mostly of tiny companies with names like Free1040TaxReturns.com. As a result, Intuit could tailor its Free File offer just the way it wanted.

But the next year, Intuit began to lose control of its creation. A scrappy competitor, TaxAct, decided to use Free File to stand out. The company decided it would try to pick up as many new customers as possible and then charge them for ancillary services. Instead of following Intuit's lead and constraining its offer to a subset of low-income taxpayers, TaxAct went the opposite direction.

"Why not go for an offer that's much simpler to understand?" is how Lance Dunn, the president of the maker of TaxAct, described the strategy in a later court hearing. It began advertising a pitch for "free federal online tax preparation and e-filing for all taxpayers. No restrictions. Everyone qualifies."

TurboTax's offer on the Free File page, meanwhile, was more difficult to parse: "if you are eligible for EIC, are age 22 or younger, age 62 or older, or active Military with a W2." (EIC stood for the Earned Income Tax Credit.)

Read More An internal document and current and former company employees show the companies steered customers away from the government-sponsored free option and made them pay.

TaxAct's ploy was a smashing success. The company's volume exploded.

Alarmed, Intuit tried to get the other companies not to offer their products for free to too many potential customers, according to Dunn. Such a request could be collusion, a violation of antitrust law, Dunn said. "Intuit asked the Free File Alliance members that we should restrict offers, which I believe is probably not legal for that group to restrain trade," he said.

ProPublica asked Intuit about Dunn's accusation, but the company did not respond.

Dunn, who declined to speak with ProPublica, made these remarks during sworn testimony in 2011. The hearing was part of an antitrust case by the Justice Department against H&R Block after it tried to buy TaxAct. The U.S. argued that, by eliminating a competitor, the merger would create a duopoly of Intuit and H&R Block. Although the Justice Department ultimately blocked that takeover, the market has grown even more consolidated in recent years. In 2019, according to a ProPublica analysis of IRS data, the two giants accounted for 81% of all individual returns filed using tax prep software.

On the defensive, Intuit and H&R Block matched TaxAct's "no restrictions" offer on Free File. Americans rushed to file for free, and in 2005, 5 million people filed their taxes through the program. Free File had become the most popular way to file taxes online.

Intuit viewed the popularity of Free File as a serious threat and took its case to Congress. That year, Brad Smith, then a senior vice president at the company and head of TurboTax, told a House committee that "the current Free File Alliance program has drifted very far from its original public service purpose and objective," as he put it. The program wasn't supposed to be for everyone, he said: It was for the "disadvantaged, underprivileged and underserved taxpayer populations."

Intuit's arguments quickly gained traction at the IRS. Already, in March 2005, the IRS had written to the Justice Department for legal advice on modifying the Free File program. The agency wanted to know: Would it run afoul of antitrust laws if the IRS barred companies in the Free File Alliance from offering a free product to everyone?

The Justice Department responded in a May 2005 letter. Clearly, wrote Renata Hesse, an antitrust section chief at the department, "any agreement among Alliance members to restrict such free service is likely a form of price fixing" and thus illegal. But there was still a way for Intuit to get what it wanted. She wrote that if the IRS itself were to impose such a restriction, it would be legal.

The IRS swooped in to beat back Intuit's competition, doing for Intuit what the company could not on its own. Despite just 5 million Americans using a program that was purportedly available to 80 million, the IRS agreed that Free File needed to be reined in.

A confidential presentation for Intuit's board showed how the company, over a decade, beat back attempts to make tax filing easier.

The agency made its reasoning clear in a previously unreported letter sent to the Free File Alliance the following year. Bert DuMars, then head of electronic filing at the IRS, wrote that there'd been a huge jump in people using Free File in 2005, but no corresponding boom in people paying for tax prep. "If this trend continued, the IRS was concerned that it could cause many vendors to go out of business," he wrote. Stock market analysts, he pointed out, had said Free File "represented a threat to future revenues and profits of the publicly traded company participants." The IRS decided to remove this threat.

The new agreement, struck between the IRS and the alliance in 2005, gave Intuit what it had sought. Companies were now expressly barred from offering free tax prep to everyone through the program. Instead, only taxpayers under an income cap, then $50,000 a year, would be eligible.

On paper, the program's eligibility had actually increased to 70% of taxpayers, or about 93 million households, up from the previous 78 million. But in practice, because broad, easy-to-understand offers were now barred, it was clear the program's use would decline.

Intuit had again bent the power of the federal government in its favor. After 2005, the Free File program was never again as popular, eventually falling to about half that year's level.


With the threat of government encroachment on ice and high-speed internet access proliferating in the mid-2000s, Intuit looked forward to steady growth and big profits. The upside of the online software business was huge, with the cost of producing each additional unit approaching zero. And TurboTax was hardly a niche product: Intuit executives still excitedly talk about the TAM, total available market, of TurboTax as every single tax filer in the country, now over 150 million households.

But TaxAct's Free File gambit had forever transformed the industry. Advertising "free" was a great lure, so TaxAct took the battle to a different venue. Barred from making a free offer to everyone through Free File on the IRS' website, TaxAct decided to make the offer on its own website in 2006. Intuit recognized a credible challenge from the upstart and countered the next year, launching TurboTax Free Edition on its website.

Confusingly, there were now two distinct options: the government-sponsored Free File and the commercial free editions.

For customers who managed to qualify, the new commercial options offered by these companies were similar to what they could get on the IRS' Free File website: The underlying software was the same, only the federal return was free, and the companies expected to make money on each customer through charging for a state tax return or other services.

But for the companies, there was a clear benefit to winning customers directly, rather than through the IRS program. The companies had complete control over how they handled customers from start to finish.

Intuit poured ad dollars into its Free Edition. Not only did the new product effectively meet TaxAct's challenge, it quickly became the major driver of TurboTax's customer growth.

How Intuit Stopped Free File From Spreading

That growth posed a challenge: how to, as internal company documents put it, "monetize free." Over successive tax seasons, Intuit unleashed teams of designers, engineers, marketers and data scientists on that problem, working at its headquarters in Mountain View and TurboTax's main offices in San Diego.

Part of the solution was to pitch users side products like loans or "Audit Defense." But it also meant misleading customers. Frequently "free" didn't mean free at all. Many who started in TurboTax Free Edition found that if their return required certain commonplace tax forms, they would have to upgrade to a paid edition in order to file.

The company came to a key insight: Americans' anxiety around tax filing is so powerful that it usually trumps any frustration with the TurboTax product, according to three former Intuit staffers. So even if customers click on "free" and are ultimately asked to pay, they will usually do it rather than start the entire process anew. Intuit capitalized on this tendency by making sure the paywall popped up only when the taxpayer was deep into the filing process.

"There's a lot of desperation — people will agree, will click, will do anything to file," said a former longtime software developer.

Every fall before tax season, the company puts every aspect of the TurboTax homepage and filing process through rigorous user testing. Design decisions down to color, word choice and other features are picked to maximize how many customers pay, regardless if they are eligible for the free product. "Dark patterns are something that are spoken of with pride and encouraged in design all hands" meetings, said one former designer. In the design world, "dark patterns" are tactics to get users to do something they don't necessarily mean to do. (ProPublica previously documented dark patterns encountered by people trying to file their taxes for free.)

On TurboTax's homepage, for example, the company carefully chooses how it describes the different editions. Prominently featured next to Deluxe Edition, which costs around $100, is the phrase "maximize your deductions."

If users initially click on the Deluxe software, they are never offered the choice to go to the Free Edition even if the no-cost option would produce the same return. "Maximize your deductions" was legendary at Intuit for its effectiveness in steering customers eligible for free filing to buy the paid product, according to a former marketing staffer.

Intuit's Share Price Has Shot Up in Recent Years

Another celebrated feature, former staffers said, were the animations that appear as TurboTax users prepare their returns. One shows icons representing different tax deductions scrolling by, while another, at the end of the process, shows paper tax forms being scanned line-by-line and the phrase "Let's comb through your returns." What users are not told is that these cartoons reflect no actual processing or calculations; rather, Intuit's designers deliberately added these delays to both reinforce and ease users' "Fear, Uncertainty, and Doubt." The animations emphasize that taxes are complicated but also reassure users that the technological wizardry of TurboTax will protect them from mistakes.

In a statement, the Intuit spokesman said, "The process of completing a tax return often has at least some level of stress and anxiety associated with it. ... To offset these feelings, we use a variety of design elements — content, animation, movement, etc. — to ensure our customers' peace of mind."

The 2007 launch of Free Edition started a period of rapid growth for TurboTax. Within two years, use of its web products had almost doubled, and over the past decade, its website has grown each year by an average of 2 million more customers. The company reported this year that TurboTax online had handled 32 million returns. In a statement, it said around a third of that number used Free Edition.

The government's Free File program, meanwhile, has mostly faded into the background, drowned out by Intuit's and other companies' "free" offers. The IRS did try advertising campaigns, spending around $2 million some years to spread the word. But compared with the reach of Intuit, this was a pittance: The company reported this year that it spent $800 million on advertising. With its budget slashed by Congress, the IRS has spent no money at all to advertise the program in recent years.


Amid its success, Intuit has sometimes had to put down insurgents bent on reforming the tax filing system. In 2007, the same year Intuit launched its Free Edition, Barack Obama, then a candidate for president, took aim at the tax prep industry. In a speech to an audience of tax wonks in Washington, he promised that the IRS would establish a simple return system. "This means no more worry, no more waste of time, no more extra expense for a tax preparer," he declared.

But the Obama administration, as Bush's had before, found that it was no match for Intuit.

Again, Bernie McKay, the lobbyist who had joined Intuit in the late 1990s and outlasted multiple CEOs, led the company's campaign. In response to the Obama threat, McKay and Intuit's small army of outside lobbyists turned to Congress, where lawmakers friendly to the company introduced a series of bills that would elevate Free File from a temporary deal with the IRS to the law of the land.

Republicans have historically been the company's most reliable supporters, but some Democrats joined them. Rep. Zoe Lofgren, the California Democrat whose district includes part of Silicon Valley, has introduced or co-sponsored five bills over the years that would codify the Free File program, with names like the Free File Permanence Act. Lofgren's spokesperson told ProPublica that the congresswoman believes the IRS, because of its role as tax collector, should not also be the tax preparer.

Hedging its bets, the company also sought to make sure the IRS could not spend a single dollar creating a public filing system. One internal document says Intuit would "advance legislative language in House Appropriations for 'No Funds' restriction on IRS spending" on such a system. It worked. Within a few years, Congress passed a 3,000-page appropriations bill that included a single sentence crucial to Intuit's financial future: "No funds," the law decreed, could be used "to provide to any person a proposed final return or statement."

Another important aspect of Intuit's influence strategy during the Obama years was covertly enlisting minority and women's groups to press its case.

The internal 2014-15 "encroachment strategy" document discloses plans to "leverage trade groups to support House/Senate Free File bills." It goes on to list the groups Women Impacting Public Policy, The Latino Coalition and the National Black Chamber of Commerce.

Intuit has given money to all of those groups over the years. All have signed letters urging Congress to make the Free File deal permanent. "The Free File program has been a clear success," said one letter signed by The Latino Coalition and the Hispanic Leadership Fund.

A spokesperson for Women Impacting Public Policy said it has received $70,000 from Intuit. The amounts given to the other groups are unknown, and they did not respond to requests for comment.

Company documents also outline plans to "mobilize" a "coalition" that included think tanks and academics, who published op-eds.

Will Marshall, president of the pro-business Progressive Policy Institute, opposed return-free filing in an op-ed in The Hill because doing one's taxes is "a teachable moment [that] prompts us to review our financial circumstances."

Anti-tax activist Grover Norquist, the most consistent champion of Intuit's policy positions, warned that "big spenders in Washington, D.C. want to socialize all tax preparation in America."

It is unclear whether they were paid by Intuit or the Free File Alliance. Norquist didn't respond to a request for comment, and a Progressive Policy Institute spokesman declined to say whether Intuit gave the group money.

Whatever external challenges to the status quo Intuit has faced, the company has been able to rely on the IRS' continuing enthusiastic support of the Free File program. Every few years, the IRS and the industry got together to renew the deal.

In part, that was due to the relationships Intuit had developed with high-ranking IRS officials. One, Dave Williams, served as the agency's top negotiator on the Free File program for several years and "was very commercially sensitive," said Mark Ernst, the CEO of H&R Block until 2007. Ernst, who later held a senior role at the IRS, told ProPublica that Williams "didn't want to offend the industry," noting that "he was particularly open to having sidebar conversations with key people where he could imagine himself landing some day."

Today, Williams works at Intuit, where he's held the title of chief tax officer since 2013. He is one of several former IRS employees who have gone on to work there. In a statement, Williams told ProPublica he did not have discussions about future employment with Intuit or other companies until after he left the IRS. He added that his career in government was focused on "what is best for the taxpayer" and that he "joined Intuit for the same reason: to help the American taxpayer."

Despite Free File's declining use, the IRS often claimed that the program was nevertheless meeting one of its original goals: driving more people to file electronically instead of on paper. Ernst, who served as a senior official at the IRS from 2009 to 2010, didn't believe that a program used by so few people was having any such effect. "It was a talking point that got trotted out all the time to justify the Free File Alliance," he said.

Internally, IRS managers have also argued that the program is, in a way, a success, because it created "a free marketplace," as one internal management report in 2017 put it. Apparently, customers weren't the only ones taken in by the word "free."


In 2018, Intuit faced rare scrutiny from inside the IRS. The agency asked its Advisory Council, a group of outside experts, to take stock of Free File. To the company's alarm, it soon became apparent that the council's report might be sharply critical.

That July, council chair and University of California, Davis, law professor Dennis Ventry wrote two pieces criticizing an Intuit-backed bill in Congress that would make the program permanent. His op-ed in The Hill was called, "Free File providers scam taxpayers; Congress shouldn't be fooled."

In response, the IRS again rose to Intuit's aid. It rushed to assure the company that Ventry's power to affect the program was limited, according to emails to the Free File Alliance obtained through a public records request.

"The Commissioner has met directly with Mr. Ventry," IRS official Ken Corbin wrote to Steve Ryan, a lobbyist for Intuit who also represented the alliance. "Mr. Ventry will recuse himself from participating or contributing to the topic of Free File."

Corbin heads the IRS division that processes most Americans' tax returns and negotiates the Free File deal with Intuit and the industry.

A few days later, Ryan arrived at the IRS' Constitution Avenue headquarters in Washington to mount a defense of the program. A former Democratic Senate aide turned lawyer-lobbyist, Ryan is known on Capitol Hill for taking on politically fraught clients, including Trump attorney Michael Cohen and the government of Qatar. He helped create Free File in the early 2000s, and it was now his job to secure its future.

Ryan's PowerPoint presentation at the IRS rehashed arguments that the company had been making for the past 15 years. It also highlighted a 2013 study by Brown University professor John Friedman, a former Obama National Economic Council official, to make the point that the program had been successful in generating "Free Tax Returns Outside of Free File." The presentation did not mention that Friedman's study was paid for by the Free File companies and was not published in an academic journal. Friedman declined to say what he was paid but told ProPublica he "wrote the piece based on my analysis of the issues, which I stand by."

Ventry, who attended the meeting, got a call the next day alerting him that a California public records request had been filed for his emails — they were subject to such a request because he's an employee of a state university. It came from the Free File Alliance, as The New York Times later reported. The request, Ventry believes, was designed to "freak me out."

In early October, the council sent a version of its final report, which included a harsh appraisal of the Free File program, to the IRS to seek responses before releasing it publicly the following month.

But in mid-October, just weeks before the report saw the light of day, the Free File industry group fired off an "urgent" request to meet with IRS officials. The goal was to re-sign and "improve" the memorandum of understanding that governed the Free File program, according to the emails. The current agreement wasn't expiring for another two years, but Ryan cited the "time urgency to make changes that will benefit taxpayers" in the coming tax season, adding, "I have not darkened your door in 2018 and need your ... attention to this opportunity."

The IRS' Corbin signed the new deal on Oct. 31. Two weeks later, the Advisory Council report was released, with a damning indictment of the program: "The IRS's deficient oversight and performance standards for the Free File program put vulnerable taxpayers at risk," the report found.

The expert body recommended that the IRS negotiate a series of new provisions designed to increase the use and oversight of the program, including mandating advertising by the companies. But it was too late. A new deal had already been signed with modest changes. As it had in the past, Intuit and the alliance had effectively insulated the program from reform. Members of the council, Ventry said, were "pissed off."

A spokesman for the Free File Alliance said the group had pushed to renegotiate the deal in 2018 because of the looming 2020 presidential campaign. "The reason for the timing of the extension of the agreement was the political season," he said. The group had not seen the report before its release, he added.

(In August, ProPublica sued the IRS to get more correspondence between the agency and Intuit's lobbyists. In response to our Freedom of Information Act requests, the agency has withheld over 100 pages. The case is ongoing.)

The new deal included rules that barred Free File companies from offering extra products to the relatively small number of users who access the program. This makes it much more difficult to convert those users into paying customers.

Read More Come along as we try to file our taxes for free on TurboTax!

At around the same time, the industry took steps to make the program more difficult to find. Both Intuit and H&R Block added code to their Free File websites that shielded them from search engines such as Google. The Intuit spokesman said the company increased paid search advertising for Free File "by nearly 80 percent" over the last year and has data showing more people found the program through online search this year than last year, but he declined to provide specific figures.

What is clear is that Intuit's business relies on keeping the use of Free File low. The company has repeatedly declined to say how many of its paying customers are eligible for the program, which is currently open to anyone who makes under $66,000. But based on publicly available data and statements by Intuit executives, ProPublica estimates that roughly 15 million paying TurboTax customers could have filed for free if they found Free File. That represents more than $1.5 billion in estimated revenue, or more than half the total that TurboTax generates. Those affected include retirees, students, people on disability and minimum-wage workers.

Customers, meanwhile, remain confused by Intuit's myriad uses of "free," and internal documents show the company knows it. Over just a two-week period this past filing season, Intuit received nearly 7,000 TurboTax customer calls in which the phrase "supposed to be free" was uttered, according to a company analysis. One customer complained that Intuit charged him even though "it says 'free free free' on the commercial." The TurboTax representative responded: "That ad has been the bane of my existence."

Even as TurboTax's business thrived, 2019 has been a rocky year for Intuit's long-running war against government encroachment. In April, the company was close to finally succeeding in its long-held goal to make Free File permanent. A bill called the Taxpayer First Act was sailing toward almost unanimous approval in Congress. But after ProPublica published a series of stories about the program, including a story showing that military families and students were particularly affected by Intuit's business tactics, the bill stalled. Congress ultimately removed the provision that would have enshrined Free File in law.

After having enabled Intuit for so long, the IRS finally responded to the pressure. It hired a contractor to review the Free File program. But the contractor had previously argued against the IRS offering its own tax prep option, and the review did not recommend major changes. The agency has not yet announced its plans for the future of the program.

The agency's inspector general also launched an audit, which is ongoing. Other investigations and litigation followed, ranging from class-action complaints, alleging that consumers had been deceived by Intuit's tactics, to investigations and lawsuits by regulators and prosecutors in New York and California. Intuit has denied wrongdoing, saying it "has at all times been clear and fair with its customers."

Read More The move by TurboTax maker Intuit to charge more lower-income customers has helped boost revenue.

Despite the scrutiny, Wall Street has continued to embrace the company's business model. The company recently announced it made $1.5 billion in profits for its fiscal year. It expects its TurboTax unit to grow by 10% next year. Last year the CEO was paid $20 million. The share price hit an all-time record.

The company has returned to its old strategy: stay the course and take its case directly to the IRS and Congress. Its allies in the Senate have again advanced an appropriations bill that would bar the IRS from developing its own tax filing system. In the spring, Sasan Goodarzi, a former head of the TurboTax unit who took over as CEO of the entire company in January, sought to reassure employees.

"Our view is this will be in the press until there is a resolution with the IRS," he said, according to the video obtained by ProPublica. "And we're working with them and we feel very good about where this will end."

Doris Burke contributed research to this story.




All Comments: [-] | anchor

r0m4n0(4073) 2 days ago [-]

A few weeks ago I started an LLC in New York. I followed all the simple instructions online and the process was actually very straightforward and easy. I had an official digital copy of proof my LLC existed within an hour. I went to sleep with and thought that there is hope for our government.

Within a few days I received a letter from a private company informing me that unless I pay them $1000 to publish the name of my LLC in two public newspapers for six weeks, one daily newspaper and one weekly newspaper, in 120 days my LLC would need to cease doing business in NY. I pull out my phone and did a quick online search. Lo and behold, there is a law that says the same.

In this golden age, the government has chosen to require me to pay a private archaic company to print in ink thousands of times on paper the silly name of an LLC I just came up with, then proceed to distribute it by truck to a bunch of old people that then proceed to throw it in the garbage.

No fancy tax algorithms would be required for NY to simply drop this nonsensical requirement. Maybe this process exists because there are groups of people making a small fortune from it?

nickjj(2568) 2 days ago [-]

Don't forget about the publication certificate fee too:

> After you get done publishing, the newspaper will provide an Affidavit of Publication, and you send that to the New York Division of Corporations with a Certificate of Publication that carries a $50 filing fee.

I live in NY but never filed an LLC, but I Google'd that. I can't believe such nonsense exists to get money in a state that already has high state / sales taxes.

GeneralTspoon(4209) 2 days ago [-]

In the UK, if you want to close a company you must do the same (pay ~£300 to post notice in a paper that your company is closing).

I think the idea is that anyone who the company owes money to can file a claim before any remaining funds are distributed.

Of course there are much better and more cost-effective ways of doing this (even pre-internet). It's just absurd rent-seeking by unproductive leeches on the economy as far as I can tell.

nojvek(3909) 2 days ago [-]

There's a ton of this little silly companies delegated by government. It's akin to lobbying in a way. They depend on inefficiencies.

Best argument I've heard is "gotta keep people employed somehow, can't be too efficient"

0xffff2(3888) 2 days ago [-]

Maybe I'm missing something obvious, but why can't you just call up the newspapers directly and list whatever it is that the law requires?

delfinom(10000) 2 days ago [-]

Ah, this.

1. Please ignore almost all the mail you get initially. There are public companies that troll NYS business registration logs and send TONS of false and highly misleading if not illegal advertisements. 'You totally need this $900 OSHA poster you can just print for free and paste on the wall'. 'Ignore the fact we copied NYS' letterhead but aren't a governemnt entity', etc.

2. The NYS tax department will send you one real piece of mail as a questionnaire.

3. Yes, the newspaper requirement is real and fucking terrible. It's a very old holdover however politicians don't want to get rid of it because quite literally, it's a handout program for newspapers. It would make alot of 'local' papers nobody reads, close up shops.

4. Shop around, there are a few ways to save money on it. Honestly, I made a LLC 2 years ago in NYS and haven't even used it business wise. I have yet to do the publicatinon requirement. You just can't legally do business until you do so BUT it doesn't give a deadline.

justaguyhere(2279) 2 days ago [-]

I wonder how many laws/rules there are, that made sense long time ago but not anymore, or those that never made sense to begin with.

bashallah(10000) 2 days ago [-]

Don't falter that easy. If the law is stated so clearly, then start 3 more LLC's and offer the news posting service for $10 in your new publications that only publish new LLC's.

moftz(10000) 2 days ago [-]

In VA, if you want an alcohol license for your restaurant, you are required to print an ad in a local newspaper stating that you intend to get a license. I'm guessing it's so some prohibitionist can call up the ABC and complain? It's completely ridiculous. I wondering what will happen when print newspaper ceases to exist.

isbwkisbakadqv(10000) 2 days ago [-]

Every episode of "Nathan for You" is amazing. The episode about this phenomenon is no exception. "The Diarrhea Times"...lol

xivzgrev(3953) 2 days ago [-]

Credit karma tax is (actually) free. I've used for past 3 years.

Why is it free? Because credit karma uses your financial data (including tax returns) to make better credit card / loan recommendations to you. So it ultimately benefits me as well, that's why I share my return with them.

BUT you don't have to share your tax return with CK to file. There's a box you can uncheck when starting each year.

dentemple(10000) 2 days ago [-]

I gave Credit Karma a shot last year, and it worked out pretty great.

It's definitely a good alternative to Turbo Tax

nan0(4101) 2 days ago [-]

Who is going to build this 'free' tax filing system? I hope not the IRS [0]

0: https://spectrum.ieee.org/riskfactor/computing/it/irs-predic...

ratata(10000) 2 days ago [-]

There is a branch of government in charge of large software projects: https://www.usds.gov

n-exploit(3977) 2 days ago [-]

'On 17 April 2018, the final day for U.S. citizens to file 2017 tax returns, the U.S. Internal Revenue Service (IRS) suffered a major system failure related to the hardware supporting its 58-year old, 20-million-line Cobol-based Individual Master File system [PDF], which is still being used today to process the vast majority of individual tax returns. As a result of the failure, the IRS extended by a day the filing due date.'

Maybe that's because a decades-old system hit capacity, and has nothing to do with the system design?

flyosity(3872) 2 days ago [-]

The crux of this article is a PowerPoint slide from 12 years ago? Really?

hudibras(1743) 2 days ago [-]

If you hover your mouse cursor over the underlined words in the article (these are called 'hyperlinks' or, more informally, 'links') and then hit the left mouse button, your World Wide Web browser will display other documents which ProPublica used for their reporting.

ejstronge(3538) 2 days ago [-]

> The crux of this article is a PowerPoint slide from 12 years ago? Really?

The article definitely focuses on more than a single Powerpoint slide. In fact, in the same sentence you cite, there's a reference to a 2014 slide.

gjvnq(10000) 2 days ago [-]

Just for comparison: in Brazil the _Receita Federal_ (IRS equivalent) provides a free multi-platform* Java desktop app for filing your taxes.

The UI isn't great* and if you do complex stock trading it can be a pain to add the necessary information, but for the vast majority of people it works just fine.

It also shows if it is better to use the standard deduction or the itemized deductions for your particular case and also calculates your effective tax rate.

In fact, the Java App has for years been the only way to file personal income federal* tax. The paper forms were abolished because (almost) no one used them and they had a large rate of mistakes.

* The Java desktop app runs on Linux, macOS and Windows. (And possibly others) * There is a separate mobile app but I haven't actually used it. * In macOS there was bug in last year's version: it used ctrl instead of command for things like copy and paste. * There is no state income tax in Brazil.

Reedx(606) 2 days ago [-]

There are proposals in the US to do this.

I think the best I've seen so far is Andrew Yang's plan:

https://www.yang2020.com/policies/making-taxes-fun/

Opt in to have your taxes automatically done. National celebration of projects completed. Ability to specify where 1% of your taxes go.

So not only make it easy to do, but try to make it fun and shift the mindset from burden to societal contribution.

codeulike(2906) 2 days ago [-]

UK provides a web form. Pretty well done one considering the complexity. I think you can use your own software too if you like and theres some sort of submission api.

The only out-of-date thing I've noticed on the UK one is stuff like 'this PDF of your tax return is 1.5mb, on a dial up modem this might take 5 mins to download' etc. And I guess some people might still be on that kind of link so fair enough

edit: And yeah in the UK you dont have to file at all if your situation is simple (e.g. you just have one job and one source of income).

Nursie(10000) 2 days ago [-]

In Australia a sort of spreadsheet-like app is published each year. It pulls in data from a variety of sources (banks and employers mostly, IIRC), fills in everything it can and then lets you do the rest.

Windows only last time I did it (7 years ago?), things may have improved further since then.

xtracto(10000) 2 days ago [-]

Adding a data point with Mexico:

All invoices are digital. Every person who gets a TaxId also gets a Private/Public keypair and generating invoices means signing the document with that key. The amazing thing is that now accountants talk about the XML, which is the legal representation of the invoice.

As a result, every year if you file your taxes, you go I to the government freely provided website , get into your account and most of the data is there for the exercise, including deducted money that you can get back.

If you are a standard worker earning less than certain amount (very sensible, so the vast majority of workers fall in there) you don't even have to file your taxes.

That's one thing that is good about this country.

liziwizi(10000) 2 days ago [-]

I also want to brag about Tax-On-Web for Belgium. Development started over 15 years ago, it's been steadily improving over the years, it even works for linux users, it already fills in all tax details they already know (like wages)

I love it!

Illniyar(3400) 2 days ago [-]

In Israel employee taxes are calculated and paid by the employer.

Only self-employed, people with side-jobs or people who make over a certain threshold need to file their taxes manually (and in that case things are a bit complicated).

Healthcare, pensions and benefits are paid as part of the paycheck if you are employed (they are universal and mandatory, so not the same as in the U.S.), there are no other deductibles that a regular person can claim.

evsasse(10000) 2 days ago [-]

I've used the mobile app on Android, it wasn't a joy (it still lacks some polishing), but it was certainly VERY straightforward.

Personally I don't have lots of stuff to file, but I did everything under 1 hour, while still discovering how to use the app and learning a bunch of taxes nomenclature.

Cthulhu_(10000) 2 days ago [-]

Similar in NL; it used to be a desktop app (which IIRC was even distributed via diskettes or a CD-ROM but later just a download), nowadays it's a webapp. Also prefills most information from employers (who already pay your income tax for you every month), mortgages, banks, etc.

The only complexity for me came when I bought my house (some costs are deductible) and when I had to report Bitcoin held at Coinbase (and for that I had to report the (EUR) on January 1st 2018 specifically).

lqet(4209) 2 days ago [-]

Fun fact: the official tax filing application in Germany which has been around for a few decades is called 'ELSTER' (apronym for Elektronische Steuererklärung, 'Electronic Tax Return') which literally translates to 'MAGPIE', as in 'thieving magpie' ('Diebische Elster'). This is a form of self-irony which is quite unusual for the German tax offices.

OS portability has greatly improved with their online app [0], which is actually very good and easy to use. Before, the only stable way to run their Desktop application on Linux was through wine.

[0] https://www.elster.de/eportal/login

cjslep(4193) 2 days ago [-]

As an American living in Kanton Zürich, I used both (paid) TurboTax and (free) Privat Tax (straight from the cantonal government).

Swiss one was way easier to understand by far. As a side effect of doing those taxes, it was easy to see my actual wealth and my actual net income for the year. Which in America is a sin.

iagovar(4105) 2 days ago [-]

Similar case in Spain.

52-6F-62(3611) 2 days ago [-]

Canada doesn't release its own software, but it certifies a selection of free and paid software of varying complexity.

I'd like it if we could take up other, simpler models of sorting out income tax but I use software out of this pool† every year and its served me just fine:

https://www.canada.ca/en/revenue-agency/services/e-services/...

† usually StudioTax. I donated to them last year. It's not pretty software but it's effective and easy to use. https://www.studiotax.com/en/

'Our business model is similar to that of a street performer. You can use and enjoy our software and later decide if the experience was worth it and you can afford a donation. However, based on many users' feedback, you should be wary of recent free offerings from big commercial organizations. To use our previous analogy, they are big circuses that send their clowns out to the street to attract unsuspecting customers to their tents and pressure them in paying for their overpriced shows.'

trauco(2441) 2 days ago [-]

Same in Chile. If you have a regular, salaried job, your employer reports your earnings and all income taxes that have been paid, your bank reports your interest income, etc.

It takes less than 5 minutes to do your taxes unless you have unusual sources of income.

cascom(3494) 2 days ago [-]

I was under the impression - perhaps falsely - that taxes in Brazil were nightmarishly complex?

amq(3689) 2 days ago [-]

In Ukraine there is a government webapp where you can fill all possible tax forms, and they come mostly pre-filled. The only small hurdle is getting the private/public key, it's fast, but only happens upon request or if you have a bank account at supported banks. If you were only on a payroll, you don't need to declare anything at all, it happens automatically.

jib(10000) 2 days ago [-]

In Sweden, the government does the work for most normal cases. They send you a suggestion of "this is all we know about" and you just say "yeah, seems right" via SMS.

In Ireland you don't do anything in the usual case, unless you think it's wrong, everything is taxed at source by the employers/banks.

Both cases different if you're a company ofc.

Al-Khwarizmi(10000) 2 days ago [-]

In Spain it used to be a Java desktop app as well, the last incarnation of a software that started as a DOS program distributed in diskettes back in 1988. Now it's no longer a standalone program, but a website (not sure which technology, but it runs in current browsers without requiring Java installed, AFAIR).

The government makes a draft declaration for you, if it's OK you just have to digitally sign your agreement and provide a bank account to pay or receive your return (if you haven't already done that in the past). Otherwise you need to edit it adding whatever they've missed, but this is only needed if you do complex stock trading, operate with money abroad, etc. For most people it's just skimming through it in case there is an obvious error, and confirming the draft.

Once you confirm the draft, it takes only around 2-4 days for the tax return money to appear in your bank account.

The website's UI is good. Spanish e-administration webapps are a bit hit-and-miss, some are dreadful (digital certificate request, I'm looking at you), but that particular one is fine and has always worked well for me.

cesarb(3170) 2 days ago [-]

> Brazil the _Receita Federal_ (IRS equivalent) provides a free multi-platform Java desktop app for filing your taxes.

Other fun stuff about that desktop app...

* Originally, it was a DOS-only program (yes, it's that old). Then for a while, it was Windows-only, then for some time (probably due to the Linux users complaining) there was both the Windows-only version and the Java multi-platform version, and later the Windows-only version was discontinued.

* The original way of sending the output of this program to Receita Federal was to write it to a floppy disk and take it to a bank, which would read and upload it. Later functionality was added to send it through the Internet instead, and some time later the floppy disk option was removed.

vgetr(10000) 2 days ago [-]

Ultimately the problem is that the tax code is altogether too complex. I filed by hand up until a couple of years ago when I had to start accounting for things like capital gains and dividends from stock sales, and at that point it became too difficult and I started paying for TurboTax. At the end of the day they solve a problem that is entirely manufactured by government.

The main area where the system seems to be gamed from a software standpoint is that the federal government provides software if your income is below a certain threshold. More than that and you're on your own. _That_ has lobbying written all over it.

alexmingoia(10000) 2 days ago [-]

A flat income tax as the sole tax would eliminate tax filing complexity. It would also eliminate the inequality in the tax code (sales tax is a larger income tax on lower incomes, tax credits disproportionally benefit people in lower cost of living areas, etc.). A flat income tax is equal in that everyone pays the same rate, and fair in that the more money a person earns the more they pay.

djsumdog(1073) 2 days ago [-]

When I lived in New Zealand, you didn't have to file at all. You could create an account on a website with your tax ID and see all the reports employers sent in. You could requests corrections if things were wrong and have any return transferred to a normal bank account.

In Australia, they had a desktop application you could download, provided by the government. It wasn't pretty, but it was straight forward and took less than an hour to go through the whole thing.

I once met an Aussie at a security meetup who worked at immigration and I told him how surprised I was that I got my visa in like two days. He was like, 'oh I worked on that piece. Yea that use to take two weeks; but we were able to fix a lot of that process.' I wouldn't be surprised if Australia taxes were all web based and even faster today.

GordonS(567) 2 days ago [-]

Surely that's only for employees though, not for companies? This is how it works in several countries, including the UK - your employer tells the tax service about your salary and pension payments, and you only need to fill a tax return if you have 'something else' to tell them about, such as another source of income.

ttn(10000) 2 days ago [-]

Can someone from US explain how come there is still no open source alternative to TurboTax?

washadjeffmad(10000) 2 days ago [-]

The exact conditions that would enable a simple FOSS tax software or portal is what TurboTax has been lobbying to make sure stays broken and can't easily exist.

Also, taxes in the US are really complicated, and while our governments know how much we owe, the burden of determination and filing is placed wholly on the citizen.

Should we be able to log in to our IRS profile page and track and manage our taxes ourselves throughout the year? Yes. Is this techically possible today? Also yes. Would this make TurboTax and some other bloodsucking middlemen less TurboRich?

Well, you can see why it's impossible.

gok(671) 2 days ago [-]

There are dozens of alternatives. However, tax prep software needs to be certified for correctness, and that's expensive and onerous. This is made harder due to the many many different tax jurisdictions in the US.

JMTQp8lwXL(10000) 2 days ago [-]

The biggest problem is tax laws change annually. You can't write the open source software once and be done with it. It needs to be updated every year.

pwg(302) 2 days ago [-]

> Can someone from US explain how come there is still no open source alternative to TurboTax

Actually there is one (depending on your definition of 'alternate'): http://opentaxsolver.sourceforge.net/

I have used it for the last two years to do my Fed and State taxes.

cjoelrun(10000) 2 days ago [-]

Probably because accountants aren't silly enough to work for free.

robbiemitchell(3446) 2 days ago [-]

It takes work to keep up with changes to the tax laws in every state and at the federal level. I don't know many open source accountants.

danso(4) 2 days ago [-]

I'm a TurboTax user myself (b/c I almost always procrastinate to the last day/week), but my guess is because tax forms and rules change every year. Maybe the general situation for free filers is easy enough to code for, though? Though one of Turbotax's purported features is that it'll tell you how easy/complex your taxes can or should be, depending on your situation.

Also, one of TurboTax's features is e-filing. Maybe there are free e-file alternatives (as in free, with no upsell or strings-attached whatsoever), but it seems you have to be an 'authorized IRS e-file provider' to do this, i.e. there's not a developer API: https://www.irs.gov/filing/e-file-options

jrockway(3433) 2 days ago [-]

Typically open source projects come from two places; a company needs software, or an individual thinks writing it would be fun or at least save them time. Preparing individual tax returns falls into neither category; companies have no interest in individual tax returns (they don't pay that kind of tax), and individuals only have to do it once a year so it's not time-efficient to write a program to do it.

throwaway_law(10000) 2 days ago [-]

There are...you can download any tax form for free and fill it in by hand or in a PDF program.

You will see articles regularly attacking TurboTax because SV has been wildly unsuccessful in producing a startup that can compete with TurboTax, so instead a lot of effort is spent attacking them in the media.

Google, Microsoft, Facebook all spend significantly more on lobbying to ensure market hegemony. Don't be surprised at the pattern, attack TurboTax in the media, then up pops a new startup that will disrupt the industry. Rinse and repeat.

ac29(2955) 2 days ago [-]

From a previous HN discussion on US taxes: http://opentaxsolver.sourceforge.net/

I haven't used it, so cant comment on how well it works.

lincolnq(3790) 2 days ago [-]

Anyone want to start one?

The project is not easy. It requires carefully reading tax codes and implementing them, handling presumably thousands of explicit legal requirements plus whatever edge cases exist that may or may not be explicit in the law. Not to mention keeping up with annual changes. If you screw it up, whether or not you are legally liable, people are probably coming after you anyway. And the support nightmare...

That said, I think a programming-languages approach might be a good one to try to execute on this. Keeping the source code as close to the tax code as possible seems the only way to stay sane with respect to long-term maintenance.

paxys(10000) 2 days ago [-]

It isn't the software that is complicated, it's the intricate knowledge of the tax laws at the Federal and state level that goes behind building it. If you can find a team of lawyers and tax accountants willing to work on it for free (and update it every year) then building something like TurboTax wouldn't be too difficult.

Accujack(10000) 2 days ago [-]

There are 'tax help' open source programs out there, but online filing (which is the main goal here) is impossible without a corporation working with the IRS to set up access, negotiate protocols, and such.

There's no open standard/API/infrastructure or protocol for online filing, so the only option for an open source program to help with that is 'print out the whole thing and send it in' which is usually a lot more trouble (more expensive in time and money) than just paying Turbotax or similar services.

Turbotax and the rest of the tax prep industry have lobbied for years to prevent easy/free options for citizens to file taxes... it isn't just them preventing other companies from building products to do it, they're suppressing all competition and modernization, basically anything that would threaten the money they get from charging people to prep taxes.

At this point the entire system is set up with high barriers to entry, and open source solutions/free software don't typically work well in that kind of setting.

Essentially, no open source program has any chance of facilitating online filing in the US because of corruption in the US government.

HeWhoLurksLate(10000) 2 days ago [-]

Maybe I'm totally wrong here, but I don't see much of a difference between extorsionist tax filing software and scam callers warning me about the IRS suing me.

Can the government please make something that's easy to use? Filing taxes isn't something that requires a separate company to help with, and I see no reason why these products have a right to exist (and especially not be canonized), especially with how exploitive they are or can be.

Shivetya(569) 2 days ago [-]

How many pages of the supposed eight thousand plus pages of tax law apply to the majority of those who file taxes? The IRS certainly can tell me when I am wrong but they won't do it except when the result is in their favor; in my case I was improperly not filing HSA expenditures which would have resulted in me getting more back.

It comes down to this and I do not blame Turbo Tax or any other company in this area. The Federal and State Tax codes are weaponized by politicians and until that stops the system will be too complex for many to do the the filing on their own and some of that is because there is always that 'did I forget something' feeling.

At least the changes made recently helped the lower and middle classes out a lot, that doubling of the personal deduction was a god send in reducing the work needed as many will never need to itemize again

kansface(4138) 2 days ago [-]

> Can the government please make something that's easy to use?

Many law makers don't want the Government to make this process easy on the assumption that the easier it is for the Government to tax, the more it will do so.

cvhashim(10000) 2 days ago [-]

Tax lobbyists like Intuit benefit too much from all of this really. Planet money podcast episode that outlines some of this.

https://www.npr.org/sections/money/2017/03/22/521132960/epis...

Also Reply All did an episode recently on the Propublica Intuit story.

https://gimletmedia.com/shows/reply-all/6nhgol

sschueller(2698) 2 days ago [-]

I find it interesting that these scam callers warning about IRS suing only works in countries where making an honest mistake on your taxes can result you being threatened with or going to prison.

cpwright(3843) 2 days ago [-]

Filing taxes does not actually require a separate company to help with. You can download the PDFs (or go to the post office), read the directions and fill them out yourself.

I have never actually done that; but my parents did well after I was using software, which sure is more convenient.

mywittyname(10000) 2 days ago [-]

> I see no reason why these products have a right to exist (and especially not be canonized), especially with how exploitive they are or can be.

But if we remove the annual pain of dealing with taxes, then Americans will get complacent and may stop being bothered by having to pay taxes.

- The serious ideology of a major American political party vis-a-vis taxes.

skeptical900067(10000) 2 days ago [-]

I use the free fillable forms. Happy with it every year so far. If turbo tax makes them go away, I'll get real politically involved real quick.

jedimastert(4104) 2 days ago [-]

> Can the government please make something that's easy to use?

To be frank, the government has issues making things usable, and 'usable' is a pretty small target when you're talking about the entire US workforce.

MR4D(4064) 2 days ago [-]

Ok, I have a really dumb question...

Why does everyone say you can't file taxes for free? The IRS has a site for it here: https://www.irs.gov/filing/free-file-do-your-federal-taxes-f...

I get that Intuit is fighting tooth and nail to keep their business, but 'hiding' a page from google - seriously, is that to be considered evil now?

Seriously, I'm missing something in this conversation.

xivzgrev(3953) 2 days ago [-]

You CAN, if...

1) you are under a certain income threshold

2) you go seek out that particular page (vs searching free file in google and clicking on intuit)

Or you go directly to intuit and qualify as a "free filer" which you only qualify for if you have a dead simple return - w2 and no deductions or extra forms.

The problem isn't that you can't file free. The problem is that intuit is doing what it can to bury it while promoting its conveniently named "free" edition that bumps you up into a paid edition faster than you can say "what?".

twoodfin(3677) 2 days ago [-]

This seems to make the hn front page every 60 days or so. Is there anything new here?

elygre(10000) 2 days ago [-]

No, it's just still amusing to the rest of the world. We're like 'This is 2019, not 1920'.

mars4rp(3852) 2 days ago [-]

I am the developer in charge of CalFile, the free filing software that everyone asks for here for state of California.

https://www.ftb.ca.gov/file/ways-to-file/online/calfile/inde...

I am pushing so hard to make it better and more useful to more people, like being able to add capital gain that is not currently supported.

these are the push backs that I am getting:

1. Last time they tried to do what all of you ask, a ready return. TurboTax and others hired bunch of lobbyist and put so much pressure on people here and killed the project. People are still afraid and don't want to do too much to grab the attention of their lobbyist!

2. Use of CalFile goes down every year and if trend continues, it would be killed in the near future.

when you have your Federal return it is much easier to pay a little more and file your state tax with it too. No body even knows CalFile exist because we don't have a marketing budget to promote it.

3. Even people here buy the argument that free software exist and people could file their taxes for free.

imagiko(10000) 2 days ago [-]

I thought calfile was the go-to place for cal state taxes? I use it every year and I think it's a fairly nice tool

thorwasdfasdf(10000) 2 days ago [-]

You sir, are Awesome! Keep up the good fight!

Now that I know about this, I'm definately going to use it. Can I use it for federal too? or do I need to use something else for federal, then I can use CalFile for state.

thatfrenchguy(10000) 2 days ago [-]

Thanks for the info, will definitely use it next year !

spoontoeat(10000) 2 days ago [-]

Hats off to you, sir. Please don't stop the work you do. Thank you!

After college I quickly weaned myself off TurboTax in favor of filing taxes for free with CalFile & Free Fillable Forms with an occasional paper mailing of PDFs that I download and fill out. Much like my opposition to using proprietary word processors, my disdain of paid tax return services that own your data is deep and profound.

While a bit painful and limited in feature set, the mere existence of a free filing option is crucial to keeping Intuit from forcing me to pay to file my returns. I find Intuit to be a rather evil company. Filing a return is mandated by law with steep penalties. They intentionally keep the process complicated and convoluted. It's a plundering of everyone's time for unproductive work and overspending on something that should be quick and easy.

onlyrealcuzzo(4130) 2 days ago [-]

Is there anything like this for of the other big states like NY, Texas, and Florida?

musicale(10000) 2 days ago [-]

> 1. Last time they tried to do what all of you ask, a ready return. TurboTax and others hired bunch of lobbyist and put so much pressure on people here and killed the project. People are still afraid and don't want to do too much to grab the attention of their lobbyist!

This is insane. Where are the actual citizens at the negotiating table?

anigbrowl(71) 2 days ago [-]

Who is/are Turbotax's lobbyist/s? Who do they put most pressure on? It's hard to take action without knowing who the players are.

xivzgrev(3953) 2 days ago [-]

Dude I salute you. There are so many edge cases to support.

For number 2 can you import fed return with calfile?

IRS free returns going down too so you are not alone: https://taxpayeradvocate.irs.gov/Free_File_Program_Is_Failin...

This is a large reason why Prorepublica keeps digging at this - it's fucking crazy actual free filing is going down while these paid shenanigan products are holding strong with their ginormous marketing budgets.

komali2(10000) 2 days ago [-]

Can you expand on why people are 'afraid' of lobbyists? The way you write that makes it sound like the elected officials the lobbyists are trying to convince, are afraid of those same lobbyists. What I assume you meant is that ground troops like you are afraid lobbyists will notice your work and pay more attention to your given team of elected politicians.

Can you help me understand?

8ytecoder(4156) 2 days ago [-]

Given that open-source has given tough competition to every area of business software what, in your opinion, is the reason for a lack of open-source options to prepare taxes?

failrate(10000) 1 day ago [-]

I tell everyone I know about CalFile. It is awesome. You are awesome for working on it.

mixmastamyk(3510) 2 days ago [-]

This year it took me over a week of full-time study to file my federal taxes! Correspondingly, my Calfile submission was done in perhaps 20 minutes. Thank you for your service. Reminds me of filing in New Zealand, a breeze.

Question, the site and now you mention it does not support Capital gain/loss. I used it anyway, it asked for Adjusted Income from federal return, which takes those into account? So it appears to support it after all. Still confused about that.

(I refuse to share my financial info with a malicious third party, and continue to.)

One nitpick on the Calfile UI, it puts the 'Continue' buttons on the left instead of the right side, which seems odd for folks that read left to right.

ptrklly(4221) 2 days ago [-]

You're doing great work! I'd love to try it next year. If I do, what do you recommend for filing my federal?

excerionsforte(4062) 2 days ago [-]

Posted it on Twitter. I encourage others to do so too. #freepromotion

jandrese(4221) 2 days ago [-]

Virginia used to have free online filing until the tax prep industry got one of their people elected and killed it off. Now our state efile options are more expensive than federal efile options.

winrid(10000) 1 day ago [-]

Thanks for your work! I've used it and it is simple and just works. No fake loading bars to boot.

You literally have office politics. I salute you.

The linked site is not mobile friendly BTW as text is cut off (OnePlus 5 Chrome).

khuey(3714) 2 days ago [-]

Is CalFile an open source project that anyone can contribute to (and if so where do I go?) or is the FTB paying you to develop it as a closed-source product?

samnwa(4170) 1 day ago [-]

You are amazing for building and running this program. People like you are what give me hope that we can build a better government! Thanks for all you do!

GCA10(3564) 2 days ago [-]

I had no idea CalFile existed, and I've been filing taxes in California since 1997. Thanks for weighing in!

The Pro Publica reporting about TurboTax's efforts to suppress the free option is eye-opening.

CobrastanJorji(10000) 2 days ago [-]

Hey, your work is super interesting and I'm wondering if you have blog posts or papers somewhere that talk about the work involved in writing an application with complicated legal scenarios like this. I assume lawyers and tax professionals are involved with requirements and approving workflows, but I'd love to read about specifics.

fedups(4140) 2 days ago [-]

I like propublica's reporting on this but geez, this needs an award for one of the top HN obsessions. Within the past 7 months:

TurboTax's 20-Year Fight to Stop Americans from Filing Taxes for Free; 211 points -- 54 minutes ago

TurboTax to charge more lower-income customers; 144 points -- 3 months ago

Congress Scraps Provision to Restrict IRS from Competing with TurboTax; 82 points -- 4 months ago

Listen to TurboTax Lie to Get Out of Refunding Overcharged Customers; 171 points -- 5 months ago

TurboTax Uses a "Military Discount" to Trick Troops into Paying to File Taxes; 170 points -- 5 months ago

TurboTax and H&R Block Saw Free Tax Filing as a Threat; 355 points -- 6 months ago

TurboTax Hides Its Free File Page from Search Engines; 881 points -- 6 months ago

TurboTax Uses Dark Patterns to Trick You into Paying to File Your Taxes; 608 points -- 6 months ago

How the Maker of TurboTax Fought Free, Simple Tax Filing (2013); 462 points -- 7 months ago

edit: if you _still_ haven't gotten enough TurboTax reporting, there's a great Reply All episode that covers propublica's reporting as well https://gimletmedia.com/shows/reply-all/6nhgol

draw_down(10000) 2 days ago [-]

Come on

jjulius(10000) 2 days ago [-]

That's an average of 5.28 months ago, which would've been around the beginning of May, less than a month after taxes were due to be filed. Three of the articles are from 6 months ago (mid-April), and one from 7 months ago (before filing was due). It shouldn't come as a surprise that taxes, and the cost to file them, would be fresh on peoples minds at that time.

delfinom(10000) 2 days ago [-]

So what? Fuck TurboTax trying to rent seek mine and anyone's ability to pay taxes. What's next? Feudalism?

detcader(2348) 2 days ago [-]

It's an issue that affects almost every American, why not obsess? I've seen two stories in the past day about Google Nest which affects maybe 10%? What a bizarre obsession.

mixmastamyk(3510) 2 days ago [-]

This is a site for 'disrupting' anti-competitive bullshit, what did you expect?

grecy(2178) 2 days ago [-]

Americans are starting to wake up to the fact they live in what is by far the richest country, but that the systems in place just make the rich richer, while nothing is done to improve the lives of the middle and lower classes.

In this particular case, the rich are actively using the system to stop an improvement to the quality of life of regular people.

What's the point in living in the richest country in the world when your quality of life is lower than many countries that don't have nearly so much money?

ISL(1625) 2 days ago [-]

And yet the cost of implementing a government algorithm has not yet fallen to near-zero.

Furthermore, the government has surely implemented the algorithm itself in order to verify submitted tax returns, so the implementation cost of pre-computing everyone's tax should be comparatively small.

vonmoltke(2099) 2 days ago [-]

ProPublica seems to be singularly focused on Intuit to the exclusion of all the other powerful interests who benefit from a complex and opaque tax code. They may be fighting efforts to make filing easier under the current rules, but they aren't really doing anything to keep those rules a clusterfuck.

lidHanteyk(10000) 2 days ago [-]

Who else lobbies like Intuit? Who else spends millions of dollars on defeating measures intended to make paying taxes simpler and easier?

Ensorceled(10000) 2 days ago [-]

In this article on how Intuit is working to prevent free filing, ProPublica focuses on Intuit? Why is this surprising?

ProPublica seems really broad in the topics they cover: https://www.propublica.org/topics/

They also have multiple articles covering other types of tax reform and on closing tax loopholes if you search their database.

Aloha(2404) 2 days ago [-]

No one else really has the same lobbying might as intuit. Even HR Block is smaller (and targets a different market)

rolltiide(10000) 2 days ago [-]

Turbotax is my cheapest option to doing taxes.

My tax lawyer would cost me probably $40,000 to do the same thing just for the hours and their obsession with analyzing parallel legal theories and case law. I use my tax lawyer to construct the whole business model before any tax filings actually have to occur. Yes I could also consider a different lawyer and cpa for actually filing the tax return. They would also be more expensive than Turbotax and I would be paying two lawyer then.

Reading comprehension gives the biggest advantage with taxes. It is worth it for me.

The IRS is more of an ally than an adversary when you are using them to basically approve exactly how you wont pay them taxes, in advance.

Barrier of entry is pretty high to unlock that level, I dont mind, its what I always aspired to do.

danso(4) 2 days ago [-]

They've done various stories on the IRS and tax policy:

https://www.propublica.org/article/irs-political-dark-money-...

https://www.propublica.org/article/irs-now-audits-poor-ameri...

https://projects.propublica.org/nonprofits/

The reporting on Intuit has been going on for years, but the recent onslaught is likely because their recent initial report on Turbotax obfuscating Google search results led to federal and state investigations, and possibly attracted whistleblowers who passed along the inside documents that make up this newer report (i.e. the snowball effect)

Disclosure: I used to work at ProPublica years ago. My tenure overlapped the authors', but I never worked on these stories.

justin66(2881) 2 days ago [-]

> They may be fighting efforts to make filing easier under the current rules, but they aren't really doing anything to keep those rules a clusterfuck.

ProPublica is trusting the reader to understand that responsibility for the current state of US tax laws lies with the US government.

juped(10000) 2 days ago [-]

This topic hits people in the wallet and makes them stupid as a result - I think they're being sincere, not malicious.

danmg(10000) 2 days ago [-]

Most of the other people gaming the system are doing it to benefit themselves personally. Intuit is gaming the system so it can continue its rent-seeking behavior against the American tax payer at large. There is an element of direct unjust enrichment against the tax payer at large which is absent from other forms of tax code manipulation.

tombert(4137) 2 days ago [-]

I really don't understand why I have to file my taxes to begin with.

I have a W2 job, and my employer presumably has sent my paystubs and tax information to the government already. I then have to copy down the information from my W2 into some kind of tax filing software, and then they have to check to make sure that what I typed matches what they have, presumably paying someone to do that if I make any kind of serious mistake. Why do I have to do anything?

I'd understand if I have a lot of complex deductions or if I were running my own business, but as it stands I do the basic standard deduction (or whatever the default options on TurboTax are).

I'm genuinely asking this...is there some utility that I'm missing here to me doing the taxes myself?

pwinnski(10000) 2 days ago [-]

You are dealing with a consequence of TurboTax's 20-Year Fight to Stop Americans from Filing Taxes for Free.

There have been proposals to mail cards to taxpayers with their financial information (as known to the government) pre-printed, and if everything is accurate, the recipient does nothing at all. TurboTax (and a few others) lobbied heavily to make sure such efforts were defeated, and so you have to keep doing your own taxes.

But please, stop using TurboTax to do so!

crisnoble(3353) 2 days ago [-]

Think of it like a high school test, you submit your test to the IRS, who has the answers, and they tell you if you are correct. Also, the US is the only class who's teacher doesn't either show them the key or skip the test entirely.

When I realized that some other countries have no concept of a tax day and don't understand the yearly hype train of US taxes, that's when I got really bummed out on how broken our system is.

0xffff2(3888) 2 days ago [-]

It's not terribly difficult to construct circumstances in which the government doesn't know how much you owe them. It's simpler to have a single policy for everyone. Therefore, everyone has to do their own taxes. I really don't think it's more complicated than that.

djsumdog(1073) 2 days ago [-]

Nope. In New Zealand you log into a website and it shows all your employers filings. You can send in requests for corrections if things are wrong, but it shows you taxes paid and anything you owe the IRD. It has an interface for transferring money to/from your bank account for payments/returns.

It's a little more complex if you own a business, but I think you can file and maintain all your business taxes throughout the year using an IRD website as well.

The US IRS has all the information to do the exact same thing. It's lobbying from the tax software companies that's kept them from doing so.

u801e(10000) 2 days ago [-]

Free fillable forms[1] has been around for a number of years and van be used by anyone regardless of income level.

[1] https://en.m.wikipedia.org/wiki/Free_File#Free_File_Fillable...

grepthisab(4106) 2 days ago [-]

Haven't read the article yet, but I assume they're probably talking about the lobbying to remove free filing or TurboTax's attempts to hide the free option as much as possible with a thousand ways to upsell you by using dark patterns during the tax filing process.

justin66(2881) 2 days ago [-]

It does not have the dialogs to walk you through the weird parts of filing a tax return. It's not really comparable to something like TurboTax.

ejstronge(3538) 2 days ago [-]

> Free fillable forms[1] has been around for a number of years and van be used by anyone regardless of income level.

The article focuses on many issues, and the existence of such forms doesn't diminish from its importance.

JMTQp8lwXL(10000) 2 days ago [-]

Free File Fillable Forms is made by Intuit.

peter303(4220) 2 days ago [-]

IRS has your W2s and 1099s by February. It could compute and suggest your takes based on your previous year age & family profile. It already does this verify your filing.

dangus(10000) 2 days ago [-]

They are still an unnecessarily painful solution.

I did do these last year to finally kick TurboTax to the curb. However, they're shamefully complex especially considering how they're already a piece of software. They barely do basic arithmetic.

The restrictions are entirely artificial, and the system is gamed so that you either spend a lot of time learning tax form rules and lingo or you just pay TurboTax to do it for you.

The IRS could very easily send a pre-filled tax form for most Americans to verify and sign like other countries do. All the relevant information is already reported to the IRS. Only business owners and people with exotic tax situations would ever have to fill anything out - especially considering our now-higher standard deduction that most people don't hit.

Despite my complaints I still highly recommend everyone use free fillable forms. You'll actually understand what is happening with your taxes by the end of it. And without that understanding you have no idea if TurboTax is doing it right.

peter303(4220) 2 days ago [-]

You could miss forms like AMT and NET with FFF. Until recent tax reform I was borderline most years, so defer to tax software to determine if I had these taxes. ACA penalty a mess too.

mattferderer(10000) 2 days ago [-]

Sadly you can't even file a simple quarterly estimate tax return online without paying a fee. If you want to do it for free, you have to print it out & mail it. Not sure how that benefits citizens or the IRS.

Intuit's main argument has always been that if the government provided such a service many people would overpay their taxes.

Honestly it's much cheaper, easier & better to just find a good accountant to do your taxes instead of Turbo Tax.

The government could do a lot to make filing taxes & starting businesses easier. All the politicians talking about being pro-small business could really start at these 2 steps. Otherwise, imho, their talk is nothing but empty promises. Creating hurdles like this is just a way of gate-keeping lower incomes out of small businesses.

gwd(10000) 2 days ago [-]

> Intuit's main argument has always been that if the government provided such a service many people would overpay their taxes.

If that's true, then most people would use Intuit anyway, once they realized how much they'd save.

anticensor(4120) 1 day ago [-]

> if the government provided such a service many people would overpay their taxes.

This is tax payer's problem, not tax office's.

candyman(10000) 2 days ago [-]

I guess this is why some voters are eager for someone like Elizabeth Warren to get in office and stop some of this anti-competitive behavior. It's odd to me because TurboTax is good software (at least when I used it) and it saves time. So to me it's worth the money. But to force someone who as a job and no real deductions to not simply file a 1040 EZ online is just a rent seeker making sure they get paid.

Accujack(10000) 2 days ago [-]

>Elizabeth Warren to get in office and stop some of this anti-competitive behavior.

Warren is a run of the mill middle of the pack Democrat. She only looks good because at present the GOP looks so bad.

The government needs to be reformed, electing one party or another President isn't going to change much, unless you set the bar really low (Trump)

cat199(10000) 2 days ago [-]

Your statement:

> ... it saves time. So to me it's worth the money.

is (unless I'm missing some alternative), based a in comparison to paper-based forms, which are in turn a horribly inefficient way for the IRS (which all taxpayers pay for) to operate.

The question is not only the anti-competitive behavior of entrenched 'supported electronic filing services', but that these are 'required' and facilitated due to horribly inefficient processes that the IRS refuses to fix properly by simply creating a way for individuals to e-file in the first place.

js2(590) 2 days ago [-]

I recently learned about this volunteer maintained spreadsheet for helping with your 1040.

https://sites.google.com/site/excel1040/

snackematician(10000) 2 days ago [-]

I've been using this for last couple years and it's a great piece of 'software', I prefer it over TurboTax.

The dedication of the maintainer to this project is truly impressive.

I'm on linux but it works fine in the browser version of Excel.





Historical Discussions: Open-source apps removed from Google Play Store due to donation links (October 16, 2019: 1430 points)
WireGuard restored in Google Play Store after controversial removal (October 18, 2019: 36 points)

(1434) Open-source apps removed from Google Play Store due to donation links

1434 points 3 days ago by regecks in 3833rd position

lists.zx2c4.com | Estimated reading time – 2 minutes | comments | anchor

WireGuard removed from Google Play Store, rectification in progress

Jason A. Donenfeld Jason at zx2c4.com Wed Oct 16 10:48:21 CEST 2019


Hi,
After waiting several days for Google to review our app -- apparently
the process is manual now and they're quite backed up -- I was
delighted to learn this morning that the app was approved. Then, 20
minutes later, I received a letter saying that the entire app listing,
not just the latest version, has been removed and delisted from the
Play Store.
They said it was because we're in violation of their 'Payments
Policy', presumably because we have a link inside the app that opens
the user's web browser to wireguard.com/donations/. I appealed using
their website appeal form. Thirty minutes later (was this automated,
unlike the manual app review process?), I received a rejection of the
appeal.
I immediately made this commit -- f0bab44b -- and uploaded a new
version. We're now waiting for a new, presumably manual, review of the
app. Until then, it is entirely unavailable on the Play Store.
Sorry for the inconvenience. I'm sure many users are just as annoyed
as I am. In the interim, luckily F-Droid has our app.
I'll send an update once we're reinstated on the Play Store.
Regards,
Jason


More information about the WireGuard mailing list



All Comments: [-] | anchor

hnarn(10000) 3 days ago [-]

> Sorry for the inconvenience. I'm sure many users are just as annoyed as I am. In the interim, luckily F-Droid has our app.[1]

Thank heavens there's an alternate way of distributing apps on Android. Say what you want about Android vs. iOS, but at least there's an alternative if you as a consumer have a beef with Google Play.

[1]: https://f-droid.org/en/packages/com.wireguard.android/

degenerate(4106) 3 days ago [-]

Even if the Play Store and F-Droid both disappeared, the replacement could be a community-run open directory of APKs sorted by version. At least we have options on Android.

reacharavindh(3082) 3 days ago [-]

This is really unfortunate. If Google is ready to arm twist a popular project like Wireguard, smaller indie open source apps stand little to no chance than to bend over. This App store gatekeeping should be illegal. Selling the hardware/OS and marketplace by the same person is more like online dictatorship at this time.

Hamcha(10000) 3 days ago [-]

Honestly. I'm fine as long as sideloading 3rd party shops is not actively sabotaged. I tend to search stuff on fdroid first and Play store later, and that's something I just can't do on iOS.

unnouinceput(10000) 3 days ago [-]

Quote: 'They said it was because we're in violation of their 'Payments Policy', presumably because we have a link inside the app that opens the user's web browser to wireguard.com/donations/'

C'mon people, is Google we talk about, you know, those who bow to might $$$ only. You should've made a donation link INSIDE google, so they can charge their 30% before you get the money, that's how you keep your app in their store </sarcasm>.

Spivak(10000) 3 days ago [-]

I mean you're being sarcastic but this is how basically every other app accepts payments to support the developer. There are plenty of 'buy me a coffee' buttons in free apps.

edhelas(3987) 3 days ago [-]

My social-network/IM app (https://movim.eu/) got also removed from the Play Store because I had a link to the F-Droid app.

The Android app is basically a web-view pointing to one of the many instances of the social network https://nl.movim.eu/?login.

Regarding the time (and also money) spent to publish the APK on the Play Store and the hassle it is to discuss with the Google moderation team I didn't bothered putting it back.

kragen(10000) 3 days ago [-]

That's interesting — not a donation page, but an F-Droid page (which as far as I know doesn't accept donations)? When did this happen?

Santosh83(2075) 3 days ago [-]

A general comment: the need of the hour is to educate the billions of 'ordinary' computer users about the basics of software, privacy, security and trust. We must not fall into the narrative being parroted around off-late that considers end-users as pretty much dumb content consumers who should not be expected to ever know even the minimum about the underlying tech and therefore every decision needs to be made for them by a small minority of corporations, govts, and the tech elite acting as gatekeepers and overseers. This is a dangerous road to go down unwittingly. Too much de facto power is concentrated in the hands of the few while the rest are gradually trained into normalise being helplessly manipulated and told to defer to authority.

orbital-decay(10000) 3 days ago [-]

I've tried to educate many people on a personal level and failed miserably almost every time, being perceived and dismissed as an evangelist for some weird ideology they don't care about (and I took it very slow and tried really, really hard not to sound as one). Maybe they are right, after all? Who am I to dictate or even influence what they should care about, especially if they don't experience that clearly. People never become tech literate until it bites them in the ass. They have to face the consequences, that's how it works.

Meanwhile in 2019, most people still find 'nothing to hide' attitude perfectly reasonable.

peterwwillis(2589) 3 days ago [-]

What you're asking for is technology literacy, similar to science literacy. Adults generally don't care to learn this stuff. But you can teach the next generation.

lm28469(4222) 3 days ago [-]

> that considers end-users as pretty much dumb content consumers who should not be expected to ever know even the minimum about the underlying tech and therefore every decision needs to be made for them by a small minority of corporations, govts, and the tech elite acting as gatekeepers and overseers.

That's what we do in every other domains, what's the difference here ?

szczepano(10000) 3 days ago [-]

In my opinion it's not the problem of software but the problem of hardware. You can more easily write software then produce phone hardware. From consumer perspective it's quite easy to make custom computer and use custom os on it cause most of drivers are open and you can see there are plenty of computer operating systems in the world, on the other hand most of phones hardware and os is proprietary what was not an issue when we were using it for calling not using it as alternative to traditional computer. So first we need to define what is broken, from my perspective broken is phone hardware.

krageon(10000) 3 days ago [-]

> who should not be expected to ever know even the minimum about the underlying tech

I have worked in this industry for long enough to know that even assuming a minimum of knowledge at all is a bad idea. You can fight this fight as you want, but this comment has nothing in it to convince be otherwise.

IAmEveryone(3666) 3 days ago [-]

I know plenty about software, privacy, security, and trust.

And I'm happy with these stores and couldn't care less about your cyberpunk libertarianism.

This isn't 1984. The moment they seriously misbehave, the law will come at them, and customers will leave. Therefore they don't.

You're indulging in fantasies because you would like to believe that what interests you is all-important. That's normal: all epidemiologists belief we should talk about the flu a lot more.

But it's just a tiny slice of what matters. So don't disparage people as stupid just because they spend their energy and time with more important things than recompiling kernels.

hawski(3831) 3 days ago [-]

I like the way that the author of Simple Mobile Tools [0] made donations. Applications are free and open source, but there is a paid app Simple Thank You [1].

That's also how I was thinking about doing 'donationware' on a mobile app store.

Side note: They are also available on F-Droid.

[0] https://play.google.com/store/apps/collection/cluster?clp=ig...

[1] https://play.google.com/store/apps/details?id=com.simplemobi...

KingMachiavelli(10000) 3 days ago [-]

How is that better than just having an IAP donation through Google? Either way Google is taking their cut right?

Nux(2868) 3 days ago [-]

Yes, this is quite a good workaround, though probably less lucrative.

postsantum(10000) 3 days ago [-]

This is against policy. Not sure if it is enforced though

https://play.google.com/about/spam-min-functionality/min-fun...

chrisballinger(3976) 3 days ago [-]

I guess this will get lost in the noise, but a few years ago Apple relaxed its restrictions on IAP donations and allows developers to offer a 'tip jar' where IAPs don't unlock any specific features. I've been shipping a free and open source app with a donate button on the App Store since mid-2017 [1].

1. https://chatsecure.org/blog/sustainable-open-source-starts-w...

wavefunction(10000) 3 days ago [-]

Does Apple take their customary cut? I am not interested in 'donating' to Apple.

hosteur(10000) 3 days ago [-]

Interesting. As someone who considers making an open source app with a tip jar like that I would be very interested in hearing how much that actually nets you if you want to share that info (Maybe via PM if you prefer)

segfaultbuserr(2584) 3 days ago [-]

I use WireGuard daily since 2017, and it's one of my most favorite computer programs in the entire world - robust, reliable, state-of-art cryptography, and seamless integration into the system.

I've never donated to WireGuard before, and I didn't think about donation (I installed WireGuard from my package manager and didn't realize a donation link exists), until now.

Just donated 25 USD.

Thanks Jason. No thanks, Google.

neop1x(10000) 3 days ago [-]

Me too. Just donated. Awesome software.

irjustin(10000) 3 days ago [-]

I know this will be against deeply against HNews but I'm open to discussion.

I don't see anything wrong with this. Charging vs Donations, I don't see what the difference is. It's people paying money for good/services/software. The primary difference is the open ended pricing model.

Google, Apple put up networks and there's value in that network. Whether someone chooses to release their app for free or charge money for it, that app is dictated by the network fees. This network is extremely powerful in terms of its distribution reach. Does it warrant 30%? That's what the market is willing to put up with, but if we're debating 30% - it should be for the whole network not just because someone has an open ended pricing model.

If this app is registered as a non-profit (which it did not look like based on the website?) then I am 100% in the wrong and do believe the app should be able to operate under different pricing schemes.

At a philosophical level - what is the difference between a free-to-use app that has open ended pricing vs a FOSS with donations?

If there is a difference, let's discuss it.

usr1106(4202) 3 days ago [-]

It is useless to discuss market or even fairness as long Google and Apple are (de-facto) monopolies for the respective phone owner.

It's a major failure of governments to allow monopolies.

dessant(3373) 3 days ago [-]

There is no need for a philosophical debate, because Google Play In-app Billing simply doesn't support donations.

https://play.google.com/about/monetization-ads/

wtfwhateven(10000) 3 days ago [-]

>Charging vs Donations, I don't see what the difference is.

Charging: Person paying gets something in return Donating: Person paying does not get something in return

>The primary difference is the open ended pricing model.

What does 'an open ended pricing model' mean?

Terretta(1702) 3 days ago [-]

While it doesn't touch on donations, for the "value prop" angle I strongly suggest developers read and internalize Apple's point of view as expressed in "iOS App Store Principles and Practices":

https://www.apple.com/ios/app-store/principles-practices/

Also on the value prop front, worth noting a variety of digital app stores at scale have found operation costs 12% to 18% fees before the hands on "40% rejection" level of oversight that Apple is providing to end users.

eMGm4D0zgUAVXc7(10000) 3 days ago [-]

I have done open source development for well over a decade, thousands of hours, tens of thousands of lines of code, and I can tell you:

Considering the amount of donations I got, it would have been much more profitable to clean toilets, to just go begging in the streets, collect bottles from public trash cans for the deposit, or just sell my body.

In fact let's have a look how much I got during the past 12 months for 250 hours of work:

$0.

Had I been working as someone who climbs onto roofs and cleans the dirt off of them I would have gotten this much for that in this country:

$16500

And then, as a cherry on top, you now demand that a giant monopoly, which is trying very hard to soak up as much of people's data as possible - without paying them for it! - should get a cut from donations for my work which I genereally give away for free.

Now I can understand that you did not know this when you wrote down your opinion!

But given that you do now, do you understand how insulting this feels? :) It actually itched me so much that I created this account solely for the purpose of telling you, and I would be very happy if you could change your mind :)

yulaow(3421) 3 days ago [-]

In most legislations they are taxed (and deducted) very differently and must respect specific characteristics (eg donators must not receive nothing of value in exchange, must not be asked to donate a minimum amount or a repeated amount, etc)

esotericn(4222) 3 days ago [-]

You're stating, essentially, that anyone who wants to publish FOSS software with a donation link should be forced to go through some convoluted procedure and register as a charitable organization. (where? in the US? in their specific region? does it matter? how much time will it take for them to figure out? is this even possible in all regions? does this require ongoing effort e.g. accounting, document submission, general bureaucracy?)

This is simply not how the real world works. If it ever does, you've created a dystopia.

Google are being bureaucratic twats here. There's plainly and obviously a difference between WireGuard and the 10,000,000th iteration of some bullshit F2P casual game, and there's obviously a distinction between donations and purchases - the intent of the user, for one.

That they put prole-tier support on the case and 'accidentally or otherwise' sling the banhammer at pivotal pieces of software is their problem to figure out.

EGreg(1773) 3 days ago [-]

One of our apps (https://Yang2020.app) were rejected by Apple for the same reason - having a Donate link. Ultimately we removed the link on native apps (kept it on the Web) and the app was accepted.

However, before doing this, we corresponded with Apple Developer Support / App Review Team and asked them about the Acceptable Business Models of their developer guidelines, which states:

(vii) Apps may enable individual users to give a monetary gift to another individual without using in-app purchase, provided that (a) the gift is a completely optional choice by the giver, and (b) 100% of the funds go to the receiver of the gift. However, a gift that is connected to or associated at any point in time with receiving digital content or services must use in-app purchase.

They scheduled a "specialist" to call us from Apple and discuss this. However when they called a few days later, they repeatedly declined to give a single example of how this paragraph would ever apply. We tried asking many different ways, but they kept saying we would need to submit the app with the business model to get it reviewed. They would not give a single example of an app they approved with this business model. I personally was on the phone and said that we simply wanted to be compliant and it was very expensive to build something just to be rejected, so we wanted guidance about their policies. But the representative did not know anything. I asked if anyone else on their team knew but they basically said no.

So Apple seems to have certain "platitudes" in their Guidelines which companies can't take advantage of.

Has anyone here seen a single exception to the 30% cut? I know ApplePay may be used to purchase "real-life" items and where you can't afford to pay 30% less to the vendor. But what about services, like booking time with a teacher? What about digital services such as hosting a videoconference? Apple takes a 30% but you have to pay Twilio. That means any web-based app would cost 30% less!

We were hoping that maybe there can be some link to buy things with Ethereum and since Apple doesn't offer that facility we can try to argue that we need transactions to be non-reversible etc. Doesn't Apple allow apps that allow the user to send and receive bitcoin payments to some address? BOOM!

Another thing we can do is micropayments, maybe, and tell users how they can purchase credits elsewhere. But that last part is against Apple's rules, isn't it?

EGreg(1773) 3 days ago [-]

Screw the rules, I have cryptocurrency!

dessant(3373) 3 days ago [-]

WireGuard was reinstated after Jason has submitted a new version which does not contain a donation link. The issue has not been solved, and Google did not yet contact maintainers to clarify whether they would be allowed to have a donation link in WireGuard.

Several other open source apps are still missing from Google Play, including andOTP.

While WireGuard was bought back online, version 0.0.20191013 was briefly made available (with a donation link) before it got updated to version 0.0.20191016. Again, the current version of WireGuard does not contain a donation link.

This is the original version of the post, which has become inaccurate minutes after it was posted:

> WireGuard has been reinstated [1], and the donation link is in the app menu (version 0.0.20191013).

> Google has reversed its decision to remove an app that contains a link to a donation page, likely because of public scrutiny.

> I'm hopeful that people will continue to seek answers for why Google has begun to remove these open source apps, there are several of them mentioned in this thread alone. Please don't allow this to go unnoticed, the future of a bunch of less popular open source projects may depend on how this issue is perceived by the public, and addressed by Google.

> [1] https://play.google.com/store/apps/details?id=com.wireguard....

fouric(4109) 3 days ago [-]

If it really is because of public scrutiny, IMHO we shouldn't actually treat it as if it happened at all - a corporation that only responds to public pressure like this should be shamed and avoided.

(slightly expanded version at https://news.ycombinator.com/item?id=20829422)

zx2c4(1744) 2 days ago [-]

An update on this: they have reinstated the app WITHOUT the link, and there has been no change in policy, according to insiders. Officially, Google has not communicated at all with us about it.

neonate(2396) 3 days ago [-]

> Google has reversed its decision to remove an app that contains a link to a donation page

Are we sure that this is a decision reversal, as opposed to just reinstating one (or a few) apps?

zx2c4(1744) 3 days ago [-]

Could you double check 0.0.20191013? My phone just updated, and I'm seeing the APK version that has the donation link removed, 0.0.20191016.

I haven't received any further emails from Google (yet?), so I'm still uncertain as to whether I have any sort of green light to republish a version with the link.

MelodyConcerto(10000) 3 days ago [-]

At least they heard us this time.

Too often they refuse to listen when there isn't an angry mob at the door.

dessant(3373) 3 days ago [-]

It's disheartening to witness some of the arguments in this thread defending Google, saying that disallowing open source developers from linking to their Patreon pages is a legal requirement, because Google is legally liable (while the rest of the internet apparently isn't) for the payments that may happen on Patreon by following those links.

thayne(4220) 3 days ago [-]

But was it a version with the donation link removed?

jrockway(3433) 3 days ago [-]

I am not sure why it needs to be called a donation. Just let the in-app purchase unlock some cosmetics in the app like games do. You weren't going to deduct the donation from your taxes anyway.

I am curious how this ends up working out. Google takes a cut of the donations now... but the process of buying is probably easier. So maybe it will increase donation revenue, and make it easier to support free software.

(Taking a step back; it is pretty weird to have apps opening links to web pages that take your credit card number. The app store operators are right to view that with suspicion because I'm sure that for every legitimate open source project, there are a million phishing scams.)

amscanne(4215) 3 days ago [-]

> You weren't going to deduct the donation from your taxes anyway.

Unless I'm mistaken, this is only possible if you're donating to a registered charity. Many view open source as more noble, but OSS developers don't magically get a different rule book :)

(I don't disagree with your perspective here, but the prevailing sentiment in this thread seems to be that OSS developers should get a different rule book.)

ThrowawayR2(10000) 3 days ago [-]

> 'Just let the in-app purchase unlock some cosmetics in the app like games do.'

Doesn't Google take a whopping 30% cut of the purchase price for both in-app purchases or direct purchase of the app itself? That's a pretty big bite out of the donation.

siruncledrew(4060) 3 days ago [-]

An easy way to let people know how to donate without links is just use plaintext on a page.

At the bottom of an "About" page:

Paypal: email-adress

Venmo: open-source-proj-maker

BTC/ETH: <address>

If someone is not a 501(c)3 org and just a solo person, why do donations have to go through a particular website link?

It seems like the same thing (getting money) can be accomplished without needing to use links from an app.

heavyset_go(10000) 3 days ago [-]

> Just let the in-app purchase unlock some cosmetics in the app like games do.

I contribute to projects that are committed to never locking away features behind licenses or in-app purchases, because most open-source projects in that space eventually went down that route.

ryanmercer(3623) 3 days ago [-]

>I am not sure why it needs to be called a donation. Just let the in-app purchase unlock some cosmetics in the app like games do. You weren't going to deduct the donation from your taxes anyway.

My immediate thought at reading the title 'So sell a supporter animation that plays in an about menu'.

dessant(3373) 3 days ago [-]

Open source developers shouldn't be forced to lock away additional functionality in their apps, or resort to special tricks just to be able to receive donations for their projects.

Google could make a positive impact by introducing new features that help users seamlessly fund their favourite open source apps, right from their Google accounts. GitHub has recently introduced a similar initiative called GitHub Sponsors [1].

> Taking a step back; it is pretty weird to have apps opening links to web pages that take your credit card number.

There is nothing weird about accepting payments outside of Google Play, and it is allowed if the payment does not result in buying a product that is used within the app. Buying a movie ticket, or operating an online store with a third-party payment processor are perfectly fine use cases which are allowed on Google Play.

[1] https://github.com/sponsors

chx(812) 3 days ago [-]

Deleted. Wrong thread.

tadzik_(10000) 3 days ago [-]

Wrong thread? :)

Tepix(3968) 3 days ago [-]

If only there was an alternative to the Apple app store like F-Droid is for Google Play...

driverdan(1482) 3 days ago [-]

There is, F-Droid on Android. Stop using devices you don't have root access to.

Andrew_nenakhov(10000) 3 days ago [-]

Actually, F-Droid is not panacea. F-Droid apps do not have access to Push notifications. Coupled with crippled background app performance in recent Android version, it extremely limits certain classes of apps, especially all messengers.

nyuszika7h(4164) 3 days ago [-]

AltStore is a thing, assuming you have a Mac...

gallexme(10000) 3 days ago [-]

Does Cydia etc no longer exist?

trulyrandom(3805) 3 days ago [-]

Apparently andOTP was removed for the same reason a couple of days ago: https://github.com/andOTP/andOTP/issues/396.

As mentioned in the announcement, you can switch over to F-Droid (for the time being) to get the latest version: https://f-droid.org/en/packages/com.wireguard.android.

bubblethink(10000) 3 days ago [-]

>(for the time being)

Why do you have that caveat ? You can use f-droid forever.

Yolta(10000) 3 days ago [-]

I wouldn't want the app from any other place than Fdroid!

izacus(10000) 3 days ago [-]

For anyone else publishing mobile apps: Neither Apple nor Google allow bypassing the store's purchase system by linking to external donation pages. No matter the app type, it will result in rejection.

alt_f4(10000) 3 days ago [-]

that plus app store monopoly on each platform should mean several billion dollars in fines, if we actually enforced antitrust laws

dessant(3373) 3 days ago [-]

Google Play In-app Billing does not support donations.

> Here are some examples of products not currently supported by Google Play In-app Billing:

> One time-payments, including peer-to-peer payments, online auctions, and donations.

https://play.google.com/about/monetization-ads/

kevin_b_er(10000) 3 days ago [-]

They both say they allow it, but don't.

Both are lying to you and everyone else. They instead demand their private tax while lying to you about it beforehand.

NightlyDev(10000) 3 days ago [-]

Donation pages, maybe. But there is a lot of categories on the play store where you have to use your own payment method or can if you want to.

Some examples are purchases that can be used outside the app, eg. cross platform purchases and things like transportation and finance.

ismaildonmez(10000) 3 days ago [-]

The same app has the same link on its iOS app and just in the last week was updated twice.

tyfon(4215) 3 days ago [-]

I wonder what would happen if you link your your home page and you have a donation link there.

How deep does it have to be before they cut you off? Can you link to a page that is only linking to another page where the donation happens? How much extra text would you need to 'pass inspection'. I find these kinds of arbitrary rules very unsettling. It's definitely time to break up the tech giants.

It is odd that they remove this though, I use at least two more open source apps that asks for a donation and will send me to their donation page if I click.

supergilbert(10000) 3 days ago [-]

When is an alternative Android play store coming?

To all the entrepreneurs reading this: please create one. To all the VCs reading this: please fund one.

It's badly needed. I heard Huawei is working on one but I wouldn't bet on it.

phoe-krk(3078) 3 days ago [-]

F-Droid already exists.

zimbatm(3806) 3 days ago [-]

The problem of donation links from Google's perspective is that they are not getting their 30% cut. You might think that this is unfair, but the app is using their distribution platform.

It's possible to make open source applications and still ask for payment on Google Play, either as the asking price, subscription or by adding in-app payments for donations. This is the right way to do it, and will probably also have a much better return since the payment workflow is much easier.

blauditore(10000) 3 days ago [-]

Donations through the Play store's payment platform are prohibited, since all purchases need to give an additional value in return. The only reliable way to get donations in a compliant way is by linking to a website from the app and have a donation button there.

black_puppydog(4220) 3 days ago [-]

I can even go along your last point. But it's a different argument to say 'going through google is more efficient' or to say 'you can't not go through google'. Frankly, they're using their market dominant position to force free developers into a distribution model that serves them as the payment processor. I'd not go so far as to argue those are separate businesses, but it's really a blatant abuse of their power.

Arguing that this is somehow related to software quality, safety or whatever is pure BS> There is simply no other explanation for this than 'it will make us money and we can get away with it.'

fwn(10000) 3 days ago [-]

> This is the right way to do it

Might be convenient and great for Googles revenue, but it's the wrong way to do it if your priorities are to reduce the projects platform-dependence and to support the project with your full donation.

Usually, once I decide to contribute money to a project I want them to be as free as possible and I want them to get as much of that money as possible. Independent from the specific app store I pulled it from.

I think it's wrong that the Google Play Store treats donations like purchases.

gowthamgts12(4219) 3 days ago [-]

Going by your point, shouldn't Amazon be blocked since they are collecting the payment info and charging without any interference from Google? How are donation links being considered to be in violation? Just wondering!

dessant(3373) 3 days ago [-]

It's unfortunate that the developer resubmitted the app without the donation link, open source projects having a donation link in their apps does not violate Google Play policies.

> Here are some examples of products not currently supported by Google Play In-app Billing:

> One time-payments, including peer-to-peer payments, online auctions, and donations.

https://play.google.com/about/monetization-ads/

IshKebab(10000) 3 days ago [-]

My app was also removed a few months ago for this. In the email they say you have to use Google Play In-app Billing, however:

> Donations to 527 designated tax exempt organizations are also permitted.

So if Wireguard becomes a proper tax exempt organisation it would be ok.

If you want donations but aren't tax exempt there is basically no way to do it. However you could easily work around it by offering something trivial like a badge (games can use IAP to purchase trivial things like hats). The downside is that Google take their 30% cut.

iicc(10000) 3 days ago [-]

>Apps that employ in-store or in-app purchases must comply with the following guidelines

A donation is not a purchase.

zx2c4(1744) 3 days ago [-]

It indeed is unfortunate. But what other recourse did we have? The first objective, before we even consider donations or anything, is to make our software available and functional to our users. So we had to resubmit. And it's not like the resubmission process is instantaneous; as of writing, the app still isn't back in the store.

But anyway, now what? Let's say things return to normal, but we still want to put the silly link in the app. Our appeal was rejected. We could send letters, I suppose. Perhaps this article on HN will get the attention of the right people? We could appeal the appeal, maybe. And then appeal the appeal for the appeal. At what point does this become a waste of energy that could be spent coding and improving the app in other ways? It's hard to know where to draw the line and which battles are worth fighting. I'm happy to hear folks' opinions on the matter.

I certainly agree, though, that it is unfortunate.

mirages(10000) 3 days ago [-]

Can someone explains me why PayPal's app is then allowed per this policy ?

Crazyontap(3930) 3 days ago [-]

A lot of people justify Google and Apple's 30% because they provide an app store and distribution.

But Android is Android because of these apps. Without them it would be like the Windows phone which was nice but didn't get enough apps on it and so it failed.

These companies depend on our apps as much as we depend on their app store for distribution.

Fuck apple for starting this walled garden and creating a parasitic relationship rather than a symbiotic relationship.

bagacrap(4221) 3 days ago [-]

TBH Google Maps was the killer app that smartphones needed to get off the ground. Out of 13 apps on my home screen, 7 are Google apps. Both Apple and Google did a little more than building the framework and then sitting back...

So how do you decide what a fair cut is? Over the course of a month, given all your app and system usage, and all the pieces of software you've touched or relied on, what percent of the engineer hours required for that stuff to work came from Google/Apple vs coming from third party app makers? Is that how you decide the split? Because I have a feeling the time it took to build the 15 third party apps I might touch in a month don't come close to the level of effort required to have built the system and first party apps.

sjwright(10000) 3 days ago [-]

'Android is Android because of these apps' but 'Fuck Apple'?

I don't understand.

gorgoiler(10000) 3 days ago [-]

Ugh, app stores!

No one needs them for discovering new software. You go to the app store to get a specific piece of software you already know about and whose reputation you've already verified.

No one needs them for code signing. If you want the real firefox.apk you can download it from https://firefox.com/ and notice a lack of SSL alarms going off. What's the code signing for — to let me know that HTTPS works?!

All I really want is an OS facility that only executes code that has been installed explicitly by me, and always in a sandbox. That'll stop that dodgy email from executing r00t.sh via an exploit in my mail client.

App stores and code signing popups are proxies for that kind of OS facility, but I'd rather it was just part of the OS explicitly.

App stores leave me with all sorts of unwanted properties, namely Google / Apple throwing their unwanted opinion around. Search for Instagram in the iOS App Store: the top UI element is "InstaMap", thanks Apple!

LennyWhiteJr(10000) 3 days ago [-]

Code Signing and HTTPS are two completely unrelated concepts...

londons_explore(4200) 3 days ago [-]

> All I really want is an OS facility that only executes code that has been installed explicitly by me, and always in a sandbox.

Androids app sandbox is pretty leaky. It has a massive surface area, and the chances of finding exploits to get out of it to find root are high. Also, even from within the sandbox, there are lots of evil things you can do (for example, start on boot, and go fullscreen and use all the ram so the back and home button SystemUI process gets killed and your app is unkillable.).

The 'badness' of the android app model is a big chunk of the reason Google wants to be gatekeeper - otherwise android apps could just load and unload like webpages do. That and profit.

ur-whale(4149) 3 days ago [-]

>Ugh, app stores! >No one needs them for discovering new software

I so badly wish I could agree with you :)

Sadly, observing the non-tech folks around me, some of which don't even know what an app is, and some of which think the only apps they'll ever use on their phone is the ones it came loaded with ... it's not true.

What is needed is an android app. store that is decentralized / distributed, where the 'trust' of an app is a consensus-based score and that has the popularity of whatsapp.

pjmlp(298) 3 days ago [-]

Street Joe and Jane lack the technical skills to understand https://firefox.com/ from https://firefox,.com/, or what installing an app means.

'Isn't just clicking _next_ until the app starts?'

AnIdiotOnTheNet(3914) 3 days ago [-]

Hear Hear. I'll take that in a non-phone version too please.

dewey(1273) 3 days ago [-]

> No one needs them for discovering new software.

There's also people that don't know that they need 'Firefox', they just want a browser. Or a tool to print envelopes or crop an image. So they search in the app store and they know it'll probably be okay.

If you search for this online you'd get a bunch of SEO spam sites, shady tools among other things. At least due to the developer fee and reviews there's at least some barrier of entry (Also if it's not perfect).

> App stores leave me with all sorts of unwanted properties, namely Google / Apple throwing their unwanted opinion around. Search for Instagram in the iOS App Store: the top UI element is "InstaMap", thanks Apple!

Apple's search is notoriously bad and the perfect case to apply Hanlon's razor.

swiley(4061) 3 days ago [-]

SSL during download really isn't a substitute for code signing.

But we have fdroid and the GNU/Linux ecosystem is going to eat android's lunch anyway.

black_puppydog(4220) 3 days ago [-]

The very same thing happened to manyverse (android ssb client) the other day. Got de-listed, resubmitted without donation page, now it's back up.

How google tries to claim any sort of support for open source is beyond me. What open source app doesn't need a donation link? Those approved and funded by google don't, okay. But other than that?

segfaultbuserr(2584) 3 days ago [-]

The strangest fact is, those donation pages only have some links, there is no in-app payment. I don't understand what Google is thinking about when they banned donations...

Wait. I suddenly understand something I've seen previously...

There is a workaround! You simply upload two versions of the app, the free edition and the supporter edition. The support's edition is a paid app. You can ask the users who want to support you to install the paid app in your free app. The program itself is still 100% free and open source, there is no functionality restrictions in the free app, the donation notice can be turned off - just as a way to encourage donations. And it's nothing against the rules of the store.

giancarlostoro(3177) 3 days ago [-]

Seems to have happened to AndOTP:

https://news.ycombinator.com/item?id=21269542

This is ridiculous. May as well only publish open source apps on F-Droid and similar.

usrusr(10000) 3 days ago [-]

Oruxmaps even had a donate version that went through proper Google channels when the free version was forcibly removed either for a donation link in the app that was incompletely removed from the Play store build variant or because of a donation link on the product website. It never came back, now the paid version on Play is effectively a billboard 'you could save those dollars by going over to F-Droid where we are payment optional'.

sniuff(4175) 3 days ago [-]

Open source app can be compiled and installed without Google play store. If you are hosting on their platform, you adhere to their rules, you are not paying monthly for your free app download bandwith and storage do you ?

esotericn(4222) 3 days ago [-]

$50 sent, thanks Jason, no thanks Google.

A small amount relative to the value that you're giving to the world, but hopefully a few pints won't go amiss.

Fuck gatekeepers.

karambir(4033) 3 days ago [-]

Though I am not using android app, I just made a $10 donation. WireGuard has been a great software for me on Desktops.

Cheers

alias_neo(10000) 3 days ago [-]

I too made a 'fuck Google' donation to the WireGuard project today, after seeing an update and then the app disappearing entirely.

I'd love to use the F-Droid version but it's not up-to-date, and isn't published by Jason, so it's not signed by him either.

Publishing it on the website with a built in update-check might be one solution I'd be happy with, it'd only need a text file hosted on the website and a check, perhaps each time WG is activated to see if there is an update. Modern Android versions can be toggled to allow installing software from external sources, so the app could 'install itself'.

dang(179) 3 days ago [-]

We merged a similar thread, 'AndOTP Removed from the Google Play Store Due to Donation Link', into this one. Perhaps we'll update the title to cover both.

https://github.com/andOTP/andOTP/issues/396

https://news.ycombinator.com/item?id=21269542

dmix(1368) 3 days ago [-]

Just to add clarity these were two different apps that were taken down: Wireguard and AndOTP (among others mentioned in this thread)

Wireguard had 50k+ installs according to its (now dead) playstore page: https://webcache.googleusercontent.com/search?q=cache:kSsZsb...

AndOTP had 10k+ installs: https://webcache.googleusercontent.com/search?q=cache:buaKcF...

hanniabu(3932) 3 days ago [-]

Hmm, any idea how Google automatically catches a donation link? I feel like there's so many ways to do it that they must only be catching a handful.

zx2c4(1744) 2 days ago [-]

[Prior reply got detached in some cleanup, so adding again here.]

An update on the WireGuard situation: they have reinstated the app _without_ the donation link. From what I can tell from talking to Google insiders, there has been no change in policy. Officially, Google has not communicated at all with us about it, so we're not sure what's up. But at least our app is available again to users, which is great.

Fnoord(3929) 3 days ago [-]

Can we just stop distributing 'donation-ware' FOSS and use F-Droid for that instead? It is not as if Google Play Store is a credible source anyway given all the hostile content (for both privacy and security). Today I learned they even remove the orig. signature [1]

[1] https://news.ycombinator.com/item?id=21268717

bogwog(10000) 3 days ago [-]

I can see the headlines already

> Google Project Zero researchers find 100 ZERO-DAY vulnerabilities in unofficial app store 'FDroid'

yladiz(4120) 3 days ago [-]

Why? Google Play Store has a much larger audience and so the apps will be seen by many more people. Many people use Android and they also should be able to use FOSS apps, regardless of the platform's privacy and security issues. Is there a reason to not join the Play Store besides something philosophical?

dessant(3373) 3 days ago [-]

Google Play In-app Billing does not support donations [1], so open source developers should be allowed to link to their own donation pages. Google Play policies do not spell out that linking to donation platforms such as Patreon is forbidden.

Disallowing developers from using in-app billing for donations, while also barring them from linking to external donation platforms, puts open source projects in a difficult position, and appears to be overly hostile towards funding open source development.

Google must clarify their position on whether they allow donation links in apps published on Google Play. The app takedowns [2] of the last few days show that this is a significant development, and it is of great importance for all open source developers.

It's best to publicly document the messages you receive during app takedowns, since app store policies are often vague, it's easy for reviewers to make mistakes, and having a clear reference for these decisions also helps the public assess the validity of the takedown.

[1] https://play.google.com/about/monetization-ads/

> Here are some examples of products not currently supported by Google Play In-app Billing:

> One time-payments, including peer-to-peer payments, online auctions, and donations.

[2] https://news.ycombinator.com/item?id=21268389

enriquto(10000) 3 days ago [-]

> (...) puts open source projects in a difficult position, and appears to be overly hostile towards funding open source development.

This is the problem of developing free software on a non-free platform. They have absolute control of your work under their own terms, no matter how ridiculous they are. I never understood why the google play store is a socially acceptable idea for free software developers.

anilshanbhag(3675) 3 days ago [-]

In this case, can't you simply have a one time purchase instead of a donation ?

silentguy(4153) 3 days ago [-]

Why can't open source app developer have companion donation app like many freemium android apps do?

raxxorrax(10000) 2 days ago [-]

I just cannot comprehend how there are thousands of predatory apps on the store and these apps get banned.

Aside from that it should highlight problems with app stores as a model for software distriubtion in general, although I don't have much hope that this will happen in the immediate future.

That there even exists such a rule is ridiculous and unacceptable in my opinion.

chapium(10000) 3 days ago [-]

> Google Play In-app Billing does not support donations [1], so open source developers should be allowed to link to their own donation pages.

Ok, I had to stop there. They have no claim to this as long as they distribute within the store. It would be nice if google allowed this, but there is no obligation. App Stores are monopolies and should be regulated to allow open distribution.

krn(1834) 3 days ago [-]

> Disallowing developers from using in-app billing for donations, while also barring them from linking to external donation platforms, puts open source projects in a difficult position, and appears to be overly hostile towards funding open source development.

Google Play has been very clear about this[1]:

> Fund solicitations

> Examples: Donation solicitations from parties without a valid 501(c)(3) tax exempt status clearly displayed to the public; solicitations from parties without valid proof of exempt tax status or proof of registration with the relevant country's regulatory bodies and authorities; and political organizations that have registered with the Federal Election Commission (FEC)

For instance, Wikipedia displays an external 'Support Wikipedia' link in the main menu of its Android app[2] without violating Google Play's terms and conditions.

[1] https://support.google.com/faqs/answer/75724?hl=en

[2] https://i.imgur.com/MEut6yC.png

rozab(10000) 1 day ago [-]

Perhaps the solution would be to allow users to buy a 'virtual currency' in the app that actually doesn't do anything? It could be argued that this is a vanity item, like cosmetics, and so is exempt.

jammygit(3596) 3 days ago [-]

I don't believe cable tv, for how locked down and awful it is/was, ever went so far as to try and limit any spending outside their own platforms. Perhaps it was a lack of innovation, but the internet and the world of software licensing is gross by comparison

iamleppert(4220) 3 days ago [-]

In an ironic twist, our app that Google supported via Summer of Code was removed.

Whoever came up with this policy and started enforcing it at Google is completely tone-deaf, needs to be fired and publicly called out so we can set an example as a community that this sort of behavior will not be tolerated.

We have contacted engineering leadership at Google and they are in the process of figuring out who it is internally.

psychometry(10000) 3 days ago [-]

If you think that's an appropriate and proportional response to an employee making a mistake or misunderstanding an internal policy, I sincerely hope that you're not in charge of anyone at your day job.

dessant(3373) 3 days ago [-]

Please share more details about your app and its takedown, it's important to publicly document the issue.

ummonk(4170) 3 days ago [-]

As crazy as this policy is, firing and publicly shaming an employee is not the right response to a poor decision by an employee.

lawnchair_larry(2797) 3 days ago [-]

Yes, let's publicly shame and fire people over business decisions we disagree with. Cancel culture!

RedComet(10000) 3 days ago [-]

Ah yes, the multi-billion dollar megacorp trying to prevent goodwill donations. Dear regulators: BREAK UP GOOGLE.

jldugger(4183) 3 days ago [-]

You think a breakup will change anything?

panopticon(10000) 3 days ago [-]

To play devil's advocate, I'm sure Google's intent is to prevent garbage apps from spamming or tricking users into paying money and the resulting PR backlash from having an app store full of shady apps.

As evidence of this, Google does allow donations if the publisher is a 501(c)(3) (see https://support.google.com/faqs/answer/75724?hl=en).

teslafanboi(10000) 3 days ago [-]

go live in china you commie

kevin_thibedeau(10000) 3 days ago [-]

They're trying to prevent money laundering.

h4waii(10000) 3 days ago [-]

Who do you trust more, to secure their infrastructure? Google or F-Droid / a bunch of volunteers who may or may not be very security-conscious?

Not to downplay the effort that F-Droid contributors (or any Android FOSS groups) make, but it's a very legitimate concern.

Would an attacker have an easier time infiltrating Google or a member of F-Droid the development community?

Are users of F-Droid higher value than users of Google Play, to warrant a direct attack? If one wants to pick off users who are fairly technically inclined, other developers, system administrators, etc...F-Droid is a pretty good place to start.

vsviridov(10000) 3 days ago [-]

At least read up on how f-droid does their reproducible build infrastructure before you spread FUD.

behringer(10000) 3 days ago [-]

Easy answer. F-Droid.

yccheok(10000) 3 days ago [-]

Google (Or Apple) is pretty serious regarding their app stores rules and regulations.

Google owns the app store. So, developers should abide rules imposed by Google although the rules do not seem reasonable all the time.

My suggestion is, developer can still implement in-app billing, removing 'donation' wording. For the in-app item, make it as good-to-have feature like 'unlocking additional app color theme'.

With such, he will still abide the rule, yet able to get his monetary reward.

oarsinsync(10000) 3 days ago [-]

> Here are some examples of products not currently supported by Google Play In-app Billing:

> One time-payments, including peer-to-peer payments, online auctions, and donations.

Source: https://play.google.com/about/monetization-ads/

jumbopapa(4157) 3 days ago [-]

How does AndOTP compare to FreeOTP+?

mtgx(152) 3 days ago [-]

Aegis authenticator is much nicer than both. It also has some extra security features:

https://www.reddit.com/r/androidapps/comments/b45zrj/dev_aeg...

https://beem.dev/

djsumdog(1073) 3 days ago [-]

The Apple/Play stores are essentially a totally broken implementation of something that had existed for over a decade before: package repositories. It's embraced and extended Linux/BSD package/ports trees, except without things like the ability to add a 3rd party repository with other authors signing keys, pin certain versions, etc.

https://battlepenguin.com/tech/android-fragmentation/#packag...

We could go on about how regular users wouldn't understand or care and blah blah blah and we'll keep getting software that's neutered, tracks us and becomes more like the proverbial car with the hood welded shut; where people who know about cars have to attach several sidecars just to keep our vehicle doing what we want it to do.

baxtr(3108) 3 days ago [-]

"Totally broken implementation" is very close to BS, at least strongly exaggerated. They're both highly successful distribution models. Of course, every system has downsides. You're free not to participate

m-p-3(10000) 3 days ago [-]

Thank goodness F-Droid exists and allows additional repos to be added, and even put donation links in the app listing when possible.

kyle-rb(10000) 3 days ago [-]

Doesn't F-droid (and other stores) count as a '3rd party repository'?

caconym_(10000) 3 days ago [-]

The primary goal of package repositories is to make available the software users need. The primary goal of Apple/Play/whatever stores is to make money.

Steltek(10000) 3 days ago [-]

Don't conflate Apple's monopoly with Google's store. Anyone can go install F-Droid or Amazon or whatever on their Android phone. The only thing stopping people is satisfaction with the status quo and/or ignorance.

However, Apple totally rules their walled garden, with some added gross exploitation to boot (recent 'Sherlocking' story for Luna Display).

Pxtl(10000) 3 days ago [-]

I do wonder how crappy an opensource package repository would get if it faced the kind of sustained attack by con-men that Play & Apple have to deal with.

aranelsurion(10000) 3 days ago [-]

For a brief moment in time, there was an alternative in mobile space having best of the both worlds. Back then Nokia was still a thing, Meego/Maemo OSes on some -very few- Nokia devices had apt & rpm. Nokia repository would be there by default, and you could easily go and add any package repository through their UI. Just like good ol' Linux package managers, because it was one.

In fact this capability even enabled the community of these OSes to support the platform much longer than Nokia intended to. They made emulators, enabled hidden functionality (like FM Radio), even the bootloader was unlocked so you could go and install Android -but drivers were not open-source- if you wanted. It was liberating.

Too bad it got killed prematurely, and from its ashes Windows Phone was born.

ken(3749) 3 days ago [-]

Huh? The 'hood welded shut' analogy has always been about proprietary software. That's 99% of the Apple/Play stores already. We're already there.

whalesalad(351) 3 days ago [-]

Your analogy is certainly interesting... but expecting everyone to want or know how to look under the hood of their car is a little delusional. Most people don't even know or care to check their oil. They drive from A to B and only take their car in for service when a light is illuminated on the dash or if it's making a funny noise.

Personally, I do all my own maintenance and love to look under my hood. My wife? She'd drive a car until it was devoid of oil or coolant and the block seized up without even batting an eye lash. I'm not passing judgement here, just offering a more realistic viewpoint.

The same is true for users of apps. There's a reason the stores have grown to be so successful, and a reason the Google store is evolving to be more and more like Apple's.

I don't think that they are perfect, but I also would never consider the open source package systems to be perfect either. Everything has a trade off, you can't have your cake and eat it too.

UncleMeat(10000) 3 days ago [-]

Package repos don't provide analytics for developers, search for discovery, malware and abuse detection that scales to millions of apps, and loads of other features that the app stores offer. They aren't equivalent.





Historical Discussions: I just lost my wallet on the way home from work (October 15, 2019: 1380 points)
Someone lost his wallet. Finder got in touch via the guy's bank account (October 14, 2019: 1 points)

(1383) I just lost my wallet on the way home from work

1383 points 4 days ago by isp in 1006th position

twitter.com | Estimated reading time – 1 minutes | comments | anchor

Welcome home!

This timeline is where you'll spend most of your time, getting instant updates about what matters to you.

Tweets not working for you?

Hover over the profile pic and click the Following button to unfollow any account.

Say a lot with a little

When you see a Tweet you love, tap the heart — it lets the person who wrote it know you shared the love.

Spread the word

The fastest way to share someone else's Tweet with your followers is with a Retweet. Tap the icon to send it instantly.

Join the conversation

Add your thoughts about any Tweet with a Reply. Find a topic you're passionate about, and jump right in.

Learn the latest

Get instant insight into what people are talking about now.

Get more of what you love

Follow more accounts to get instant updates about topics you care about.

Find what's happening

See the latest conversations about any topic instantly.

Never miss a Moment

Catch up instantly on the best stories happening as they unfold.




All Comments: [-] | anchor

neor(10000) 3 days ago [-]

I found a wallet once. It has a bankcard in it, so I called the bank, telling them I had found the wallet of one of their customers and asking them if they could help me return in to the owner.

They weren't allowed to give contact details of the owner to me, but they were allowed to forward my contact details to the owner of the wallet.

Through this route we eventually got the wallet back to its owner. Took half a day to get everything in order, but man were they happy to get the wallet back.

jen729w(4183) 3 days ago [-]

Same find, in Australia. But they (Commonwealth) wouldn't entertain any swapping of details. I was just on the phone to the service centre and I thought all was lost ... but of course they said "just take it to any of our branches and we'll sort it out".

So I lose the satisfaction of closure — who knows what happened to that thing after I handed it over — but this does feel like the most-safe option.

chirau(1066) 4 days ago [-]

Yeah yeah, they coincided with a good one. I have lost 2 passports, 4 laptops and 6 cellphones on the subway in NYC. Half of those were literally stripped off of me. The other half I was drunk and negligent.

I don't believe in good samaritans. People only return wallets when they have no value or gains to be realized from them. The good folk leave it there, the ones with varied motives take it away.

Again, congrats for recovering your wallet. I'd never dream of it.

MayeulC(10000) 4 days ago [-]

I've seen some studies[1] that suggest wallets are returned more often if they contain money. Suggested reason: we tend to surestimate the value of money vs the other contents when we find a wallet.

[1] could be https://text.npr.org/s.php?sId=734141432

tonyedgecombe(3991) 3 days ago [-]

I hope it's not you that comes across something I've lost.

nostromo(3167) 4 days ago [-]

It reminds me of the automated collect calls of yore.

'You have a collect call from 'GAME IS OVER PICK ME UP MOM THANKS'. Would you accept the call?'

austinjp(4084) 4 days ago [-]

Or 'three rings means I'm staying over, any more means come pick me up'.

astura(10000) 4 days ago [-]

That's a classic Geico commercial, one of my favs.

https://youtu.be/jWPlfWwgFKI

97b683f8(10000) 4 days ago [-]

Another solution is to look up the BIN (bank account identifier, i.e. the 6 first digits of a credit card number):

https://binlist.net/

Then drop the wallet at the bank's closest office.

Symbiote(4197) 4 days ago [-]

Can't you just look at the big logo on the front of the card that says 'Halifax Bank' or whatever?

es-hn(10000) 4 days ago [-]

Coincidentally, I was able to return four peoples' IDs and other cards just this afternoon. Yesterday I was walking in East Oakland and found a pile of credit cards, debit cards, and IDs. Oakland Police Department wouldn't take them so I brought them home and started googling names and addresses.

Of the four IDs I found three belonged to adults; the third was a state ID for a minor related to one of the adults. I was able to find a contact number for all three, via relatives, and had reached them all by afternoon.

I learned that the person who'd dumped the cards had committed a series of car break-ins in San Jose yesterday morning and had fled to Oakland. They stole purses, laptops, and an iPhone. I wish OPD had taken an interest in the theft or at least in returning the stolen property.

djrogers(3621) 4 days ago [-]

> wish OPD had taken an interest in the theft

Sadly, in California most theft is merely a misdemeanor, and busy PDs do not have time to chase down criminals to give them a ticket.

droithomme(10000) 4 days ago [-]

> I wish OPD had taken an interest in the theft

Been there, in a different city, a few times. Police are somehow in collusion with the crime syndicate.

Don't trust police. Ever. All police are corrupt and can not be trusted. Are there exceptions? Only to the extent they don't actively commit crimes but stay silent and refuse to go against their criminal 'brothers in blue'. Which makes them complicit as well. As I just said, all police are corrupt. Including anyone replying to this to tell me they are the one exception.

es-hn(10000) 4 days ago [-]

Update for anyone who's interested:

After OPD refused the IDs/cards I contacted a handful of local news outlets and explained the situation. One of them, ABC 7, began reaching out, possibly incessantly, to the media relations office at OPD.

By 4:30 PM today I had a text from one of the ABC 7 producers telling me the OPD media relations officer wanted to get in touch and including her desk phone number. When I spoke with her she apologized, acknowledged that procedure had not been followed, and told me they were in touch with SJPD regarding the break-ins. She also told me she would follow up with OPD leadership.

Not sure how applicable this strategy would be in other communities/countries but it appears to have worked for me. I'll also note that I reached out to the mayor's office before contacting news orgs and got no response.

throw0101a(3223) 4 days ago [-]

While this is neat and kind, wouldn't it have been easier to see which debit / credit cards are in the wallet, and simply take the thing to the nearest branch?

detaro(2090) 3 days ago [-]

Easier than sending those transactions online, possibly from their phone?

astura(10000) 4 days ago [-]

Stupid American question - how does someone get your bank account information from your wallet in the UK?

chrisseaton(3077) 4 days ago [-]

You bank account information is written on your bank cards, which are in your wallet.

beefalo(10000) 4 days ago [-]

The bank account number is written on the card. European bank accounts make a lot more sense than in the US, your account number is a public number that can be used to receive money only and there is a transfer number for making payments.

epx(4215) 4 days ago [-]

If social media posts cost 0.01, things would be way better.

kart23(10000) 4 days ago [-]

this might actually be a decent idea for an instagram competitor. Sounds dystopian, but I'm sure a good marketing department could make it seem normal.

mburns(3875) 4 days ago [-]

Further incentivizing Influencer culture? Sounds awful.

lucb1e(2135) 4 days ago [-]

Email spam.

I would not think twice about a cent, but people would definitely think twice about spamming a mailing list or sending newsletters to people that you know don't want them.

badrequest(4054) 4 days ago [-]

Agreed, but let's not stop at a penny.

oil25(4078) 4 days ago [-]

Does Twitter not allow connection from the Tor network any more? What a depressing reality.

> 403 Forbidden: The server understood the request, but is refusing to fulfill it.

julianz(10000) 4 days ago [-]

Nah I think the tweet is gone, I can't see it from a browser. (edit, weirdly I tried again and it worked. Definitely errored with a strange error the first time)

journalctl(10000) 4 days ago [-]

This is neat! I once found someone's debit card on the street (in the US). It had a small face shot on it, so I found them on Facebook and sent them a message. Of course, Facebook had already started doing the "messages from non-friends are really hard to find lol" UI dark pattern, so like two years later the woman responds thanking me, but also saying she just canceled the card. Which is what I would've done, but I at least wanted to provide some peace of mind or closure that it wasn't stolen or anything.

umvi(4218) 4 days ago [-]

You can do a picture search on Facebook?

hinkley(4121) 4 days ago [-]

We had a woman visit a group I belong to from out of town, and I gave her and someone else a ride to a meeting. A couple hours later I found her debit card in my back seat.

I hate having to cancel cards. I'm still dealing with the echos of losing mine back in February[1] (right before moving!) and so I'm calling everyone in the group trying to get her contact info.

Turned out she hadn't even noticed it was gone yet.

[1] AT&T keeps silently cancelling my auto-pay on the new card and nobody can figure out why.

technofiend(10000) 4 days ago [-]

Nine hundred miles from home I spotted a driver's license in the snow from a chairlift on Snowmass Mountain. It was still there when I skiied back down. Imagine my surprise when it was for a neighbor who lived two blocks from me! It was only surpassed by my neighbor's surprise when I knocked on her door and handed her her son's driver's license.

nindalf(10000) 4 days ago [-]

> "messages from non-friends are really hard to find lol" UI dark pattern

You think you would do something different here? And you have considered all the possible ramifications of doing so? No doubt you've considered the impact on the number of spammy/scammy interactions that everyone experiences, the number of harassing messages that people around the world receive and you've made an informed trade-off between that and meaningful social interactions people have with the folks who find their wallet.

Seems like a pretty strong statement to call it a dark pattern, implying malice, when it could just be a good thing.

mav3rick(10000) 4 days ago [-]

'Dark pattern'. Do you know how many unsolicited messages women receive on Facebook ?

andreygrehov(2560) 4 days ago [-]

A few years ago, I found a wallet at a New York subway station. The wallet had some documents (including driver's license), weed and a note saying 'You are fool'. I found the guy on Facebook. He didn't know anything about the note, but we figured it was written by someone who found the wallet before me. He or she took all the cash and left a note, lol. I threw out weed and mailed the owner all of his documents. He was the happiest man in the world on that day.

abathur(10000) 4 days ago [-]

I've found quite a few things over the years. Phones. Car keys. Cards. Wallets. I've run after people who left purses, bags, chargers, and even a guy who straight-up forgot his laptop on a table. The vast majority of the time, these things happen in some establishment with a lost-and-found I can turn items over to.

I have, though, found a half dozen credit/debit cards in the past two years outside of any good place to leave them (parking lots/garages, streets, sidewalks...). I guess I'm a good-samaritan with a healthy dose of paranoia/skepticism/cynicism. When there's no obvious place that I think the owner will know to look, I tell someone I know that I found it, and promptly cut the card up into fine pieces, mix them, split the pieces into a few groups, and dispose of them in a 4+ trashcans in different locations (this is probably excessive, but my goal is to dispose of the card with as much caution as I would get rid of my own card if I knew it could still be charged to.)

I have weighed trying to return them, but I ended up triangulating against the chances I'll: waste time tracking down someone who has already canceled the card anyways, send the card to the wrong person, completely creep someone out by tracking them down, or get caught with a card that has been reported stolen and have a hard time proving I didn't steal it.

consp(10000) 4 days ago [-]

This is very nice use of the (now) mandatory message space for SEPA transfers!

Ah, this reminds me from the days when Dutch postbank transfers were instantaneous if not send to another bank. A friend and I did a chat with the short messages you could send. We even got a printout of all the 1 ct messages as that was still a thing back then.

OJFord(3422) 4 days ago [-]

It's (UK standard) 'Faster Payments' l, not SEPA. I don't know if it's mandatory, but all the (five that I recall) banks I've used since its introduction have supported the reference/message field. It's usually pretty instant, especially if intrabank.

artsyca(10000) 4 days ago [-]

Why couldn't they call his bank and get them to contact this fellow?

jandrese(4221) 4 days ago [-]

The bank might be suspicious that it's part of some kind of scam. Like they might trust the info more because it's coming from their bank's official number.

esseti(4203) 3 days ago [-]

Has anyone understood how this has been done? how can you place a message in a transaction like this?

If not mistaken this was done via contactless payment, right? how is that possible to add a message? (maybe you can in some country)

stef25(10000) 3 days ago [-]

When I send a payment I can either include a structured message (a code in the format xx-xxx-xxxx) or a 'free text' message in which I could easily write 'found your wallet contact me on [email protected]'

Liquid_Fire(10000) 3 days ago [-]

It's not a contactless payment, it's a bank transfer. The person who found the wallet sent payments of £0.01 to the owner. When you make bank transfers you can include a short payment reference which can be any text you want, and it appears on the other person's statement.

The UK has the Faster Payments Service which enables most bank transfers to be completed in a few seconds, so they would have seen the messages immediately as soon as they logged into their bank account to freeze the lost card.

lkrubner(792) 4 days ago [-]

In terms of the kindness of strangers, I'm reminded of a friend who decided she wanted to hitch hike around Mexico when she was 18. She was poor but she wanted adventures. She wanted to get away from the beaten path, away from tourist traps, and go out into the rural areas so she could discover what Mexico was really like.

Everyone who picked her up warned her that what she was doing was very dangerous. She was lucky, they said, to get a ride from them, and not the criminals all around them. She got ride after ride like that. Family after family picked her up and took her along on their trip and all of them said, "You are in danger! You are lucky we are the ones who picked you up!"

In this manner, she traveled around Mexico for a month, and she never had a dangerous ride.

lazyasciiart(10000) 3 days ago [-]

Another woman did that recently(ish) in Europe, not just for adventures but to prove the world was a safe place or something - she was murdered though https://en.wikipedia.org/wiki/Pippa_Bacca

mbar84(10000) 3 days ago [-]

This reminds me of a story about how a tribe in the amazon jungle picks their camp site. They check the surrounding trees to make sure they are in good health. I imagine you would have to spend 1000 nights next to a tree they would regard as unsafe, before there was even a chance for something to happen to you. When it happens though, your family will remember it.

jaco8(10000) 4 days ago [-]

In a bus traveling south of Mazatlan I met a family and they invited me to stay at their home in Mexico City. This place, as it turned out, was right in the middle of the largest slum and consisted of a metal frame with roofing and side walls made of tarpauline and a mud floor . The first few days there was a queue of people outside just to oogle the foreigner, but it was all good with lots of smiles. The morning teeth brushing and face washing ritual by young and old was fascinating to watch, with everyone awaiting their turn at the single faucet for hundreds of people.Food was simple, mostly made on small kerosene stoves. I stayed for 8 days and at no time did I feel unsafe , despite my host explaining about many pickpockets , expert thieves and robbers living in this area who went to work the tourist spots during the day , like other people would go to an office , but this was home. My host had a real job in the forestry department of the city government , but he said he could not afford to live else where.

tgsovlerkhgsel(10000) 4 days ago [-]

The problem with hitchhiking is that you don't pick someone at random, someone decides to pick you up.

If you pick a random person, they'll almost certainly be a normal, honest person.

If you advertise yourself with a giant sign 'I WILL GET IN YOUR CAR IF YOU STOP', then a small percentage of random strangers, but a large percentage of opportunistic criminals, will stop.

I'd argue she got very lucky.

zantana(10000) 3 days ago [-]

This is literally survivorship bias.

wil421(4074) 4 days ago [-]

They were picking her up to protect her. I've been in Central America and had these types of things happen, never foolish enough to hitchhike. Many times people on buses have explained these types of things to me. Where I need it watch out for my stuff, how to not get ripped off, and which places I should avoid. One example I can remember[1].

[1] https://www.washingtonpost.com/news/morning-mix/wp/2015/08/2...

EA(4217) 3 days ago [-]

Survivor's Bias...

ericmcer(10000) 4 days ago [-]

I frequently walked home from work in Oakland at 1am when I worked a night shift. I always wore headphones too and people would tell me I was being stupid. After about a year and a half of this I was coming up a hill to a four way intersection when someone popped out from behind a fence just before I passed it, grabbed my shirt and stuck a gun in my stomach.

He ended up being not the worst and let me keep my keys and license when I asked.

It only takes once, and the consequences for her would probably have been much worse.

Moru(3950) 3 days ago [-]

That's not how it works. People travel around like that the whole time I'm sure. If every one would end up dead after a month, there would not be anyone left. But that one time it happens, be it after 10 years or 20, you are still dead. Sucks to be that tiny percent that it happens to.

SI_Rob(10000) 4 days ago [-]

Every so often my pessimistic tendencies get a healthy slap across the face by the actions of someone who took the high road when they easily could have chosen otherwise.

It's a couple summers ago, at the height of the summer tourist season and I'm cycling up 4th ave SW in DC. Somewhere between the NASA HQ and the National Mall my camera bag - packed full with a fairly new DSLR, a few lenses, and a secondary cell phone - came unbuckled from my messenger bag and tumbled to the sidewalk. Probably a few thousand dollars of gear, not counting the considerable hassle of resetting 2fa and credentials for every possible account that could be tied to my phone (it was password-locked but I have no idea how well that would survive a determined attack).

I was booking it pretty hard trying to catch a metro, so I didn't notice the loss for a couple more blocks. After the only genuinely involuntary (and painful!) facepalm I've ever given myself, I hurried back home (lived in town close by) and immediately started cancelling every account when my main cell phone rang. The bag was waiting for me, all contents undisturbed and intact, in a hotel lobby a couple blocks away. An anonymous samaritan had picked it up, brought it in and gave it to the concierge without a word, then walked away. Concierge called me using my contact info in a business card that was also in the bag.

In one instant, some unsung karmic superhero single-handedly erased the work of several hundred asshole double parking jobs.

Al-Khwarizmi(10000) 3 days ago [-]

A few weeks ago, my partner and I, together with our 1-month-old baby, were rushing to a courthouse as she needed to do some paperwork for a case she's involved in. We were under high pressure as the courthouse would close in like half an hour, but our son was especially hungry that day and we had to stop several times to feed him, etc.

In one of these stops, probably due to the stress and rush, we left her handbag behind. We noticed when we got to the courthouse (with around twenty minutes left until it closed). This not only implied losing her credit cards, house keys, healthcare card, some money and her phone, but also her ID card which was needed for the paperwork in court. The deadline for the procedure was the next day so we might still have a chance by going to the police and asking for a temporary ID or something, but I'm not even totally sure it would be possible, and in any case it would have been an absolute mess and we would have spent the next morning running here and there.

One or two minutes after we realized the handbag was missing and we were feeling like crap and powerless to do anything, my phone rings. Someone had used my partner's phone [1] to call her mother (because 'Mama' is usually a reliable contact, I guess) and she told them to call me. I met the finders in a nearby square five minutes later. They returned everything, and refused any compensation. We completed the paperwork in time that day.

They will probably never know how big a favor they did to us (of course returning cards, phone, money, ID, etc. is always a big favor, but in this case it was even more important than normal due to the court issue, having very little time due to a newborn baby, etc.) and how grateful we are.

[1] In case you are wondering, indeed she didn't have her phone locked with a PIN number, pattern or similar... I know, this goes against every security recommendation. It's just laziness. In this case, it probably helped by accelerating the return, though.

qxfys(10000) 3 days ago [-]

I live in Sweden. Lost my wallet during my bike to work time in the morning. Not sure why it fell from my pocket, but it might because of the bumpy road due to some construction.

As most of the things inside the wallet can be replaced in a couple of days (bank card, and some ID's), I did not worry that much. But, a nice guy from the neighboring office took the high road by calling the contact point of my office, which ended up in my boss hand.

faith in humanity restored. :-)

epx(4215) 4 days ago [-]

Sometimes it happens, and sometimes not. I lost my wallet 4 times in total, 3 in Brazil (which is $HOME), 1 in Germany. Got it back 3 out of 4 times. Guess where.

roenxi(10000) 4 days ago [-]

There is an interesting selection bias at work.

1) If someone on the street approaches you, they are probably up to something.

2) If you randomly select someone on the street you have probably picked a moral and upstanding person who would love to do you a good turn.

Because of (1) people tend to be a lot more defensive than they need to be when assessing what the average stranger is likely to do.

FineTralfazz(10000) 4 days ago [-]

While I was in college, there was a summer I couldn't find any tech work and ended up doing property claims for a tour company. It paid minimum wage, but it was pretty fun. What really surprised me was how many valuable items (camera bags full of lenses and an expensive-looking camera, CPAP machines, etc.) we would find with absolutely no identifying information whatsoever. We were able to track down some of the owners, but not all of them. If you're traveling, you should definitely have your name and contact information not just on, but inside your bags. It could come in handy someday.

njovin(10000) 4 days ago [-]

Not too long ago after a series of unfortunate events I spent the night on the floor of the international terminal of the Atlanta airport. The next morning I (exhausted) was riding the tram back to my departing flight's terminal and noticed a kid left a little dinosaur action figure on the tram. I didn't notice until the next stop. I grabbed the toy, hopped off the tram, sped-walked back to the last stop, and walked through a handful of gates looking for the kid.

In the end I didn't find him and left the toy on a counter, hoping he'd find it. The whole thing bummed me out. My own kid's lost a few toys and I know how devastating it can be. Granted I was sleep deprived and emotional, but it put a damper on my day nonetheless.

Everyone I know who's worked in restaurants or as a delivery driver tips very well. In my limited experience, teachers make some of the best parents when it comes to student interaction and parent participation.

I'd bet whoever found the wallet had lost their own at some point. Empathy is a powerful, powerful thing. I wish that more people would recognize that and work to instill it in their children.

js2(590) 4 days ago [-]

I left a MacBook Pro behind in the bin at a security checkpoint at Miami International Airport a few years back. I realized it when I got home, looked up the airport's lost & found number, called them, and they had it. I was shocked. They held it for my dad to pick up.

erikbye(4212) 3 days ago [-]

Why is every top comment on HN a personal anecdote?

shaneprrlt(10000) 4 days ago [-]

> In one instant, some unsung karmic superhero single-handedly erased the work of several hundred asshole double parking jobs.

Perfect balance. Yin and yang.

DoreenMichele(203) 4 days ago [-]

I turned in more than one lost wallet or debit card to a librarian (or similar) while homeless.

I didn't want to call them myself or whatever because I was homeless and I knew I was automatically suspect as a thief because of it. So I wanted someone the world would trust/believe to deal with it.

Well before I was homeless, attempts to return wallets or found ID cards or whatever were consistently met with suspicion. One woman tried to run from me and my husband on the assumption that we were up to no good for trying to catch her because we happened to see her drop her wallet just before she got in her car. But, obviously, we must have been muggers or something. No other explanation was possible in her mind.

phendrenad2(10000) 3 days ago [-]

A lot of people don't want to become a criminal by taking a bag that doesn't belong to them. Not exactly shocking.

alexis_fr(10000) 4 days ago [-]

Is it actually legal? In France, a bag in a public place is deemed to be a bomb and the military has to come, secure the area and destroy it. Happens on a routine basis, not even worth the newspapers. You don't have such laws in Washington DC?

ojosilva(3474) 3 days ago [-]

Someone registered and paid for a new Netflix account with my gmail address plus a point somewhere, which Gmail ignores. So I logged into the Netflix account, thinking it was actually mine and this was some misunderstanding. I realized that was not the case, so I to tried to get in touch with the person but no other info was available, everything was done from his phone apparently according to the account logs. I figured that reporting it to support would not do much since they too did not have his info and no phone number was on record, they would cancel the account and maybe the amount paid would be lost.

Finally the only way I found to notify the user was by creating/modifying Netflix users with usernames as short telegraphic messages such as 'you registered', 'using my email', '*@gmail.com is mine','contact me or change it'. That message would be visible as the user opened up any of the Netflix apps or web app. It apparently worked as a couple days later my email stopped being primary on the account and never got any Netflix emails at that gmail inbox.

peter303(4220) 3 days ago [-]

gmail has this weird bug that if I am [email protected], I also get email from [email protected]

sleepychu(4112) 3 days ago [-]

How did you log into it? By changing the password?

chris_overseas(2670) 3 days ago [-]

I have this problem too - only, it happens a LOT. I've had people use my Gmail address but without the '.' to sign up (or try to) for Playstation accounts, Apple accounts, Paypal, Skype, Bank accounts, plus it's pretty common that I receive wills and other sensitive legal documents, property deals, you name it. The legal docs etc are likely due to other people mishearing/mistyping someone's email which I can understand, but I'm constantly baffled by how often people must type in their OWN email address incorrectly for important situations too.

Often I can just ignore the signup confirmation emails and it doesn't go any further, but it's surprising how often there is no confirmation link sent out and the account gets opened without verification of the email address. At that point, it can be incredibly difficult to get the problem rectified. Most companies don't seem to have any procedure in place to deal with this situation - Many online companies only have an easy way to contact them if you are one of their customers, so I often have to resort to resetting that user's password to get into their account and use the company's contact form to try and get things straightened out from there. Sometimes even that isn't possible, e.g. because 2FA prevents me from resetting the password, so about the best I can do is unsubscribe from their marketing emails and leave the account active.

donkeyd(4216) 3 days ago [-]

I have this same issue with one of my junk email addresses. It's pretty short, it's a famous person's last name (I was a fan), but it's not a very common last name. I've received all kinds of stuff, from many people, but one lady in particular. I only recently noticed this and when checking, I found out that this had been going on since 2008. She's used my email for: car service appointments, plane tickets, Walmart orders, forwards from her work email, turbo tax, customer support, change.org, hotel reservations, etc. Pretty much every aspect of her life has ended up in that mailbox.

I'm truly happy for her that I'm not a malicious person and when I figured it out I sent her a message. I hope she manages to stop using my email for stuff.

zantana(10000) 3 days ago [-]

It does make a case for not using common user names. I used to always try to use my first name in the early days of the Internet, but it was almost always taken.

The one time I got it was with one of the Maxis online accounts and for the next couple of years I seemed to get a bunch of password reset requests from people who thought it was their account.

While there's something to be said for not using the same user name across different systems using a very common name is not a mistake I'm going to make again.

einpoklum(10000) 4 days ago [-]

Life lesson:

Keep a note with your email address in your wallet - clearly marked 'wallet owner's email'.

Paranoid version of the life lesson:

Keep a note with an anonymized email address, with forwarding set up for the message to go to your actual mailbox, in your wallet.

simonebrunozzi(460) 4 days ago [-]

> Advanced/paranoid version of the life lesson:

This! You never know.

lolc(10000) 4 days ago [-]

I stick my email-address on all valuables I could conceivably lose. Just last month the police sent me a message about a camera found in the road with my address on it. I'd been wondering for weeks where I misplaced it. It must have fallen off when I passed there with my bicycle.

The poor thing must have seen at least one bout of rain and a lot of sun judging by the ablated jacket. And it still works perfectly!

caf(2911) 4 days ago [-]

Even better - put a photo or even a drawing of your face on the note as well, to try and evoke some empathy from the finder, increasing the chances it'll be returned to you.

periferral(4153) 4 days ago [-]

I assume your drivers license is in your wallet. You are concerned about anonymized email?

floatrock(4103) 4 days ago [-]

I hate it when my cynicism alarm goes off, but this feel-good tweet about hacking payment transfers for good is written by a guy who's twitter bio says he's a PM at Transferwise ('We're building the best way to move money around the world.')

All the replies are people contrasting shitty experiences they've had with their own wallets and money transfer services.

His top pinned tweet is:

> Reasons to work at @TransferWise:

> 1. Irreversibly change the world of finance to be fairer

> 2. Excellent office dogs [video of cute dogs running around]

I mean, I'd love this to be true, but the coincidences are uncanny.

dmix(1368) 4 days ago [-]

If it wasn't a real story it's a brilliant bit of marketing.

But this may be more commonly known 'hack' in the UK with their banking system than it appears.

braythwayt(956) 4 days ago [-]

Be all that as it may, in a world where people use their Wi-Fi SSIDs to say nasty things to their neighbours, I totally buy that this story is plausible.

lucb1e(2135) 4 days ago [-]

Isn't this a success story of a regular bank account though? I would find this super far-fetched as some sort of weird promotion, especially where OP's company is never mentioned and not actually posed as better than a regular bank account (or a regular bank account as bad).

Let's just enjoy nice stories that we come across.

chrisseaton(3077) 4 days ago [-]

But this is a standard bank account and transfer? This is the old way of doing things, not his new way.

smcl(3924) 4 days ago [-]

Don't ignore your bullshit-ometer on social media platforms, it's rarely wrong.

MildlySerious(10000) 4 days ago [-]

It's not noteworthy all of itself, things get unproportional amounts of attention all the time on the internet, but I find it interesting that this story then ended up on the very top of HN

timcameron(10000) 4 days ago [-]

Hi, I'm the guy who lost his wallet! I agree it's a convenient overlap and understand your cynicism but I assure you that it's genuine. I can even provide you with the receipt from me riding a jump bike around for an hour looking for my wallet! :)

EDIT: spelling + I did pin the tweet about transferwise after it blew up, and I also changed my bio to share my small side project: Podmast.com - check it out :)

woutr_be(10000) 4 days ago [-]

To be a bit more cynical, every time I see those generic 'change the world' and cheeky 'office dogs' things listed as a reason to work there, I can't help think; 'you really couldn't come up with any meaningful reasons to work there?'

Maybe I've already gone through enough shitty startups to not fall for it anymore, or maybe I'm just a cynical guy in general.

bpsh(10000) 4 days ago [-]

Just want to chime in to tell any prospective new Transferwise customer that my experience with the company has been dreadful. They can for no reason deactivate your account with funds still remaining. That's not the problem though, it is that their appeals teams takes over 2 weeks to respond per message. Yep, waited 3 weeks to get an answer for deactivation. Another 3 weeks to convince them it was an error. Now it has been a month since I sent in my supporting documents and no reply yet. Think I might have to file a formal complaint at this point.

dang(179) 4 days ago [-]

There's a softer version of this intuition: a guy who works for Transferwise would be more likely to write a blog post about something like this when it happened to him.

incompatible(3530) 4 days ago [-]

Hacking payment transfers to send messages can be used for 'good or evil'. I recently read about somebody using it to harass an ex-partner.

puranjay(3888) 4 days ago [-]

I understand the cynicism, but at this point, Transferwise isn't a scrappy startup and I can't see any reason why they'd want to pull off such a PR stunt. They're valued at $3.5B. $3.5B companies don't 'fake lose' their wallets to get to the top of HN

wyxuan(10000) 4 days ago [-]

Transfer wise does forex not traditional bank transfers like this

patio11(954) 4 days ago [-]

For what it is worth, I saw this story and thought 'Oh, I love this; creative use of metadata in the payments system', principally because I am professionally involved with payments. All the 'coincidences' here are explainable by 'people who geek out on X will tend to see a lot more X in their daily lives than civilians and will become focal points for other people who geek out on X, who see a lot more X in their daily lives than civilians.' You can trivially reproduce 'OMG banks!' on any social media feed you control; just kvetch in public about a banking story.

Also, there are people on HN who do marketing for a living (checks business cards including me?), and to the extent that you model someone as going into their planning meeting with 'OK guys I have this great idea for how to hit our Q4 numbers: lets do stealthmarketing on our personal Twitter accounts', you should know that that would not receive the reply 'Genius Bob, why do we employ teams of expensive professionals to do this when you come up with such pearls of wisdom for free.'

Mentioning this half because excessive cynicism is a failure mode for geeks and half because many people on HN will, at some point in their life, actually have to get good at causing people to adopt a product or service, and HNers should correct the mental model that would suggest that this is in any way aimed at or effective at causing adoption of a product or service.

hayyyyydos(10000) 4 days ago [-]

Interesting that there's an 18 character limit. In Australia, with the NPP (equivalent to UK's Faster Payments), your description can be up to 280 characters including, of course, emoji.

Out of curiosity, does Faster Payments allow you to pay to any other identifier (email address, phone number), or only bank account number? More importantly... does it support emoji?

Liquid_Fire(10000) 3 days ago [-]

> does Faster Payments allow you to pay to any other identifier (email address, phone number)

There is a system called Paym which lets you send money to (the bank account associated with) a phone number, but you have to sign up for it explicitly to associate your phone number with your account, and I personally have never used it nor heard of anyone I know using it.

> does it support emoji?

I believe it's limited to some subset of uppercase ASCII only. However, some banks allow emoji for transfers to other accounts within the same bank (e.g. Monzo).

debt(4089) 4 days ago [-]

What's a 'reference'? You can enter arbitrary info during a transaction?

Genuinely curious. Is this a UK thing or available everywhere?

hansjorg(3925) 4 days ago [-]

If you're a European teenager transfering money to someone you know, it's customary to enter something like 'Payment for the drugs and the firearms'.

quickthrower2(1314) 4 days ago [-]

You can in the US. Just turn the check over and write on the other side :-)

chrisseaton(3077) 4 days ago [-]

When you pay someone, how do you let them know what it's for without adding a customer number, or a note saying 'thanks for dinner' or something? That's a reference.

markdown(3418) 4 days ago [-]

It's always been a thing in the countries I've banked at: Fiji, New Zealand, Australia.

detaro(2090) 4 days ago [-]

At the very least with european SEPA transfers and their predecessors, but I would have assumed pretty much all bank transfers where you send money support some way to put a message - how else would you map incoming transactions from previously unknown senders to what they're for?

Scarblac(10000) 4 days ago [-]

In the Netherlands, for at least the last 35 years, transaction forms have a field 'Description' where you can enter arbitrary text. Originally on paper, now electronically.

Is that in any way remarkable?!

jagged-chisel(10000) 4 days ago [-]

'Reference', 'memo', 'note' - some referrer or reminder about the transaction in question. Invoice number, PO number, 'movie tickets' ... it's just a free-form text field.

smoe(10000) 4 days ago [-]

Wouldn't it be much easier for the finder to just get in touch with the bank and ask them to notify the owner?

I once got a call/mail from my bank giving me the contact details of someone who had found my lost keys that had an RSA SecureID token generator thing on them and the finder called the hotline imprinted on it.

I don't remember much details around it since it happend in the mid 2000s.

wues(10000) 4 days ago [-]

No it wouldn't. Sending three or four transfers from mobile banking app is for sure faster than finding bank number, calling them, listening to the info that the conversation will be recorded, listening to the menu, then listening to music interrupted with 'we will be with you shortly' etc. I guess I could send 20 transfers at least during this time.

crimsonalucard(10000) 4 days ago [-]

Only in rich countries.

Rebelgecko(4177) 4 days ago [-]

Even in poor countries, a wallet with money in it is actually more likely to be returned to its owner than an empty one

https://www.npr.org/2019/06/20/734141432/what-dropping-17-00...

marcosdumay(4221) 4 days ago [-]

A while ago, there was a fashion of making lost wallet studies around the world. It looks like every factor is more relevant than country (time of day, wallet stile, place, phone number being available, anything), except for those countries where people absolutely do not let things out of place and those where people think everything is a bomb.

dang(179) 4 days ago [-]

Would you please stop posting flamebait to HN?

We detached this subthread from https://news.ycombinator.com/item?id=21263246 and marked it off-topic.

tim333(1252) 3 days ago [-]

It's a shame Apple makes it so hard to return their devices. I've tried with a locked phone and AirPods but thought they have serial numbers and probably Apple's databases have owner info they totally refuse to forward any 'it's been found' type messages.

pflenker(10000) 3 days ago [-]

I think with 'find my phone', it's possible to display a message on the phone - provided it is switched on and connected.

midvar(10000) 4 days ago [-]

Am i reading too much into this?

The twitter user just so happens to works for a company that uses the technology the 'good Samaritan' used. I just glanced at who the guy was, and found it funny that he works in the industry. Who knows, maybe that's why he found it so awesome.

'In April 2018, TransferWise joined Faster Payments as the first non-bank payment service provider to be a directly connected settling participant,[23] after being the first of its kind to gain access to Bank of England's Real-Time Gross Settlement (RTGS) system.[24]'

https://en.wikipedia.org/wiki/Faster_Payments_Service

'Money transfer firm TransferWise has become the first non-bank company with direct access to Britain's Faster Payments Scheme, which the start-up said on Wednesday would help it compete with banks on sending money overseas.'

https://uk.reuters.com/article/uk-transferwise-britain/start...

peterburkimsher(3328) 4 days ago [-]

It is coincidental that the person who lost their wallet is working in that industry, but it probably helped the story go viral instead of just disappearing.

I happen to be friends with the guy who found the wallet - we met at the CouchSurfing meetup in Kaohsiung, when he visited Taiwan.

This was his personal reaction:

'I found a wallet in the middle of the road while cycling home from work today. I couldn't find the guy on Facebook, but one of his bank cards had his sort code and account number printed on it, so I got in touch by transferring him 4p and including my phone number in the payment references. It worked, and he tweeted about it, and it's gone viral! Unfortunately so have I (with a cold) but this is making me feel much better'

'A fitting end to perhaps the strangest day of my life. More reflection tomorrow once I've wrapped my head round it all x' (with a link to a video from 7News Australia about it).

Symbiote(4197) 4 days ago [-]

The system has existed for over a decade, and the slower predecessor for over 50 years. It's a normal thing in Britain, and doesn't need to be advertised.

I don't think his work has affected his reaction here.

madeofpalk(3874) 4 days ago [-]

'the technology the 'good Samaritan' used' is just what anyone in the UK uses to send money to someone else. It's been standardised for like... forever, and is nothing special or worth astroturfing for.

p0llard(4135) 4 days ago [-]

Yes.

Pretty much any domestic bank transfer initiated electronically via mobile/online banking will be cleared through Faster Payments up to a limit of about £25,000 ($32,000) for most banks (the theoretical limit is an order of magnitude higher), making it the de facto way of transferring money in the UK. When I was a student, bank transfers by Faster Payments were more common than cash payments for settling small debts between friends.

In addition, BACS/CHAPS transfers (the other two main electronic clearing systems in the UK) typically incur fees, and would normally only be used if the value exceeds the limit for Faster Payments. In addition, Faster Payments normally clears 'instantly' (i.e. in seconds); it's literally the most convenient way I know of to transfer money domestically in the UK, and is accordingly ubiquitous: cash requires you to have exact change and have cash to hand, which is increasingly rare in Europe; BTC addresses are too long compared to a UK account number/sort code.

Having recently moved to the US for a year, I've noticed that the financial institutions tend to be years/decades behind Europe in terms of electronic payment processing and clearing.

JenBarb(10000) 4 days ago [-]

A few weeks ago I found a working iPhone X on the side on the sidewalk. I thought about using siri to call a recent contact and try to get it back to the owner, but I found the language was set to Chinese (which I cannot read or speak) and I gave up and put it back on the sidewalk in case the owner came back.

In retrospect, I don't think I could have done anything even if the language was English apart from wait for the owner to call their phone. That said, I'm curious if anyone has any suggestions about how I could have been a better Samaritan.

njx(3570) 4 days ago [-]

You could have given it to some near by store saying someone dropped it and maybe he or she may come looking for it..

Long time ago, I came out from Denny's during lunch and just after walking on the right edge of the parking lot, found a bundle of 100 dollar bills. There were several restaurants in the same location. I walked into the same Dennys and gave it to the cashier saying I found it in the parking lot and probably belongs to someone who might have visited here and then left. Next day I get a call from Dennys saying the owner of the cash wants to thank you and reward you. I said thanks but not needed.

ghaff(3499) 4 days ago [-]

Probably contact Apple with the serial number.

astura(10000) 4 days ago [-]

Drop it off at the local police station.

faeyanpiraat(10000) 4 days ago [-]

Use google translate text-to-speech function, and make your phone ask the other phone in Chinese to 'call mom'.

isp(1006) 4 days ago [-]

See also ancient bash.org lore: http://bash.org/?814243

blunte(4160) 4 days ago [-]

When I see bash.org, my first thought is, 'I put on my robe and wizard hat.'

degenerate(4106) 4 days ago [-]

Would be amazing if that's where this guy got the idea, from a Bash.org post in 2000

mileycyrusXOXO(4089) 3 days ago [-]

I forgot about bash.org - there goes 2 hours. Well spent.

tobr(2386) 4 days ago [-]

My bank only seems to allow 12 characters.

Why is that? What am I even supposed to do with 12 characters?

journalctl(10000) 4 days ago [-]

It's enough for a phone number, at least!

lucb1e(2135) 4 days ago [-]

Mine allows about an original tweet in length, but forces a line break in the middle, and it disallows dangerous characters. Just the usual suspects that always break everything and cause database tables to be dropped or commands to be injected. Exclamation marks are super dangerous.

I never understood this.

codezero(1798) 4 days ago [-]

send them the name of a file in your very old version of DOS?

wruza(10000) 4 days ago [-]

Beware of lost wallet frauds.

At some countries they drop a wallet on the road, wait for you to find it and then appear and claim that few hundred bucks (or a local equivalent) are missing. First they try to threat you, and then resort to the police. Your fingerprints are on the wallet.

If you want to help, make sure you have a stranger witness at least, who is willing to help, or better look for a policeman nearby.

wruza(10000) 4 days ago [-]

Similar fraud: a guy who looks like a bad guy "finds" the wallet near you, makes sure you seen it and then inclines to split it. "Don't even try to call cops!", take your cut, etc. If you take it, even out of confusion, bad guy vanishes and their friends come and claim they seen all in reverse and that you took most of the money. Banknotes with known serials / pen marks are in your pocket, threats, police.

EpicEng(3973) 4 days ago [-]

>First they try to threat you, and then resort to the police

How could they pull this off? It's 100% he said, she said. What court is going to see such a case through and find in favor of the fraudster?

spoondan(4144) 4 days ago [-]

Hmm. That's neat. But usually someone has a bank card or other membership card with a support line on it. Is there some reason to not just call one to report the found wallet?

The times I've found a wallet, I've called one of those support lines. The support person was able to contact their customer to give them my phone number. The customer called/texted me, and we arranged for them to pickup the wallet. This has worked every time to get the wallet back to its owner within a couple hours of my finding it with minimal inconvenience to me.

royce(10000) 4 days ago [-]

It depends on the organization. Some larger financial institutions are sometimes more likely - or even bound by internal policy - to simply trigger their standard card-cancellation procedures.

Instead, one could present the financial institution with the hypothetical question, and then only give the specifics if the institution's policies are non-destructive.

If the finder is a decent sort, and has the time to try alternatives, I'd rather get the cards back intact than to have to do the cancellation dance.

lewisjoe(3181) 4 days ago [-]

The first thing I do with a new wallet is to put a piece of paper in it containing my name and phone number. It stays there forever useless, until it gets lost. I stole the idea from a reddit thread.

thrower123(3005) 4 days ago [-]

This is a perfect use case for those free address stickers that you get in the mail.

m-i-l(3025) 4 days ago [-]

Nice hack.

The 'official' way of dealing with lost property in the UK is to take it to your local police station. If no-one claims it within 6 weeks, you might be able to keep the item. I certainly remember doing this a few times when younger. Not so sure many people know this nowadays.

Anyway, my mother lost her purse a few months back in the small town in Scotland where she lives. Someone suggested she try her local police station (actually not so local any more thanks to the current government having closed over 600 police stations[0]) and lo and behold someone had handed it in there!

[0] https://www.thetimes.co.uk/article/600-police-stations-shut-...

tapland(10000) 4 days ago [-]

Going to a local bank also works. And you want to do that to get their cards replaced/blocked as well. And they have the owners contact details.

OJFord(3422) 4 days ago [-]

The one time I tried, Bobby wasn't interested!

I think my father managed to find the owner from the wallet's contents in the end.

marai2(4007) 4 days ago [-]

Two weeks ago in Maui at the visitor center at Mt. Haleakala a distraught young woman came in saying she had dropped her wallet somewhere between the visitor center and the peak. When she went to the women's bathroom in the visitor center to check if she might have dropped it there, she found someone else had forgotten their iPhone by the sink. Within a few minutes an older couple walked into the visitor center and said they had found a wallet on a hike but had lost their iPhone :-) Two individuals each finding the others lost item was pretty karmic!

mlthoughts2018(2828) 4 days ago [-]

and at that moment the volcano erupted..





Historical Discussions: My Favourite Git Commit (October 18, 2019: 1191 points)
My Favourite Git Commit (October 11, 2019: 5 points)

(1225) My Favourite Git Commit

1225 points 1 day ago by robin_reala in 31st position

fatbusinessman.com | Estimated reading time – 6 minutes | comments | anchor

I like Git commit messages. Used well, I think they're one of the most powerful tools available to document a codebase over its lifetime. I'd like to illustrate that by showing you my favourite ever Git commit.

This commit is from my time at the Government Digital Service, working on GOV.UK. It's from a developer by the name of Dan Carley, and it has the rather unassuming name of "Convert template to US-ASCII to fix error".

A quick aside: one of the benefits of coding in the open, as practised at GDS, is that it's possible to share examples like this outside the organisation that produced them. I'm not sure who first introduced that idea to GDS – it was well-established by the time I joined – but I'm forever grateful to them.

Why I like this commit

I've lost count of the number of times I've shared this as an example of what commit messages can do. It's fun because of the ratio of commit message to code change, but that's not why I think it's worth sharing.

In a different organisation, from a different developer, this entire commit message might have been change whitespace, or fix bug, or (depending on the team's culture) some less than flattering opinions about the inventor of the non-breaking space. Instead, Dan took the time to craft a really useful commit message for the benefit of those around him. I'd like to step through a few of the ways I think this is a really good example.

It explains the reason for the change

The best commit messages I've seen don't just explain what they've changed: they explain why. In this instance:

I introduced some tests in a feature branch to match the contents of
`/etc/nginx/router_routes.conf`. They worked fine when run with `bundle exec
rake spec` or `bundle exec rspec modules/router/spec`. But when run as
`bundle exec rake` each should block failed with:
    ArgumentError:
      invalid byte sequence in US-ASCII

Without this level of detail, we could hazard a guess that this commit fixed some kind of parsing error in some tool or other. Thanks to the commit message, we know exactly which tool it was.

This kind of information can be really valuable to document, and is all too easy to lose as people forget the original context behind their work, move on to other teams, and eventually leave the organisation.

It's searchable

One of the first things in this commit message is the error message that inspired the change:

ArgumentError:
  invalid byte sequence in US-ASCII

Anyone else who comes across this error can search through the codebase, either by running git log --grep 'invalid byte sequence' or by using GitHub's commit search. In fact, from the looks of the search results, multiple people did so, and found out who had found this problem before, when they came across it, and what they did about it.

It tells a story

This commit message goes into a lot of detail about what the problem looked like, what the process of investigating it looked like, and what the process of fixing it looked like. For example:

I eventually found that removing the `.with_content(//)` matchers made the
errors go away. That there weren't any weird characters in the spec file. And
that it could be reproduced by requiring Puppet in the same interpreter

This is one of the areas commit messages can really shine, because they're documenting the change itself, rather than documenting a particular file, or function, or line of code. This makes them a great place to document this kind of extra information about the journey the codebase has taken.

It makes everyone a little smarter

One thing Dan did here that I really appreciate was to document the commands he ran at each stage. This can be a great lightweight way to spread knowledge around a team. By reading this commit message, someone can learn quite a few useful tips about the Unix toolset:

  • they can pass an -exec argument into find to run a command against each file found
  • that adding a \+ onto the end of this command does something interesting (it passes many filenames into a single file command, rather than running the command once per file)
  • file --mime can tell them the MIME type of a file
  • iconv exists

The person who reviews this change can learn these things. Anyone who finds this commit later can learn these things. Over enough time and enough commits, this can become a really powerful multiplier for a team.

It builds compassion and trust

Now the tests work! One hour of my life I won't get back..

This last paragraph adds an extra bit of human context. Reading these words, it's hard not to feel just a little bit of Dan's frustration at having to spend an hour tracking down a sneaky bug, and satisfaction at fixing it.

Now imagine a similar message attached to a short-term hack, or a piece of prototype code that made its way into production and set down roots (as pieces of prototype code love to do). A commit message like this makes it much easier to remember that every change has a human on the other end of it, making the best decision they could given the information they had at the time.

Good commits matter

I'll admit this is an extreme example, and I wouldn't expect all commits (especially ones of this size) to have this level of detail. Still, I think it's an excellent example of explaining the context behind a change, of helping others to learn, and of contributing to the team's collective mental model of the codebase.

If you're interested in learning a bit more about the benefits of good commit messages, and some of the tools that make it easier to structure your changes around them, I can recommend:




All Comments: [-] | anchor

carapace(2884) about 23 hours ago [-]

Gah... About a third of my commit messages are just 'minor cleanup' or a variation. Should I be ashamed?

sixstringtheory(10000) about 20 hours ago [-]

I think shame is an awful motivator. Instead I would praise you for identifying an opportunity to improve :)

hleszek(10000) 1 day ago [-]

To avoid those kind of issues, non-ascii characters are forbidden in our code base. They are automatically verified in a pre-commit git hook.

unfunco(3661) 1 day ago [-]

It should be checked as part of the CI too, if you're doing that, some people might never install the hooks, and some people might git commit --no-verify.

BlueTemplar(10000) 1 day ago [-]

This issue would never have happened using modern software. And if your text (including source code) still isn't in UTF-8, you're doing it wrong.

(I guess unless you're using some specific, very performance-conscious system that has to use 7/8-bit characters and is never going to connect to the Internet anyway.)

rimliu(10000) 1 day ago [-]

It's like preventing the headache with a guillotine.

Symbiote(4197) 1 day ago [-]

I intentionally add non-ASCII characters to our code, so that an incorrectly configured IDE or bad tool fails.

75% of the development team has at least one non-ASCII character in their name, so it would be pretty rude otherwise.

It's much better to knowingly reject a tool at the start, since it can't handle ordinary characters, than find out a year later with the first e.g. British customer that it can't handle '£'.

bloak(4222) 1 day ago [-]

If you don't allow non-ascii then you may have to mangle some people's names in the copyright header.

It's good to check files for unexpected characters, though. Here's some Perl to do it:

  perl -e 'binmode(STDIN, ':utf8'); binmode(STDOUT, ':utf8');
  foreach (split(//, join('', <STDIN>))) { ++$c{$_}; }
  foreach (sort(keys %c)) { printf '%8d %s\n', $c{$_}, $_; }'
I tested that on a text file I was working on ... and I discovered that the file contained a BOM (U+FEFF), not at the start of the file, but at a random point in the middle of the file. I've deleted it. Who knows what problems it might have caused for me later?

You could have a pre-commit git hook that refers to a whitelist of allowed non-ascii characters, or also allow all alphabetic characters, or something like that.

codegladiator(10000) 1 day ago [-]

Should have been in a doc or wiki instead of commit message.

I have never seen any dev searching for error messages in commit messages.

For the rest of the points (makes smarter, builds trust and compassion), if it's so worthy put it on the blog (like this blog post itself) so it can has a potential to reach some reach some audiance.

chris_wot(1457) 1 day ago [-]

Why not put it in both?

kace91(4190) 1 day ago [-]

agreed. My team used to like of extensive commit messages, so that if you had trouble with a piece of code you could just git blame it and take a look at the referred commit.

The problem with that approach is that it doesn't survive as well as you'd like, because fixing a typo in a line would get you 'ownership' of the line (since only the last person to change it is blamed). It's even worse in semi-major refactors due to moved/renamed files being treated as new....

Far better to have docs apart.

lucideer(4167) 1 day ago [-]

> I have never seen any dev searching for error messages in commit messages.

I've seen many devs search for error messages in Github search. That often turns up results in people's comments in issue threads, but the search also includes commit messages.

clinta(10000) 1 day ago [-]

I think this should have been an issue. Open an issue with the original error, document the troubleshooting in the comments of the issue, then close the issue with the commit.

beefhash(746) 1 day ago [-]

I think my favorite (in terms of humor) is a commit from mpv complaining about locales and encodings. You can practically feel the committer's sheer frustration.

[1] https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f02...

badsectoracula(10000) about 3 hours ago [-]

This is why i stick with ASCII :-P.

RHSeeger(4205) 1 day ago [-]

That is... wonderful. I've spent some time dealing with locales in C and other places that depend on the things being discussed in the commit. Just reading it bring back some of the rage I felt.

mikehollinger(10000) 1 day ago [-]

My favorite (in terms of dark humor, if we're honest) is YOLO, one of the more interesting deep learning object detectors. [1] It is the exact opposite of yours in every way. The code is brilliant however.

Even the papers are snarky. [2]

[1] https://github.com/pjreddie/darknet/commits/master

[2] https://arxiv.org/pdf/1804.02767.pdf

Daemon404(10000) 1 day ago [-]

Not a commit but from the same author, so you may enjoy: https://github.com/wm4/dingleberry-os/blob/master/README.rst

It does go off on a few tangents, but it's an interesting read.

mark-r(4210) 1 day ago [-]

Agreed, that's an over the top commit message.

I do disagree with the assertion that it takes a lot of code to convert between the various UTF variants, 3 pages is an overestimate. https://stackoverflow.com/a/148766/5987

jplayer01(10000) 1 day ago [-]

Final paragraph:

> All in all, I believe this proves that software developers as a whole and as a culture produce worse results than drug addicted butt fucked monkeys randomly hacking on typewriters while inhaling the fumes of a radioactive dumpster fire fueled by chinese platsic toys for children and Elton John/Justin Bieber crossover CDs for all eternity.

I also nominate this commit.

oefrha(4042) 1 day ago [-]

"mpv", "sheer frustration"... I didn't have to click the link to know it was authored by wm4.

dmitryminkovsky(4062) 1 day ago [-]

> "Those not comfortable with toxic language should pretend this is a religious text."

btilly(743) 1 day ago [-]

Oh. Locales. The remembered pain.

Save a file in Notepad. Open in vi. See that it is different. Find data in the database, no clue the weird characters were originally supposed to be. And so on and so forth.

I once wrote a reasonable program and sent it as a bug report to the maintainer of the Perl module DBD::File. He sent it as a bug report to BerkeleyDB. They said they never thought about it but yes, that would be silent data corruption with no way to recover. The program? Maintain an address book in a BTree sorted in the current locale. Enter names. Change locale and insert something else. Voila! Lost data with no way to recover!

testplzignore(10000) 1 day ago [-]
twic(3509) 1 day ago [-]

This message says much more about the author than it does about the commit.

cryptonector(4197) 1 day ago [-]

This needs to be REQUIRED READING at the Open Group and the ISO C standards committees.

I'll quibble just a bit and say that:

  a) the C locale should be a UTF-8 locale...
     that tolerates invalid sequences (because
     the C locale historically is a just-use-8
     locale),
  
  b) even with new functions that take a locale
     handle, we need functions that use a global
     one, however that global one should be set
     once and NEVER changed in the life of the
     process, and it should be set either before
     main() starts, or before main() does anything
     that needs a locale, or starts any threads.
thcz(10000) 1 day ago [-]

I do feel like Git commit descriptions are severely under-utilised for sure, but I believe there is a reason for that which until fixed, will prevent rich and contentful commit descriptions for flourishing.

In the article order: the screenshot is from a commit detail page. How often do you land on this page? You need to specifically click through. If you are in a commit list, the only thing that sets title-only commits and commits with description apart is an ellipsis link which practically blends in with the background. It is not very well integrated nor discoverable. Also I don't believe the commit descriptions render as MarkDown (unlike issues) which is also a shame as it feels like less a doc then. But I might be wrong on this. But even outside of GitHub, how many other UI/IDE plugins and other kinds of Git tools restrict commit display to just the title and put the description on the sideline? Most of them. I think this further leads to the currently low value of the commit message being searchable. Since it is exceedingly rare for good commit messages to exist, no one thinks to search them. People default to Google, when their own project's codebase/knowledge base could hold the answer to their query. I don't have much of an opinion on the commit message telling a story / having a human touch. I mean, it doesn't hurt I guess, but until _full_ commit messages become more 'mainstream' (for a lack of a better word), they can be as human as they can, but they will live in solitude.

jmiserez(10000) 1 day ago [-]

If only important/tricky commits have long messages it's fine. Most IDEs will show you the full text if you hover over the line, so when working on something it's easy enough to check them. I'm huge fan of the timemachine-like 'Show Git History for Selection' view that IntelliJ has, so usually there's no reason to go to a Github/Gitlab commit page.

ryukafalz(3919) 1 day ago [-]

Where they're really valuable, IMO, is when you're tracking down when/why a change was made with `git blame`. When you're looking for the reasoning behind a change, it's extremely helpful if there's a detailed commit message going along with it.

Darkstryder(4006) 1 day ago [-]

In the same vein, I wish there would be a standard workflow for putting code inside commit logs.

The typical use case would be database migration scripts : IMO they are always a pain to version properly because fundamentally Git and all the other software versioning tools let you describe the 'nodes' (in the graph theory sense of the word) of a codebase ('at commit A the code was in this state, at commit B it was in the other state'), but severely lack when it comes to describe the edges between nodes ('in order to go from state A to state B, you need to execute this code')

I think the temporal dimension of software engineering is still poorly understood, and severely undertooled.

residentfoam(10000) about 21 hours ago [-]

In a professional context (and not only), one would think that this should be the normal expected good practice. But unfortunately, that is not the case.

It is so surprising (well not really) to see how, in most cases, developers put so little to close to zero effort in writing proper commit messages and more in general to have a clean commit history. They simply don't care and you keep seeing garbage commits with non-sense to close to empty message and description. Sadly enough this is seen as normal and just accepted.

Every single team I have been working with from small to large organizations I always had to pick up on the 'write proper commit history' fight. And even after extensive explanations on why you should do that, people simply don't care and they keep pushing stuff like: 'fix', 'updated class z' and stuff like that.

Commit history does not seem to be part of the review process.

Sometimes it is just so depressing to see how so unprofessional software engineers are.

ImaCake(10000) about 21 hours ago [-]

If it makes you feel better, biologists are often no better. Our equivalent to git commits is labelling tubes and keeping little excel databases of what has gone where. Often databases stop being updated or people give their tubes esoteric labels that are meaningless to those who look at them a year later. As a research assistant in a large lab, I discovered blood tubes with literally no labelling, and often spent hours searching for samples in the labyrinth of freezers in that lab. It is also not uncommon for papers to be retracted because the original authors lost the raw data!

derefr(3684) 1 day ago [-]

I'm torn about whether I'd like to see this kind of information in a commit message, vs. Something more like:

"Remove parser-unsupported character. Closes #340295."

...where ticket 340295 (wherever, not necessarily Github Issues) goes into more detail about the cause, investigation, and resolution process, as a history of the evolution of said process across a conversation.

pojzon(10000) 1 day ago [-]

IMHO you should pick a right medium for some kind of messages and I dont think git commit message is the right medium in OP example. Most companies have ticketing tool to keep track of additional information like: how long did it take to solve that issue, was there any input from other people, long 'marked up' message explaining what was wrong, how it was fixed and tested (markup is a lot easier to read than console log).

Using ticket number already forces you to use 'the right tool' to view the message.

IMHO I would consider OP example as a bad practice and your example as a better solution.

rustbear(10000) about 22 hours ago [-]

The problem with this approach is when the team changes its ticketing system or the codebase goes to a different team (that has a different ticketing system), meaning all that context is lost. Happens all the time.

danmur(10000) 1 day ago [-]

I wish Git wasn't like some generic term for source control. A little diversity is good!

BlueTemplar(10000) 1 day ago [-]

Good point - but what is the use case where another source control system (like SVN?) works better ?

CharlesColeman(4211) 1 day ago [-]

I really like this commit message. I've found that switching to git from more traditional version control systems requires a lot more discipline in some ways. A lot of people just commit, commit, commit lots of incremental changes with no context or story to them. I've seen pull requests with dozens of tiny commits together make up a cohesive effort, but individually are just useless. I've been really having to push my team to spend time to cleaning up their commit history before getting their pull requests merged.

I think it's really important to capture the context and indent behind changes. I may be weird, but when I'm fixing an issue, often try to find when it was introduced, which often provides really useful information for the fix. That's much harder to do if the commits aren't cohesive and the messages aren't descriptive.

seanbarry(10000) 1 day ago [-]

Implement and enforce conventional commits: https://www.conventionalcommits.org/en/v1.0.0/

generaljelly(10000) 1 day ago [-]

I'm one of those people. I make lots of little commits, it gives me space to really make a mess of coding going down some rabbit hole and performing 'reset --hard' when I get too away from myself, and track what I'm doing locally. As long as each commit isn't causing a problem with CI/CD, and my pull request to master is well documented what is the value added of cleaning up commits?

(Junior developer here, looking to be convinced!)

strictfp(3936) 1 day ago [-]

OR you could just write

Replace invalid ASCII char. Fixes rake error 'invalid byte sequence in US-ASCII'.

I don't want your entire life story in my commit log.

lazyant(4199) 1 day ago [-]

do you think a junior developer, or maybe somebody not vary familiar with Linux would not learn anything or benefit from reading those comments?

Obvious point is that commit messages can be used besides what was done as a form of documentation and teaching tool (why, how).

JeffRosenberg(10000) 1 day ago [-]

Then use `git log --oneline` and you don't have to see the lengthy details, until the inevitable day when you find you need them.

Cthulhu_(10000) 1 day ago [-]

For critical applications, I for one would like to know the story behind a commit, preferably in the commit itself and not a reference to an external system like idk, Jira.

My favorite examples of commit messages are the Linux kernel, where you can tell that they're being specifically crafted instead of just used as a work log to be ignored. This means that ten years down the line, people can still see when a change was made and why, who was involved, who signed off on it, etc. Have a look at the commits at https://github.com/torvalds/linux/commits/master

Supermancho(10000) about 17 hours ago [-]

I, too, would rate this a substandard git comment. Dave basically vomited a bug ticket of information, which is highly contextual and irrelevant ... like the lines he was faced with, which tell us nothing in the future nor anything we could not see in the change. The error is known, from the ticket being addressed. Documenting what error a bundler throws in the application deployment, within git seems...silly, since it will likely not apply to all points in time. That's why we have separate issue tracking.

There was a whitespace encoding issue AND the developer didn't really understand the issue, since they ended with 'One hour of my life I won't get back.'. Over my 20 years, I've seen this EXACT scenario multiple times across multiple companies. Some jr engineer gets stuck with some troublesome weird error in a corner-case that ends up being a non-standard whitespace. It's a learning opportunity and he lamented it because it was different and nobody told him 'we could stop this from happening again, generate a new issue'.

There are salient improvements that the git commit would benefit from both comment changes and additional code:

1. Include a (new) feature ticket that is linked to this issue - to create a process that doesn't allow for this again (eg fix a linter)

2. Include the name of the bug ticket (Convert template to US-ASCII to fix error) in the commit title, that was being addressed.

3. Create a test to specifically enforce the us-ascii encoding or add necessary rules to a linter.

m_sahaf(3967) 1 day ago [-]

On the other end of the spectrum you get ImageMagick useless commit messages[0].

That extreme aside, I'd rather have commit messages that delve into the why-and-how the commit alters the behavior to the better rather than cryptic message as 'Replace invalid ASCII char'. Now we have documented reasoning and thought process that can aid future debugging. They can also be beneficial for new devs hacking on the project, or students learning how to implement and improve systems.

Personally, I enjoy reading these. The Go commits often have commit messages like these, and they are shared on HN often for a reason. They're learning material. They can't go on a wiki because they're tied to particular set of changes in a particular point in history. They also can't be comments on the code because they're tied to particular lines in different files, and code comments can only cover a set of consecutive lines in one file.

One recent example I could find is this[1]. Yeah, it fixes ^Z, but why didn't the old approach work? Why did it work for some time then didn't? How did it change? Why is this commit optimal, if it is? All of this along with scenarios to reproduce the issue.

Give me your life story anytime over cryptic message.

[0] https://github.com/ImageMagick/ImageMagick/commits/master

[1] https://github.com/golang/go/commit/610d522189ed3fcf0d298609...

jordigh(458) 1 day ago [-]

> I don't want your entire life story in my commit log.

Why not? Where else do you want it? Is something forcing you to read the full commit log?

There's no length limit on commit messages and commit messages are mostly out of the way. Most VCSes have a way to only show you the first line. So if you want summaries, that's what the first line is for. If you want the full story, that's what the body is for.

Combined with annotate/blame, commit messages can be very helpful source-level documentation. Nobody has ever complained about too much documentation, and commit messages are the perfect time to document what happened because it's one of the few times where our tools actually force us to write something in order to proceed. As long as we're being forced to write something, write something good and informative.

DJHenk(10000) 1 day ago [-]

That's why most guidelines for commit messages prescribe a short description and an optional long description. The message in the article does not have a short description, which would have been easy to include. For that reason, it's not 'My favorite Git commit message' either.

nailer(414) 1 day ago [-]

This is good, I'd add:

> Replace invalid ASCII char. Fixes rake error 'invalid byte sequence in US-ASCII'. See #123

So people can get the life story if they want it.

vanderZwan(3088) 1 day ago [-]

"I didn't have time to write a short commit message, so I wrote a long one instead."

gwd(10000) 1 day ago [-]

> I don't want your entire life story in my commit log.

I agree with this, but I think yours is too short.

Scientific papers typically introduce enough information such that a person familiar with the field but not an expert in that particular area can understand generally what's going on.

That's my ideal for a commit message as well: someone generally familiar with the codebase but who hasn't looked at this specific code (or perhaps not in a few months) should be able to understand what's going on; then the job of the reviewer is basically just verification.

My 'template' is normally something like: 1) What's the current situation 2) Why that's a problem 3) How this patch fixes it. So in this case, it might look something like this:

---

Convert template to US-ASCII to fix error

$functions use `.with_content(//)` matchers to do X. These matchers require ASCII content. The $foo template contains a non-ASCII space; this results in the following error:

ArgumentError: invalid byte sequence in US-ASCII

Fix this by replacing the non-ASCII space with an ASCII space.

---

No need for a life story, but still searchable, and has enough information for even a casual contributor to do a useful review.

konsnos(10000) 1 day ago [-]

This is true when you can reference the commit to an issue. Then, seeing the simple commit message you can select if you want to dig up what happened by reading up the comments at the issue.

On the other hand it really gets into my nerves when people don't use the task/issue/whatever manager system appropriately. Recently, I lost a couple of days trying to figure out how to compile a c++ framework because the other guy didn't document his pipeline. In general I'm really disappointed by the majority of my colleagues for the lack of comments inside and outside of our codebase and this is a persistent issue, at all the companies I worked for. Me along with other similarly irritated people, always ask for documentation if it is not given.

leejo(3771) 1 day ago [-]

> I don't want your entire life story in my commit log.

I[1] want enough debug information in the commit log to be able to reproduce the issue without having to go on web hunts to understand the problem. Especially when the change appears to be trivial on the surface, because these are the ones that can turn out to be rabbit holes.

I don't want to have to interrupt you to get this information because you didn't write a good enough commit message, and you probably don't remember anyway. I don't want go look at an external issue tracker that i may not have access to, or may not even exist anymore.

[1] Where 'I' is: me, your future self, a future maintainer, a junior dev, an open source contributor.

mschwaig(10000) 1 day ago [-]

Generally speaking sure, there's no need to make things more complicated than they are, but the author even found some evidence in the history that indicates other people found this message useful.

The powerful thing about this is having everyone put this kind of info in the same place IF they think it might be useful to the next person.

EugeneOZ(3530) 1 day ago [-]

> I don't want

Who cares.

Hope you enjoy this tone - your was the same.

CGamesPlay(4055) 1 day ago [-]

Still, you've left out the details that you've confirmed that there's no other instances of this in our codebase. I'm also firmly in the 'all commit messages should include a test plan' camp, so you should at least say how you found the error ('bundle exec rake was run before and after').

I get you're being terse for demonstrative purposes, but even eschewing verbosity we should still convey all the pertinent information.

marvin(3813) 1 day ago [-]

You can write the brief summary in the first 80 characters, like OP did. Then write details in the body below, in case someone needs the context. Most tools display only the first 80 characters unless you expand the body.

This case is probably longer than necessary, but I've saved a day of debugging on multiple occasions due to someone (also myself) leaving some lines of context, reasons and reasoning after the high-level description.

celticninja(4203) 1 day ago [-]

That belies the effort that went into the fix

EliRivers(4192) 1 day ago [-]

That's great for you. You don't want that. However, if you code in a team, doing everything for your own wants rather than considering the needs of the team (present and future) is just bad software engineering.

deividy(10000) 1 day ago [-]

I figure that so few people read commit messages that in most part of time this is kind of useless. Specially in an early stage of a project. Things will change as faster as I can type a message such as this.

Code is simple. Humans overthink. I would prefer a commit message such as: 'Fix invalid byte sequence in US-ASCII when running bundle exec rspec.' than a dev that keep stucked trying to write a cool message and never fix the issue.

avian(4060) 1 day ago [-]

> few people read commit messages

That very much varies. In a project where 90% of commits are "fix", "foo", "commit", etc. then yes. Nobody will ever read that (or do pretty much anything else useful with a VCS).

On the other hand, when every commit message is on the level, the yes, people do read them. Actually, first step when investigating any problem or trying to understand some code is to look at the commit log.

See e.g. Linux kernel or some of the Google-related open source projects (chromium, webrtc, etc.) for examples of good, long-form commit messages.

navinsylvester(2071) 1 day ago [-]

I use the following convention to start the subject of commit(posted by someone in a similar HN thread):

    Add = Create a capability e.g. feature, test, dependency.
    Cut = Remove a capability e.g. feature, test, dependency.
    Fix = Fix an issue e.g. bug, typo, accident, misstatement.
    Bump = Increase the version of something e.g. dependency.
    Make = Change the build process, or tooling, or infra.
    Start = Begin doing something; e.g. create a feature flag.
    Stop = End doing something; e.g. remove a feature flag.
    Refactor = A code change that MUST be just a refactoring.
    Reformat = Refactor of formatting, e.g. omit whitespace.
    Optimize = Refactor of performance, e.g. speed up code.
    Document = Refactor of documentation, e.g. help files.
ShteiLoups(10000) 1 day ago [-]

Yes. As is convention, commit messages should be a one line header, then and empty line an a body (if necessary).

The whole thing should be width limited to 80 or 100 characters.

And the subject line should complete the sentence 'If this commit is applied, it will...'. It should start with a capital letter, then move to lowercase, and necessarily will start with a verb.

blaisio(10000) about 22 hours ago [-]

Yes! I thought this was a great idea.

The other thing I do is add a 'MODULE:' prefix to my commit message. It makes them way more readable. Lots of other people do this too; you can see this in Redis, Linux, and Go codebases, for example.

So your message might look like 'router: add support for path vars'. Much easier to read than 'Add support for path vars to router'

pricci(10000) 1 day ago [-]

I also use something like this. A variation of Angular's commit convention[1]. There are tools like commitizen[2] that can help new adopters to build commit messages.

[1]: https://gist.github.com/stephenparish/9941e89d80e2bc58a153

[2]: https://github.com/commitizen/cz-cli

wh-uws(3328) about 22 hours ago [-]

If someone made enforcing this convention a git hook I would use it in a heartbeat.

I.e. Love using husky to enforce code conventions through linters...

Only issue is it would be hard to completely enforce in code and obviously require some review but still nice

lugg(3073) about 20 hours ago [-]

Why? (Honest question, hear me out)

Is it a vocab consistency thing?

I see pretty consistent vocab used across orgs anyway, so given there is shardd domain knowledge / language at play, im not sure what the goal for standardization is?

Don't take this the wrong way, I use a lot of these when appropriate, but I don't think I could agree that refactoring must be just a refactor, and I don't think I want to limit anyone's commits to this list of change types either.

Forcing changes into these words means a lot of stuff is pointlessly bucketed when more appropriate wording could be used.

Say I want to push a commit 'prioritise shipping route a over route b'.

So I have to put it under start? Optimise? Fix? Why? Prioritise is the right word, why not just use that? Why play mind games when we have a whole dictionary to draw from?

klodolph(4214) about 19 hours ago [-]

Some additions I personally use:

- Change: change functionality or behavior.

- Cleanup: more than formatting, less than refactoring. No functional changes.

- Rename: refactoring that changes a name but does nothing else.

- Extract: refactoring that creates a new module or class by moving code out of an existing module or class.

pnako(10000) about 11 hours ago [-]

Isn't that convention just the English language?

groestl(10000) about 22 hours ago [-]

I like the convention, but I challenge the idea that optimization can be just a refactoring.

webscalist(10000) about 21 hours ago [-]

Every commit message must start with JIRA ticket number

hyperpallium(2718) about 21 hours ago [-]

Formatted for my fellow HN-disenfranchised mobile sufferers:

Add = Create a capability e.g. feature, test, dependency.

Cut = Remove a capability e.g. feature, test, dependency.

Fix = Fix an issue e.g. bug, typo, accident, misstatement.

Bump = Increase the version of something e.g. dependency.

Make = Change the build process, or tooling, or infra.

Start = Begin doing something; e.g. create a feature flag.

Stop = End doing something; e.g. remove a feature flag.

Refactor = A code change that MUST be just a refactoring.

Reformat = Refactor of formatting, e.g. omit whitespace.

Optimize = Refactor of performance, e.g. speed up code.

Document = Refactor of documentation, e.g. help files.

simonh(4121) 1 day ago [-]

My favourite Github commit was someone removing their password from a test list in a penetration testing tool, because they didn't want anyone to know their password. I just tried, but couldn't track it down. The subsequent comment trail was hilarious.

ilitirit(3676) 1 day ago [-]

This really just depends on your team/company/culture.

Lengthy commit messages are not really required if you have associated tickets in a bug-tracking or project-management system. More often than not, you'll just be duplicating info.

Tactic(10000) 1 day ago [-]

This is true until the company changes the bug-tacking and project-management software without properly porting over everything because they use different identifiers.

I tend to put a link to the external system with a very brief explanation, allowing someone to quickly assess the what and why with the ability to dig elsewhere for more detail.

drej(4201) 1 day ago [-]

This reminds of a Markdown issue I've had many many many times - sometimes (and only in some engines), headings would not render and I'd only get '# foobar' instead of '<hx>...'

It took too long for me to track the issue. When I write '#' using alt-3, I then write a space and oftentimes I don't lift alt soon enough and alt-space creates a non-breaking space (on macOS). And some/most Markdown engines don't recognise '#nbsp;text' as a heading.

I suspect something like this happened in the commit linked here.

misnome(4041) 1 day ago [-]

This happened to me all the time, especially with python2 code without encoding declared (which caused a failure to parse the file because of the comment).

I've since switched my editors to highlight such characters.

gwbas1c(3917) 1 day ago [-]

Honestly, I think better rules are:

- Whenever someone asks a question in a pull request, answer it by putting a comment in code. - Include a ticket number (for whatever ticketing system you use) in the commit

Why? I find that commit messages are black boxes. They only come out when doing a git blame, but they don't show up in my IDE. Instead, I'm more likely to run across messages like this when they're comments in code or discussions in our ticketing system.

I think if we had tighter integration among our IDE, git, and the ticketing system, detailed messages commit messages like this would be extremely useful.

adwww(10000) 1 day ago [-]

Code comments have a habbit of getting lost during future tidy ups or refactors.

If the code is that unclear it needs a comment, refactor it into a named method, where the method name describes exactly what it's doing.

luord(2168) 1 day ago [-]

What I gather from this is that commit messages are a better place for documenting stuff like this than polluting every line of code with often redundant comments.

I like it.

sethammons(4058) 1 day ago [-]

I think the git log is an interesting place to document this kind of thing. Also, there should never be redundant comments. I do think though that lots of folks put things in the git log that _should_ be a code comment. For myself (as most of the people I can think of that I work with), we don't reference git commits unless we are actively investigating a previous change. You should not hide reasons for code being the way it is in the git commit; it should be exposed for direct observation to someone who might edit the code. Ex: '// the following delete call is required to remove the item from the DNS cache to ensure the test validates non cached dns items'. That should be in the code.

powjfjti83(10000) 1 day ago [-]

To an extent I find this lazy and even a humorous take is giving it attention it doesn't deserve

I can complain about anything

Can this person build a language that's still as broadly useful as C that no programmer will find issue with

"Oh boy I encountered a particular odd and annoying thing. I won't bother to offer an alternative.

I'll just complain about others efforts while ignoring they ultimately enabled me to realize my media player project."

Talk about sheltered suburban kids

Bet he also tossed his SNES controllers against the wall, blaming Nintendo when he ate shit in FZero

Since I have no use for MPV let's go on a rant about writing more tools when we already have numerous open options to do the same thing

Oh right because life is relative and relative to me this is grand standing baby talk

So sick of tough guy coders. Oh rly you're good at seeing the plot holes in text driven computer code? I mean you sit in your basement all night focusing on it so you're a normal human that gets better at they spend time on

Meanwhile, not really contributing in a lasting way to growth of humanity with a media player or your other efforts

But of course the efforts that do live long like a general purpose performant language are the kids toys

If there's a joke here it's this persons life perspective

simias(4157) 1 day ago [-]

The commit was complaining specifically about the way locales were designed, not C as a whole. While C was very successful, I don't think you could argue that C locales ever reached the same level of popularity.

That being said I do agree with you that this complaining is not massively productive. Dealing with localization and non-ASCII text is notoriously difficult. Look at Java, Python 3, Windows, PHP 6 and how many 'misconceptions about Unicode' articles you can find online. You could spend hours pointing out how each approach has tons of drawbacks but clearly the perfect solution doesn't seem to exist so a compromise had to be found.

In particular I'm not sure I agree with his complaints about locales being global state. How else would you handle them? You need to have some kind of global config flag somewhere to decide what local the user wants to use. Having explicit versions of the stdlib taking locales as parameters could be nice I suppose.

This bit in particular seems to completely miss the point:

>Idiotically, locales were not just used to define the current character encoding, but the concept was used for a whole lot of things, like e. g. whether numbers should use ',' or '.' as decimal separaror.

Of course if this programmer assumes that locales and charsets should be the same thing they'll end up frustrated.

dahart(3788) 1 day ago [-]

It's a shame you see it only as complaint, the brilliant thing about the text you're replying to is that it educates the reader as to the bigger problem, as well as being very funny. Does your complaint do either? Understanding why C locales are broken may very well contribute to the lasting growth of humanity. Did you note that Eric Raymond fully agreed?

SirensOfTitan(4127) 1 day ago [-]

This comment comes across really mean and judgmental. I didn't read all of the original commit, but it read to me as informative and passionate with all of the frustrations solving a difficult problem comes with.

Honestly, this comment is so over the top I can't even detect whether you're serious or not.

dang(179) about 23 hours ago [-]

We detached this subthread from https://news.ycombinator.com/item?id=21290517.

drej(4201) 1 day ago [-]

I love these commits. Then don't have to be this verbose, but they have to tell a story of why things were done. I can sort of deduce the what from the code itself, but the why is sometimes shrouded in mystery.

I started with these explanatory git commits a few months ago and they are super useful, even if you're just reading your own commits from some time ago.

tigershark(10000) 1 day ago [-]

To me they are red flags because it means that very likely who writes this kind of commit messages doesn't use a proper issue tracking system.

Ensorceled(10000) 1 day ago [-]

I often review commit logs of my teams, especially while we are tracking down problems or I'm making sure the release notes capture everything.

There has to be a balance between this and 'WIP'; I'm imagining trying to page through the commit log to see what changed when every 1 line change has a 35 line commit associated with it.

UK-Al05(10000) 1 day ago [-]

I believe there's a standard format for this.

The first line is the title, then there's a description.

These heavy descriptions of commits, are used quite heavily in linux kernel dev

pricechild(4214) 1 day ago [-]

Would `git log --oneline` not help?

1f60c(10000) 1 day ago [-]

That's the beauty of it, the first 50 characters of the first line (IIRC) will be shown as a summary, and this one summarizes 35 lines in seven words.

sethammons(4058) 1 day ago [-]

I'm glad my near-exact pain has been experienced by others. I had an undefined function call of ' ' in a ruby script years ago. Finally, I turned to a hex editor at the suggestion of a colleague. The culprit was non-ascii whitespace that ruby decided should be a function declaration. Copy pasta error out of a hipchat code snippet.

Dumblydorr(10000) 1 day ago [-]

Sorry, beginner here, can you explain more simply the difference in the whitespaces? Are they just encoded differently?

davestephens(10000) 1 day ago [-]

Pasta and computers are always a bad mix...

messe(3509) 1 day ago [-]

For anybody wondering, the likely origin of the invalid character is somebody using an Apple Ireland/UK keyboard layout where # is Option-3 (AltGr-3), and non-breaking space is Option-Space (AltGr-Space).

C1sc0cat(10000) 1 day ago [-]

Ah I would have bet good money on an apple product being involved.

22c(10000) about 5 hours ago [-]

I recently added a Rake task to one of our builds which checks for the exact problem mentioned in GP, after having 3 separate occasions in the last 6 months where OSX 'smart' characters have changed the encoding of a file consumed by things expecting pure ascii.

Unfortunately it is a bit of a hack that shells out to 'file -i', but I'll take it over hours of frustration.

laurent123456(1242) 1 day ago [-]

I don't know how many times these non-breaking spaces caused problems. I think linters should prevent commits that contain non-breaking spaces. And if really one is needed, it should be encoded as `&nbsp;` or with whatever encoding is relevant.

kissgyorgy(2098) 1 day ago [-]

> I don't want your entire life story in my commit log.

Then I never want to work with you ever (or any code base you ever touched) because of your laziness and lack of experience working with huge code bases where long commit messages are life-saving. Also you don't care about your colleagues and long-term maintainability.

Debugging bugs for months and weeks because of lack of proper commit messages is the most frustrating experience.

kyberias(4139) 1 day ago [-]

>> I don't want your entire life story in my commit log. >Then I never want to work with you ever (or any code base you ever touched) because of your laziness...

What an incredibly aggressive response.

mepiethree(10000) 1 day ago [-]

So you never want to work with... any junior dev ever? I can't think of a single person we have hired out of college who used Git well.

dang(179) about 23 hours ago [-]

You've repeatedly crossed into personal attack on HN recently. That's not cool. I've banned this account until we get some indication that you've read https://news.ycombinator.com/newsguidelines.html and sincerely want to use HN in its intended spirit. That spirit is intellectual curiosity and kind, thoughtful conversation.

Comments like this one, https://news.ycombinator.com/item?id=21276666, https://news.ycombinator.com/item?id=21204403, https://news.ycombinator.com/item?id=21149400, and https://news.ycombinator.com/item?id=21123836 are not ok here. It's a pity, because you've also posted quite good comments. But they are not worth the cost of the worst ones.

We detached this subthread from https://news.ycombinator.com/item?id=21290076 and marked it off-topic.

deaps(10000) 1 day ago [-]

This gives me ideas.

My commits are usually short and sweet - to the point. I document my code very well, however.

One of my strengths in a previous life as a Master Automobile Technician was the ability to document the entire process -- from duplication of a concern, to troubleshooting, to correction, to verification...it's literally how I got paid (which I never understood why so many automotive techs took short cuts while documenting, especially for warranty concerns where you deserve to be paid (by the manufacturer) for everything you did that was necessary to fix the concern the first time).

I could be mistaken (life-long coder, former network engineer / architect for nearly a decade, but I'm currently in my first-ever role as an actual backend developer). I think I was told to keep commits to one line unless absolutely necessary. I'll have to bring this up though. I like the idea of searching through the git logs for specifics, as opposed to having outlook search through the git commits for actual pieces of code changed, or error codes which might not actually be there.

In either event, at least more descriptive, yet still short, messages such as what strictfp suggested 'Replace invalid ASCII char. Fixes rake error 'invalid byte sequence in US-ASCII' Although I really like having some reasoning and logic - or how/why in those easily-searchable logs as well.

I think some people remember everything and some people don't (although we're all probably roughly similar logically). I have a hard time remembering what I ate for lunch yesterday - that's why I count on good documentation to function as efficiently as possible in the future. I've worked with people who have an absolute uncanny ability to remember 'stuff'. That's impressive, but I do not have that ability myself.

luhn(3837) 1 day ago [-]

> I think I was told to keep commits to one line unless absolutely necessary.

The advice I've heard: Your first line should be a concise summary of the commit. This is because a lot of UIs only show the first line up front. (GitHub, git log --pretty=oneline, etc.) However, it's okay (and often encouraged) to go into further detail on subsequent lines.

chungy(3832) 1 day ago [-]

> I think I was told to keep commits to one line unless absolutely necessary.

I think I would phrase something more usefully as: keep it as short as possible, but no shorter.

I find it to be fairly rare that a commit is so self-evident that only the summary line can do.

Groxx(4081) about 14 hours ago [-]

My standard has been:

Commit messages should describe why you're doing something ('X asked', or '[reams of supporting evidence why this needed to be made faster but more confusing]'). It provides context to current reviewers, and future archeologists who wonder what you were drinking at the time. Perhaps you had a good reason for doing [insane thing X]! Perhaps you didn't. If you didn't write it down, they might change or leave it, and break something or prevent something from getting a proper fix.

Code comments should be notes to code-readers that are relevant at all times until changed or deleted. 'How to use this', 'beware changing X', 'Z is hot garbage and should be replaced if used for Q'. Ideally you'll have asserts or tests or something that actually enforce this, but of course that's not always a realistic option. Comments in code will follow the code around, and don't require chasing code history through N layers of refactoring and indentation-wars, which is what makes commit messages mostly inappropriate for needs like this.

kgwxd(2421) 1 day ago [-]

I had a similar issue with zero-width space recently. Why in the world does that character even exist?

lolc(10000) 1 day ago [-]

I've used it to hint good line-break positions in a text body where the soft-hyphen does not apply.

Chris2048(3484) 1 day ago [-]

Should this go into a commit message, instead of an issue/ticket?

adamlett(4105) 1 day ago [-]

I say yes. I like to keep information about the code as close to the code as possible. Issue trackers come and go, and even if you keep the same issue tracker around, how are you going to relate the change in the code to the particular issue down the road?

FWIW, I also prefer READMEs to Wikis.

Sharparam(10000) 1 day ago [-]

Since this is describing the commit and what was done and why, the commit seems like a better place.

In tools like GitHub, if you make a PR with this commit, it will also automatically put the text in the PR description.

I would much prefer this at work over what I usually see with inconsistent commit message styles and not explaining properly what was done, and not following the recommended max length per line.

shakna(3138) 1 day ago [-]

The commit message still logs in the usual way, but it carries the whole set of information with it, in a way that a centralised ticket system doesn't.

When a developer is looking at logs for solving some problem, they can easily review the rationale for changes.

I'd much prefer the log explaining everything, rather than having to look to a ticket that may no longer exist.

torgard(10000) 1 day ago [-]

If you put it in a pull request, that'll just put it in the merge commit instead.

I'd prefer it being on the commit itself.

Dzugaru(10000) 1 day ago [-]

A great question! I'm still wondering why we all mostly use separate tools for tickets and version control (and knowledge base, and collaboration, and management/hiring, the list goes on), given we already know a lot about software development.

jimws(10000) 1 day ago [-]

Absolutely yes. As a developer I can:

  git log | grep <error message>
  git show <commit id>
I don't have to go looking for an issue tracker and figuring out how to search it effectively.
the_duke(3525) 1 day ago [-]

This is a big discussion.

In a professional setting, companies usually want this information to live in the issue tracker. Mainly to provide insight to managers/other teams without looking at commit messages.

But it removes the information from the code: you now need to look at the issue tracker to make sense of changes, eg when looking at the history of a file, or with git-blame.

I'd argue that all relevant information that affects the code and architecture of the specific repo should live in commits. They should not only tell you what was changed, but also why; and provide enough context to understand the change in the scope of the repo.

All information that is not directly tied to the code, eg cross-repo/product/etc concerns can go in the issue tracker.

Of course this only works well with a `git commit --fixup` and squash + rebase / squash + merge workflow.

And with monorepos it also becomes a tooling problem.

tigershark(10000) 1 day ago [-]

In my opinion absolutely no. It should go in a Jira issue and the commit should have the Jira id in the message. In this way when blaming you are just one click away to the full issue description, with properly formatted text and screenshot if needed. It will also be visible to all the team in the Jira board and they don't have to click on the specific commit to notice that problem.

tomstuart(2916) 1 day ago [-]

Yes, because it lives with the code.

katzgrau(4146) 1 day ago [-]

I'm always intrigued when developers complain that it was '[surprisingly short amount of time] of my life I won't get back.'

Maybe I'm not as clever, but I'm lucky if I fix an issue like that within a few hours. It can sometimes derail a workday. In fact, fixing in a few hours would be something worth celebrating!

Pxtl(10000) 1 day ago [-]

Seriously. I'd count my lucky stars if I squished that one in under 4.

withinboredom(4204) 1 day ago [-]

One of my favorite pranks is to put control characters in the commit message (like the bell) and then you get an auditory notification anytime anyone nearby opens your commit messages.

govg(10000) 1 day ago [-]

Does this actually work on modern editors / browsers?





Historical Discussions: Gitlab: don't discuss politics at work (October 16, 2019: 1155 points)

(1159) Gitlab: don't discuss politics at work

1159 points 3 days ago by devilcius in 3410th position

www.theregister.co.uk | Estimated reading time – 6 minutes | comments | anchor

Updated GitLab, a San Francisco-based provider of hosted git software, recently changed its company handbook to declare it won't ban potential customers on 'moral/value grounds,' and that employees should not discuss politics at work.

The policy addition, created by co-founder and CEO Sid Sijbrandij and implemented as a git pull request, was merged (with no approval required) about two weeks ago. It was proposed to clarify that GitLab is committed to doing business with 'customers with values that are incompatible with our own values.'

Advertisement

Such a declaration could run afoul of legal boundaries in some circumstances. While workers have no constitutional speech protection in the context of their employment, federal labor law requires that employees be allowed to discuss the terms and conditions of their employment and possible unlawful conduct like harassment, discrimination, and safety violations.

But it's perhaps understandable given how, over the past few years, workers in the tech industry have become more vocal in objecting to business deals with entities deemed to be immoral or work that conflicts with declared or presumed values.

At Google, for example, employees have protested the development of a censored search engine for China, the company's Project Maven AI protect for the Pentagon and its provision of cloud services to federal agencies like US Customs and Border Protection (CBP), US Immigration and Customs Enforcement (ICE), and the US Office of Refugee Resettlement. And currently Googlers are sounding off on internal message boards about the company's decision to remove content supportive of democracy protests in Hong Kong.

Meanwhile at Microsoft's GitHub, employees at both companies have objected to GitHub's business with ICE, not to mention Microsoft's government contracts. Employees at Amazon have also urged the company not to sell its facial recognition technology to police and the military.

And recently workers at devops biz Chef raised similar objections to doing business with ICE. Three weeks ago, Chef, after refusing to bend, decided not to renew its contracts with CBP and ICE.

In what appears to be an effort to avoid such protests, Sijbrandij amended his company's handbook to state: 'We do not discuss politics in the workplace and decisions about what customer to serve might get political.'

And what reason does Sijbrandij's pull request provide to support this position? It says, 'Efficiency is one of our values and vetting customers is time consuming and potentially distracting.'

Historical precedent

If you can see how people might respond to IBM, infamous for providing technology that helped the Nazis in World War II, saying, 'Who has time to look into the source of this hard German currency?' you can imagine how GitLab's policy amendment has been received.

Drew Blessing, a staff engineer at GitLab, in a discussion of the policy change, appears to be incredulous that Sijbrandij would word the statement in a way that suggests the company will take anyone's money.

'Is there no scenario we can envision where we would choose not to do business?' Blessing asked. 'I understand that it may be a rare occasion where we would want to, but it seems like we may want to reserve that right rather than say we never will? Obviously we could always change the policy if that occasion arises, but it seems like a strange signal to send to say we won't ever exclude a customer.'

Sijbrandij responded by saying the company can change its strategy at any time, though it would honor standing commitments to customers.

Unsatisfied with Sijbrandij ducked his question, Blessing pressed for an answer. 'Can you comment on this question, please?' he said.

'Is the timing of this update coincidental or in response to what's happening in the Chef community? Unless it's entirely coincidental it seems like we're signally that we would accept a similar contract and would hold our ground if people disagreed.'

Sijbrandij responded by stating, 'as this [merge request] indicates we would do business with any entity that we're allowed to do business with.' He also said the policy change wasn't a direct response to what happened at Chef 'but that situation did cause me to think about it and make this [merge request] to explain the reasons better.'

Some people are fine with business deals divorced from moral considerations. Ben Fellows, a cloud computing consultant, argues that GitLab is just a tool maker shouldn't be accountable for how its services are used, just as an automaker would not be expected to ensure only law abiding citizens drive its vehicles. If ICE has violated the law, he argues, there are legal processes to deal with that.

'The problem with the whole 'activism' mindset is it doesn't actually target the people who created the problem, it just creates lots of noise – and the problem with noise is facts get lost,' Fellows said.

But as others point out, GitLab has made statements about its values. It has a Code of Conduct, in which it talks about empathy for others. Similar statements about values landed software biz NPM in hot water earlier this year after it fired several workers who sought to unionize, making a mockery of said values.

Advertisement

As a commenter identified as 'casiotone' observed, 'If your values aren't used to inform who you're doing business with, why do you bother pretending to have values at all? This [merge request] demonstrates that you don't have any values except 'we want to make money, and it doesn't matter who gets hurt.''

The Register asked GitLab to comment. We've not heard back. ®

Updated to add

GitLab has performed what's technically known as a reverse ferret.

Sponsored: How to get more from MicroStrategy by optimising your data stack




All Comments: [-] | anchor

xurias(10000) 2 days ago [-]

I think it's concerning that everybody seems to think it's okay for companies to clamp down on political discourse. Where do people think we're supposed to talk about this shit? Who's supposed to champion these causes if not companies? Who has actual power in this world other than companies? Companies dominate the media, the way we consume media and everything that we see.

All this means is nothing changes, nothing improves and companies are free to profit off of suppressing rights in authoritarian countries without blowback from people using their platform (like Blizzard/Riot) or people that work for them. Companies are not and never have been neutral entities. If your goal is profit maximization to the detriment of the social fabric and moral principles, then sure, go ahead and pretend to be neutral. But I'm not sure that should be the goal and we shouldn't be helping companies pursuing this goal.

themacguffinman(10000) 2 days ago [-]

> Where do people think we're supposed to talk about this shit?

With your personal friends and family? With NGO or activist groups and forums? Town halls and local political events? Literally anywhere outside work?

> Who's supposed to champion these causes if not companies?

Democratically elected representatives? NGOs, activist organizations, and citizen groups? Since when were for-profit fiduciaries the primary battleground for politics?

imgabe(1639) 3 days ago [-]

Did I miss something? Is there a reason why you have to qualify that you're not advocating for Joe Rogan? Did he get 'cancelled'?

draw_down(10000) 3 days ago [-]

I think it's ok not to advocate for Joe Rogan.

dang(179) 3 days ago [-]

We detached this subthread from https://news.ycombinator.com/item?id=21275269 and marked it off-topic. The comment originally said something like 'Not advocating for Joe Rogan but...'

anm89(3804) 3 days ago [-]

To clarify I don't dislike Rogan. I don't always like him either. I think he occasionally makes great points and sometimes says profoundly dumb stuff. The point is he's divisive and I don't wan't this to become a Joe Rogan debate.

chipotle_coyote(4168) 3 days ago [-]

Is there a reason someone shouldn't make such a clarification if they choose? 'I may not agree with what you say, but I'll defend your right to say it' gets brought up an awful lot in free speech discussions on HN, and that's fine, but we need to emember that it definitionally includes defending the right to say, implicitly or explicitly, 'That guy over there is saying really stupid/dangerous things and this is why he should stop and why you shouldn't listen to him.'

justinmchase(10000) 3 days ago [-]

People try, they label him various things and will call you 'alt right' if you don't qualify your reference to him. Its a pretty desperate label but they try.

kodt(10000) 3 days ago [-]

There is a contingent out there who are very much against him if only for the reason that he has given a platform to people they don't like.

ineedasername(4192) 3 days ago [-]

There's really two mandates to workers here:

1: Don't discuss politics at work. This seems reasonable, when the discussion has a good chance of getting heated and making it more difficult to work with colleagues.

2: Don't vet potential customers (in part because it might get political)

This second one is less defensible. Avoiding politics is fine when it's unrelated to the work, but when it intrudes on work, it becomes relevant, and avoiding the issue is a lapse of ethical responsibility. That people might disagree on the boundaries of that responsibility is precisely why the discussions are needed.

I'm not saying this is easy, that it won't lead to conflict, but it can be done in a structured fashion that minimizes the ability for conversation to blow up into flame wars. Again, not easy, but necessary to avoid abdicating moral responsibility for your actions.

ineedasername(4192) 3 days ago [-]

For the downvoters: maybe you haven't worked somewhere that the work has explicit political context. I have. There are ways of dealing with that fact that don't degenerate into shouting and acrimony. There are ways of professionally, maturely, addressing difficult issues relevant to the work you do.

ascertain(1537) 3 days ago [-]

They've since changed #2 to the point of walking the whole thing back.

gkoberger(746) 3 days ago [-]

I disagree with GitLab on this one, but that title is one crazy leap in logic... 'Blood money is fine with us, says GitLab'

shadowgovt(10000) 3 days ago [-]

The Register has somewhat clickbaity editorial standards, to be sure.

(Edit: media name correction. Egg on my face!)

Fr0styMatt88(2642) 3 days ago [-]

Discussing politics at work has NEVER been a routine thing for me and as far as I know that's always been the cultural norm, at least where I live. It's handled very carefully. So I just assumed that this was the norm and discussing politics at work was the exception.

michannne(10000) 3 days ago [-]

It is the norm, in the non-SV world.

malvosenior(2948) 3 days ago [-]

I'm sure everyone who has to earn a living to support their family will appreciate this move. Nothing is worse than politics at work, where you may lose your paycheck if you don't align with the correct factions.

pfranz(10000) 3 days ago [-]

I'm well aware I'm in a very privileged position, but I've tried to make a very strong effort throughout my career to live below my means. Maybe I'm pessimistic about losing my job or my company going under, but the freeing thing is if I'm unhappy for whatever reason I have options for leaving.

I try hard not to assume or expect this from others, but I very strongly encourage it.

SolaceQuantum(2683) 3 days ago [-]

'Nothing is worse than politics at work, where you may lose your paycheck if you don't align with the correct factions.'

I should bring up in this light that its still legal to fire people for being gay in much of the USA and there's a current supreme court case coming up to determine federally if discrimination on the basis of orientation or gender identity is covered under current antidiscrimination law. In that case you're at risk for losing your paycheck if you aren't born the correct orientation and are found out.

claudeganon(10000) 3 days ago [-]

If wage labor, "work," is the condition of surviving in society, how could it avoid being political? Who is included in it, privileged by it, how it is organized, etc all have serious consequences that determine, again, one's, baseline survival.

reaperducer(4022) 3 days ago [-]

'There are three things in life I learned never to discuss with people: religion, politics, and The Great Pumpkin.' — Linus

Hamuko(10000) 3 days ago [-]

Well, some countries have laws against that kind of a thing.

Tehchops(10000) 3 days ago [-]

What are the correct factions?

JDiculous(4163) 3 days ago [-]

The problem isn't the discussion of politics, it's the fact that that would even matter or influence your job in any negative way.

onion2k(2004) 3 days ago [-]

While everyone does have to earn a living and it's fair to think that's very important, if you believe your employer is doing something wrong and yet don't either speak up or look for a new job you're effectively condoning your employers actions. That isn't particularly admirable and your peers would be right to tell you as much.

Cpoll(10000) 3 days ago [-]

I wonder if they're trying to pre-empt the issues Chef has been facing recently in selling to ICE.

the_watcher(3996) 3 days ago [-]

Discussed in the article. The CEO was asked, specifically, about this.

baalimago(4184) 3 days ago [-]

Reminds me of Peter Handke's recent Nobel prize in literature, where the committee were/are heavily criticized for judging only by literature, and not by Handke's political viewpoints.

Maybe politics doesn't need to be involved in every single part of society, only the political aspects of it.

crispyambulance(3887) 3 days ago [-]

> Maybe politics doesn't need to be involved in every single part of society, only the political aspects of it.

Sure, but when the 'political aspect' involves denying a known genocide that's something that many of us aren't willing to ignore.

And I say that as someone who has enjoyed Handke's work.

Sorry, but he shouldn't get a pass for what he said in the 90's. He, of all people, should have known better.

download13(10000) 3 days ago [-]

Giving someone public recognition also gives recognition to anything that person might say. If they are known for believing that killing certain groups of people is okay, then giving them recognition is inherently political.

g_sch(4222) 3 days ago [-]

I tend to think of everything in life as political in some way. After all, politics is the fundamental question of how power is distributed in the world. However, even if you don't subscribe to this view, I would argue that work and the workplace are one of the more political spaces.

I agree that it isn't easy to be political in the workplace, and it is often better to keep your head down. But think about all the posts here on HN about workplace issues: compensation, toxic bosses, dubious or unethical product and design choices...and so on. All of these things are political because they involve people exercising their power on others, and other people organizing to take that power back.

claudeganon(10000) 3 days ago [-]

What are the bounds of the "political aspects" of society though and how are they negotiable? If we're to preclude "politics" from the workplace, we would have to have also explain how most of the rights we enjoy as workers are the direct byproducts of labor struggle and its "politicization" of the economy and business practices.

einpoklum(10000) 3 days ago [-]

It's a diametrically opposite situation.

Handke was hand-picked by a panel to receive some special benefit.

Gitlab offers a mass service which essentially anyone can use; any denial of the service would require searching out individuals to penalize from among the masses.

sonotathrowaway(10000) 3 days ago [-]

Can you expound upon which aspects of society are political and which are nonpolitical? The Supreme Court is currently considering whether or not companies can discriminate on sexual orientation, just as it had done for race and religion. What do you say to such people whose identities are inherently political?

alexanderlabrie(10000) 3 days ago [-]

Banning staff from talking politics at work shouldn't be allowed under the First Amendment.

notadoc(4114) 3 days ago [-]

That same logic would suggest that no company should be able to ban any topics, content, speech, or people from their platform, no? What is free speech and expression, and what is censorship?

I think society and corporations are wrestling with these concepts right now, and I'm not sure anyone has it right yet.

chrisseaton(3077) 3 days ago [-]

That's not what the First Amendment is about - it's about preventing the government from making laws that infringe free speech. But this isn't a law, and GitLab isn't the government, so the First Amendment does not apply.

> It is a common misconception that the First Amendment prohibits anyone from limiting free speech, including private, non-governmental entities. It is applicable only to state actors. (Wikipedia)

isostatic(3852) 3 days ago [-]

If Congress passed a law banning discussion of politics at work, yes that would be a First Ammendment issue.

  Congress shall make no law ....
All the amendment does is specify what Congress can't do.
alexanderlabrie(10000) 3 days ago [-]

Re: everyone who commented, I said 'shouldn't,' not 'isn't.'

teraflop(10000) 3 days ago [-]

As a general rule, the First Amendment only restrains the government. It means the government can't ban speech (except in narrow, 'content-neutral' ways) and courts can't punish you for it (except for things like defamation).

But the US follows at-will employment, under the more general principle of freedom of association. An employer does't have the same duty to treat its employees 'fairly' that the government has to its citizens. Your employer can fire you for pretty much any reason, including differences of opinion, and the First Amendment has nothing to do with that.

The exceptions to this rule are better thought of as exceptions to at-will employment, rather than extensions of First Amendment protection. For instance, you can't fire an employee for discussing wages or working conditions with their co-workers, in the same way that you can't fire someone for being the wrong religion or for reporting illegal behavior.

(If you're saying that the courts should change their interpretation of the First Amendment to include a right to discuss politics at work... well, that would be a major break with precedent, and a huge reach beyond what the text of the amendment says, with wide-ranging ramifications. Instead, some states have passed specific laws that protect workers from discrimination on the basis of political opinion, without needing to invoke the First Amendment.)

sonotathrowaway(10000) 3 days ago [-]

'If your values aren't used to inform who you're doing business with, why do you bother pretending to have values at all? This [merge request] demonstrates that you don't have any values except 'we want to make money, and it doesn't matter who gets hurt.''

It sounds a lot like GitLab has a set of values that it doesn't feel comfortable openly championing, so it just lies.

buboard(3489) 3 days ago [-]

that black or white thinking sounds fallacious

kemenaran(3692) 3 days ago [-]

Technical professions are generally held responsible for what they enable. Biology: researchers can't do anything they want on plants, animals or humans. Medicine: doctors are expected to follow strict guidelines with their patients. Even engineering: implementing a cheating system for diesel tests will (rightfully) get engineers into trouble.

But software engineering seem to have a pass on how the things we build is actually used.

Dark patterns, facial recognition, mass surveillance? Just some code, a nice puzzle. It's like the profession as a whole feels only distantly connected to what they actually do.

I welcome the trend of developers taking more moral responsibility for their work. I want more ethics courses, more difficult questions, more uneasiness about what we do with our code, and what we enable. Not to be sure we build the right thing, but at least to be less wrong–and, sometimes, to know when we clearly build the wrong thing. I want us to be accountable; like other technical professions in the world.

And if in the workplace, the place where the majority of software in the world is usually produced, we can't have these discussions, what use will it be?

scarejunba(10000) 3 days ago [-]

This fetishization of these 'professional careers' has got to stop. Particularly this fictional glamourization of the philosopher professional who is held responsible for all outcomes of his work. To this day, Dr. Jeffrey Glassberg roams free despite mistaken forensic DNA tests putting so many innocent people in jail. That is one, but I could produce hundreds. As a matter of fact, I challenge you to name a single inventor (who, for sake of this argument, we may choose to be also a 'professional') who is innocent by this metric.

randyrand(4199) 3 days ago [-]

Your examples aren't very convincing for biology and medicine. The counter examples seem way more numerous.

Biologists enabled chemical warfare. Same harm has been done with mathematics. Physics.

I think we largely let researches do research, and don't hold researchers accountable for harm enabled with that research.

In other words, self policing of customers seems like a newer and uncommon phenomenon.

darawk(10000) 3 days ago [-]

> Technical professions are generally hold responsible for what they enable. Biology: researchers can't do anything they want on plants, animals or humans. Medicine: doctors are expected to follow strict guidelines with their patients. Even engineering: implementing a cheating system for diesel tests will (rightfully) get engineers into trouble.

Biology researchers are held responsible for what they do. Not usually what they enable. The examples you gave are biology researchers themselves doing something. The relevant analogy would be if we held the inventors of CRISPR responsible for some future super-virus engineered using it.

nradov(886) 3 days ago [-]

Software development is not actually a profession, it's just an occupation. That's not necessarily a bad thing but let's not pretend that it's something that it isn't. In order to be a profession on the level of law, civil engineering, medicine, arms, accounting, etc it would need — among other things — a clear code of ethics. Years ago the IEEE made some tentative steps toward turning software development into a profession but they didn't get much traction.

justapassenger(10000) 3 days ago [-]

How does arguing if politician X or politician Y is better makes tech more responsible?

Supporting politician X doesn't make person more moral or ethical. Same as believing in god A over god B.

Politics discussions are just tribal wars.

buboard(3489) 3 days ago [-]

> hold responsible for what they enable.

That's irrelevant. The rules are for conducting responsible research, not for pondering the potential misuse or evil use of whatever technology. Physics and bio have a far greater record of evil uses, and it was never questionable. It was accepted that scientists do science, and ask questions later. The before-the-fact crusades are a very recent (like, 10 years) phenomenon limited to the west coast of the US

RcouF1uZ4gsC(4022) 3 days ago [-]

> Technical professions are generally hold responsible for what they enable.

Not really. Doctors are expected to do the best for their patient without worrying about societal consequences. If Hitler came to a doctor in need of medical help, the doctor would be ethically required to do their best for him, desolate the negative consequences of Hitler to society.

Same with lawyers. Even if their client is a serial murderer and has co fessed their guilt to them, they still have to do their best in representing them.

vowelless(3861) 3 days ago [-]

I am glad Dang changed the title as the original one was unnecessarily inflammatory.

I don't understand why it's encouraged (in some companies) to discuss politics at work in a way that leads to internal issues. Purely from a commercial stance, team cohesion has a positive impact on people and product. Why do anything to disturb that?

> Such a declaration could run afoul of legal boundaries in some circumstances. While workers have no constitutional speech protection in the context of their employment, federal labor law requires that employees be allowed to discuss the terms and conditions of their employment and possible unlawful conduct like harassment, discrimination, and safety violations.

This seems like a false dichotomy. There is a difference between being amoral and illegal.

The nazi example they gave is also quite egregious. America has sanctions. If the public wants corporations to not interact with certain countries, they can ask their legislators to fascilitate passing of sanctions (like we have today against Syria, making it illegal to provide services to them).

Being held to a political moral standard is tricky if you are not in the mainline political stance. That would make me quite uncomfortable. I go to work to support myself and my family. Don't make that hard for me to do due to politics.

shuuut(10000) 3 days ago [-]

You are right. We should not allow any discussion at work that is not about the work itself.

There are so many things that can disturb team cohesion! Even non political ones like talking about if you prefer rain or sun, or fries with mayo or ketchup.

Thank you! And now let's go back to work...

com2kid(4163) 3 days ago [-]

I had plenty of conversations at work about differing tax policies, libertarian vs authoritarian policies in regards to land ownership, drug legalization, military action, etc.

The problem is the willingness of people to engage in civil discourse has fallen off a cliff. 'Thought leaders' on both sides are making serious $$$ by convincing their followers that the other side is to be truly despised and that all opposition must be met with aggression of some sort (if not physical, verbal, spiritual, etc). Creating an us vs. them mentality has proven (yet again in history) to be very profitable.

Every time this happens, the results are regrettable and 50 years later everyone is looking at history books going 'what the holy hell were they thinking?'

tl;dr the problem isn't discussing politics at work, the problem is that society in general has turned into a team sport where people pick sides and start acting like hooligans when they meet someone from the other side.

microtherion(2571) 3 days ago [-]

> Purely from a commercial stance, team cohesion has a positive impact on people and product

Do you think the Gitlab CEO's policy declaration helped or hurt team cohesion?

rtkwe(10000) 3 days ago [-]

> I don't understand why it's encouraged (in some companies) to discuss politics at work in a way that leads to internal issues. Purely from a commercial stance, team cohesion has a positive impact on people and product. Why do anything to disturb that?

The issue is the null stance is functionally equivalent to support for the status quo which at this moment includes a lot of stuff people would not actively sign on to but often feel very content with passively signing off on it.

> If the public wants corporations to not interact with certain countries, they can ask their legislators to fascilitate passing of sanctions (like we have today against Syria, making it illegal to provide services to them).

This ignores both just how much the business tail wags the legislative dog in the US and the mountain of other factors that prevent or slow Congress from taking action against specific countries.

Nullabillity(4193) 3 days ago [-]

> I don't understand why it's encouraged (in some companies) to discuss politics at work in a way that leads to internal issues. Purely from a commercial stance, team cohesion has a positive impact on people and product. Why do anything to disturb that?

The health of the democracy is far more important than your company's profit margin. And besides, enforcing silence is still a political move that strongly favours the opinions that the power structures imply with their actions.

eropple(2750) 3 days ago [-]

> The nazi example they gave is also quite egregious. America has sanctions. If the public wants corporations to not interact with certain countries

You're talking countries--that's one case but it's not the only case. America also has Nazis. Here. They are Americans. Even if they weren't, it's an abdication one's core moral responsibilities to one's community and society with this kind of 'oh, ask your [bought] legislator [about something unrelated]' do-nothingism. But they are.

I'd make damned sure that an executive team knew it was them or me before I took one thin dime that might've come from Richard Spencer or one of our other homegrown Nazis. When I was a consultant I fired clients for less and I slept great for doing so. It's literally the minimum duty I have to my community and to the human race at large--the denial of custom to hostis humanis generis wherever they rear their heads? A bare minimum of decency.

x86_64Ubuntu(10000) 3 days ago [-]

Why do titles have to be sanitized for this audience? I can't remember of too many other sites whereby the admins have to essentially scrub the original titles. That makes me question the userbase more than it does the writers of said article.

archi42(10000) 3 days ago [-]

> I don't understand why it's encouraged (in some companies) to discuss politics at work in a way that leads to internal issues.

There is a middle-ground between 'talking about politics' is 'encouraged' and 'forbidden'.

shantly(10000) 3 days ago [-]

> I don't understand why it's encouraged (in some companies) to discuss politics at work in a way that leads to internal issues. Purely from a commercial stance, team cohesion has a positive impact on people and product. Why do anything to disturb that?

I'm not usually a 'the people in power are actively trying to keep the workers divided amongst themselves and seeing one another as the enemy, so they don't organize against the people who are actually keeping them down' sort, but in this specific case... well, there might be a bit of that going on, at least for what 'political discussion' usually means in the context of SV-type workplaces.

hannasanarion(10000) 3 days ago [-]

>America has sanctions. If the public wants corporations to not interact with certain countries, they can ask their legislators to fascilitate passing of sanctions (like we have today against Syria, making it illegal to provide services to them).

This is a very confusing position to me. It is a bad thing to pick and choose your customers based on moral agreement, unless that morality is defined by the government in which case it's good?

repolfx(10000) 3 days ago [-]

I don't think even Google actually encouraged people to discuss politics at work. They never did when I was there. It was more a side-effect of decentralising the power to create forums. Anyone could create discussion lists there without asking permission, this was highly useful for coordination, making it easy to spin up new projects and avoiding siloing of information into people's private mail accounts. The flip side was nothing stopped anyone creating lists to discuss whatever divisive topics they wanted.

There's also been a cultural shift over time. When I first joined Google I remember feeling that it was a pretty libertarian sort of place. In my team several colleagues were openly and proudly very pro-capitalist, anti-communist, small government types. If anyone was left wing it wasn't visible and I don't recall encountering identity politics, ever. There was a very strong cultural commitment to neutrally serving every user with every query, and ranking being completely algorithmic, to the extent that the company agonised over hot-patching 'Googlewhacks' where pranksters manipulated results for unusual searches to offend their ideological opponents.

This didn't seem strange, it was just how it was. I have a vague feeling I thought other tech firms were the same. Twitter declared they were the free speech wing of the free speech party after all, and anything internet or web related had a strong vibe of 'communication will make us stronger'.

This is a very far cry from the modern Google where arguing maybe girls find geeky stuff boring gets you fired, where they manipulate ML models to believe the world is some intersectional 'utopia' as part of social engineering programmes, where employees spend huge amounts of time erasing gendered pronouns from source code comments.

I suspect the shift was caused by Google running out of experienced software engineers to hire, and switching to growing through recruiting new college grads at much greater numbers. The hard shift leftwards in academic institutions has been well documented. Even in 2006 I remember a recruiter telling me they'd 'run out' of candidates in the USA and that's why they were expanding more into Europe. The concept you could 'run out' of candidates in a place as big as America was pretty shocking to me - on probing, he explained that almost all people with the skills they needed were in well paid happy jobs already and few were interested in moving, even with the great perks on offer. Whilst there were always a small flow of people coming onto the market due to natural churn, 'everyone who cares knows Google is hiring' he said, and they'd already interviewed vast numbers of them.

This change in culture would inevitably lead to this situation due to the way that conservatives see leftists as naive, but leftists see conservatives as evil. Combined with leftists generally understanding conservatives much worse than the other way around (this has been shown by large scale ITTs), and together it means flooding your firm with new grads is a surefire way to create massive internal conflicts.

southerndrift(10000) 3 days ago [-]

>I don't understand why it's encouraged (in some companies) to discuss politics at work in a way that leads to internal issues. Purely from a commercial stance, team cohesion has a positive impact on people and product. Why do anything to disturb that?

Fail early. You can achieve cohesion by letting go of everybody who cannot discuss politics in a civilized manner. Then you can be sure that the remaining team can discuss anything without any problems.

On the other hand, is it cohesion when the team is held together by outer influences?

silasdavis(10000) 3 days ago [-]

> I don't understand why it's encouraged (in some companies) to discuss politics at work in a way that leads to internal issues

I am not familiar with companies that encourage discussing politics.

Politics is about how people ought to be organised. I find it strange that this would not be something to discuss in the context of many persons primary organisational structure.

Part of my remuneration is the effect I may have on the world. Positive or negative.

oarsinsync(10000) 3 days ago [-]

> The nazi example they gave is also quite egregious. America has sanctions. If the public wants corporations to not interact with certain countries, they can ask their legislators to fascilitate passing of sanctions

While you correctly note that governments can pass sanctions against countries to facilitate trade restrictions, it's worth considering the full context of the article when considering their descent into proving Godwin's law.

They explicitly state the example of Chef working with arguably contentious and/or politicized domestic organisations. Given that recent employee protests are based around domestic government organisations, that have (rightly or wrongly) been compared to nationalist and/or protectionist organisations in history, what is the correct course of action there?

If you disagree so strongly with your own government, and/or believe your own government cannot be trusted to tell you the truth, what is the correct response? In recent history, we had marches against going to war on what wasn't considered believable grounds. Despite the best intentions of the leaders at the time, who ultimately ignored the peaceful and legitimate protests, the protestors were later proven to be true.

Given that backdrop, and the corporate / commercial capture of legislative interests, how is employee activism not a logical next step of 'vote with your wallet' combined with 'change it from the inside'?

ljm(10000) 3 days ago [-]

I don't think the issue is politics itself really, it's more of a lack of maturity or compassion when it comes to having the conversation.

If people engaged in the conversation aren't prepared to take every participant seriously, no matter their politics stance, then it's a sure-fire way to create a toxic work environment. You can't go into it with the assumption that everyone needs to be a left-leaning liberal, or that it's okay to bully a Trump voter (or vice-versa with right-leaning and Obama).

If at that point you can hold a conversation about politics without losing your shit as soon as someone supports something you don't like, then encouraging it is fine. If you're still in the mindset of having compassion for only the people on your own side then some growing up needs to happen first. Otherwise all it does is alienate those colleagues with marginal beliefs. Not exactly diverse and inclusive then.

And it doesn't necessarily follow that you'll enable far-right nutjobs or neo-Nazis as a result. HR tends to frown on things like racism and such like, it wouldn't really get very far.

tom_mellior(3681) 3 days ago [-]

> Being held to a political moral standard is tricky if you are not in the mainline political stance. That would make me quite uncomfortable. I go to work to support myself and my family. Don't make that hard for me to do due to politics.

This cuts both ways: If your employer is known to take millions from ICE to support what ICE does, and your opinion is that ICE is an evil organization, that can also make you 'quite uncomfortable'. It might also make you want to leave that employer, which would make it harder to 'support yourself and your family'.

legostormtroopr(3979) 3 days ago [-]

Good to see that the very non-controversial opinion of "be neutral at work and treat customers equally" is being used to compare GitHub to IBM during Nazi Germany.

Why can't GitHub just provide code version control systems without concerning itself with who is writing code?

Given the recent implosion of StackOverflow, less politics in tech is probably a good thing.

rubbingalcohol(3151) 3 days ago [-]

I'm not sure why you're getting downvoted but no comments addressing the points you raised.

Can anyone provide an example of Nazi code being hosted on Gitlab? It might be nice to justify what otherwise looks like an absurd comparison made by this article. What would nazi code even look like? Or is 'nazi' in this case just a catch-all for wrongthink?

dangthecoward(10000) 3 days ago [-]

The stated policy is that they would take nazi money.

dang(179) 3 days ago [-]

The Register has a special mix of technology reporting and satire. As with the title, that bit is obviously there to maximize provocation. Yet the article itself has some diligent research behind it, poking through commit logs and internal threads that few other journalists would. As a result it appears to be a scoop. Googling doesn't reveal any other media piece on this. I bet there will be others after this thread, though.

Basically, it's their job to provoke and our job as a community not to take the bait.

dvt(1708) 3 days ago [-]

GitLab are being hypocritical[1][2]. Let's not pretend the 'women in tech' and LGBT movements aren't political. Frankly, I think more women in tech and LGBT representation are probably good things, but GitLab is being profoundly disingenuous with their guidelines.

To clarify my point: if you open the gates to the marketplace of ideas, make sure it's an actual marketplace.

[1] https://about.gitlab.com/blog/2018/10/08/stem-gems-give-girl...

[2] https://shop.gitlab.com/products/womens-rainbow-shirt

daxelrod(4072) 3 days ago [-]

When setting workplace norms, there may be a substantial difference between discussing, for example, social issues, policy, partisanship, governance, and ethics. Depending on context, any of these may be 'politics'.

It feels uncharitable to call them hypocritical without understanding what they mean.

sgustard(3080) 3 days ago [-]

Topics like 'who should we hire' are relevant to the business, and that overrides politics. Obviously topics like 'what business should we be in,' 'what countries should we sell to,' 'what customers do we want' and on and on can have political components. But those are nonetheless important conversations to have at work.

anarchodev(10000) 3 days ago [-]

The difference here is the potential for revenue generation stemming from the ostensibly political stance. Gitlab's goal is to make money, and they're signalling that this goal is firm regardless of who is hurt by their customers, using their tools. So they're happy to sell you a rainbow shirt to support women in tech. If there was a similar shirt they could sell about ICE and CBP (and that would be the end of their involvement in said political stance) they'd sell that too. Instead, the moral stance is obviously to not work with these organizations, and that will potentially cost them money.

This is pretty standard neoliberal politics. Just as with similar decisions at Microsoft and Palantir, this change will never come from executives. It's up to the workers to change the company's direction, and I hope they will.

SpicyLemonZest(10000) 3 days ago [-]

I think it's fair to call women in tech initiatives nonpolitical. There's no significant political opposition to them; things like the INSPIRE Women Act (the women in tech initiative for NASA) regularly get passed with no real debate.

codetrotter(3852) 3 days ago [-]

Could you clarify why you mean that women in tech and LGBT movements "are" political?

Movements are broad, and one way of reaching the goals that people in movements seek may be through political means, but that goes for a lot of things, and there is always a lot of of other ways of reaching those goals that one can work on without bringing politics into the picture.

For example, political legislation is one way that we may bring more women into tech, but there are many other ways too. The GitLab page you linked is focusing on inspiring the next generation of women to work in tech, and telling stories about successful women inside of GitLab. I did not immediately see anything inherently political on that page, though I only skimmed through it briefly so I may have overlooked something, in which case feel free to point out any examples of anything political that you see there.

Certainly, blanket saying that talking about politics is not allowed may well turn out to lead to situations where something becomes challenging to talk about because politics may be intertwined. For example, a speaker might go into politics territory in some talk, and how should they handle that? But that could happen in any area of discussion.

Equal opportunity, equal rights, and equal treatment, which is at the core of both the women in tech and the LGBT movements, is not political. It is about human rights.

I don't see any hypocrisy in simultaneously wanting to talk about certain issues at the company level while at the same time saying that national politics are to be avoided. But like I said, the two may intertwine more than we'd like and it could be difficult to distinguish the two in some or even a lot of cases.

dang(179) 3 days ago [-]

Those are great questions. But please omit the flamebait. (Edit: dvt kindly took the flamebait out—see below.)

If flames take over, only the people who enjoy flaming will remain in the debate. That's a strict loss for thread quality, which is one reason for the guideline I asked everyone to remember above: 'Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.'

https://news.ycombinator.com/newsguidelines.html

mosselman(3926) 3 days ago [-]

What do you mean? I am honestly asking, not flaming or anything. What is the point you are making?

djsumdog(1073) 3 days ago [-]

It's hard to define what is and isn't political right? It can get ontological .. (i.e. if everything is political than nothing is political and if nothing is political than everything is political).

It seems like in this case Gitlab is trying to avoid some of the Google/Amazon blowback for taking questionable contracts. In those cases .. tech people are in high demand. It's not that difficult for us to find work elsewhere.

That being said .. it's almost impossible to be moral in customers. Everyone company does questionable stuff, has questionable clients, mixes normal with military contracts, etc. Should Gitlab not allow projects from churches who create tools to help minister to homosexuals (in trying to convert them away from homosexuality)? Is being intolerant of the intolerant .. tolerant?

There are really complex questions here.

brlewis(1511) 3 days ago [-]

Thank you for providing those links. In both cases gitlab is partnering with a non-profit organization that AFAICT is not lobbying to pass or repeal laws. I understood 'politics' in the context of this article as being about election candidates, government actions, or changes to the law. These gitlab partnerships are only 'political' in the sense that people who like to argue about politics also like to argue about topics that these partnerships touch on.

vowelless(3861) 3 days ago [-]

How is women in tech a political thing? Just seems like an effort to make the work place more welcoming to people.

daliusd(10000) 2 days ago [-]
NietTim(3798) 2 days ago [-]

I like it, just wish they'd drop the 'against protected groups' from point three. Encouraging violence or discrimination against some other groups are okay? (no) Odd.

ergothus(4219) 3 days ago [-]

I have a pet theory that we (that is, the culture I'm familiar with, so I don't really know how widespread it is, but at least the white U.S.) have spent too long saying 'don't discuss politics, sex, religion, and politics'.

We literally have no practice handling differences regarding the very items people are passionate about.

The answer isn't to get into massive drama-filled flame wars, nor to drive people with minority opinions into hiding, but at the same time I don't think continuing the 'enforced silence until it's considered common knowledge' is the right way to handle it.

See another comment pointing out that issues like gender equality, cognitive diversity, and workplace rights regardless of sexual preferences are ALL 'political', but also directly impact the workplace. How do we decide workplace issues if not at the workplace?

Karunamon(3248) 3 days ago [-]

>We literally have no practice handling differences regarding the very items people are passionate about.

Looking at it from the employer side, that passion is precisely why these policies are being enacted.

Were I employing someone, I'm required to not have a hostile work environment, which includes certain jokes (because people may be offended), and now includes politics (because people may be offended), and people being offended means I can get sued.

The safest thing I can do, that any company can do, is say 'check your politics at the door' and then fire anyone who does it anyways.

It's a garbage solution for the reasons you bring up, but unless we change how those regulations are set up, it's the option that many will take.

rawland(3067) 3 days ago [-]

There is an issue opened to revert this:

    > [...] I understand the motivations behind the
    > policy change but they are misguided: GitLab,
    > please show that the concerns of your community and
    > staff are our first priority by reverting
    > this change.
See: https://gitlab.com/gitlab-com/www-gitlab-com/issues/5579
whoisjuan(4105) 3 days ago [-]

An update was merged a few minutes ago: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

milesward(10000) 3 days ago [-]

Human rights aren't politics, they're non-negotiable.

fipar(4139) 3 days ago [-]

I was born under a dictatorship (that fortunately became again a democracy while I was still a child) and I can tell you human rights don't exist unless you have enough people willing and able to stand up for them.

More broadly, laws aren't worth the paper they're written on if you don't have a police and armed forces willing and able to uphold them.

I understand what your point is; I've talked to several people who voice the same opinion when discussing my country's dark past, but trust me, when the bullies get to power, you can't shout 'it's a human right!' back at them. You either have to fight back, hide, or run away.

Hamuko(10000) 3 days ago [-]

Human rights are massively political. Saying anything else is either ignorant or naive.

Soviet bloc countries didn't vote for the Universal Declaration of Human Rights because of the right of citizens to leave their countries. And let's not get into the human rights of Uyghurs and the worldwide approval/disapproval to China's re-education camps.

einpoklum(10000) 3 days ago [-]

There are few things more political than rights, their extent, their upholding or abrogation, their contradiction etc.

jlawson(10000) 3 days ago [-]

Right, and when different human rights conflict, then we resolve the conflict through a political process.

Simple example: Religious freedom/freedom of speech and gay rights. So should the Christian baker be _forced_ to make a cake with words explicitly against his religious beliefs? Which human rights come first here?

notadoc(4114) 3 days ago [-]

People disagree on what exactly qualifies as 'human rights' though, and perhaps that's where it becomes political.

Smithalicious(4215) 3 days ago [-]

Human rights exist only as a political construct formed by many treaties and agreements which have been thoroughly negotiated. I could not come up with anything more political than international law if I tried.

smudgymcscmudge(4154) 3 days ago [-]

I might agree with you that human rights are non-negotiable, but we're probably going to get into a fight when it comes time to list those human rights.

yawgmoth(10000) 3 days ago [-]

Agreed. I am reminded of MLK Jr's Letter From A Birmingham Jail: https://www.africa.upenn.edu/Articles_Gen/Letter_Birmingham....

The salient point being that indifference at scale is worse than individuals with explicit bias.

kazinator(3824) 3 days ago [-]

Rights are absolutely negotiable; it's part of your right to be able to trade some of your rights for some benefit.

If you're employed at a desk job, you've traded much of your human right to be wherever the heck you want between the hours of 9 and 5, Monday through Friday, in exchange for a salary.

claudeganon(10000) 3 days ago [-]

While this seems self-evident, this claim is only made possible by a mass politics and organization to defend its "non-negotiability." Arendt wrote about this problem as the "right to have rights"

http://criticallegalthinking.com/2019/07/12/hannah-arendt-ri...

oh_sigh(10000) 3 days ago [-]

Who said otherwise?

duxup(3967) 3 days ago [-]

I feel like discussing politics at work has always been something most folks try to avoid.

On the other hand your company telling you not to is a whole other situation. I'm not sure a policy really solves much. Anyone who wouldn't think twice before discussing such things ... probabbly isn't going to read / care about the policy.

It's one of those rules that no matter how well intentioned, I'm not sure it works well as a policy.

PunchTornado(4162) 3 days ago [-]

we discuss brexit every day at work. in my previous job too. i don't believe there is an office in Britain where brexit is not mentioned.

eric_b(10000) 3 days ago [-]

I personally have no problem with them making this decision. I wish more companies would take stands like this. At least it's honest. Unlike the NBA and every other company with value statements they honor only when convenient or advantageous.

Look, if you morally object to this, then don't use GitLab. If you think businesses should be held to a higher ethical standard than they are currently, get active with local or even national politics. Start a letter writing campaign to your elected officials. Do something productive.

You know what isn't productive? Getting outraged on Twitter or Hacker News, making emotional outbursts of regurgitated sound bites and platitudes.

pdonis(3976) 3 days ago [-]

> If you think businesses should be held to a higher ethical standard than they are currently, get active with local or even national politics. Start a letter writing campaign to your elected officials. Do something productive.

I would say the productive thing to do would be to start your own business that holds itself to a higher ethical standard, and then out-compete the other businesses.

saghm(4181) 3 days ago [-]

> You know what isn't productive? Getting outraged on Twitter or Hacker News, making emotional outbursts of regurgitated sound bites and platitudes.

Genuine question: you're making a strong statement on Hacker News expressing your disagreement with how other people are acting; what do you think distinguishes this from the behavior that you're describing that makes it more productive?

oneepic(10000) 3 days ago [-]

O/T: the 'Blood Money' part of the article title is why I hate news media in general. Sensationalizing, editorializing headlines marked as 'news' and not just opinion.

phjesusthatguy3(10000) 3 days ago [-]

This is The Register. Their headlines are fairly hyperbolic and/or tongue-in-cheek (or both) about everything.

buboard(3489) 3 days ago [-]

They should have tried something far worse, this is not funny enough. I suggest 'Gitlab: Eating babies is fine'

mprev(3675) 3 days ago [-]

I'm not defending it but The Register is, avowedly, aiming to be the tabloid of tech news.

RcouF1uZ4gsC(4022) 3 days ago [-]

I think this sounds very reasonable for the kind of tool that is being produced.

Joel Spolsky has a great article on Big Macs vs Naked Chef

https://www.joelonsoftware.com/2001/01/18/big-macs-vs-the-na...

In there, he talks about bespoke development vs mass market products, using McDonald's Big Macs vs a chef.

I think this provides a reasonable framework for approaching the present controversy.

For example, if McDonald's sold a Big Mac to <villain>, no one would bat an eye. However, if a chef went to work for <villain> there would be questions.

In the same way, if you are making a mass-market, generally useful tool, nobody should be excluded from using your tool. However, if you are doing custom, hi-touch development, then it might be more justified to screen your clients more.

I think Gitlab falls more into the mass-market, generally useful tool.

archi42(10000) 3 days ago [-]

I think that the point is McDonalds doesn't empower $villain to do $badstuff (that is, unless he is literally starving - in which case it would be inhumane to not give him food). However, technology empowers us to do things that would not be possible without it.

Sometimes, a company can not choose who uses their tools - e.g. if it is open source, you can hardly prevent $villain from using it. This is similar to a hammer: If $villain buys a hammer at a hardware store, the hardware store is not to blame if $villain smites people (unless maybe $villain is known to be villainous, but that's not the point right now).

On the other hand, if a known $villain comes to you, and asks you to help him commit $atrocities: You can make the decision not to help.

(So, to put it short: I think your analogy is bad).

pimmen(10000) 3 days ago [-]

The analogy avoids the difference that food is a consumable that everyone, from minimum-wage workers and billionaires, uses and getting better quality food doesn't make you orders of magnitude more powerful than everyone else. Computer technology is different, it enhances the differences between people. An artist working for a villain to make propaganda posters doesn't raise as many protests as Facebook being open to villains because through Facebook your campaign can get orders of magnitude stronger.

If Happy Meals came with an assault riflevpeopke would probably be more angry at McDonalds for catering to bad guys.

Illniyar(3400) 2 days ago [-]

Gitlab falls into both - what you are referring to is the cloud offering, the self-serve one.

I doubt anyone (employee or otherwise) would bat an eye if ICE or the U.S. military would open a private repository on Gitlab cloud.

But Gitlab also does enterprise sales (I assume it does enterprise sales at least since they have enterprise software) - which usually means customized integration, bizdev and account management. More often than not organizations like Gitlab approach the companies to sell to them. In the case of government agencies they often can't even approach legally, Gitlab would have to apply into an open bid contract.

pfranz(10000) 3 days ago [-]

It's tough to make appropriate analogies (which are hyperbolic) without the topic being blasphemous. But companies like IBM, Hugo Boss, and Volkswagen still get talked about as supporting Nazis during WWII. BASF even has a page about their involvement manufacturing Zyklon B gas [1]. I do think you have a point of bespoke vs mass market, but I still think there's a line. Even in the bespoke category you hear about celebrities taking large paychecks to attend parties for questionable people and often get defended because they're not formally supporting them but are draining their wealth by some amount.

I think the difference might be a commodity relationship; nobody would question <villain> using Windows, but a business account with Microsoft would. Even with <villain> using Windows, if it was a prominent and ongoing thing I imagine people would expect Microsoft to intervene.

[1] https://www.basf.com/global/en/who-we-are/history/chronology...

secretdark(4220) 3 days ago [-]

What I find most amazing is less the politics aspect and more the moral/ethical issue. For a company to have an explicit stated policy of _actively_ never taking into account the moral/ethical impact of their work is just... astounding.

It is difficult to imagine a more inhuman and bloodless statement.

shadowgovt(10000) 3 days ago [-]

The company isn't run by robots. Someone is still making judgement calls.

This policy just says if you're the wrong somebody, you can be fired for taking about it.

dangthecoward(10000) 3 days ago [-]

It really is just shocking.

We know what kind of person the CEO is and it's not good. Not a person i would ever want to associate with.

Hamuko(10000) 3 days ago [-]

Didn't Cloudflare take that stance prior as well?

umvi(4218) 3 days ago [-]

I think that's a bad faith interpretation.

It's more like a pressure cooker company saying 'We build kitchen tools. We don't care who buys them. We will not spend any resources vetting customers to make sure they won't blow up a marathon with our product. Yes, even neo nazis are allowed to buy our product even though we executives disagree with them politically/morally/etc.'

That's an entirely reasonable stance.

ga-vu(2351) 3 days ago [-]

They're Ukrainians. Slav people are pretty blunt about these issues. Simply put, they don't care. Work is work. Politics is not work.

gdy(10000) 3 days ago [-]

That's racist

anm89(3804) 3 days ago [-]

I personally think this is awesome. I don't wan't some git hosting startup to be the arbiter of morality for society. The engineers, designers, and PMs shouldn't have an outsized voice in society because they have a specialized useful skillset and ended up on a successful product.

If these users are breaking laws, then put them out of business via the courts and sieze the assets (the repos in this case) via legal means. Otherwise why would I wan't gitlab to have anything to do with this process?

The tech unicorns screwed themselves over BIG TIME, the second they stopped claiming they were just infrastructure and platforms and got into content moderation. They will now forever be a pawn of whoever has some power and has some agenda. It's an obviously unwinnable game for everyone involved besides maybe some politicians.

I don't want this to become a Joe Rogan debate but Naval Ravikant got this exactly right on his Rogan Interview: https://www.youtube.com/watch?v=3qHkcs3kG44&t=3661

brnt(10000) 3 days ago [-]

'Arbiters of morality of society' is only possible because American capitalism is rigged to favor gravitation towards one or a few consolidated entities. This need not be. When Americans ask, where's Europe's Google, I smile, because the landscape, for now, seems to prevent consolidation into a handful of olicharch, and therefore no one company can arbit anything. There's always loads of choice, or there should be.

Next, the bystander effect. Is it commendable a company watches as bad actors use its platform up until a court ruling puts an end to it/ Another area where Europe has history and Americans would do well to learn.

mayneack(1786) 3 days ago [-]

> The engineers, designers, and PMs shouldn't have an outsized voice in society because they have a specialized useful skillset and ended up on a successful product.

The size of every group's voice in society isn't a magical balance. I don't want rich people or corporations to have an outsized voice either. It's a zero sum game, so the only way to reduce that is to give more voice to other groups. Employees should try to leverage their numbers more, not less, because that's the only tool they have. Gitlab is based in San Francisco, so their votes probably don't matter either.

pnw_hazor(10000) 3 days ago [-]

Yes. I agree that companies shouldn't become adhoc moral arbiters. beholden to social media campaigns, or the like.

However, the bigger problem is that companies with international presence are going to have an increasingly difficult time navigating legal sanctions or directives from the various countries they may sell to or operate in.

The challenge comes when a company is faced with being closed out of a large market if they don't comply with another country's legal directives. While, the best/right answer might be to ignore those countries, the reality of having to withdraw from valuable markets would be challenging.

Though, while it may seem like new problem, non-digital industries have been dealing with this kind of thing for a long time. Maybe the WTO, or the like, will develop fair trade/IP regime to guard against enabling a country to block or sanction international digital/media companies for violating local laws. But I wouldn't expect anything like to happen soon because countries are not to keen on giving up sovereignty.

Maybe spontaneous digital trade wars will become a thing.

edit: typo as usual

wpietri(3351) 3 days ago [-]

'Don't discuss politics at work,' is a political statement. It rules particular views in bounds and rules others out.

Similarly any business ends up making moral choices. It's unavoidable. To act in the world is to have impact. To consciously act is to intend particular impact. Commerce is inherently social; you choose to serve certain people in certain ways, with particular outcomes intended. These choices are inherently moral.

The notion that we should ignore that sometimes is of course a strongly political view. And when it pretends to be non-political, I think it's also incoherent.

KirinDave(641) 3 days ago [-]

But don't you think individual employees have a responsibility to recognize when they're contributing to causes they don't like, and possibly resigning?

Doesn't this rule therefore forbid employees from discussing and acting collectively over this, which in the US is a federally guaranteed right?

theon144(10000) 2 days ago [-]

>I don't wan't some git hosting startup to be the arbiter of morality for society.

Me either! The point is everyone should be the arbiter of morality for society. Abstaining from moral judgements because it's 'inefficient' is just alibistic, and leaves society worse as a whole. Tech companies are participating in society just as any other company, there's no reason they should be exempt from ethical considerations.

Technology is not neutral, never was, and pretending it is isn't going to change it - it's just going to pave the way for 'inevitable' unethical programs and moral lapses (see: mass surveillance, Amazon->Ring cooperation, ICE cooperation...)

xenocyon(4194) 3 days ago [-]

Counterpoint: many tech workers want some agency over what they are building and don't want to be forced to work on nefarious applications. I don't seek an outsized claim on society but I do reserve the right to decide what my hands will - or won't - build.

empath75(1877) 3 days ago [-]

I, as a user, and as an employee am going to refuse to do business with nazis or do business with people who do business again. It's a free country and I'm allowed to do that, and companies like GitLab can do whatever they want with that information.

I'm not the only person like me, and they're going to have a choice as to whether they want to do business with nazis or so business with people like me.

They don't get to just wave their hands and make these sorts of decisions go disappear. They have to make a choice.

anarchodev(10000) 3 days ago [-]

> I don't wan't some git hosting startup to be the arbiter of morality for society. The engineers, designers, and PMs shouldn't have an outsized voice in society because they have a specialized useful skillset and ended up on a successful product

Refusing to serve a customer when you disagree with that customer's goal is pretty far from being a 'moral arbiter for society.' Keep in mind that anyone is free to use other services (or roll their own) and gitlab can't do anything about that. Neither would it be overreaching for the workers building that product to request a say in how it's used.

> If these users are breaking laws, then put them out of business via the courts and sieze the assets (the repos in this case) via legal means.

Refusing someone a service you provide is a completely legal action. This has never been illegal afaik. In many cases I can think of the users wouldn't be actually breaking any laws, which isn't the same as saying that their actions aren't immoral.

> The tech unicorns screwed themselves over BIG TIME, the second they stopped claiming they were just infrastructure and platforms and got into content moderation. They will now forever be a pawn of whoever has some power and has some agenda.

In this last sentence, who are you claiming has power? It seems to me if I had power I wouldn't bother trying to persuade my boss not to do business with certain agencies, I'd just make it illegal and force them to change. Does that seem like a course of action available to gitlab employees?

3xblah(10000) 3 days ago [-]

Maybe they are not 'infrastructure'. Maybe they are actually middlemen.

InfinityByTen(10000) 3 days ago [-]

Finally I read a comment on a platform and I can listen to ideas and deep understanding. Thanks for sharing that interview!! You won Naval a fan already :)

gameswithgo(4103) 3 days ago [-]

>The engineers, designers, and PMs shouldn't have an outsized voice in society because they have a specialized useful skillset and ended up on a successful product

So do you also feel that billionaires shouldn't have an outsized voice in society?

dwild(10000) 2 days ago [-]

> If these users are breaking laws, then put them out of business via the courts and sieze the assets (the repos in this case) via legal means.

What if it's legal where's it's happening? Let's say that China use Gitlab to host their repo for their software that track US citizens. Would you be happy with that?

> The tech unicorns screwed themselves over BIG TIME, the second they stopped claiming they were just infrastructure and platforms and got into content moderation.

That's absurd, it was always true for ANY industry. You can decide not to support a company by not buying there because you don't support their moral standards. That's perfectly fine. Would you still be with an ISP that is against net neutrality while there is another one that is for net neutrality? Now replace 'net neutrality' with any other moral decision and it's still valid.

That's what make company decide to takes moral stands. It's purely economics.

> They will now forever be a pawn of whoever has some power and has some agenda.

That was always true and still is even with Gitlab decision. Even in this case, Gitlab decision makes them a pawn to whoever has the means to do business with them. They want to still be able to get the business of ICE, China, etc... and you.

LegitShady(10000) 3 days ago [-]

I had no idea who Naval Ravikant was until I randomly read one of his tweets and thought 'this dude has good thinking' and went to look up him. Interesting character.

brianpgordon(4069) 2 days ago [-]

Companies do not magically become exempt from needing to behave ethically just because they're participants in the economy. If, for example, some government is committing human rights violations, companies have an ethical obligation not to enable it, even if the services they offer aren't under specific legal sanctions. This idea of a company just doing business with no social or ethical responsibility to anyone other than stockholders is an ancap fantasy.

And Gitlab isn't some unconcerned third party passing judgment on morality for the rest of us. We're not talking about your local Dairy Queen franchise gratuitously taking a position on Brexit for no reason. The only power Gitlab has is to withhold their own service. If you're going to argue that their service is so important for society to function that withholding that service constitutes being an 'arbiter of morality for society,' to the extent that's even true it only increases their responsibility to make sure its products aren't used for evil. A Dairy Queen franchisee doesn't have much power so they don't have much responsibility, but a Google or an IBM has enormous power to make a difference and therefore they have enormous responsibility.

I fundamentally don't agree that corporations should be given carte blanche to use any legal options to maximize profit with no regard to the consequences. The political process and legislation/regulation is probably a better way to address companies behaving unethically, but given that in many cases our political process has failed to put those laws in place we need to hold corporations accountable ourselves.

graeme(2852) 3 days ago [-]

You'd be surprised at what is 'not illegal'. Every community moderates.

You can make an argument that some sites have moderated too much, or been too overtly political.

But every community moderates, and has to. This can be demonstrated with a simple example. Guess what isn't illegal? Spam!

It should be obvious that FB and Twitter etc have to take spam down though.

Then there's abusive behaviour. Stalking, harassment, etc. Often not illegal, but hurts a platform. Better take that down.

What about propaganda? They should let it all go? That sure sounds like a....phone network. Actually wait, it sounds more like the news media, which has been regulated for decades. A zero moderation policy would likely have led to demands for regulation, too.

And then there's the massive category of topics which are not illegal, but horrifying. Have zero moderation, and you end up as 4chan or worse.

--------

I'm generally in favour of keeping politics out of things, but your 'we take stuff down only for a court order' is a naive view. Literally every forum moderates, they have no choice.

neonate(2396) 3 days ago [-]

Wow, that is such a clear analysis that someone should transcribe it. Here's the main part of what Naval says:

'If Google, Facebook, and Twitter had been smart about this, they would not have picked sides. They would have said 'We're publishers. Whatever goes through our pipes goes through our pipes. If it's illegal, we'll take it down. Give us a court order. Otherwise we don't touch it.' It's like the phone company. If I call you up and I say something horrible to you on the phone, the phone company doesn't get in trouble. But the moment they started taking stuff down that wasn't illegal because somebody screamed, they basically lost their right to be viewed as a carrier. And now all of a sudden they've taken on a liability. They're sliding down this slippery slope into ruin, where the left wants them to take down the right, the right wants them to take down the left, and now they have no more friends, they have no allies. Traditionally the libertarian-leaning Republicans and Democrats would have stood up in principle for the common carriers, but now they won't. So my guess is, as soon as Congress (this day is coming if not already here...)... the day is coming when the politicians realize that these social media platforms are picking the next president, the next congressman. They're literally picking, and they have the power to pick, so they will be controlled by the government.'

adamsea(4214) 3 days ago [-]

To quote someone quoted in the article, 'As a commenter identified as 'casiotone' observed, 'If your values aren't used to inform who you're doing business with, why do you bother pretending to have values at all? This [merge request] demonstrates that you don't have any values except 'we want to make money, and it doesn't matter who gets hurt.'''

dominotw(2050) 2 days ago [-]

> The engineers, designers, and PMs shouldn't have an outsized voice in society because they have a specialized useful skillset and ended up on a successful product.

isn't that true for all voice in the soicety though.

jtms(4219) 3 days ago [-]

I would advocate for Joe Rogan, he is a voice of openness and willingness to talk about things rather than just shutting down conversations because you might not agree. He is also a great listener. The world needs more people like Rogan.

gbanfalvi(10000) 3 days ago [-]

> The engineers, designers, and PMs shouldn't have an outsized voice in society because they have a specialized useful skillset and ended up on a successful product.

They built these services. They are, ate least in part, responsible for how they work and the consequences of what they built. The content they're serving and data has a meaning to us.

You can't pretend that a social network doesn't have a role in bringing anti-vaxxers together and enabling them to organize.

You can't pretend that an advertising platform doesn't have a role in distributing misleading information by business or other interests.

> If these users are breaking laws...

Laws aren't the ultimate arbiter of everything. There's a reason they change and evolve over time. Personal responsibility is not a law, yet we expect everyone to have some.

p4bl0(320) 3 days ago [-]

That's one part of it. The part were employee cannot discuss politics at work is very problematic however. What about unionizing?

burtonator(2084) 3 days ago [-]

I wish it was that simple honestly.

If Google is doing business with nazis then as a consumer I have absolutely every right to boycott Google.

ProAm(1508) 3 days ago [-]

While I largely agree, history has shown this wont fair well for humanity either. Look at IBM and the Holocaust [1]

[1] https://en.wikipedia.org/wiki/IBM_and_the_Holocaust

kissgyorgy(2098) 3 days ago [-]

You are the kind of guy who would see a murder, would says 'Not my problem.' Not great. Everyone should be the moral compass of everyone else. The society nowadays seems fucking terrible.

jka(4206) 3 days ago [-]

The trouble with libertarianism is, ultimately, that libertarians only care about themselves.

The kind of policy that GitLab is proposing here is fine - or even aspirational, under a certain lens - until one day you wake up and find that the same software is being used to destroy your community and compatriots.

How could that happen? Isn't this free speech to max utility? Isn't that flawless?

No, it's free utility to anyone - and you might eventually find that the people who are using it don't have you and your neighbours interests at heart.

The design of technology is _not_ ideologically neutral. It contains embedded biases and beliefs - even if they were subconscious or part of a community mindset - and those can and will be changed and overruled over time.

Careless licensing of technology will lead to the overriding and subversion of those ideologies, in ways that many of us would find abhorrent.

If you'd like to support that on the basis of free speech, you should ask yourself whether the worst re-applications of these technologies would still allow you to speak freely.

Smithalicious(4215) 3 days ago [-]

Good, this makes me glad to be using Gitlab. Gives me confidence that I can rely on them even if I ever get into hot water politically. I also think that it's good to assert that yes, it's fine for a git host to just host people's repositories and not have to be in charge of making moral judgements. We have other tools for judging morality; things so immoral that society shouldn't allow them should be handled by the law, and things that individuals do not morally agree with should be boycotted by those individuals.

bluntfang(10000) 3 days ago [-]

>Gives me confidence that I can rely on them even if I ever get into hot water politically.

Rely on them for what, exactly? What makes you think they would bend for you and not this other political entity that has you in hot water?

peterwwillis(2589) 3 days ago [-]

Has anyone read the pull request? For those that didn't, here it is:

  + We do business with customers with values that are incompatible with [our own values](/handbook/values/) for the following reasons:
  + 
  + 1. Our mission is 'everyone can contribute', while there is a [code of conduct for contributing](https://about.gitlab.com/community/contribute/code-of-conduct/) we want to get as close to everyone as possible.
  + 1. We [do not discuss politics in the workplace](/handbook/values/#religion-and-politics-at-work-) and decisions about what customer to serve might get political.
  + 1. [Efficiency is one of our values](/handbook/values/#efficiency) and vetting customers is time consuming and potentially distracting.
  + 1. It maps to the MIT expat open source license we use that [doesn't discriminate against fields of endeavor](https://apebox.org/wordpress/rants/456).
Of the 'we do not discuss politics in the workplace', this is the quote from the handbook:

  We generally don't discuss religion or politics in public forums because it is easy to alienate people
  that have a minority opinion. It's acceptable to bring up these topics in social contexts such as
  coffee chats and real-life meetups with other coworkers, but always be aware of cultural sensitivities,
  exercise your best judgement, and make sure you stay within the boundaries of our Code of Conduct.
Here's the problem: this policy can be used to protect people, and disenfranchise people.

On the one hand, you can use this policy to prevent white people from loudly supporting a white supremacist political leader in an office with a very small percentage of people of color. Banning talk of politics here will make it easier to stop dog-whistle racism before it even starts.

On the other hand, you can also use this policy to shut conversations by minority groups who want to support politicians who are improving the lives of people of color. If one goal your company has is, for example, to make your company more ethnically diverse, it would perhaps behoove you to allow different ethnic groups to discuss the political issues they face, and raise awareness of issues critical to them. Banning such speech makes it much harder for them to advocate for better treatment, and educate people in the workplace about the issues they face.

Everyone who thinks this is controversial is falling into the trap of trying to judge a complex issue with emotion, rather than complex rational thought. If you literally decided your opinion about this within 30 seconds, chances are it wasn't very well thought out. I'm willing to bet the CEO is just as guilty of such rushed decision-making.

--

But there were three other points in the PR!! The last of which I think is really worth considering: 'The MIT expat open source license [..] doesn't discriminate against fields of endeavor.'

The most common example given is that 'you cannot stop an abortion clinic, or an anti-abortion activist, from using the source code'. This is an incredibly important part of our society that protects minority groups.

In 2012, in Colorado, a bakery discriminated against a gay couple by refusing to make their wedding cake, because the bakery owners didn't approve of gay marriage. This was (rightly, I think) found by the courts to be illegal discrimination. But you could also consider this case a form of discrimination of field of endeavor, if what you're endeavoring-for is to be married while gay. You can't refuse to make the gay couple a wedding cake - so should you be able to refuse a white supremacist from using your version control tool?

This is where we walk into ethical quicksand. There is a l o n g philosophical rabbit hole you can fall down trying to figure out how to treat people you disagree with. People have spent their whole lives going over these issues and literally nobody has figured it out for certain. And that's why I think the policy, based on the third reason, is acceptable.

The third reasoning, 'vetting customers is time consuming and potentially distracting', is clearly true. We could spend our entire lives arguing the moral philosophy of how to treat people we disagree with, but we'd never get any work done. Sure, you could start an ongoing process of defining who can and can't be a customer, but if it takes any amount of discussion at all, and impacts your business, you're losing money and time, and not necessarily achieving an increase in value, either for society, or your customers/stockholders.

So for the sake of expediency, for simplicity, for being the Switzerland of open source, this one company can allow that particular conversation to go on outside its walls, and continue to just bake cakes and write version control tools, and let history be the ultimate arbiter.

DATACOMMANDER(4047) 3 days ago [-]

What part of the world do you live in, where there's a real risk of a white supremacist politician receiving even 1% of 1% of the popular vote?

aaomidi(4118) 3 days ago [-]

FYI: If you say you don't care about politics and you ban political speech in your workspace - that on its own is a political statement that you're fine with the status quo.

Steven_Vellon(10000) 3 days ago [-]

I disagree. There are a variety of reasons to prohibit political speech in the workplace even if one desires to change the status quo:

* A company might find that it's employees become toxic when political disagreements happen.

* A company might find itself alienating people with minority viewpoints.

* A company may find itself losing customers because political views discussed in the workplace is attributed to the company.

This mentality that lack of political discussion in the workplace implies agreement with the status quo has negative consequences, and makes employees feel pressured to voice support for things they don't actually support in order to avoid losing face with coworkers.

Smithalicious(4215) 3 days ago [-]

Is it? You can not be fine with the status quo but still have a multitude of reasons to ban political speech in your workspace:

- you think political speech isn't useful for changing the status quo

- you think the political speech that will probably take place will lead to a world worse than the status quo, in the opposite direction of which you want it to go (this doesn't just mean 'I think my coworkers are of the wrong party' but also 'I think the world should be less politically divided')

- you think political speech will change the world for the better, but you also just hate political speech even more than you hate the status quo

sbarzowski(10000) 3 days ago [-]

Not necessarily. You may just think that the costs of the discourse outweigh the benefits in some specific context.

throw_xyzyz(10000) 3 days ago [-]

Which one? The one in the wider world, the one in the US, the one in SV?

Udik(3611) 3 days ago [-]

Which status quo? Say what your interpretation of the 'status quo' is, and I'll read it as a political statement. Say that you shouldn't accept it, and I'll read that as a political statement too.

commandlinefan(10000) 3 days ago [-]

Not really, though - the "status quo" at the moment is that everybody should shun U.S. government agencies, whereas gitlab appears to be taking the exact opposite stance; that is, bucking popular opinion.

wbronitsky(10000) 3 days ago [-]

There are many places and times that people do not want to discuss the politics of the outside world. Examples could include while giving birth, while shopping alone at the grocery store, while sitting alone on a city bus, while performing surgery, while discussing your raise with your boss.

Just because someone does not want to discuss something here and now, doesn't invalidate that they have original opinions. Not everything is a dichotomy all the time. Plenty of people who are not fine with the status quo don't want to discuss it at work.

justinmchase(10000) 3 days ago [-]

This article seems to be laced with innappropriate bias. It appears to be judging gitlab for this decision.

Beyond that, its weird that this is not only an uncontroversial position to take but anything other than the only legal option. The government should be the only arbiter of morality with regards to business, where judging your business or actions to be illegal and/or issuing a court order to a company to stop service or remove content should be the _only_ legal way a company can refuse you service.

I do not think putting these kinds of decisions into the hands of unelected people or companies is a good idea.

Kaveren(4215) 3 days ago [-]

Hypothetically, why is it so unreasonable that you would ban someone who advocated for genocide from participating on your platform?

The government is not supposed to legislate all morality. There's plenty of awful things you can do that are perfectly legal. Companies can and should be able to choose who they do business with.

dang(179) 3 days ago [-]

This is a divisive topic, but the story seems to pass the interesting-new-phenomenon test (https://news.ycombinator.com/newsguidelines.html). If you comment, note this guideline: 'Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.'

We've changed the title in accordance with the site guidelines, but if anyone can suggest a better (i.e. more accurate and neutral) title, we can change it again. Edit: I took 'won't exclude customers on moral grounds' out of the title above, since they apparently no longer have that policy. https://news.ycombinator.com/item?id=21275311

kgwxd(2421) 3 days ago [-]

If they no longer have that policy then it no longer passed the interesting-new-phenomenon test and almost all of the article is irrelevant.

midnighttoker(10000) 3 days ago [-]

> We've changed the title in accordance with the site guidelines

You've change it away from the story title, which is normally the site guideline isn't it?

sterileopinions(10000) 3 days ago [-]

You literally are being dishonest.

falcolas(10000) 3 days ago [-]

I don't normally agree with such editorial changes, but this one is really bad. Thanks.

sytse(1757) 3 days ago [-]

Someone at GitLab just made a suggestion to change the wording of the policy that I merged https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

djsumdog(1073) 3 days ago [-]

> we encourage diversity of ideas. We respect the uniqueness and autonomy of others, even if we do not always agree with them.

I like this. I like this a lot.

I've never been a fan of the way the Contributors Code of Conduct does explicitly allow for people to have opinions of their own outside the projects so long as they don't try to represent those opinions as being a representative of the project.

If it had, and offered remedies, the guy in the Opal situation could have just removed Opal from his Twitter profile, instead of being asked to leave the project.

iamasoftwaredev(10000) 3 days ago [-]

So you'll still do business with say, nazis, you just aren't gonna brag quite as loud about it.

You're a coward.

remram(10000) 2 days ago [-]

This reads like a complete 180 on the previous policy. It's very far from a 'change of wording'.

Don't get me wrong, I welcome it, and if the most we can get I feel like it is acceptable. However calling it a 'change of wording' is a bit dishonest, really you reverted it completely in response to the negative feedback.

ricardobeat(3526) 3 days ago [-]

The new text is on a slippery slope: 'Derogatory staments', 'threats' and 'discrimination' are all very subjective and up to the interpretation of whoever-has-power or feels offended by anything, and we've seen this go wrong so many times recently. It's almost a complete reversal from the previous statement.

midnighttoker4(10000) 3 days ago [-]

Does that in any way change the meaning?

Would you still do business with say, turkey as they move in the kurds?

India as they continue to oppress kashmir?

Or gab?

stormfront?

do you have a line beyond legal minimums?

archi42(10000) 3 days ago [-]

This still leaves the 'talking about politics is banned'-point-of-criticism. Since you're the CEO: How did problems discussing politics manifest at GitLab that you feel this step is necessary? Or is this a 'preemptive strike'?

repolfx(10000) 3 days ago [-]

That merge appears to change the policy in ways that render this discussion largely irrelevant (given the current subject line).

It used to say:

We do not currently exclude anyone from being a customer based on moral/value grounds.

followed by a list of reasons why not. It now says:

In some circumstances, we may opt to not work with particular organizations, but we assess those on a case-by-case basis. Some reasons we may choose not to work with certain entities include, but are not limited to ... Making derogatory statements ... Encouraging violence or discrimination against protected groups.

proc0(10000) 3 days ago [-]

The argument 'if you don't engage in politics then you might be part of the problem', is itself a political argument, therefore it is irrelevant in the context of a no-politics rule.

lazyasciiart(10000) 3 days ago [-]

Then the initial demand to not discuss politics must also be political and therefore self-contradictory, so it should be ignored.

cbuchanan(10000) 3 days ago [-]

For those who are a little confused about the no politics at work thing...

If you read the policy we link to about politics, we mention people talking politics during coffee chats, 1-on-1s, etc, but as a general rule we don't lead with those topics.

We focus on inclusion, and we have been very successful at that by respecting others and making work a collaborative, judgment free place. As someone who sometimes has the minority opinion I have absolutely loved this policy, and I feel closer and more accepted by my colleagues as a result.

I'm very proud to work at GitLab.

bluntfang(10000) 3 days ago [-]

what marketing firm do you work for? Oh you're literally the content marketer for gitlab lol (as per your comment history). you're literally getting paid to say this.

shadowgovt(10000) 3 days ago [-]

So when do you imagine they'll be implementing Navajo language support in their UI?

Or is someone going to have to make the judgement call that Navajo users don't constitute a big enough political block to influence their l10n priorities?

Someone has to make those calls

dang(179) 3 days ago [-]

We detached this subthread from https://news.ycombinator.com/item?id=21275269.

repolfx(10000) 3 days ago [-]

Navajo speakers aren't a political bloc, they're a linguistic bloc, and support for languages is driven by revenue potential in any software company I've ever seen.

throw_xyzyz(10000) 3 days ago [-]

>So when do you imagine they'll be implementing Navajo language support in their UI?

What is the roi for implementing Navaho?

jswizzy(10000) 3 days ago [-]

When was it ever okay to discuss politics at work?

cassianoleal(10000) 3 days ago [-]

It may not be in the US (I couldn't tell) but in all other countries I've been it usually ranges from slightly awkward to perfectly acceptable or normal.

It should be ok to discuss politics in any and every space as it affects every single aspects of everybody's lives. Considering most adults spend around half of their awake time at work it seems silly to avoid it there.

archi42(10000) 3 days ago [-]

We actually do this quite often, and that's even though our field is not concerned with politics. But maybe that's a cultural thing? (Edit: Also, views at work vary. In US terms the spectrum is probably from Bernie-Sanders-left to pre-Trump-Reps-right.)

jumbopapa(4157) 3 days ago [-]

It's only okay to discuss left-leaning policies at work. I'm libertarian leaning and keep my mouth shut because if I said how I felt about half of the things my co-workers discuss I would be ostracized.

koolba(638) 3 days ago [-]

Back when it was possible to have a conservative viewpoint on social issues without being subject to public shaming by a disruptively loud minority. It wasn't that long ago either.

Most people spend more time at work than with their actual families. This leads to all kinds of conversations, in the office, in the break room, or at a bar. Not everything is appropriate for every setting but pretending differing opinions don't exist is ludicrous.

sokoloff(3938) 3 days ago [-]

Whenever you have reasonable adults working together?

I think I've, on appropriate occasion, discussed politics with friends and colleagues at every workplace I've ever been at. [Edit: some senior leadership] openly supports the DNC and has commented to that effect at town halls.

To pretend that people are apolitical or to ban all political discussions because some people can't seem to handle disagreements in a professional manner is one way to handle the problem, but perhaps not the first nor optimal solution to implement.

JohnFen(10000) 3 days ago [-]

Nearly every place I've worked at (in the US) over the last 20 years has had an explicit prohibition against discussing politics or religion in the workplace.

I'm not saying if it's OK or not as a philosophical matter, but it does help make the workplace more pleasant.

kevingadd(3785) 3 days ago [-]

Depending on your field, it can be unavoidable. Look at the stuff Facebook is having to deal with now, deciding whether to reject specific advertisements from US presidential candidates. It's impossible to tackle that without ending up just having regular old political discussions in the workplace, if only to figure out what actions to take.

Thankfully most jobs won't touch this but if you're running internet services it'll probably come up.

JoshTriplett(171) 3 days ago [-]

When things some people classify as 'politics' directly affect work. When you're trying to get your company to make improvements to the state of the world. When you're talking about corporate culture, inclusion, and solving problems. When you're arranging for your company to make a donation, or sponsor an event, or advising against doing so due to potential ramifications. When you're making travel plans and need to know more about your destination. When you're discussing salary or investments or retirement or health or hobbies and the potential impact of future legislative or political changes on those.

We're not just talking about the stereotype of people standing around a water cooler declaring affiliations and throwing political talking points past each other.

dijit(3530) 3 days ago [-]

I say this as a very left-leaning person (by British standards at the very least), but in my company strong left leaning values are part of the culture and it's expected that you promote LGBTQ+ issues, promote Women in the workplace and promote immigration.

It's not a bad thing in my mind, it's heavy handed at times, I can see how it would rub some traditionalists the wrong way in an extreme way. If you held some form of conservative view then you'd almost certainly be a pariah in our company/studio.

But left-leaning politics is definitely promoted and discussed in the office.

scarejunba(10000) 3 days ago [-]

Depends on people, I suppose. I'm easily capable of disengaging and leaving people not upset so I have discussed politics (anarcho-capitalist views, full paternalism views, just-world views) safely and without rebuke everywhere I've worked.

golemiprague(10000) 3 days ago [-]

Many things were ok to do at work not too long ago and are still ok in most work places around the world. That includes forming romantic relationships, talking about politics and all the other adult human interactions people do also outside work.

traderjane(3121) 3 days ago [-]

If I talk about whether it's legal to fire people for being gay, is that bringing politics to work? How can I be focused on work if I keep hearing about gay people? Also, isn't it obvious that homosexuality is seen as a severe spiritual evil to a large (if not majority) segment of the population?

What is the professional intuition on productivity when a large segment of the population considers homosexuality to be spiritually depraved?

fortran77(3626) 3 days ago [-]

It's astounding that people do! We talk about work at work.

SkyPuncher(4209) 3 days ago [-]

This is what I'm baffled by in all of this discussion. Work is not a place for politics. Maybe some light-hearted stuff, like discussing a ribbon cutting or some generally agreeable policy change (like a change to paid parking).

Even with very, very close co-founders, politics was an extremely rare discussion. When it did come up, it was generally in the context of how certain things affected the business. Even then, it was with a deep understanding that the other parties where smart, well-meaning individuals.

rustyconover(4219) 3 days ago [-]

In my mind it has always been ok to talk about your politics as long as you're respectful and you respect the differences of people who disagree.

To make it against whatever workplace 'rules' there are seems overbearing and thought-controlling. Our personal beliefs and political experiences make up a lot of who we are, our individual perspective on the world and somewhat how we think.

How can you hold your organization to socially responsible policies and actions if you can't discuss what should be socially acceptable? Political beliefs form the ideas of what is the scope of the organization's responsibility.

In my recent experiences at a hedge fund and a very large investment bank, politics were discussed pretty openly in both environments (alas this was only in NYC). I found the discussions worthwhile and informative as it helped me understand my own beliefs more.

It's unwise to hold beliefs that you never have to justify. You should put in the intellectual work so that you can articulate and argue your own politics it will make sure your beliefs are well founded.

Rebelgecko(4177) 3 days ago [-]

Maybe it's a SV thing? I saw some of the documents from a recent lawsuit against Google and was pretty astounded by the sorts of things that people were willing to not only say but actually put into writing where anyone on their intranet could see (things along the lines of 'I won't let anyone on my team if they have political belief X or voted for Y')

ajaimk(2858) 3 days ago [-]

This is a biased article written in an inflammatory style. Also, there is a mistake in the first 5 letters (Gitlab is not 'San Francisco based' being remote-only)

sl1ck731(10000) 3 days ago [-]

The company entity is headquartered in San Francisco, even if its only a PO box that is its 'address'.

PunchTornado(4162) 3 days ago [-]

a company has to be registered somewhere.

aledalgrande(3820) 3 days ago [-]

Politics could be discussed at work, if parties didn't decide to go for super extreme, polarizing positions. These days it feels like you cannot speak with the other side of the wall without getting into a fight. What happened to pragmatism and centrists? Politicians need divisiveness to keep their chairs.

That said, I would exclude any customer that has extreme views. Why support their platform, especially with a product like Gitlab, and help them spread hate?

buboard(3489) 3 days ago [-]

> and help them spread hate

because hate has been extended to include whatever people don't like, i.e. hate. Hate is a particularly bad choice of sieve. Violence is better

ProfessorLayton(4207) 3 days ago [-]

Without trying to be inflammatory in any manner, I will say that it takes a certain level of privilege to say one shouldn't discuss politics at work.

If you disagree I'm happy to discuss this viewpoint rather than being downvoted to oblivion.

Lots of issues are deemed "political", but imagine you fall into one of the marginalized groups:

— lgbt: Don't discuss the possibility about being fired for your sexuality because it's too political.

— Women in tech: Nope, let's not go there, too political.

— Underrepresented minorities in tech: Sorry it's a pipeline problem, don't bring politics into this.

— Education: Too political to discuss the fact that schools are trying to balance their admissions in the face of very uneven opportunities amongst their applicants. Never mind the fact that school admissions were never fair to begin with.

We can't improve without discussion, and it's unfortunate that these type of issues are so divisive.

Again, If you disagree I'd love to understand your viewpoint as to why.

Sacho(10000) 3 days ago [-]

I don't understand how any of these conversations(besides the first one, in perhaps an HR meeting), could possibly be topical at work? The company isn't paying me to preach my political agenda.

Have you also considered that while a 'marginalized' person might be keenly aware, researched and emotionally invested in discussing these things, a 'privileged' person may feel ambushed and distracted by these questions while they are trying to focus on work? The language and obvious expectation in many of these 'discussions' is a with-us-or-against-us recruitment for the agenda, often backed up with moral blackmail(e.g. 'we're fighting for our lives here and you won't even help us'). At work we try to afford each other respect, politeness and professional courtesy, but from what I've seen, these 'discussions' are anything but.

I don't think imposing oneself onto other people like these 'discussions' do is effective. Surely you've seen the hyperbole, the aggression and vitriol that goes with them and can imagine how it seems to someone who isn't invested in the agendas? After all you ask to not be downvoted even in your meta-discussion about the topics, what about when discussing the topics themselves?

I agree with the progressive arguments - that you can't just say 'no politics in X', that politics are inherent to what you do. What I absolutely disagree with is the tactics employed, and I think they are not just ineffective, they have a negative effect on engagement. People may want to help, but they don't want to feel like they're being coerced to. People may want to discuss, but they want to do it on their terms. People may want to listen, but they don't want to be lectured.

I think this is why there is such a strong push to disengage and ignore these problems - and I agree that they are problems that need discussion and need solutions. The in-your-face unapologetic advocacy, the constant motte-and-bailey argumentation, the extreme emotional reactions to any dissention, the preaching from a perceived moral high ground - these are not a conductive environment for changing minds.

So I agree with you. I'll go even further. It's bullshit to say 'no politics at work', and it's hypocritical. I will keep supporting that policy, because it's the lesser of two evils, because I think progressive strategy is counter-productive and a recipe for disaster. I'm happy to have any of these discussions, on a neutral, preferrably anonymous forum, where there is enough distance that perhaps we could have a productive argument. Work is none of those things.

legostormtroopr(3979) 3 days ago [-]

> We can't improve without discussion, and it's unfortunate that these type of issues are so divisive.

Except Cancel Culture is making it that these can't be discussed without complete agreement.

Take for example, 'Women in Tech', personally I don't see underrepresentation of women in tech as a problem that can be or should be 'solved'. For the better part of 15 years, there has been a massive movement to encourage women in STEM. There are hundreds of Women in Tech meetups, scholarships, Womens only courses... yet the numbers have barely budged in more than 10 years. Personally, it looks like in aggregate it will be difficult to get 50/50 representation of women and men in tech. To make it clear, we should definitely support everyone who is in tech, and make it an inclusive environment, but the continued push for 50/50 isn't going to happen so perhaps its not worth the huge money sink it is.

At the last place I worked that opinion was flat out branded 'sexist', and if you didn't vocally agree with every women in tech initiative people asked why.

So I would say the ability to speak openly about politics was shut down long ago, and not by the people you think.

eadmund(2681) 2 days ago [-]

I suspect very strongly that each of your 'marginalised' groups would have much greater luck discussing politics at works than its opposite. I can't imagine any tech company these days allowing folks to use work time to support the traditional view of marriage, traditional gender roles or old-fashioned racial views (that last, in particular, is completely unimaginable — and that's a really good thing!).

I think the privilege right now belongs to those in those 'marginalised' groups, who have the privilege to advocate for their views.

icelancer(4072) 3 days ago [-]

>> Underrepresented minorities in tech

>> If you disagree I'd love to understand your viewpoint as to why.

I'm a underrepresented minority in just about everywhere but tech, yet you draw the line 'at tech' for some arbitrary reason (as if my place of work or work opportunities defines who I am).

This is why I disagree. Someone is making arbitrary groups of underrepresentation and/or minority harm who have no real claim over said power.

Also:

>> I will say that it takes a certain level of privilege...

I am one of said underrepresented minorities 'not in tech' (I also don't work in tech anymore, so I guess I'm in the clear now) who have no interest or desire to talk politics at work. Not when I was poor and drawing on EBT/SNAP/WIC and not when I've been well off, or anywhere in between.

There is no absolute reason you have to be privileged to not want to talk about politics at work. This is a majorly false assumption.

joey_bob(10000) 3 days ago [-]

'I will say that it takes a certain level of privilege to say one shouldn't discuss politics at work.' Can you clarify the meaning? As it reads to me, if one belongs to a marginalized group, one would suffer disproportionately from making such a statement. I'm not quite clear on the semantics I suppose.

On another note, anecdotally, how many people have had a work relationship or project suffer due to discussion of politics at work, or alternatively, improve? (This being a reason not to discuss, it would be beneficial to sample for how likely it is to occur negatively.)

awinder(4221) 3 days ago [-]

I'll take a shot at it, with the caveat that I too would like to not be downvoted into oblivion xD.

So discussion for the sake of discussion is not exactly what I would see as a high need for the workplace. If you are trying to address these types of issues in a way that has business impacts, I think you're outside of what I would term as "discussing politics". From that list — discussing inclusivity initiatives has clear workplace benefits. Discussing a highly charged political event involving lgbt issues is adjacent to that, and has a highly likelihood of being inflammatory, and isn't directly correlated to anything actually going on with the business.

I think that framework would also help everyone stay on point. You don't go to work to solve the worlds ills. But you can address some of the ills where it's actually impacting your life at work. Further, it should prevent backlashes from other groups, and help humanize by getting out of ephemeral arguments and getting into discussions about building a collective environment where the business thrives and everyone is equally entitled to feel safe at work. Making work a better place — good. Making work look like Crossfire — bad.

friedman23(4022) 3 days ago [-]

It seems to me what you want is not to discuss politics at work but dictate politics are work. Unless you are perfectly okay with your co worker coming in with a MAGA hat every day and explaining to you why they think LGTB people don't deserve the right to marry.

zarkov99(4203) 3 days ago [-]

Because tech culture has skewed so insanely towards the authoritarian left that any challenge to the orthodoxy your post suggests is immediately weaponized. There is simply no way to honestly discuss the points you bring up without putting one's reputation and eventually career in danger. The Damore memo is a perfect example of this. The only safe thing to do in our culture right now is virtue signal and nod along with the rest of the mob. Perhaps one day the madness will pass and we will be able to again ask hard questions in public and try, together, to get to the bottom of things. Until then, lets just not talk about these things, so that a great many of us are not forced to wallow in hypocrisy for the majority of our waking hours.

zamalek(10000) 3 days ago [-]

I'll openly admit that I don't have an answer for the deeper issue here (Gitlab selling software to evil customers), in terms of specifically politics:

I don't think politics should be discussed at workplaces that do not participate in political systems in an official capacity (lobbying, PACs etc.). If your employer engages in politics, then that should be an allowed topic and/or other accommodations should be made for employees who regard their employer as morally reprehensible.

> LGBTQ+, women in tech, underrepresented minorities in tech, education

I don't consider these political in any way. Employees should be able to talk about how the company can improve its own diversity policies, irrespective of what the government decides what to do. The line of thinking here is similar to 'separating church and state,' and also that it's easier to change company policy: changing company policy provides quick and 'easy' improvements to social progress, while the longer and more difficult fights at the political level occur.

austincheney(3771) 3 days ago [-]

I don't mean to be inflammatory but you are subjectively biasing the subject of 'political discussions at work' out of concern for marginalized groups. In my experience this bias often has the opposite of intended result by providing what might be a marginalized group into a specialized class deserving extra-special care instead of equality, which eventually results in entitlement and possibly resentment.

If you want to protect marginalized groups write and enforce policies that focus on inclusion, equality, and anti-discrimination regardless of politics or preference. In other words: when nobody is special everybody is equally special.

ryanackley(3935) 3 days ago [-]

By 'discussing' do you mean a bunch of people getting together to agree with each other on these topics? I ask that because I can't see a socially well-adjusted person openly opposing any of these ideas. This is the curse of political correctness. The topics you list carry the weight of such moral certainty in most peoples' minds that opposing someone's opinion on the topic will immediately trigger a negative reaction. This makes it 'political'

For example, if someone at an office in downtown SF openly announced they didn't want to work with an lgbt person, would their coworkers look at that person and be like 'Aww, poor thing, lets discuss this with him or her so I can understand his way of thinking' or would they just think 'bigot! I knew something was off about him'

Btw, I think these are all important topics and I agree that we should be trying to educate people via discussion and empathy.

jcranberry(10000) 3 days ago [-]

I would think the most influential voice in these cases would be managers/leadership adding a provision to company guidelines and the like, in order to attempt to cultivate and safeguard a productive day to day culture.

But this particular perspective isn't necessarily a function of their 'privilege' in the sense of cultural capital and an advantageous background etc, but just of their own self-interest in this exact moment.

AnimalMuppet(3739) 3 days ago [-]

Take underrepresented minorities in tech (presuming yours is a tech workplace): You can work on that in a 'nonpolitical' way. (It is, as you say, still politics, in a way, but it's not being done like it's politics.) You say, 'You know that Android developer position that's been open for four months, that we can't find anyone for? I wonder if there might be some non-traditional candidates that would fit that role. How could we go about finding them?' That's done in a non-political way. The emphasis is on 'this could help the organization'.

Or, you could address the same issue by saying 'We need to pressure HR to require them to present at least one under-represented minority candidate for each open position.' That's a much more political approach to the same problem.

On a more general level:

> We can't improve without discussion, and it's unfortunate that these type of issues are so divisive.

Sure. But don't do the discussing at work. That's not what work is for. (The exception is when the problem does affect work. If the workplace is trans-hostile, say, you need to address that at work, and there's probably not a non-political way to do it.)

weberc2(4188) 3 days ago [-]

These are the _only_ kinds of politics that are acceptable in many workplaces. We have diversity departments and minority groups and women's groups and consultancies and recruiting agencies advertise their ability to find minority candidates. People are afraid to publicly question these policies.

> Education: Too political to discuss the fact that schools are trying to balance their admissions in the face of very uneven opportunities amongst their applicants. Never mind the fact that school admissions were never fair to begin with.

Women outnumber men in universities and pretty much throughout the pipeline (EDIT: to clarify, I mean the broader education pipeline, not the tech subset thereof). This whole post is so disconnected from my experience and any experience I've heard about that it feels like it's from inverse universe.

opportune(10000) 3 days ago [-]

I absolutely agree that politics is everywhere and picking and choosing which type of political discussion is allowed is itself political, (I've also never been convinced that URMs in tech isn't primarily a pipeline problem but ofc wouldn't want to shut off debate on that)

I guess the difference in what is allowed is between "positive" vs "negative" politics. By that I don't mean positive politics is absolutely the best whereas negative is stuff I disagree with, but saying "We want to empower marginalized groups" is positive and doesn't bother that much people who.. don't think it's necessary, because it's not a political position you can disagreee with per se. But negative politics might be something like "we are going to ban X because we find it abhorrent" whether X is racist stuff, antivaxx stuff, or LGBTQ stuff you are directly prohibiting beliefs.

Note that I don't necessarily agree that positive vs negative (or, un-disagreeable vs actively disagreeing) is exactly where you should draw the line where politics is allowed, but I'm just trying to rationalize and say maybe it's not inherently hypocritical.

ryandrake(4132) 3 days ago [-]

In all of your example topics (and others), there are clear sets of 'acceptable' and 'unacceptable' opinions, and if you share an unacceptable one, you face all kinds of potential work-impacting repercussions ranging from 'sad faces' to getting fired.

In the spirit of imagining you're in the out group: Imagine you're a Trump supporter and work in a big-name SV tech company. Someone asks you at work 'Boy, Trump is a bonehead, isn't he?' Well, now you're put in a no-win situation: You have to either lie about your beliefs to keep everyone happy, share your beliefs and risk getting fired or some other type of discipline. Or, you'll need to dodge by saying 'I don't think it's appropriate to talk about politics at work,' leaving it to the asker to speculate about your views.

Remember 'Invasion of the Body Snatchers?' At the end where most people are aliens and the few real humans have to put on a fake persona in order to walk around freely? If you slipped up and dropped the persona for an instant they'd all point at you and make that alien noise, and you're done for! That must be what it feels like to hold unorthodox political views working in a Silicon Valley tech company these days.

voxl(10000) 3 days ago [-]

Preach. Politics being silenced in a work place is so short sighted and dumb. We should strive for a world where disagreement is not the end of a good working relationship, not one where crypto-politics is the new normal in the workplace. Because if you genuinely belief people want try to identify who is and isn't racist in a 'conforming' way then you're delusional.

overgard(3998) 3 days ago [-]

Well, the problem is that practically anything can be 'politics', I don't think it's a useful way to frame thinking around what you can or can't say at work.

Generally though, when people say something is 'politics' what it means is it comes down to a value judgement, rather than there being a middle solution, or the issue is too complicated for there to be a general consensus. As an example that probably wouldn't come up at work, consider the abortion question. If you're of the fundamentalist religious sort, your values practically demand you believe that aborting a baby is wrong. If you're not, then it's a lot more grey. Something like that is a value issue, there isn't really a compromise that doesn't involve one side changing fundamental beliefs about the world and the universe.

To answer your specific issues though:

- lgbt: not relevant. It's illegal to discriminate on that, so... either your company is following the law or it's not. Also: nobody needs to know. It doesn't need to be discussed, it just either needs to be litigated or not litigated. But no smart company wants their employees talking about their personal views on that because that could be a huge liability if someone has intolerant personal views (even if they don't apply them professionally).

- Women in tech & underrepresented minorities: ... well it IS a pipeline problem. I'm pretty sure the vast majority of companies would love to have more diversity, but you can only hire the people that apply. I see a lot of the SJ types make offices miserable hammering on this stuff that frequently can't be fixed locally. I mean yeah, if someone is scaring away people, then address that behavior, but that's more of a 'having respectful competent management' issue.

- Education - Never really heard of this being an issue that isn't allowed to be talked about. I don't think anyone would really get upset about this though.

Also, as others have mentioned... a lot of times when people say these issues should be 'discussed' they don't really mean that. The far left generally hates 'discussion' on their pet issues if there's even a hint of not towing the party line, and it quickly escalates into people being called racist. So frequently, avoiding 'politics' is just saying, 'please pick your fight somewhere else'

skybrian(1703) 3 days ago [-]

The problem is getting hooked on politics that doesn't actually affect you personally, other people on your team, or how you do your job. Just like we do on Hacker News, you can have employees reading about stuff happening elsewhere in the world, commenting on it, and getting upset over it.

At large companies, this can include things going on in different parts of the company. You don't know them or how they work, but you start taking it personally because it affects the company's reputation.

I'm retired, but if I went back to work, I would want to pay more attention to helping my co-workers and accomplishing the team's mission (which of course involves some politics) and pay a lot less attention to other, less relevant politics.

You can say it's privileged to want to shut out the world and concentrate on improving one thing, but I prefer to think of this as an aspirational goal.

BlueTemplar(10000) 3 days ago [-]

Yeah, taken in the most widely possible way, politics is anything that you say or do outside of your family - so it becomes the absurd injunction of 'no communication or work allowed at work'...

amoitnga(4222) 3 days ago [-]

Here's my 2c.

Why do you need to discuss anything at all? Work is where I work. I'm being paid for my time and professional expertise, not my opinion on anything else. I don't get why would I be allowed to spend that time on conversations not related to work.

Being allowed to have some short talk, maybe discuss something quickly, like a game last night or something, is and should be more of a nicety. Nobody gonna stop me from doing this. But I don't see why we should be allowed spending long periods of time discussing other important issues.

There are meetups, we can get together after work, go for a beer etc... And have those conversations in our personal time if we deem it important. But not during business hours.

Work is for work. So banning conversations is reasonable imo, and not only political.

imgabe(1639) 3 days ago [-]

To me 'don't discuss politics at work' would apply to management as well and imply that people should only be hired or fired for performance based reasons and not because of their sexuality, race, or gender.

TeMPOraL(2647) 3 days ago [-]

Others made good general and specific arguments for avoiding such discussions at a workplace, but I'd like to offer a thought experiment that can help understand it better.

Imagine yourself noticing that people around you start to reveal themselves as strongly religious - and not in your religion, if you subscribe to any - and very much into proselytizing. Not a day goes by without someone approaching you to convince you to believe in their god. Even when not explicitly preaching, they keep invalidating everything you worry about. 'You fret about your 401k as if you were to use it. Armageddon is coming soon, and if you believe in God, you'll get the best 401k there is!'. 'I wonder how can you spend so much time on that hobby, when so many people still haven't heard the Good News.' 'Yes, this disease is a tragedy, but God's Kingdom will soon cure it, and all others too. These doctors are wasting their time.' 'You're depressed because of climate change? Don't be! God won't let anything happen to Earth!'

At some point you would really start to prefer people to stop talking any and all religion at workplace.

This isn't an arbitrary example. As a teenager, I've been that preacher. People who do that believe strongly in what they talk about - it's literally the most important issue in their lives. The conflict stems from failure to understand that just because they care about an issue, doesn't mean everyone else does too. Failing to recognize that, doubling down on efforts to make others pay attention, breeds contempt, resentment, and otherwise achieves the exact opposite of what's intended.

I've seen both sides of this, and the way the topics you mention are approached today very much reminds me of such religious groups. There's a time and place for all discussions, but they need to happen in contexts where all parties are at least a little receptive, and not by performing DDOS on someone's attention, or invalidating everything they care about that's not related to the important issue.

sonotathrowaway(10000) 3 days ago [-]

I must confess that over the past few years I have been gravely disappointed with the white moderate. I have almost reached the regrettable conclusion that the Negro's great stumbling block in his stride toward freedom is not the White Citizen's Counciler or the Ku Klux Klanner, but the white moderate, who is more devoted to 'order' than to justice; who prefers a negative peace which is the absence of tension to a positive peace which is the presence of justice

- Letter from a Birmingham Jail [King, Jr.]

xeonoex(10000) 3 days ago [-]

To me, there is a big difference between discussing politics and preaching politics. I do want my company to tell me not to discuss politics with coworkers who are willing to discuss it. Not dicussing politics, especially with people who have a difference of opinion, is what leads to rage filled echochambers that are so common now. However, I do not want people preaching politics unsolicited at work though.

I think politics being a taboo topic is an attempt to keep workers in order and ignore any political or environmental impact that their own company might have.

iliketocomplain(10000) 2 days ago [-]

> Without trying to be inflammatory in any manner, ...

I'm not a racist, but ...

jedberg(2257) 3 days ago [-]

As a wealthy white male, I couldn't agree more. For people like me, politics is a choice. But as my female friends of color often remind me, politics is not a choice for them.

ahbyb(10000) 3 days ago [-]

Nobody is saying 'don't get involved in politics', just 'don't bring it to work'.

to11mtm(10000) 3 days ago [-]

> I will say that it takes a certain level of privilege to say one shouldn't discuss politics at work.

My counterpoint would be that it takes a certain level of privilege to be able to discuss politics at work. Or, to be more specific, anything other than the job you are working. (And, of course, the things your employer can't legally stop you from talking about.)

Or, to put my point more specifically: Almost all of the discussion I have seen ignores that most of these problems are, at their root, class based problems. I suppose what is more saddening from a humanist standpoint is the way the extremists on both sides wish to eventually put the other under their heel.

That said, touching specifically on my personal experiences in Education, the issue is that any Metric based solution will likely run up against Goodhart's law sooner or later. While what I witnessed in Academia was probably far from the norm, I doubt it was rare.

james_s_tayler(10000) 3 days ago [-]

Let's not forget office politics. In that case, you basically can't discuss anything at work.

djsumdog(1073) 3 days ago [-]

Have you seen problems with people in any of these groups unable to talk about these issues? I had a project owner once who had a manager not get along with her at all.

'Is it because you're a woman,' I asked her. She said, 'No, I don't think that's it.' .. and he did get along with another woman; one of our Functional Analysis.

Sometimes person x doesn't like person y because .. they don't like person y. They don't think person y is effective or useful or good at his or her job, not because y is an Asian, or woman, or trans or Mexican. ... or it could be and the person is a victim of bigotry and doesn't know it.

I know it's anecdotal, but I've worked in a lot of different places, in 3 different countries, and in my experience, I think it's the opposite. I think people are afraid to say anything remotely non-left or criticize any minority employee or say anything that could remotely be inferred as problematic.

egoisticalgoat(10000) 2 days ago [-]

These kinds of policy always scare me because people don't realize how prevalent politics are in our daily lives, even as devs. Dark patterns? Ad tracking? Hell, even something as simple as a dropdown for sex/gender is political. With these kinds of policies, you're forced not to think, just execute.

manfredo(10000) 3 days ago [-]

— lgbt: Don't discuss the possibility about being fired for your sexuality because it's too political.

— Women in tech: Nope, let's not go there, too political.

— Underrepresented minorities in tech: Sorry it's a pipeline problem, don't bring politics into this.

In my experience in tech (though only Silicon Valley tech companies, so that's a narrow sample) this the opposite of what is true. Companies are very eager to display how much they support LGBT movements and announce how they'll increase diversity by offering larger hiring bonuses for diverse hires and institute systems of reservations for women and non-Asian PoC. Both the private and public companies I've worked with have been very active in pursuing these goals. And we're not alone Microsoft offers larger bonuses for hiring managers that hire diverse candidates [1], and Intel withholds portions of bonuses unless diversity quotas are not met [2].

1. https://www.theverge.com/2019/4/19/18508013/microsoft-pro-di...

2. https://www.wsj.com/articles/intel-hits-a-novel-diversity-ta...

alkonaut(10000) 3 days ago [-]

Absolutely right. Saying "don't talk politics at work" seems to assume that politics is something that goes on on the outside. A stream of news on divisive topics.

In reality, politics happens everywhere and you can't really avoid talking about it.

"Don't talk about politics not relevant for the workplace" could be reasonable (most poltics is after all not related).

cgiles(10000) 3 days ago [-]

> it's unfortunate that these type of issues are so divisive.

The reason they are so divisive is because the loud people fall into two categories: 1) people who loudly and constantly complain about these issues and for whom no amount of improvement is enough, and 2) people who think all these problems are greatly exaggerated, part of the liberal culture wars, and find it extremely annoying that anyone would bring it up.

The 'silent majority' in these issues are people who say 'yeah, sure, it seems fair to allow gay marriage, why not' and 'let's see what we can do to improve female and minority representation but let's not jump to the conclusion that anyone in particular is to blame'.

The hallmarks of being in this silent majority are:

- being generally well-intentioned

- supportive of incremental, common-sense solutions that are not zero-sum

- dislikes intense conflict and blame games

- does not take a view of the issue that makes one group of people the 'heroes/victims' and another the 'villains'

So, if you hear the proposal 'let's allow politics at work', who do you think is actually going to speak up? That's right, the loud and annoying people that you can hear anytime by turning on the news. Ideologues for whom compromise and assuming good faith in your opponents is anathema. No thanks.

thrwayxyz(10000) 3 days ago [-]

It takes a huge amount of privilege to be able to talk about politics at work. The despised minorities, like zoophiles, would not only be fired but arrested with the whole office cheering that on.

Something I find pretty odd given that fucking a cow is a lot better for the cow than eating it.

devmunchies(10000) 3 days ago [-]

> If you disagree I'd love to understand your viewpoint as to why.

I don't disagree, but I'm 'privileged' and want to keep it that way for me and my posterity. Not ashamed in the least because its not bad to be privileged.

yourbandsucks(10000) 3 days ago [-]

If it were the case that these things were never mentioned, and anytime they were brought up, people were silenced, I'd totally agree with your point here.

But we're getting wall-to-wall coverage of basically those exact issues, all the time, with only one point of view endorsed. It's uninclusive to anybody who thinks differently in terms of methods or priorities.

manigandham(823) 3 days ago [-]

In your 'neutral' comment, you're already taking a stance on the topics and automatically defining marginalized and privileged. People are far more complicated than that.

That's why these discussions aren't a good idea. Companies are paying you for productivity and ensuring a calm environment is important. You have plenty of opportunity to discuss and create all the change you want elsewhere.

RcouF1uZ4gsC(4022) 3 days ago [-]

> We can't improve without discussion, and it's unfortunate that these type of issues are so divisive.

Around 40% of voters support Donald Trump in some way. Are you sure you really want to discuss politics at work? Do you want someone wearing a MAGA hat explaining to everyone why immigration is too high?

Because that is what real political discussion will involve. In fact, I suspect many people who are advocating for politics at work, actually only want people whom they agree with to be able discuss policies that they generally agree with.

So for me, as long as the company does not prevent political activity outside of work, I am happy to have work be a neutral zone where people from opposite ends of the political spectrum can just work together on how to make money for the business.

XPKBandMaidCzun(10000) 3 days ago [-]

> I will say that it takes a certain level of privilege to say one shouldn't discuss politics at work.

Having a discussion is hard. While you come to this well-intended, participants bring their own life experiences and hardships. They may not agree on the starting point of the conversation:

How is 'marginalized' and 'privileged' is defined? Some may not take that perspective on the above issues, and people with these characteristics may not consider themselves part of the 'group', they may believe overcoming hardship by the way of grit and merit being the ideal.

People with these characteristics who don't consider themselves part of the 'groups' may find the rhetoric being used patronizing, they don't want to be treated special or different, they want to be recognized for their contribution to the collective. That's where role models come from, to some.

They may disagree on what 'empathy' means. Casting a group as marginalized draws broad strokes about others, because who isn't marginalized? How could someone know how someone else feels or what they have faced in life? If you don't like having your hardship belittled, why potentially do the same thing to others? We all have feelings and bring our unique stories and experiences to the table.

The definition of 'inclusiveness'. To some, the qualm isn't the subject, it's how some people who cite these groups disregard the larger collective or group they're apart of (e.g. their team, their workplace).

The definition of 'discussion'. There are stories of people being fired for merely citing agreed upon academic social/psychological research. Some sense viewpoint discrimination - fear of being sanctioned for stating their perspective - or to even say they're hurt by the topic's inferences.

yodsanklai(4066) 3 days ago [-]

Why do you think LGBT rights (first item on your list) is something that should be discussed at work? There are laws against discrimination and companies have to comply, but I don't see a good reason why this kind of topic should be addressed at the workplace. Some employees are uncomfortable with discussing politics at work as it may create unnecessary tensions.

Those who like debating about politics can always gather after work.

I don't see what it has to with a level of privileges.

That being said, I also don't like being told what type of conversations I'm allowed to have.

crispinb(3222) 3 days ago [-]

Completely agree. Besides which it's a hopelessly medieval master-slave attitude to employment - the notion that they're not paying for time, expertise or work, but rather taking temporary ownership of subjects. I would have nothing to do with Gitlab knowing this, as employee or customer.

microcolonel(4207) 3 days ago [-]

Excellent policy, but they could stand to realize how political their offices have already become. Maybe the culture of the company blinds them to this, but GitLab's overtly involved in politics.

detaro(2090) 3 days ago [-]

gitlab has offices?

lazyasciiart(10000) 3 days ago [-]

The wording of this statement sounds very poorly thought out. Does a customer have to pay them? What if I argue I don't believe I should have to pay, or I have a moral objection? What if my belief in freedom says it's OK to ddos them? Presumably they'll fall back on something like 'our contract says you have to pay us to use our services'. Great. So you can and will refuse to work with a customer for a reason you chose, not an externally imposed restriction. The intended meaning is probably more like 'we will work with anyone who works within our core beliefs about how society and payment for services should work, but we have not done the work of figuring out what those are in advance'.

tengbretson(10000) 3 days ago [-]

You're conflating beliefs and actions. I can have a moral objection to paying for Gitlab's services and still pay for Gitlab's services. I can write article after article about how it is immoral to pay for Gitlab's services. As long as I pay for their services, I will get them.





Historical Discussions: Amazon's Consumer Business Turned Off Final Oracle Database (October 15, 2019: 925 points)

(925) Amazon's Consumer Business Turned Off Final Oracle Database

925 points 4 days ago by jeffbarr in 183rd position

aws.amazon.com | Estimated reading time – 7 minutes | comments | anchor

Over my 17 years at Amazon, I have seen that my colleagues on the engineering team are never content to leave good-enough alone. They routinely re-evaluate every internal system to make sure that it is as scalable, efficient, performant, and secure as possible. When they find an avenue for improvement, they will use what they have learned to thoroughly modernize our architectures and implementations, often going so far as to rip apart existing systems and rebuild them from the ground up if necessary.

Today I would like to tell you about an internal database migration effort of this type that just wrapped up after several years of work. Over the years we realized that we were spending too much time managing and scaling thousands of legacy Oracle databases. Instead of focusing on high-value differentiated work, our database administrators (DBAs) spent a lot of time simply keeping the lights on while transaction rates climbed and the overall amount of stored data mounted. This included time spent dealing with complex & inefficient hardware provisioning, license management, and many other issues that are now best handled by modern, managed database services.

More than 100 teams in Amazon's Consumer business participated in the migration effort. This includes well-known customer-facing brands and sites such as Alexa, Amazon Prime, Amazon Prime Video, Amazon Fresh, Kindle, Amazon Music, Audible, Shopbop, Twitch, and Zappos, as well as internal teams such as AdTech, Amazon Fulfillment Technology, Consumer Payments, Customer Returns, Catalog Systems, Deliver Experience, Digital Devices, External Payments, Finance, InfoSec, Marketplace, Ordering, and Retail Systems.

Migration Complete I am happy to report that this database migration effort is now complete. Amazon's Consumer business just turned off its final Oracle database (some third-party applications are tightly bound to Oracle and were not migrated).

We migrated 75 petabytes of internal data stored in nearly 7,500 Oracle databases to multiple AWS database services including Amazon DynamoDB, Amazon Aurora, Amazon Relational Database Service (RDS), and Amazon Redshift. The migrations were accomplished with little or no downtime, and covered 100% of our proprietary systems. This includes complex purchasing, catalog management, order fulfillment, accounting, and video streaming workloads. We kept careful track of the costs and the performance, and realized the following results:

  • Cost Reduction – We reduced our database costs by over 60% on top of the heavily discounted rate we negotiated based on our scale. Customers regularly report cost savings of 90% by switching from Oracle to AWS.
  • Performance Improvements – Latency of our consumer-facing applications was reduced by 40%.
  • Administrative Overhead – The switch to managed services reduced database admin overhead by 70%.

The migration gave each internal team the freedom to choose the purpose-built AWS database service that best fit their needs, and also gave them better control over their budget and their cost model. Low-latency services were migrated to DynamoDB and other highly scalable non-relational databases such as Amazon ElastiCache. Transactional relational workloads with high data consistency requirements were moved to Aurora and RDS; analytics workloads were migrated to Redshift, our cloud data warehouse.

We captured the shutdown of the final Oracle database, and had a quick celebration:

DBA Career Path As I explained earlier, our DBAs once spent a lot of time managing and scaling our legacy Oracle databases. The migration freed up time that our DBAs now use to do an even better job of performance monitoring and query optimization, all with the goal of letting them deliver a better customer experience.

As part of the migration, we also worked to create a fresh career path for our Oracle DBAs, training them to become database migration specialists and advisors. This training includes education on AWS database technologies, cloud-based architecture, cloud security, OpEx-style cost management. They now work with both internal and external customers in an advisory role, where they have an opportunity to share their first-hand experience with large-scale migration of mission-critical databases.

Migration Examples Here are examples drawn from a few of the migrations:

Advertising – After the migration, this team was able to double their database fleet size (and their throughput) in minutes to accommodate peak traffic, courtesy of RDS. This scale-up effort would have taken months.

Buyer Fraud – This team moved 40 TB of data with just one hour of downtime, and realized the same or better performance at half the cost, powered by Amazon Aurora.

Financial Ledger – This team moved 120 TB of data, reduced latency by 40%, cut costs by 70%, and cut overhead by the same 70%, all powered by DynamoDB.

Wallet – This team migrated more than 10 billion records to DynamoDB, reducing latency by 50% and operational costs by 90% in the process. To learn more about this migration, read Amazon Wallet Scales Using Amazon DynamoDB.

My recent Prime Day 2019 post contains more examples of the extreme scale and performance that are possible with AWS.

Migration Resources If you are ready to migrate from Oracle (or another hand-managed legacy database) to one or more AWS database services, here are some resources to get you started:

AWS Migration Partners – Our slate of AWS Migration Partners have the experience, expertise, and tools to help you to understand, plan, and execute a database migration.

Migration Case Studies -Read How Amazon is Achieving Database Freedom Using AWS to learn more about this effort; read the Prime Video, Advertising, Items & Offers, Amazon Fulfillment, and Analytics case studies to learn more about the examples that I mentioned above.

AWS Professional Services – My colleagues at AWS Professional Services are ready to work alongside you to make your migration a success.

AWS Migration Tools & Services – Check out our Cloud Migration page, read more about Migration Hub, and don't forget about the Database Migration Service.

AWS Database Freedom – The AWS Database Freedom program is designed to help qualified customers migrate from traditional databases to cloud-native AWS databases.

AWS re:Invent Sessions – We are finalizing an extensive lineup of chalk talks and breakout sessions for AWS re:Invent that will focus on this migration effort, all led by the team members that planned and executed the migrations.

Jeff;




All Comments: [-] | anchor

victor106(4153) 4 days ago [-]

> The migration gave each internal team the freedom to choose the purpose-built AWS database service that best fit their needs, and also gave them better control over their budget and their cost model.

Anyone know of a comprehensive resource that we could use to figure out which DB to use?

zamalek(10000) 4 days ago [-]

R&D and engineering.

jl6(4203) 4 days ago [-]

Here's an algorithm which I believe to be 95% accurate:

If only a single user/device needs to access the data: use SQLite. In all other cases: use PostgreSQL.

onefuncman(10000) 4 days ago [-]

The og-aws Slack or ##aws on freenode

But really, you should understand what kind of data store you need based on your use case is first and then decide on an implementation later...

mooreds(124) 4 days ago [-]

Depends on what you mean by comprehensive. https://aws.amazon.com/products/databases/ is a good place to start.

After that I'd do some PoCs or talk to a consultant.

tootie(10000) 4 days ago [-]

I wonder if anyone here has seen any net new adoption of Oracle DBs in the last few years? It was an absolute juggernaut 10-15 years ago and has rapidly tanked their reputation and spread their business across software, services and cloud with mixed results. Is it actually still growing or just milking their stodgy enterprise customers for more money?

rossdavidh(4125) 4 days ago [-]

I think that not very many existing customers of Oracle migrate off, but I have to think that they are no longer the default for new and growing businesses to move into. AWS seems to have become that, with Oracle not even the second-most likely option (that would probably be Google or Microsoft).

Having said that, Oracle is probably the COBOL of databases, in that it will still be around (collecting good $$) from its existing customers for a long, long time.

mxschumacher(1864) 4 days ago [-]

I believe the business model is that it is near impossible to leave the ecosystem (this is also how SAP can still thrive). It takes a lot of money and expertise to make the migration happen. Few companies have scaling problems as pronounced as Amazon and as much of a vested interest in migrating to their own custom solutions.

bluejekyll(2836) 4 days ago [-]

When they threaten you with lawsuits for using their software, and make their pricing completely opaque so you don't really know what you owe (Comcast style), why would you knowingly choose that? Postgres is pretty darn good these days.

I still love Java, but I won't be surprised if that language becomes a business risk because of Oracle's behavior in other spaces.

abraae(10000) 4 days ago [-]

We started a new SaaS company last year. We use Java + Oracle from the get go.

Last company was very reliant on Oracle, we ran the entire business on a single Oracle instance for a couple of decades.

We had our share of adventures, but over the years never lost a minute's worth of data despite server, hard drive and other issues. We've also had a few scary performance problems, but have always been able to tune the DB to get past them. And once or twice, we've been in a dark place where you are googling for cryptic error codes, and we've always come through. That's a lot of business and millions of dollars earned through having a very reliable database underneath.

For us, the cost equation is whether it's worth learning a new DB. It's not. If you can run your business on a single instance, then on AWS, it's cheaper to stick with what you know than to try and move to a new DB for a few K of saved licence fees.

What's more, using Oracle is a good answer when the PHBs at our customers ask about our stack. No one ever got fired for using Oracle.

So yes, it certainly makes sense to use Oracle for new projects if you are already Oracle experts.

Java is somewhat the same equation as well ;)

arpinum(10000) 3 days ago [-]

One area of growth - Accidental Oracle customers. Oracle makes software acquisitions and bases the new version of the product on Oracle DB. Customers are forced to upgrade and take on an Oracle DB, or get a firm push into the Oracle cloud.

thrower123(3005) 4 days ago [-]

Oracle is slashing positions in a lot of their sales divisions. There was a large layoff this spring/summer, and from what I can tell from contacts that were there, anyone who can is jumping ship to AWS or other competitors, and poaching as many of their old customers with them as they can.

solatic(10000) 4 days ago [-]

I can imagine that there are more than a few Oracle customers whose spend increases year-over-year because:

1) they have significant investment in the human capital side of Oracle, which is to say, Oracle DBAs on staff who are highly capable at keeping the lights on for their Oracle databases, but have little or no familiarity with alternatives 2) enterprise architects are required to get approval from their DBAs for the design of new systems, who won't approve non-Oracle databases that they can't support, so new systems get built with new Oracle databases 3) the financial cost of the additional databases is organizationally far enough removed from the design phase that Finance has no real power to say no, and anyway such enterprises have other areas of the business that are far more bloated grabbing Finance's attention, where the cost of additional Oracle licenses is a rounding error by comparison 4) Anybody at a high enough level to see why this is a problem has far bigger problems to deal with and so the situation persists.

I doubt that apathetic enterprises are enough to make Oracle database a growth center overall when nobody in their right mind adopts it for new companies, but I doubt that all of their customers are looking for a migration exit.

drawkbox(3196) 4 days ago [-]

In 2001/2003, I led a small team at an agency that converted Kraftfoods and 50+ brands from 25+ ATG Dynamo web/app CMS servers + 20+ Oracle servers to .NET 1.0 CMS on 3 web servers, 3 app servers and 3 Microsoft SQL Servers.

Even then Oracle tools were horrible for developers and still are. The only reason they had that many databases was Oracle's business is licensing, massive, expensive licenses, and their sales pushed them sold through upper management. Those days are almost over, developers choose the tech, the cloud is here. Without Tom [1], Oracle was basically not even a great developer community, all documentation was bad and the Java interfaces on the tools / management studio were horrendous. Oracle is fast but for most things on the web caching comes into play heavily and other databases are good enough. Oracle is good for large datasets/reporting, cursors are extremely fast when Oracle is tuned, the problem is it is like vehicle that needs tuning and optimization all the time or it drives bad. The cost and upkeep make Oracle mostly overkill and too painful.

Ellison laughed at the cloud like Ballmer laughed at the iPhone. That was a major error. But ultimately Ellison is an engineer that forgot to make sure developers like to use the product. That will ultimately be their fatal flaw as now the developers choose the tools not what is sold in above them.

All other platforms are developer centric and cloud first now. Oracle the all seeing one says the end is nigh for Oracle the database. No new projects will be using it without force. Too bad Oracle also owns MySQL, Java etc. I really wish someone else had bought Sun.

[1] https://asktom.oracle.com

binarymax(2215) 4 days ago [-]

I'm interested in seeing how much they try to milk Java in the next 10 years. I can see them getting desperate enough to really bring the hammer down on licensing, killing off openjdk, and making life miserable for everyone in the process.

matchagaucho(10000) 4 days ago [-]

Often it's the apps, like eBiz Suite, that drive the net-new adoption of the underlying DB.

The large Accounting firms handle 80% of the financial audits of Fortune 500 corporations. They are used to walking up to Oracle DBs and extracting the reports needed for compliance and reporting.

If they have to learn a new ERP or proprietary Accounting system, then the T&M costs can mount.

koolba(638) 4 days ago [-]

> Financial Ledger – This team moved 120 TB of data, reduced latency by 40%, cut costs by 70%, and cut overhead by the same 70%, all powered by DynamoDB.

Does this mean they have a custom set of books running atop DynamoDB or is this specific to aggregate reporting and financial data warehousing?

denimnerd42(10000) 4 days ago [-]

Would love to know also. My guess given the volume of data is data warehousing.

the_duke(3525) 4 days ago [-]

The video feels like a big middle finger to Oracle.

'stop oracle', and the URL https://aws.amazon.com/products/databases/freedom.

Not that I'm complaining, Oracle deserves all the hate they get for their business practices.

But associating AWS with 'breaking free' is extremely ironic. It's just as much of a source for vendor lock in as Oracle.

If you are not extremely careful/dedicated with your system design, you are never moving off of AWS ever again, or only with a huge engineering effort that almost no-one will swallow. AWS also knows this very well: discounts are harder to come by if they can tell you are totally locked in to their infrastructure.

Combine that with the fact that AWS loves to fork OS projects and build products out of them, without contributing back upstream.

AWS is a terrific platform in many ways, but it is certainly not something that will lead to freedom of choice.

AtlasBarfed(10000) 4 days ago [-]

In the same vein of oracle trying the one-ring-to-bind them, RDS is amazing on many levels, but AWS is all in on database lockin with their customers.

RDS probably affords enough flexibility to get your data out, but Dynamo... that's a different story.

mzkply(10000) 4 days ago [-]

I guess the only freedom is that of multi-year license contracts.

deanCommie(10000) 4 days ago [-]

This is a popular opinion but it simply fails to resonate to me.

'This company's software is so useful you can't help but use it because it saves you so much time, effort, and money' is NOT lock in.

You can absolutely choose to develop on AWS In a way that avoids lock-in (Containers, managed DB engines, other cloud storage providers have s3-compatible apis). People don't because it's more CONVENIENT not to - nobody is forcing the to.

jpalomaki(2962) 4 days ago [-]

This is kind of reply to Larry Ellison's comments [1] about how Amazon can't run their business on their own databases.

[1] https://youtu.be/xrzMYL901AQ?t=50

CydeWeys(4057) 4 days ago [-]

Migrating everything to the same externally-available cloud infrastructure is exactly the thing that Google needs to do, but has been failing at for going on a decade now. Yes, I'm bitter about it.

zzzcpan(3630) 4 days ago [-]

It's not even on the radar for them I believe. Maybe ask again in another ten years.

wwarner(3176) 4 days ago [-]

The claim is that some of the data was migrated to publicly available systems. I'm guessing that quite a bit of it is stored in amazon specific systems, for two reasons. First, availability. If the amazon product catalog depends on Aurora, then when Aurora has an issue, so does Amazon retail. Second, custom solutions that were developed before modern alternatives existed have an insurmountable inertia in a big company.

mattlondon(10000) 4 days ago [-]

Is it not the case that it is actually the opposite? That GCP is running on the internally-available cloud infrastructure at Google?

How would they turn that inside out? What would GCP actually run on if they ran GCP on GCP? Or do you mean run search/gmail/youtube on GCP? Isn't their internal infra basically kubernetes in all but name?

iRobbery(10000) 4 days ago [-]

'This team moved 40 TB of data with just one hour of downtime, and realized the same or better performance at half the cost, powered by Amazon Aurora.'

Would have loved to have read some technical details on how 40 Tbyte was transferred in an hour, even migrated.

Havoc(4219) 4 days ago [-]

>Would have loved to have read some technical details on how 40 Tbyte was transferred in an hour, even migrated.

Perhaps I'm missing something but that doesn't seem hard considering:

1) It's only like 3x high end consumer level 16TB HDDs

2) This is Amazon...they have AWS - really fat pipes and endless horsepower on demand

remus(10000) 4 days ago [-]

Presumably they didn't go for a 'turn it off, do the transfer, turn it on again' strategy. I'd guess they transferred the bulk of the data beforehand then applied incremental updates until the databases matched then turned off the old db.

tyingq(4179) 4 days ago [-]

Is 'Amazon's Consumer Area' deliberately specific? That is, are they still using Oracle somewhere else? (Despite 75 petabytes being significant)

Edit: I did see 'some third-party applications are tightly bound to Oracle and were not migrated'. Curious if that's all that is left, globally, at AMZN.

sillypuddy(3024) 4 days ago [-]

There are Oracle AWS offerings - https://aws.amazon.com/oracle/

danesparza(4184) 4 days ago [-]

The very fact that they are dogfooding DynamoDB (with core features of their business) makes me want to take it more seriously and use it more.

I also love that they didn't just replace Oracle with <insert competing RDBMS here> but instead thoughtfully pieced together a strategy based on what the data was going to be used for.

Well done!

mey(3930) 4 days ago [-]

Have used DynamoDB a couple times, most recently for data processing.

It is extremely expensive, for data storage and performance. Queries are limited and cumbersome to write.

If you are looking at it, put a reasonable amount of test data in and play with it too see what costs look like compared to the performance you need from it.

mjw1007(10000) 4 days ago [-]

Amazon's leadership famously pushed hard in the early 2000s for all its systems to share data only via service interfaces ( https://gist.github.com/chitchcock/1281611 ).

I wonder how much of the reason for that was actually a wish to avoid getting further locked in to Oracle (changing or removing a database that backs a single service is very much easier than changing a database that also acts as an integration point).

discodave(10000) 4 days ago [-]

If the goal of SOA was to reduce Oracle lock in then it utterly failed. After SOA practically every team had an Oracle db, so now they have to convince tens of thousands of engineers to migrate thousands of databases.

Imagine what Oracle is going to try and charge Amazon per month for every remaining DB given how publicly hostile Amazon has been.

vkaku(4174) 4 days ago [-]

Obviously, enough Amazon bots are giving me bad karma, but my point stands: No, Amazon is not a special snowflake because it did this.

This was an expensive exercise in spite, took a long time and a lot of people - and everyone knows it.

They could have done this far efficiently already if they wanted to scale out, but that company has too many conflicting internal priorities that they took a heck of a long time to move away to anything that scales.

They never try to invent things and if not Oracle, they'd use some other vendor built software. That is the true reality of this company.

So no, if you're thinking you're better than this company for moving away from vendor built software, you are mistaken. In the worst possible way.

brixon(10000) 4 days ago [-]

They did move to vendor built software. They moved to one that they could get better prices for, AWS. Everyone else will not likely get the deal that they received.

whoisjuan(4105) 4 days ago [-]

'Amazon bots'... sure dude. Because all your opinions are popular by default. Otherwise is just 'bots' downvoting you.

Why people can't admit that their opinions can be unpopular sometimes. That's not a bad thing. Not all the things you say need to be popular to have meaning to yourself and others. It sucks to be downvoted but that's part of having a healthy discourse in HN.

StreamBright(2711) 4 days ago [-]

It is hard to understand what you are saying. HN is not really an AWS loving place. Expensive in regard of what? If you end up saving 50% of your year over year cost across all the company than spending millions on this project and running it for 5 years is still worth the investment in the long run at their scale. Amazon is not looking for 1 month investments, their AWS endeavour is 13 years old[1]. They can spend a long time on something if they really up to it.

1. https://www.statista.com/statistics/250520/forecast-of-amazo...

samfriedman(3975) 4 days ago [-]

In my mind the true 'hard part' would be in migrating business logic that lives in Oracle stored procedures (PL-SQL). Are there any tools that do a decent job in translating procs to non-Oracle dialects?

metadata(10000) 4 days ago [-]

Yep, we built SQL Tran to do exactly that. https://www.spectralcore.com/sqltran. Finishing the rewrite right now which allows us to make changes much faster.

dragonwriter(4218) 4 days ago [-]

> Are there any tools that do a decent job in translating procs to non-Oracle dialects?

One of the main selling points of EnterpriseDB (built on top of and supporting development of Postgres) is a significant degree of Oracle compatibility, including PL/SQL, client interfaces, etc., so if you go that route you might not need much translation.

poxrud(10000) 4 days ago [-]

AWS has the AWS Schema Conversion tool that works pretty well.

mbell(4052) 4 days ago [-]

I haven't used it in quite awhile but as I recall ora2pg[0] has some basic auto-conversion of PL-SQL.

[0] http://ora2pg.darold.net/

darksaints(10000) 4 days ago [-]

It should be noted that the email that Steve Yegge made famous was:

1) The birth of the service oriented architecture as a standard at Amazon

2) Took place long before AWS was used by Amazon's commercial business, and most software systems were monolithic, requiring mainframe-like dedicated hardware.

3) Was inspired due to the epic spaghetti monster known as FC Software, whose entire non-architecture revolved around a massive Oracle database.

When Amazon hit the limits of scaling the Oracle databases with beefier hardware, they realized what a clusterfuck they had created. There were literally hundreds of applications, doing different things, but all connected to the oracle database. The oracle database functioned as a database, but also as a high throughput queue, as well as a message passing platform. At high loads, different applications were hitting the same rows so frequently that the transaction errors alone could bring down an entire fulfillment center.

And it should also be noted that it was the same FC Systems (the inspiration for the move towards SOA) that took 12 years to finally get rid of Oracle.

Moral of the story: you need competent software architects, and at the very least, a scalability plan. Also, fuck Oracle.

etaioinshrdlu(3733) 4 days ago [-]

I dunno about that. Maybe this could be read as a success story. They built half-assed software quickly, scaled it to ridiculous levels, made billions of $, and finally had to replace it with something better. And now they are just fine.

zzzcpan(3630) 4 days ago [-]

> The oracle database functioned as a database, but also as a high throughput queue, as well as a message passing platform. At high loads, different applications were hitting the same rows so frequently that the transaction errors alone could bring down an entire fulfillment center.

Isn't that how everyone uses Oracle? I've seen exactly the same complains from developers at banks. Although building software around big featureful RDBMSs in general encourages poor architectures and messy software.

mxschumacher(1864) 4 days ago [-]

Will be interesting to see whether Amazon can make their insight into a service. Countless companies would be glad about a migration path from Oracle to AWS's solutions.

yannis7(10000) 4 days ago [-]

wasn't Larry Ellison last year who was bragging that the whole of Amazon/AWS runs on Oracle?

bluejekyll(2836) 4 days ago [-]

He often uses Java as a means of making that statement somewhat true.

atonse(10000) 4 days ago [-]

That, and Oracle's legendary shitty business tactics, probably pushed AWS to make this a higher priority, and stick it in oracle's face with videos of oracle shutdown parties and blog posts.

Take that, Larry Ellison.

paulddraper(4026) 4 days ago [-]

To be fair, AWS does run on Java.

exabrial(4092) 4 days ago [-]

Oracle should have been doing everything to keep them as a customer and push their databases through cloud offerings. They already push their software through resellers, sort of the same thing, just automatic

WomanCanCode(10000) 4 days ago [-]

Do you know what is happening to their cloud offering?

orf(1775) 4 days ago [-]

> Across the 48 hours of Prime Day, these sources made 7.11 trillion calls to the DynamoDB API, peaking at 45.4 million requests per second.

> The team added an additional 63 petabytes of storage ahead of Prime Day; the resulting fleet handled 2.1 trillion requests per day and transferred 185 petabytes of data per day.

> Prime Day 2019 also relied on a massive, diverse collection of EC2 instances. The internal scaling metric for these instances is known as a server equivalent; Prime Day started off with 372K server equivalents and scaled up to 426K at peak.

Woah.

https://aws.amazon.com/blogs/aws/amazon-prime-day-2019-power...

apocalyptic0n3(10000) 3 days ago [-]

The largest scale I've ever needed to work at was 2 EC2 instances (really just 1 for incoming and 1 for background SQS job handling). The numbers in that post are simply staggering to me.

sofaofthedamned(4154) 4 days ago [-]

Hate oracle personally. But when Amazon mention cost savings, I assume they're not at list price for normal aws customers like us?

somethingwitty1(10000) 4 days ago [-]

I can't say either way, but they mention the average person saves even more than they do (90% vs the 60% Amazon saw). This was due to their Oracle license being much cheaper than normal Oracle customers. So if you extrapolate, even if they did get some discount being Amazon, the discount they were getting on Oracle probably makes the savings comparable to if they weren't getting large discounts on each side (regular customers of Amazon/Oracle).

nikanj(3980) 4 days ago [-]

Oracle licenses are priced with a relatively equation, taking two inputs:

1) How much money do you have 2) How much would it cost you to switch to something else

Your annual license cost is the smaller of the two inputs, minus a 10% discount.

QuadrupleA(3941) 4 days ago [-]

Article mentions that, yeah - cost savings are based on their already heavily discounted deal with Oracle.

IronWolve(3579) 4 days ago [-]

Oracle goes after customers with 3rd party companies, if you use oracle expect calls for audits and more licenses. You download virtualbox extension at work? Thats a fee per user. Use an Oracle VM in a vmware cluster? They want paid for all cores in your environment now.

Seriously, Oracle goes after their customers like criminals. I refuse their calls at work, they really are just looking for something to make you pay.

toyg(3652) 4 days ago [-]

> Oracle goes after customers with 3rd party companies

That's going to end soon - the partner ecosystem has been royally shafted by Oracle's big 'cloud push', people are moving away from a vendor who simply sold them down the river; and in their Brave New Cloud World, there is no need for license resellers and auditors.

Ironically, that might end up improving Oracle's reputation - they'll just be seen like Google, where everything is automated and support goes from mediocre to nonexistent.

generatorguy(10000) 4 days ago [-]

I tried to license virtualbox extensions and they told me to call back when I wanted at least 500 licenses.

LrnByTeach(4189) 4 days ago [-]

Here is the meat of the Report:

>We migrated 75 petabytes of internal data stored in nearly 7,500 Oracle databases to multiple AWS database services including Amazon DynamoDB,Aurora, RDS, and Redshift.

>We kept careful track of the costs and the performance, and realized the following results:

>Cost Reduction – We reduced our database costs by over 60% on top of the heavily discounted rate we negotiated based on our scale. Customers regularly report cost savings of 90% by switching from Oracle to AWS.

>Performance Improvements – Latency of our consumer-facing applications was reduced by 40%.

>Administrative Overhead – The switch to managed services reduced database admin overhead by 70%.

e12e(2000) 4 days ago [-]

> Cost Reduction – We reduced our database costs by over 60% on top of the heavily discounted rate we negotiated based on our scale. Customers regularly report cost savings of 90% by switching from Oracle to AWS.

So Amazon saves 60%, when switching to paying 'cost' for Amazon services - and they still claim customers paying cost+markup save 90%?

I'm sure there's probably savings - but that math sounds optimistic.

Or more likely they're apple to oranges - porting a service that could run on a single postgres node from oracle to anything will save you maybe 10k USD a year in licensing for which you see a pretty slim return.

jobu(4009) 4 days ago [-]

> Customers regularly report cost savings of 90% by switching from Oracle to AWS.

How the hell is Oracle still able to charge so much? Earlier this year we were looking to spin up a temporary development instance of Oracle DB for a proof-of-concept project, but the license alone was over $50k. People were already pissed about them adding licensing costs to Java, so this kicked off an effort to start migrating to a modern tech stack.

dantiberian(963) 4 days ago [-]

I read it as saying those discounts had been negotiated over the past 10-15 years.

chrisseaton(3077) 4 days ago [-]

How did they negotiate a discount when they weren't yet able to move off the database? 'Give us a discount or...' or what?

growlist(10000) 4 days ago [-]

The video strikes me as an epic troll:

https://www.youtube.com/watch?v=9yBP5gnnZi4&feature=youtu.be

fanf2(61) 4 days ago [-]

Who was the lucky bod who got to do the deed, and why did we only get to see their right arm?

kgraves(3265) 4 days ago [-]

> stopped oracle

Now Amazon should do this to every company using them.

jeffbarr(183) 4 days ago [-]

That was the actual shutdown. I was there, it was not rehearsed, and there were no retakes.

jansan(4151) 4 days ago [-]

I am pretty sure the emotions are real in this video.

jnaina(10000) 4 days ago [-]

Here's the key difference between Oracle and AWS:

- Oracle will schedule periodic audits on their customers and legally threaten them for inadvertent licensing oversights

- AWS makes it a mandatory policy that all their sales folks periodically sit with their customers to help optimise and reduce their ongoing AWS Cloud costs

If you are making a case that Oracle == AWS, you don't know what you are talking about. Yes, I have dealt with both these companies up close.

philliphaydon(2126) 4 days ago [-]

> AWS makes it a mandatory policy that all their sales folks periodically sit with their customers to help optimise and reduce their ongoing AWS Cloud costs

Their KPIs are to reduce costs for customers but also to increase usage of AWS services to make more money. The KPIs seem to contradict each other.

irrational(10000) 4 days ago [-]

We've been working on migrating from Oracle to Postgres for a few years now. We are about 2 weeks from being finished. It is not for the faint of heart, but it is totally worth it. The documentation is much much better, performance is equivalent or better, the sql dialect is saner, etc. Other than moving the data itself (ora2pg was invaluable for this), rewriting the queries is what has taken the most amount of time. Some of our tips on differences between oracle and postgres sql:

replace nvl with coalesce

replace rownum <= 1 with LIMIT 1

replace listagg with string_agg

replace recursive hierarchy (start with/connect by/prior) with recursive

replace minus with except

replace SYSDATE with CURRENT_TIMESTAMP

replace trunc(sysdate) with CURRENT_DATE

replace trunc(datelastupdated) with DATE(datelastupduted) or datelastupdated::date

replace artificial date sentinels/fenceposts like to_date('01 Jan 1900') with '-infinity'::date

remove dual table references (not needed in postgres)

replace decode with case statements

replace unique with distinct

replace to_number with ::integer

replace mod with % operator

replace merge into with INSERT ... ON CONFLICT... DO UPDATE/NOTHING

change the default of any table using sys_guid() as a default to gen_random_uuid()

oracle pivot and unpivot do not work in postgres - use unnest

ORDER BY NLSSORT(english, 'NLS_SORT=generic_m') becomes ORDER BY gin(insensitive_query(english) gin_trgm_ops)

Oracle: uses IS NULL to check for empty string; in postgres, empty string and null are different

If a varchar/text column has a unique index a check needs to be made to make sure empty strings are changed to nulls before adding or updating the column.

PostgreSQL requires a sub-SELECT surrounded by parentheses, and an alias must be provided for it. - SELECT * FROM ( ) A

any functions in the order by clause must be moved to the select statement (e.g. order by lower(column_name))

Any sort of numeric/integer/bigint/etc. inside of a IN statement must not be a string (including 'null' - don't bother trying to use null='' it won't work). Concatenating a NULL with a NOT NULL will result in a NULL.

Pay attention to any left joins. If a column from a left join is used in a where or select clause it might be null.

For sequences, instead of .nextval use nextval('')

fuzzieozzie(10000) 4 days ago [-]

If you have a DB migration challenge then take a look at www.compilerworks.com

sidcool(227) 4 days ago [-]

Super useful. Thanks.

WorldMaker(10000) 4 days ago [-]

> replace merge into with INSERT ... ON CONFLICT... DO UPDATE/NOTHING

PostgreSQL supports MERGE statements [1], and a cursory glance doesn't seem to show that Oracle deviates that much from the SQL standard. Is there a particular reason you are avoiding the merge statement or is it simply a readability preference for the non-standard 'upsert' ON CONFLICT extension?

(I'm curious because I was a big fan of MERGE statements in a past life. Part of being allowed to use them in that codebase was proving that the standard was pretty well followed across DB vendors [not that we used more than one], as well as that MERGE performance was critical to some very large dataset joins with complex update logic.)

[1] https://www.postgresql.org/message-id/attachment/23520/sql-m...

cabaalis(10000) 4 days ago [-]

Now do MSSQL.. (Gets pen and notepad)

ducktape(10000) 3 days ago [-]

@irrational, you have my sympathy. But next time you hear of anybody wanting to torture themselves with such an endeavor you might want to save their soul by asking them to look at CompilerWorks.com. Full disclosure, I work at/for CompilerWorks and will be biased since they pay the bills. But as an ultimate compliment for the kind of work we do, one of our customers brought xz0r's comment about us to our attention.

mulander(3008) 3 days ago [-]

> replace rownum <= 1 with LIMIT 1

rownum is a pseudocolumn numbering entries, sorting the results will give you records in non increment rownum. On such queries rownum <= N will give you the record that was first in the results before sorting.

LIMIT will give you the first record after sorting.

So rownum != LIMIT and if you really want to implement the same logic you would need to use select row_number() over () as rownum in the ported query, put that into a subquery and filter with where rownum <= N in the outer level.

Example:

select * from (select row_number() over () as rownum, * from pg_class order by relname) as x where x.rownum < 5;

select * from pg_class order by relname LIMIT 5;

Will give two very different results.

The first one maps to oracle:

select * from pg_class where rownum < 5 order by relname;

The second one doesn't.

koolba(638) 4 days ago [-]

> any functions in the order by clause must be moved to the select statement (e.g. order by lower(column_name))

What does this refer to? AFAIK, you can invoke functions just about anywhere including in the ORDER BY clause:

    => CREATE FUNCTION some_func (int) RETURNS int AS $$ SELECT $1 + 1; $$ LANGUAGE SQL;
    CREATE FUNCTION
    => SELECT t.* FROM (SELECT 1 AS a) t ORDER BY some_func(t.a);
    a 
    ---
    1
    (1 row)
Are you referring to not being able to reference column aliases in an ORDER BY clause? (work around is to use a subquery)

    => SELECT 1 AS a ORDER BY some_func(a);
    ERROR:  column 'a' does not exist
    LINE 1: SELECT 1 AS a ORDER BY some_func(a);
Pxtl(10000) 4 days ago [-]

A lot of this sounds similar to MS SQL Server - the MERGE one sticks out, does Postgres not have MERGE?

encoderer(3846) 4 days ago [-]

Somebody can build a consulting business from this HN comment.

ilaksh(3227) 3 days ago [-]

Wow.. someone could start an Oracle -> Postgres migration consulting business just from your comment.

I hope they name it SADL Consulting.

michannne(10000) 4 days ago [-]

Using SQL Server exclusively has given me a Never-Oracle mindset. Some of these differences sound insane. and

> any functions in the order by clause must be moved to the select statement (e.g. order by lower(column_name))

I'd pull the hair outta my head

colek42(3791) 4 days ago [-]

The best part about this comment is that any errors or new best practices will automatically be updated with the entire knowledge of the world.

sk5t(10000) 4 days ago [-]

I would note that 'foo IN (a, b, c)' can often be better done as 'foo = ANY(ARRAY[a, b, c])' in Postgres - not least because you can shift to passing in an array of variable size without having to construct a special statement for each.

For sorting on the result of a function, consider using a subselect. 'SELECT foo FROM (select fun(z) as foo from blaz) tmp order by 1'.

thibautg(3649) 4 days ago [-]

Thanks I've sent this comment to our DBAs. Very useful.

JacKTrocinskI(4167) 4 days ago [-]

I have a hard time believing PostgreSQL performance is better than Oracle, can you give an example? How big is your database and how much did migrating cost you?

vkaku(4174) 4 days ago [-]

I wonder: How long before someone adds a patch to PostgreSQL as a 'compat' patch, or extension.

thomasjudge(3454) 4 days ago [-]

What about pl/sql to pl/pgsql?

hamilyon2(4019) 4 days ago [-]

How do you develop and debug triggers?

aswanson(3784) 4 days ago [-]

It's almost like you should set up a Patreon for this comment.

ddgflorida(10000) 4 days ago [-]

Very useful - anyone using oracle should try to use ANSI keywords now to avoid work in the future, always use COALESCE instead of NVL, use ANSI date functions, etc.

TheKarateKid(10000) 4 days ago [-]

As someone who has never used or worked with Oracle DBs, can someone explain why migrating to another technology is so difficult?

With all the database engines available, is there really not another one that can match Oracle without massive customizations?

forinti(10000) 4 days ago [-]

When you have an Oracle database, you probably have a lot of PLSQL. That, in itself, is not so serious (PgPLSQL is quite similar).

But you'll probably have Forms, Reports, and Apex too. These are all designed to make building an app a piece of cake but make it impossible to migrate except if you rewrite the whole thing.

raincom(10000) 4 days ago [-]

I used to write PL/SQL long time back. In old days, every business logic was written in stored procedures, which the java code calls. Migrating stored procedures is a hard part since whoever wrote it left the company long time ago.

Whatever talent available in the company that wants to migrate off Oracle, would not want to take the risks of migration failure.

stickfigure(3129) 4 days ago [-]

It really depends on how your applications were designed. If you 'go deep' with Oracle - and if you've hired Oracle DBAs, you probably have - you probably have a lot of your business logic buried in a large body of stored procedures that aren't version controlled and rely on Oracle-specific behavior.

Ygor(4057) 4 days ago [-]

In theory SQL is SQL, even more so if your App uses a layer of indirection like an ORM. If you don't use any Oracle specific features, procedures and similar, in theory you could just replace it.

But that is not what happens.

First problem is - majority of mature enterprise systems (which tend to use Oracle), are not designed in such a schoolbook way, so that's you first problem. Codebases are ugly and teams loose the knowledge to maintain some parts that are mature and work mostly ok - often the DB layer.

Another concern is regressions in the product that is hard to properly test for. Minor details that are exposed by even small changes in DB behaviour or performance, that won't get caught in testing, but will cause an impact when your next scheduled job happens to fall on the first friday of that one month where the customer has that very special report that is just slightly so different and runs only once every 7 years.

Another major pain point are operational concerns. When dealing with DB technologies, managing, deploying, operating your back end DB servers can be quite a different beast. In theory your teams should be able to handle it, as they are experienced engineers that know how to learn new tech quickly. In practice, you don't have money for those, and your team is lazy and doesn't really want to learn that new cool tech, just let them do their job.

And finally, if it all 'just works' at the moment, it is a hard sell to switch such a fundamental part of the system and properly validate the investment versus the risks.

And this is all if you're not a super user of Oracle specific pieces, surrounding tooling or consultants/expertise.

silon42(10000) 4 days ago [-]

It probably starts with horrible things (not SQL compatible) like '' is equal to NULL...

ajkjk(10000) 4 days ago [-]

Oh man. I used to work at Amazon at a time when this Oracle deprecation was starting up. It's utterly non-trivial.

It's not really 'switching relational DBs', it's 'rebuilding all of the legacy systems to not use relational DBs'. Their scale had long since left the realm where it was reasonable to run everything on massive monolithic DBs, but the technical debt hadn't been paid down -- so they just kept putting more money into bigger and bigger hardware and larger DBA teams to support them. For instance when I last worked the there each warehouse ran on a single massive Oracle DB. That isn't solved by swapping to another database; it was well beyond reasonable.

I suspect that this article seems like a bit of self-promotion but it's actually intended as a giant middle-finger to Oracle.

AtlasBarfed(10000) 4 days ago [-]

Amazon is a slightly annoying company that I rarely root for, but Oracle is a predatory company whose customer relations strategy seems to actively involve litigating them as a default assumption.

Amazon is considered a paragon of IT, so their expulsion of Oracle is a useful tool at the CIO/CTO level of 'nobody ever got fired for IBM' level of reasoning.

Unfortunately, AWS is becoming the new IBM in that reasoning. Great that it took ten years to reach that level of agreement, now hybrid cloud needs to take over the grey haired CIO / CTO level.

Long live Postgresql!

philjohn(10000) 4 days ago [-]

Also, kudos to them for not just cutting loose the DBAs who have a lot of institutional knowledge and training them to be consultants on AWS data storage services both internally and externally.

Wistar(10000) 4 days ago [-]

Back in the early days of Oxygen Media I remember the then CTO saying that Oracle's pricing model appeared to him to be based on turning the customer upside down to empty all of the money from their pockets.

golergka(2603) 4 days ago [-]

Isn't AWS much less prone to customer lock-in? May be I don't have experience with tweaking AWS under heavy high load, but 98% of code I write for backend that I host on Amazon isn't specific to AWS or it's services in any way.

rayiner(2917) 4 days ago [-]

I don't think most people would view the companies that way. Amazon is taking on mom & pops, warehouse employees, etc. Oracle might be super aggressive, but their targets are generally larger businesses and often huge corporations that are perfectly capable of taking care of themselves.

Justsignedup(4205) 4 days ago [-]

Next up: oracle sues Amazon for not using their product.

JacKTrocinskI(4167) 4 days ago [-]

How is Oracle predatory? PostgreSQL is great for web developers and small businesses but in terms of support, security, and enterprise features I think Oracle has PostgreSQL beat. I have been an Oracle database developer for the last five years, they have a great database, not sure what all the hate on HN is about.

gopalv(10000) 4 days ago [-]

> Amazon is a slightly annoying company that I rarely root for, but Oracle is a predatory company

Amazon and Oracle couldn't be more different in this point of view.

Amazon's predatory instincts are pointed inwards and downwards, towards employees and vendors, while being mostly great towards the customer.

Everything from their warehouse workers with timers, the crying software engineers and all other things I hear about them are about running an internally facing extraction economy.

The squeeze is being put on those who draw a paycheck (or at least, a profit) from Amazon, while the consumers see instant access servers on EC2, Amazon Go for lunch breaks in a city or One hour shipping of diapers.

In comparison, Oracle seems to be going after those who pay them already.

scarface74(3825) 4 days ago [-]

Unless your company is going to spend years and millions of dollars to create their own OLAP database product for their own internal use knowing that they can then charge customer $4.80 an hour for a usable system, Amazon isn't a useful model for any other IT organization.

outworlder(3630) 4 days ago [-]

> whose customer relations strategy seems to actively involve litigating them as a default assumption

They are not nice to their partners either.

godson_drafty(10000) 4 days ago [-]

The developer carrying it out still using 2015 MBP

akhilcacharya(4196) 4 days ago [-]

Is that a...big deal? It's just 2015. Most of my coworkers use 2015 models because they prefer them.

PeterisP(10000) 4 days ago [-]

We currently have two 2018 MBPs on the shelf that were bought as 'upgrades' back then but they're sitting unused because people are sticking with their 2015 MBPs until they won't be maintainable anymore or Apple (or someone else) releases a version that's an improvement over 2015 MBP instead of a downgrade.

ravedave5(10000) 4 days ago [-]

You'll have to pry mine from my cold dead hands.

catalogia(10000) 4 days ago [-]

Many consider 2015 the best year yet for MBPs.

TimTheTinker(3995) 4 days ago [-]

And he's using iTerm2 as the terminal.

dekhn(10000) 4 days ago [-]

i miss my t42p

taf2(3748) 4 days ago [-]

Me too - still the best laptop

hanniabu(3932) 4 days ago [-]

My 2011 MBP is still my primary. This is the case for a lot of people because the newer versions just don't compare. On to of that you can't even upgrade the components in them since everything is integrated into the board.

kevan(10000) 4 days ago [-]

I'm also on a 2015 model, but it was manufactured in early 2017 so it's not that old. I and several people I know are holding out on the 2015s for as long as they can because of ports, no touch bar, and a better keyboard compared to the newer models.

zippergz(10000) 4 days ago [-]

My company has tried to get me to replace my 2015, and I am resisting until they release a better keyboard design.

andoma(10000) 4 days ago [-]

Imagine the disappointment when pressing enter and nothing happens because the key didn't register.

jedieaston(10000) 4 days ago [-]

Is this not a case study that anyone can leave Oracle now, if an Amazon scale install can?

AshwinDurairaj(10000) 4 days ago [-]

SAP and Salesforce would love to, but haven't achieved Oracle-free status to the joy of Larry.

vkaku(4174) 4 days ago [-]

It shows that a big company like Amazon, hiring thousands of SDEs still needed other vendors software than their own, because of all that time they spent in hiring and firing people and obsessing over their customers needs.

sarcasm

mumblemumble(10000) 4 days ago [-]

It is not.

Amazon accomplished the task by building their own Oracle replacement. You have to be approximately as large as Amazon for that to be a realistic option.

If you'd like to use the alternatives that Amazon built, you can, of course, migrate to Amazon Web Services. Though, if you dislike the vendor lock-in you're experiencing with Oracle, and are considering Amazon as a good place to flee to, you possibly haven't been paying attention. With Oracle, you're just renting the software. With Amazon, you're renting both the software and the hardware, and renting the hardware is a precondition of being able to continue using the software, AND they charge you a fee to move your data off of their hardware. It's one of the most well-engineered enterprise vendor lock-in traps the industry has ever seen.

ceejayoz(2116) 4 days ago [-]

Just because Bill Gates has the resources to buy a fleet of 747s doesn't mean I can.

lawnchair_larry(2797) 4 days ago [-]

Or is it a case study that you need the resources of Amazon to pull off migrating from Oracle?

StreamBright(2711) 4 days ago [-]

I think the approach can be applied to many companies. I have seen some companies turning every single data storing problem into an Oracle problem. Amazon turned many of the problems into ones that are easier to solve. Using 6 different technologies[1]. I am sure not all applies to all departments but it is a better approach to do this at the scale they got. For small companies this really does not matter. If your hot data fits a single node disk cache you obviously won't benefit from using a horizontally scalable database but Oracle's customers are often in the higher end of the scale spectrum. The question is how can Amazon compete with Oracle where the relationship is there for decades between a customer and Oracle. Good example of this is banks. They managed to get some of the financial companies to move some workloads to AWS but to move Oracle workloads to something else is a whole different game.

1. https://media.amazonwebservices.com/blog/2019/bye_bye_oracle...

mmsimanga(3636) 4 days ago [-]

> some third-party applications are tightly bound to Oracle and were not migrated

Oracle's strategy of buying companies like PeopleSoft is paying off.





Historical Discussions: Flash Is Responsible for the Internet's Most Creative Era (October 13, 2019: 894 points)
Flash Is Responsible for the Internet's Most Creative Era (October 10, 2019: 4 points)

(895) Flash Is Responsible for the Internet's Most Creative Era

895 points 6 days ago by Osiris in 1029th position

www.vice.com | Estimated reading time – 17 minutes | comments | anchor

A version of this post originally appeared on Tedium, a twice-weekly newsletter that hunts for the end of the long tail.

It wasn't always this way, but life is pretty good for the music industry these days on the streaming front.

The industry, for the first time in more than two decades, is seeing significant growth, largely off the back of streaming technology such as Spotify, Apple Music, and Pandora.

Good for them—and possibly also for consumers. But it's not hard to forget that the music industry, caught off-guard by new technology in the late 1990s, tried to force the issue of getting paid through the launch of forgotten services like MusicNet and PressPlay—and despite the similarities to the way we stream music now, it got burned, badly.

Perhaps that was a foregone conclusion, but the response to Napster is still very much worth analyzing. Why did a very similar spin on today's streaming services crash and burn, anyway?.

1994

The year that CNN first reported on the existence of the Internet Underground Music Archive, one of the first services that existed for distributing online music. The service, essentially the SoundCloud or BandCamp of its day, allowed independent artists to distribute their music over digital means. IUMA was a hugely influential idea, but struggled as a standalone company, eventually falling by the wayside around 2001 after a failed acquisition by the similarly noble digital music company eMusic. The service's music was brought back to life via the Internet Archive.

Why Primitive Radio Gods is the perfect band to explain the Napster-induced downfall of the physical music industry

Before the music industry failed to rebuild the digital music industry in its own likeness, it pulled a fast one on bands like Primitive Radio Gods.

You probably don't remember the band, but you most certainly remember the song—if not its title, "Standing Outside a Broken Phone Booth With Money in My Hand," which, let's face it, is a title only Fiona Apple could love.

Popular in the summer of 1996, the song, best known for its inclusion on The Cable Guy soundtrack, represents a low point of a trend that came to define the music industry in the pre-Napster era. It was a good song, to be completely fair to its creator, Chris O'Connor, and the tune drew the attention of Columbia Records thanks in no small part to its use of atmosphere and a particularly effective B.B. King sample.

The problem was, the rest of the album was effectively old demos produced on the cheap and rushed to release by Columbia before O'Connor had a chance to properly record them in a studio. And this fact was obvious to reviewers.

Allmusic reviewer Stephen Thomas Erlewine described the album as such: "At its core, Rocket sounds like a demo tape with one promising song."

(Their second album, for what it's worth, got better reviews.)

To be fair, the album was more symptom than diagnosis. The real problem was that Columbia would not sell "Broken Phone Booth," a top-10 hit on the Hot 100 Airplay charts, as a standalone single. Because of how the label made its money on rock music at the time, they needed an album—and rather than buying time for Primitive Radio Gods to record one, like a buzzy single might allow for nowadays, the song required the band to immediately release whatever they had lying around.

This was a great way to make money— Rocket, which cost $1,000 to record, went gold—but a horrible way to treat customers. When you're a teenager, stuck paying $16 for an album featuring one good song and nine demos, something like Napster seems hugely appealing.

Primitive Radio Gods were far from the only band to momentarily glimmer based on the kind of bad music-industry calculus famed producer Steve Albini, best known for his work with Nirvana, could see from a dozen miles away, but their trajectory highlights nearly all of its downsides. The band didn't even get a chance to release a legitimate album on a major label, because their label shut down!

If you're an "old millennial" music fan, you know what happened next to the music industry: In 1999, programmer Shawn Fanning created Napster in his Massachusetts dorm room, his idea quickly swept the internet, free music was had by all, Sean Parker found his calling, people downloaded that one Primitive Radio Gods song they liked without the other nine demos, the band Dispatch became one of the first digital-only success stories, Metallica growled publicly, and the music industry sued the service into oblivion, all in the span of three industry-disrupting years.

By the time the original Napster was gone for good in mid-2002, lots of alternatives, like KaZaA and Gnutella, had appeared. But the music industry, being the music industry, wanted to protect the good old days—the days in which they could sell their $16 CDs—as much as possible.

Enter MusicNet and PressPlay.

"What we underestimated was that the pivot to digital would be a two-step pivot. First there would be downloads, then subscriptions. We're clearly on the right side of history on this one."

— Rich Glazer, the founder of RealNetworks, noting in a 2016 interview with VentureBeat how his company's RealPlayer service, the basis of the MusicNet offering, was ahead of its time. The company, founded in 1994, played a key role in the launch of the music industry's fledgling attempts to create a legal music service. RealNetworks is still an active company in 2018—that big settlement from Microsoft helped buoy them during the lean years, as did some more recent patent sales—though the company is a minor player compared to what it once was. The firm has found success in Asia in recent years, particularly China.

An example of the Pressplay service. Image: OpenP2P

The music industry's first attempts at legal digital music were confusing, user-hostile, and kind of sucked

In the roughly 24 months between the time Napster shut down its popular free service and Steve Jobs announced the iTunes Music Store to the public, the music industry tried to create legal replacements, but the lack of precedent was a problem. Nobody could figure out exactly what a legal digital music industry was supposed to look like, or how it was supposed to work.

All the music industry knew is that it wanted the golden goose to be secured from the arms of digital thieves, so the solutions they gravitated toward were instilled with digital rights management, which was starting to come into its own around this time, thanks to both the growing sophistication of the technology, which I wrote about last year, and the 1998 Digital Millennium Copyright Act, which infamously made strides to prevent the technically inclined from attempting to legally break this technology.

"No person shall circumvent a technological measure that effectively controls access to a work protected under this title," Section 1201 of the U.S. Code states.

And with the appearance of Napster, the music industry suddenly had a reason to use these rules they pushed for. The problem? The music industry was stuck in a format war of sorts. With five major record labels at that time—it was six until 1999, when PolyGram merged with Universal Music Group—it was hard finding any common ground on a complicated issue like digital music. The short-term result of this discomfort was that the music industry effectively split the digital music industry in two.

On one corner was Universal Music and Sony Music, the two of which backed PressPlay, which was built with DRM technology from Microsoft's Windows Media Player. In the other was MusicNet, which had three labels at play—Warner Bros., Bertelsmann Music Group (BMG), and EMI.—and the backing of RealNetworks, whose streaming technology came to define the early internet.

Their offerings were slightly different—for a $9.95-per-month fee MusicNet allowed for 100 temporary downloads and 100 on-demand streams at launch, according to Billboard, while Pressplay allowed for 300 streams and 30 downloads and offered limited CD-burning capabilities at higher price points—but the tissue tying the two approaches together was DRM.

It certainly wasn't music, as the services made no effort to collaborate with one another. In the post-Napster era, the combination of limited libraries and competition from peer-to-peer file sharing services put the companies at a major disadvantage. As Billboard's Brian Garrity put it:

The big concept of 2001 has been that in order to compete with file-sharing services that offer a virtually limitless universe of free content, music companies must curb pirate peer-to-peer networks in the courts while at the same time develop similar secure music services of their own. But their offerings have two main differences: Consumers pay to access content rather than receive it for free, and content is primarily rented to consumers instead of accessed on a buy-to-own basis.

Complicating factors significantly was the landscape of the technology that separated the two services. Microsoft and RealNetworks (which was founded by a Microsoft alum) had become major rivals on the streaming front, with the conflict playing out between the two not unlike Netscape and Internet Explorer, with Microsoft's market power coloring the business conflict. (Microsoft's Justice Department settlement loomed over the situation as well.) In 2003, this would lead to a lawsuit that RealNetworks would eventually score a $761 million settlement from.

The result of this conflict for the digital music industry was that the services weren't compatible, effectively limiting the amount of music that was on each service—some major labels on one, some more major labels on the other. By the end, neither service had each of the major labels. The conflict between the two tech giants bled into the services they sold the music industry on.

'It's a crying shame that the Microsoft-RealNetworks rift has spilled over to the major labels,' Jupiter Media Metrix analyst Aram Sinnreich told CNET in 2002. "The end result is that it will be a longer time before consumers will have access to a music-subscription service that offers them enough music.'

It's not a dissimilar situation to the mishmash of availability we see with our streaming video services today, where pieces of content come and go like passing trains through the night, but in contrast to Napster or Audiogalaxy, it was a tough sell, especially given the artificial limitations the services imposed on users.

The music industry tried to strong-arm a replacement for Napster in the market, just as it strong-armed gold records out of artists who had a single and nine demos. And they were convinced that the strategy would work without any issues.

But the skepticism engendered toward both the music industry at large and DRM as a controversial new technology ensured its efforts immediately drew scrutiny. In August of 2001, just a few weeks after the free version of Napster was forced shut by court order, the US Justice Department opened up a fresh antitrust investigation into PressPlay and MusicNet.

The concern at the time was less that PressPlay and MusicNet were dominating the industry conversation without making room for third-party players. Sites that looked like potential competitors, like MP3.com and eMusic, had been recently snapped up by Universal Music and turned into affiliates of PressPlay, bastardizing the MP3-friendly approach the services originally took. (Both companies were later sold off: eMusic is active today; MP3.com is a content site that hasn't been updated in about three years.)

Meanwhile, other firms with competing DRM technologies—like InterTrust, the company that held most of the important DRM patents and would soon win a big settlement of its own from Microsoft—were suddenly competing with the major labels, that wouldn't even share with one another, let alone a third-party service.

It was clear to outsiders, and even some insiders, that what the music industry was trying to do wasn't going to work. (It was also clear that it was costing the labels tens of millions of dollars they would never get back.) The technology was too restrictive, the approach too stacked in favor of the record industry. There was too much distrust and bad blood in the air after what happened with Napster.

Stephen Witt, in his 2015 book How Music Got Free, portrayed Universal Music's then-CEO, Doug Morris, as being overly excited about PressPlay, to the point where Recording Industry Association of America President Hilary Rosen, frequently portrayed as an "enemy combatant" of the Napster era during this time, had a hard time talking him off the ledge. (Rosen, it should be said, was simply sharing the company line because it was her job.)

"On several occasions he told Rosen to stop talking to Napster, to stop negotiating with the Fannings, to stop worrying so much, because he had something that would 'make it all go away,'" Witt wrote. "In later years, PressPlay would be a reliable starting point for listicles of the 'Top All-Time Tech Busts.'"

Really, the only person who was able to talk the music industry off the edge was Steve Jobs.

In a 2011 article after Jobs' passing, Warner Music's then-VP, Paul Vidich, explained to Billboard that Jobs quickly cut through the music industry's BS ("'I don't want to talk about what you guys are doing,' he said. 'You guys have always had your heads up your expletive-deleted,'" Vidich recalled of their first meeting) and came up with a solution that every label could agree to—a situation where the technology people were in charge of the technology and the record labels got out of the way.

'We did our deal, closed it in October 2002, they then pitched it to each of the other, who signed on and they launched it on April 28, 2003," Vidich noted. "Within a month they sold a million downloads, which startled everybody."

Soon enough, the Justice Department closed its investigation into PressPlay and MusicNet. It wasn't necessary anymore.

2003

The year that Napster relaunched as a pay service, effectively a rebranded version of the former PressPlay service. The company had been purchased by Roxio after its bankruptcy and had no connection to the original service. The Napster brand, which has existed for more than 15 years outside the purview of Shawn Fanning and Sean Parker, is so strong today that in 2016, the music service Rhapsody rebranded itself as Napster.

So, it's worth pondering: Why were MusicNet and PressPlay such bad ideas, and why, in contrast, is Spotify seen as a much better one?

Clearly, if you break it down, the approaches are similar minus the download limitations put on use of the older services—especially with the paid subscription model that's common today.

But the key difference may be the intent of the offering. Spotify is clearly a service that was built for music listeners first and music labels second. This gave the company some headaches as those labels complained about things like mechanical reproduction, and it's had to change up its approach a few times as a result, but the fact of the matter is, the service has always favored the listener over the label or the artist.

Certainly, the fact that our phones made the technology more portable played a factor as well.

But it's worth suggesting that perhaps we, as consumers, changed, with some distance from Napster. We spent years in a content free-for-all, with little in the way of concern about who was going to get paid. Not because it was the right way—but because it was the path of least resistance.

In 2003, as he was announcing the iTunes Music Service, Jobs called subscriptions "the wrong path," an avenue that became the path of most resistance.

"These services treat you like a criminal," he said at the time.

As Apple Music came to life from the purchase of Beats and emerged as Spotify's most robust competitor, the line was heavily scrutinized after the fact, but it's possible that the wrong path became the right one for a single, simple reason.

We stopped treating music fans like criminals.

Correction: This article originally stated that Steve Jobs passed in 2010. He died in 2011. Motherboard regrets the error.




All Comments: [-] | anchor

mycall(10000) 6 days ago [-]

There is tons of things going on on YouTube these days.

Latty(10000) 6 days ago [-]

Yeah, while the Flash era was definitely great for animation, that was really more because animation was the only real option for video. Streaming video has largely replaced that, and cheap cameras mean animation is way costlier to do now.

It is a shame that the tooling for animation online doesn't appear to be good. I guess the reality is probably that it's just cheaper and easier to render your animation to video, or release as a game. Game engines are working harder to target browsers as a platform, so the latter option will probably come full circle.

etaioinshrdlu(3733) 6 days ago [-]

I really think from a business perspective Adobe bungled just about everything with Flash.

When the tide was turning against them with the rise of iOS, they should have full open sourced Flash player and made the authoring tools free to use, and encourage competing authoring tools.

Flash would likely have then been on a track towards full standardization and native support in browsers. Adobe's authoring tools would likely be best-of-breed and indispensable for high end web dev. All the security and performance and interoperability problems could have been solved over time. Flash is not all that different from an SVG to be honest.

One counterpoint to make: Flash sites were to my knowledge static layouts, the very opposite of responsive design. That is a big industry shift that Flash never made.

Instead Adobe threw in the towel and encouraged breaking a large portion of the old internet by deprecating it.

I think moves like these amount to hundred billion dollar drags on the economy. In a sense we're all a little bit poorer as a result. It's like digging a hole and filling it back up again. It's economic activity that benefits no-one.

ScottFree(4142) 6 days ago [-]

> Flash sites were to my knowledge static layouts, the very opposite of responsive design. That is a big industry shift that Flash never made.

Adobe Flex did data-driven responsive design just fine. It was the popular framework before es5 and html5 made it obsolete.

cft(1043) 6 days ago [-]

I read somewhere that Adobe could not opensource Flash Player due to some library codecs licensing issues. It was unfortunate however, that Adobe bought Macromedia. Macromedia Flash would not have caused wrath from Steve Jobs and Macromedia would have been nimble and motivated to work something out with Apple

seanalltogether(2013) 6 days ago [-]

Adobe bungled everything about Macromedia. My livelihood was pretty much built around Macromedia during the entire 2000s and I remember how uneasy I was about the Adobe buyout back then. Macromedia conferences were seriously creative and fun and educational and developer friendly, the first year after Adobe took over they turned it into a big marketing event and within a short time span the entire community was killed off. I think back then (and maybe even now) Adobe just saw the web as a publishing platform, whereas Macromedia saw it as a playground.

scarface74(3825) 6 days ago [-]

Well,

Adobe claimed that Apple was stopping them from supporting Flash on the original iPhone. When Adobe did finally get Flash (barely) running on Android. It required 1GB of RAM and 1Ghz CPU. The original iPhone had a 400Mhz CPU and 128Mb of RAM.

Adobe was late shipping Flash for the Motorola Xoom. Motorola touted being able to use Flash as a feature over the iPad. Leaving it in the unfortunate situation that you couldn't even visit the Xoom marketing page running Flash from a Xoom for the first six months.

Adobe could never get Flash working on mobile well.

EDIT: It wasn't until the iPhone 5 introduced 5 years later in 2012 that there was an iPhone that could have met Adobe's specs for Flash.

aikah(4033) 6 days ago [-]

What Adobe destroyed is the community existing around Macromedia products. It was just unprecedented to my knowledge. Most Macromedia solutions were extremely easy to use AND improve via plugins. You could use Javascript to easily create add-ons for Flash, or Fireworks, or Dreamweaver and there was that 'community' aspect that did not exist with Adobe product.

And then Macromedia got 'Eloped'...

It's crazy how Adobe never leveraged that community and just pissed everybody off then it died out...

There was also that kind of 'friendly' competition where teams add to come up with the most bad ass interactive experience and brands had huge budgets to promote this or that product. It was an healthy relationship between marketing and creativity. Everybody even now, remember at least some Flash websites. '2advanced' anybody? Who remembers the design of the web sites they visit today? It's all the same.

Obviously at some point, Flash ads became a nuisance, and mobile kind of killed it in the browser...

> Flash would likely have then been on a track towards full standardization and native support in browsers.

Unfortunately no, because TC39 rejected Ecmascript 4. Ironically, Microsoft who is responsible for Typescript is to blame for that. Because they had their own solution 'Silverlight', it was short-sighted.

pier25(3433) 6 days ago [-]

> Flash sites were to my knowledge static layouts

Some were, the bad ones, but most were not.

I did a lot Flash back in that time and it was all responsive.

dillonmckay(10000) 6 days ago [-]

Nobody has mentioned Youtube using Flash initially to create the experience of 'it just works.'

We had RealPlayer, Quicktime, and others, but video on the web was rather difficult.

lqet(4209) 6 days ago [-]

And they only made HTML5 playback the default mode 4 years ago.

PBnFlash(10000) 6 days ago [-]

HLS DASH are never going to be as simple as RTMP was with flash. It 'just worked' in a way the internet just doesn't anymore.

buboard(3489) 6 days ago [-]

You can set up a video chat lecture with text chat and pretty good video with the free red5 java server and something like Avchat.com . That's 20 year old tech and works today, but will be removed from the net in a year. There is no HTML5 alternative. Why?

I think Tech took a weird turn somewhere around 2010. Things became more homogeneous, designes became bland without rough edges, literally with rounded corners. Options were removed, defaults became simpler and dumber. Dev tools are more about saving programmers from themselves rather than pushing things to the limit. Flash is probably an example of it

doctorpangloss(4080) 6 days ago [-]

Unity WebGL is definitely close to where Flash was. If anything WASM adoption is growing, it's a respected standard. Once there is multi threading, heap expansion and streaming instanciation in iOS ... I'm going to guess 15, or 4 years from now, it'll be the true spirit of Flash.

I'm not sure about a "dumber" design discipline though. Really it's Windows developers being left behind, for a while now.

cookie_monsta(10000) 6 days ago [-]

I'm not familiar with red5 but it seems that webRTC does a decent job on video and text chat with the browser doing the heavy lifting in terms of encryption and permissions. Am I missing something?

vertex-four(10000) 6 days ago [-]

I threw up a livestreaming setup + chat with nginx-rtmp-module, Prosody, Converse.JS and a few lines of javascript in a weekend. I'm not really sure what you're talking about tbh.

superkuh(4024) 6 days ago [-]

> 2010. ... homogeneous ... bland without rough edges, literally with rounded corners. Options were removed, defaults became simpler and dumber.

This is the result of web devs targeting smart phones. They've made the web suck bending over backwards for the limited abilities of phones to act as computers in both the UI and network sense.

excessive(10000) 6 days ago [-]

I was never a Flash developer, but I certainly admired it in its prime. The early versions could be installed simply by putting a 47kb DLL in the plugins directory, and in a time well before the HTML canvas or JavaScript with good performance, this brought dead web pages to life in a way that was very portable across browsers and operating systems. It was much better (lighter, simpler, quicker to load) than Java applets from the mid 90s.

It's easy to imagine the various security vulnerabilities could have been fixed by a more responsible/responsive company, and the abuse by advertisement companies should've been solvable with an ad blocker.

JavaScript, CSS, and HTML have come a long way in capability, but the mashup is much more distasteful to me. Flash was elegant despite its frequent misuse.

evrydayhustling(10000) 6 days ago [-]

Something that's not easy to appreciate about Flash if you weren't there is that it is the first tool that really put multimedia interactivity within reach of non-devs -- and, almost gratuitously, made it dirt simple to distribute your creation at the same time.

A tech-savvy artist, who would previously have needed a handful of devs and a publishing company to get a fun toy to a few hundred users, could suddenly build and send something around the world in a week. No wonder it uncovered so many creative projects waiting to happen!!

vobios(10000) 6 days ago [-]

> It was much better (lighter, simpler, quicker to load) than Java applets from the mid 90s.

People often forget about those.

jugg1es(10000) 6 days ago [-]

I couldn't possibly agree more with this article. It is still possible to do the kinds of things that Flash did now-a-days with HTML5 canvas, but it requires a lot more programming abilities than Flash required.

Websites for design firms were literal masterpieces. Now, those same firms have beautiful, yet predictable, websites.

The user experience was entirely unique from site to site. No side vertical stack of buttons, no top nav bar. Designers back then encouraged exploration. You just don't see that anymore.

Overall, websites are way better now. But we have certainly lost something as the entire web has become commoditized and homogenized. No one is willing to take risks anymore. The late 90s and 2000's was a different kind of internet than it is now, for sure.

bredren(4138) 6 days ago [-]

But there is far more competition than there was before. People have devices allowing them shorter attention span to get to value than before.

This change is not only because of the loss of flash.

kaolti(4177) 6 days ago [-]

Lots of good points made, I'll add mine.

Flash got me started in web design as a kid and put me on a path to earning a good living. Lots of things changed since then but I want to hear your thoughts on one specifically.

It used to be cool to try things and put something together that in the grand scheme of things is kind of bad. I get the feeling that's not the case anymore.

Seems like there's so much pressure to sell the image of you being an expert, there is no room to try wild things and see what happens - which is the basis for learning and development btw. What we get then is everyone copying the 'expert' patterns, but hardly anyone doing anything original or truly creative, because it feels risky.

This is probably true for a lot of other creative endeavours that have big $ behind them - the movie industry comes to mind. Sure, there are more movies being made now than ever, but it sure as hell doesn't feel like we're in the most creative era and you know it's got nothing to do with the tools.

Someone please tell me otherwise.

redisman(10000) 6 days ago [-]

I feel bad for people coming of age now in the world of programming. There's a lot of pressure to follow the best practices and they sure as heck don't involve an easy way to draw a Z colored pixel at coordinate (X, Y). Or to coax a programmed beep out of the speaker. Which is basically how I learned 50% of programming.

You don't really get that visceral moment of magic reaction from web technologies or super polished game engines which is where that happens now.

zanny(10000) 6 days ago [-]

I think it has to do with the commoditization of image. In the early Internet we didn't have centralized social media so going to a new place and experimenting didn't have ramifications across your social circles via platforms attached to your real name. You could make 'bad' stick figure cartoons by night while working a 'respectable' career by day and nobody would know who you are either way.

I feel terrible for kids today - they grow up into social media not knowing what it is, and by the time they can comprehend it their creativity has been stifled for years by the implied pressures of conformity and a cultural disapproval of failure.

But theres some nostalgia in that too - its not like every 80s and 90s kid was sitting in a computer lab inventing the next tower defense game. It has always been a tiny minority of kids that get creative from a young age, and if anything the continued availability and quantity of entertainment can serve to keep kids occupied with consumption rather than creation more today than ever before.

quickthrower2(1314) 6 days ago [-]

The bad thing about being 'creative' on the web is it not being obvious how to use your site. I remember back in the day a lot of website using Flash only that were like this.

code_duck(3952) 6 days ago [-]

Flash developers were responsible for a lot of heinous interface experiences that informed every rule of good UI and resource efficiency.

neop1x(10000) 5 days ago [-]

What prevents someone to replicate the Flash experience with modern web tech like JS, HTML5, CSS or Canvas/WebGL? I mean something with similar editor functionality, producing result drawable in modern web browsers. EDIT: There actually is Adobe Animate so what prevents people from being as creative as in the Flash era?! https://www.adobe.com/products/animate.html

whatever_dude(3813) 5 days ago [-]

My theory is that the web has overall matured. A lot of what Flash did is now possible and even better/faster/easier, but the reason for those aren't there. In many ways Flash websites featured novel navigation and interaction experiences, the likes of people never had seen before, but that's a gimmick that gets old fast. No insurance company wants or needs an animated 3d intro in their website; they want and need good SEO, and information that's easy to reach. If they want to impress with something visual, they'll go to YouTube or Instagram.

rezmason(10000) 6 days ago [-]

I might have missed a few names, but I'd like to contribute a list of folks and sites from the old Flash community who seemed like rock stars (in order of how much I idolized them).

Once I typed it up, I realized it was all dudes, so let me first link to https://flashgoddess.com/spotlight/ , who shone a spotlight on the women who worked in the same space. (Shoutout to Lisa Larson-Kelley, fellow alumnus of FlashCodersNY, and Stacey Mulcahy, whose hilarious AS2-to-AS3 migration talk was fucking awesome!)

If you wish to view any of these links, I recommend pasting them into web.archive.org and setting the Wayback to the mid-to-late 2000's.

mrdoob.com (Mr. doob, later created three.js) audiotool.com, lab.andre-michelle.com (Andre Michelle) wefail.com (Jordan Stone, Martin Hughes) gskinner.com (Grant Skinner, later created EaselJS, etc) quasimondo.com (Mario Klingemann) bytearray.org (Thibault Imbert) bit-101.com (Keith Peters) vectorpark.com (Patrick Smith) hoogerbrugge.com (Han Hoogerbrugge) levitated.net (Jared Tarbell) blog.joa-ebert.com (Joa Ebert (also audiotool)) orisinal.com (Ferry Halim) hospital.apoka.com (Edouard Artus) kirupa.com (Kirupa Chinnathambi) mosessupposes.com (Moses Gunesch) gotoandlearn.com, theflashblog.com (Lee Brimelow)

Most of these folks also spoke at tech events, sharing techniques and hard-learned lessons. Like that time when Wefail gave a talk that took a memorably heavy shit on Jakob Nielsen. Hell yes.

Also worth noting the HBO Voyeur promotional site made by Big Spaceship: archive.bigspaceship.com/hbovoyeur

The Flash community was creative and groundbreaking as hell. Most of the folks who criticized it for being proprietary and insecure are lifelong users of proprietary, insecure platforms; the worst aspects of the Flash-equipped web linger still, namely ads, inaccessibility and site bloat; and let's face it, as capable as the modern web client stack is, it's lost so much ground to the walled garden platforms, that Steve Jobs's 2010 agenda can no longer be seen as anything but duplicitous.

Godspeed, FutureSplash! You were to good for this web.

jjirsa(4220) 4 days ago [-]

All of that and no 2Advanced

l_t(10000) 6 days ago [-]

I had a formative experience with Flash as a middle schooler. I loved Albino Blacksheep, Newgrounds, etc. I thought the videos were hilarious and the stick-figure-style animation was approachable. So I acquired Flash, and I was blown away by how easy it was to create these silly animations. Automatic tweening was a miracle to me.

Since then I've done a lot of video editing with different tools, but I still think back to Macromedia Flash and how I, a child with no ability to code and no knowledge of HTML or Web tech, was able to make my imagination come to life on the screen.

I believe it's that powerful experience for novices that we're missing. I'm not sure how we should get it back.

(edit: phrasing)

sircastor(10000) 6 days ago [-]

This reminds me of my discovery and learning Hypercard probably 15 years before you learned Flash. To me it was the same kind of creative spark. I was encouraged by Cosmic Osmo, and later Myst and the like. I never did learn to program with it.

I don't know where the modern equivalent is. Maybe they only pop up every couple of decades...

hunta2097(10000) 6 days ago [-]

Disclaimer: I'm an old git.

I think you have deftly described a bigger problem, as the whole industry gets more and more sophisticated and complex. The barrier for entry is so high now that I think it stifles young people's interest too quickly.

The great thing about Flash was the developer tool. Part animation studio, part simple programming IDE. It was a great balance.

Maybe there is a place in the market for framework to implement a flash-a-like in a HTML5 Canvas?

eXpl0it3r(4221) 5 days ago [-]

> I believe it's that powerful experience for novices that we're missing

I think we have more 'powerful experiences' for novices than ever before, which is part of the whole attention problem. In the early 2000, we didn't have YouTube, Netflix, WhatsApp, TikTok, etc. where tons of teens spend many, many, many hours in nowadays. Those who got access to Flash found a sink for their free time and after just a few hours will have figured out the basic elements. Try to get kids to focus for just these initial few hours on their own, without them getting distracted on some other platform or immediately trying to find a YouTube tutorial, instead of just trying things and actually learning the tool, rather than copying whatever someone else is doing in a video.

mntmoss(10000) 6 days ago [-]

I think the essential thing about this era is a gradual cultural rediscovery of ownership. We've just been through a very lengthy race to the bottom for all sorts of information - pretty much anything ephemeral and disposable is free or extremely cheap, and then heavily locked down to protect property rights. And it's built a kind of event horizon to culture where all the things pressed up closest to the digital universe just disappear into a little footnote on a wiki page saying 'yes, this happened'. Memes, blog posts, videos, games, etc.

There is a game product called 'Fortnite', and it's just had a huge in-game event, so it's clearly here, alive and well, but you'll never again be able to experience Fortnite as it was 24 hours ago - ever.

And yet future culture is, as Alan Kay puts it, 'the past and the present'. It's our reaction to that hole, where nothing really builds on anything else, that, in turn, is motivating interest in products with longer time horizons, longer stories and histories to them.

An obvious metaphor for this is video games vs pinball:

* Fundamentally digital vs fundamentally analog

* Mostly design & marketing vs mostly manufacturing

* Trivially cloned vs scarce, unique

* Black-box artifact vs maintainable assembly

* Perpetually caught in the breathless hype cycle of tech, vs increasingly existing outside of that cycle

Pinball's days as part of the traditional amusement business ended with the 1990's, but it's found a resurgence of interest in the home market as a kind of collectable furniture - something to put in a rec room or a basement arcade, that retains decent trade value if maintained. A whole array of small manufacturers have appeared this decade to serve that market. It's much easier to understand a collector's market for it being sustained 30 years out, versus video game collecting, in which any product with a modicum of popularity will have had its primary content either already preserved through piracy(if emulated), or else impossible to reproduce(if a service). It's a much stronger version of interest in vinyl records or dead-tree books taking precedent over streaming music and e-books.

Because digital media has so little physical value, it is beholden to be entirely marketing driven, front-to-back, and to treat you as either a product marketer or as the product, and sometimes both. The true form of the medium remains always hidden behind the UI. Even your personal work, done on systems you wholly control, just disappears into a collection of files, where it is easily forgotten.

And in that sense I think we are not really asking, 'Where is Flash? Where is Hypercard? Where is BASIC?' - because in different eras each of those tools did the kinds of things we wanted and expected from a beginner's tool - so much as we are asking, 'Where is the actual medium? Where can I do work and preserve the original source material? Where can I send a kid to learn to play with software and not have it all break six months later, rendering the learning useless? How can I curate software when nobody can make any promises?' Tech continues its warfare for a platform monopoly, and so on this front we keep starting from zero, over and over. It's not hugely different from the space we've arrived at in professional software development, where dependency hell and code rot is an ever-increasing concern for all codebases.

legym(10000) 6 days ago [-]

I didn't know anyone else knew albino black sheep. Browing miniclip and playing flash games was the peak of middle school for me

smcl(3924) 6 days ago [-]

Carefully worded, I suspect we both "acquired" it through similar means :-)

Mathnerd314(3648) 6 days ago [-]

Everything young people do now outside of schooling and dedicated hobbies seems to be mobile-based, so Tiktok is probably the equivalent to Newgrounds these days.

This 'novices' thing seems to me to be overblown though; Alan Kay has been beating the 'programming for novices' drum for decades with Smalltalk and basically nothing happened. I would rather have professional-quality tools with decent Youtube tutorials, like Blender, Unity, Python, etc. have been developing.

djsumdog(1073) 6 days ago [-]

I know a lot of early Adult Swing shows were made with Flash (I think Aqua Teen?). What are studios using today? Is there anything equivalent today that matches the power of Flash for young animators?

CryoLogic(10000) 6 days ago [-]

https://www.anim8.io might spark your imagination again.

Lots of animators are still making indy animation, it's just been drowned out on the major platforms due to how time consuming it is to create and it's basically not possible to monetize well outside of Anim8 now (YouTube algorithm favors frequent uploads and high video duration - not compatible with animation).

JohnBooty(10000) 6 days ago [-]

We're.... misattributing some things here, I think.

Flash was a bundle of good things and bad things, as we all more or less agree. Good for creativity, bad for usability and accessibility and security, etc.

You could, obviously, do so many things in Flash that were impossible with web standards.

But, why was there such a gap between what you could do in Flash and what you could do with web standards? Why was Flash even necessary?

This was due in large part to how Microsoft acquired a stranglehold on the web right as it was beginning to really take off -- IE4, IE5, and IE6 were the defacto standards, and Microsoft used them to quash innovation on the web as they (successfully) tried to hold on to their desktop software business for another decade or so.

Perhaps we haven't lived through the worst possible timeline, but there was certainly a better outcome possible - where Microsoft didn't stifle innovation on the web for close to a decade.

(And now, of course, we're approaching another dark age as Google heads toward IE6-like market share....)

est(2953) 6 days ago [-]

> Microsoft used them to quash innovation on the web

Microsoft didn't 'quash' any innovation, the ActiveX framework allows more innovation than any other browser on the market, Flash was invented as an ActiveX plugin, so does AJAX and it started the whole Web 2.0 era. IE had so much incredible feature that it makes modern looks like a dull corporate utility. We had VRML and vector graphics. We can playback videos with IMG tags. Heck we can even embed the mighty DirectX on Web. Chrome basically re-invented everything we already had with dHTML in the 90s.

Microsoft didn't took down ad blockers by crippling the API. In fact there's innovative abuse problem with IE, think all those toolbars!

redisman(10000) 6 days ago [-]

Flash had a VM that truly was 'write once, run on anything' and a great (software) pixel and vector renderer. Cross-browser compatibility with HTML standards was completely bonkers and HTML had none of the features Flash Games and Animations needed. Try to move or rotate a picture at 30 fps with old HTML.

someexgamedev(10000) 6 days ago [-]

I made games in flash for over a decade and miss the ease of dropping an experimental game idea onto the web where everyone could enjoy it. If you had a free weekend, you could release a game to 99% of the internet.

JavaScript game libs simply aren't as featureful, even the ones promising the scenegraph API. They don't work as well cross platform (by the end of that decade I had one codebase which could deploy to web, iOS, and Android); and that's where the audience has headed.

I hope this book spends some time on flash's contribution to video game design as well as web design.

zanny(10000) 6 days ago [-]

You can make a game today in Godot / Unity, export to HTML5, and drop it on itch.io to play in browser in minutes.

nojvek(3909) 6 days ago [-]

I only got into CS because of flash. It was simple to get started. Draw some shapes, add color and start tweening. Point and click got you very interesting animations. Deploying was a swf that worked in a ubiquitous manner. Then I learnt a bit of Actionscript on my own before even learning any sort of formal programming. It was a ton of fun to add event listeners and do basic interaction. I learnt what object oriented programming was. The sprites were the objects and I could program them and create dynamic instances.

I would spend entire holidays messing around in flash obsessively. Also then got into mxml and Flex.

Yeah I then learnt C, Java, JavaScript, Python etc but it wasn't really as fun as Flash. Flash had progressive unwrapping of complexity. You could create impressive games in Flash quite easily. I learnt a ton by decompiling existing games from miniclip.com (love that site) and changing things to see what effect they had. Kirupa.com was a fantastic resource too.

Oh man. I could talk for hours about Flash and the self learning experiences.

dluan(2107) 6 days ago [-]

I got an A in 7th grade biology because I animated the electron transport chain in a little flash movie over a holiday break. I burned the swf onto a CD and gave it to my biology teacher who then showed it on his computer during parent teacher night.

ArtixKrieger(10000) 6 days ago [-]

Flash was my favorite game design tool of all time, and it changed my life.

In 2002 I created the original AdventureQuest. Because the game was created in Flash (Actionscript 1) it was instantly cross platform. It could dynamically load any number of animated monsters, weapons, cut-scenes, towns, and backgrounds. Most importantly, those loaded items could have their own code which let them do anything-- giving them new AI, features, or letting there be a game within the game. Also, it was easy to communicate with the database through posts to webpages. Because of this, your character was persistent and the game was able to update with new things every week. (In fact, the game has been updating for 17 years and is STILL updated with new things every single week.)

As Flash evolved, we built more games: DragonFable, MechQuest, and a slew of minigames. Actionscript 2 has a special place in my heart. It was my favorite language. Not being a 'real programmer', it was really easy to use. Making things was ultra fast. Actionscript 3 was a challenge for me. our 'real programmers' loved it. But, it was so hard for me to actually make anything happen with it. When it became the new standard and we started developing with it, I stopped programming and focused on the content assembly side.

We have built several Flash Massively Multiplayer Video games: AdventureQuest Worlds, EpicDuel, Oversoul, and HeroSmash. They all benefited from the same features that the very first game did, allowing for weekly (or more often) releases of new content. The games literally grew under your feet as you played them.

Creating art, animation, and interactivity in Flash was the best. Flash 3.0 was probably my favorite version. Flash CS6 being a close second.

We are currently in the process of converting our most popular game, AdventureQuest Worlds to Unity. Switching from vector to raster is a nightmare-- ask anyone who has gone through converting 10,000 items with no universal conformity.

Flash was just so fast and easy. We still use Adobe Animate (which is what they call the Flash tool now) for creating 2D vector art. But I really do miss Flash as a 'one stop shop' tool to create... anything.

Battle on! Artix

smush(4177) 5 days ago [-]

Man - DragonFable & MechQuest were great fun. I still think of Sir Valence and the various knights with punny names!

I hope between godot, procreate 5, and open source flash runners, long term Flash will be properly succeeded; for now it appears we have regressed unless you pay Adobe for a CC subscription.

Riverheart(10000) 4 days ago [-]

Thanks for the great games! Remember playing AQ Original many moons ago. Hard to believe how much time has passed.

Edit: Out of curiosity, did you ever evaluate OpenFL/Haxe as a migration path?

Fr0styMatt88(2642) 6 days ago [-]

Does anyone here remember an old DOS program called Cartooners [1]?

It let you do animated shorts with predefined characters, different backgrounds, music, speech bubbles, etc. I used to love playing around with that as a teenager.

I found the workflow ingenious: You placed a character on the screen, selected what action you wanted it to perform (like 'Walk'), I think you selected how you wanted it to move, then you held down the REC button and watched the character act as the animation was being 'recorded'. To do something new, you just stopped recording, set the character up for the next action, move, etc, then do the whole REC thing again. Repeat this as needed.

Adding another character? Scrub back to where you wanted that character to appear and repeat the above process.

New background? Scrub to where you wanted the new background to appear, set the background and then 'record' over the old one.

It was similar for everything. Put a speech bubble down where you wanted it, hold down 'REC' for as long as you wanted it on the screen, then just remove it and record further as needed.

It was so damn intuitive. Never seen anything quite like it since and I'd love to know if there's a clone of it or something out there that I've missed!

[1] https://archive.org/details/msdos_Cartooners_1989

smush(4177) 5 days ago [-]

The scrubbing features remind me of Microsoft 3D movie maker which was a Win 95-era product that had the same easy approach, but with 3d characters.

robflaherty(4186) 6 days ago [-]

Today's internet is far more creative than the internet of the Flash era. Creativity has moved on to more consequential things than graphic design.

whywhywhywhy(10000) 5 days ago [-]

Flash wasn't 'graphic design' it was a unique cross over of design and code in harmony and at the time most graphic designers I encountered used to say 'web designers are not real designers' they only considered print work important.

Funny how things change.

MisterTea(10000) 6 days ago [-]

I was going to add how I'm surprised that no one is trying to emulate it in js or wasm but I appear to be wrong. someone on Newgrounds is doing just that:

https://www.newgrounds.com/bbs/topic/1444275

kbumsik(1575) 6 days ago [-]

There was an effort to run Flash using JS by Mozilla, called Shumway, now in Mozilla Graveyard.

https://github.com/mozilla/shumway

metafunctor(3078) 6 days ago [-]

Flash sites were creative, in that they were occasionally built by people with amazing talent and great skill.

Most of the time, though, Flash was the multimedia CD-ROM of the web. Flash was an extension shoehorned in, didn't feel like the rest of the web, and didn't deeply integrate with the browsing experience.

Flash content was an alien rectangle, a wormhole into a strange dimension, where a mouse click would often not behave anything like a mouse click usually would.

Flash pretended to be on the web, but Flash was not of the web. That's why it's now gone.

code_duck(3952) 6 days ago [-]

I couldn't stand the lack of bookmarking, history, back and forward buttons, the use of sounds and music, and lack of ability to view source.

excessive(10000) 6 days ago [-]

Heh, this is really a matter of perspective. If you think the rest of the web was wonderful, then Flash was a turd floating in a swimming pool. If you think the web is a poorly designed shoddy mess, then you might think Flash was a raft keeping you afloat in the sewer.

HTML and JavaScript are so sloppy and incrementally improved by accretion rather than taste, I lean towards the sewer point of view.

jayd16(10000) 6 days ago [-]

People are looking back with rose tinted glasses, I think. This was an era where you could build rich media with absolute canvas values and it would look great and it was easy for an artist to take that canvas and do things.

The iPhone killed flash but not just in the obvious way. These days everything needs to be reactive and run on well on mobile hardware. Even if iOS supported flash it would need to be a different beast. Different screen sizes, an explosion of aspect rations, heavier security requirements, accessibility, touch controls and more led to the downfall.

bbayer(3675) 5 days ago [-]

Adobe Air could produce executables iOS and many other operating systems and environments. I have seen many successful mobile games done with Air. Flash Player is a virtual machine and swf is compiled bytecode. It could be ported to any hardware and OS. Death of Flash is not technical, it is political. SWF format and Flash Player is proprietary. They chose to kill it instead of supporting someone else's technology.

phaseshifter(10000) 6 days ago [-]

I remember sideloading the Flash Player APK on my Android phone around 2013 (they stopped official support in 4.1) and most of the Flash content was effectively unusable even then.

Pecking on a tiny video player with your finger to change volume, quality, or playback position was a terrible experience and the battery drain wasn't great either. Watching videos outside of YouTube was a terrible time for phone users before sites transitioned to HTML5.

phs318u(10000) 6 days ago [-]

Or Flash's heyday coincided with the explosion of the web from tech-circles to widespread public use, and critically, before the big walled gardens (FB etc), became the hub for most publication on the internet.

Mirioron(10000) 6 days ago [-]

FB and other sites like that themselves were popular places for flash games as well.

taneem(4210) 6 days ago [-]

During 1999-2000, I helped hundreds of people learn how to use Flash. I was, looking back now, probably one of the top experts on Flash 4 at the time in the world. The twist - I was a 15 year old living in a tiny African country called Lesotho.

Lesotho is pretty isolated from the world. Nobody even knows it exists. Living there, Silicon Valley might as well be on Mars.

However, we used to get issues of Wired Magazine from South Africa, and these came with shareware CDs. These CDs included 30-day trial editions of Macromedia Flash.

Flash was amazing at the time. Being able to create interactive animations blew my mind. I learned Flash 4 completely inside and out. I knew every single feature, every single quirk.

Of course living in Lesotho, there was nothing I could really do with all this. Most people around me didn't even know how to use computers. Flash was several layers of abstraction away from that.

So I used to spend all my time on Yahoo Chat's Web Design chat rooms. Mainly hanging out with nerds in the US. We used to have countless people drop by in the rooms every day asking questions about Flash. Mainly people working for web design agencies in the US. I was the resident Flash expert. Flash questions always were referred to me.

In the 2000s Flash rightly got a lot of flak. I'm not sad it's gone. But it was really something special, especially in the late 90s.

mikorym(4222) 5 days ago [-]

Was there a 'computing hotspot' where you were at the time? I think with all the school issues (and university issues) in South Africa it would be cool to have sort of (physical) computing libraries with Raspberry Pis or something like that–a sort of enabling environment where independent kids would at least have the resources necessary to build an eventual career for themselves (perhaps unwittingly at first).

countryqt30(3890) 5 days ago [-]

That's exactly the spirit! The internet makes it so easy for 'poorer' countries to level up damn quickly! Spread the word there! <3

krtkush(4024) 5 days ago [-]

Flash is the reason why I'm a programmer today. It was Flash which kindled my interest in computers and made sure I spent hours playing around with it and then other aspects of computers.

That era of discovery and expression was something special.

lawlessone(10000) 5 days ago [-]

you should write a book!

husamia(10000) 4 days ago [-]

Where are all the creatives flocking to nowdays? HTML5?

dpau(10000) 6 days ago [-]

thanks for sharing this wonderful story. it would make for a lovely short film.

cryptozeus(3590) 5 days ago [-]

Great story, all that isolation from Silicon Valley probably helped you focus on your craft and didn't provide distractions.

daleharvey(948) 5 days ago [-]

Curious and feel free to ignore, what was your screenname on the yahoo web design chat rooms? I also hung around there a lot around that time using one of those embarrassing screennames made by teens and their favourite songs (renegade_master)

I was one of the people constantly wishing flash would die. Although I am happy the open web won that battle, I was always hoping and am sad that nothing based on open standards came to replace it.

cmarschner(3952) 6 days ago [-]

So what are you doing these days?

srachamim(10000) 1 day ago [-]

I spent 3 nights at Lushoto last week. Nice place!

Aardwolf(10000) 5 days ago [-]

There is still a huge amount of amazing content in flash, tons of games and movies on sites like newgrounds.

The older ones are fully self contained games, some of which are amazing, in a single swf file that works in all major operating systems (The newer ones started side loading data from servers though.)

I hope there will always remain ways to play those.

on_and_off(10000) 6 days ago [-]

It was so special.

It was just the right mix of relative simplicity and power.

No wonder that all the creatives flocked to flash.

brylie(4169) 5 days ago [-]

Is there any open source tool/community trying to create a simple HTML5 animation studio similar to Flash? That would really open up doors for widespread creativity and the adoption of web standards.

ghostbrainalpha(10000) 5 days ago [-]

I've been asking the same question for years.

Greensock is the only thing that feels remotely like the good parts of Flash to me.

But it does less, and requires learning some Javascript.

https://greensock.com/

jrwoodruff(2449) 6 days ago [-]

In my personal opinion, Flash was ruined by Adobe when they tried to 'fix' it (improve it?) with ActionScript 3, which followed the ECMA script standard. AS3 changed everything about how flash worked, and if you weren't a software engineer, the learning curve was steep. I actually went and took some basic programming classes (literally in visual basic lol) and learned how to write proper code.

Not a bad thing, but once I learned some of those basics, I no longer had fun toying around with animations and creating, and instead spent my time writing classes and functions. It became a lot harder to 'hack' things together, and, for me, less fun.

And besides that, plenty of real coding tools already existed, and HTML/CSS/JS was improving really quickly as well. So I ended up learning those instead.

All of that was actually great for my design career, but Flash just didn't have a place anymore. And I'm still sad about that.

acdha(3745) 6 days ago [-]

> Not a bad thing, but once I learned some of those basics, I no longer had fun toying around with animations and creating, and instead spent my time writing classes and functions. It became a lot harder to 'hack' things together, and, for me, less fun.

I don't disagree with that but I think it needs to be in the larger context of Adobe being bad at almost everything related to platform support. Their APIs were not well-considered; consistency happened at other companies; the tools were expensive, poorly designed, buggy, and basically unsupported ("The Flash IDE crashes when I do X". 10 months pass. "Pay $800 for the upgrade and let us know if that fixes it "), and performance/battery impacts always lagged behind the native alternatives — all of which sucked the fun out of the creative process as you had to take time away for pure drudgery working around Adobe's slacking.

If at any time they'd invested 10% more on quality and basic software engineering they wouldn't have had all of their customers looking for alternatives. As we saw, that can be more profitable for a short period but catches up hard.

jugg1es(10000) 6 days ago [-]

AS3 was literally the best thing that ever happened to Flash. The problems with Flash had nothing to do with the actual tech (aside from security) and everything to do with product management. It's just a shame that AS3 came so late in the Flash lifecycle.

whatever_dude(3813) 5 days ago [-]

AS3 was indeed a bad investiment given the direction ECMAScript ultimately went, but I do not believe for one second that it contributed to Flash's loss of popularity over time. Learning it after AS2 was easy and it was a much better language/platform to use, not to mention faster.

If anything I applaud Adobe by starting fresh instead of, say, accumulating warts in the language/VM the way Java did.

sirspacey(4221) 6 days ago [-]

This one got me in the feels man. Same.

woutr_be(10000) 6 days ago [-]

I actually quite liked the changes in AS3, and it being more programatic, I think it paved the way for what I do now.

After AS3, I remember switching to Flex, and basically using it like any current language. Although I absolutely hated the MXML syntax. I was pretty sad at the time, since AS3 was my main language, but looking back at it now, I think it was for the better.

smt88(4211) 6 days ago [-]

It is very, very silly to suggest the Flash era was more creative than the YouTube/indie-game era.

In general, the Web is more commercialized now, so creative content doesn't always rise to the top, and there are incentives to be a low-quality content factory.

But there are thousands or millions of online videos (some, like Flash, are animated) and indie games on major platforms that are just as creative as the most experimental, interesting stuff on Newgrounds.

buboard(3489) 5 days ago [-]

youtube and indie games literally started as flash

mattkevan(3138) 6 days ago [-]

As far as I'm concerned, the whole field of web design peaked in 1999 when this website was made:

http://superior-web-solutions.com/

Enable Flash, turn up the sound and whatever you do, do NOT skip the intro.

ktpsns(4103) 6 days ago [-]

Wow, clicking this link I just realized my Firefox/Linux installation still has the Flash player plugin. So I could view this gem of 90s Flash-only website.

It shows all the problem the web had back then: It took me four clicks to see actual text about what the authors are trying to say/sell. Everything before was crying 'wow look at the amazing technology we can build'.

For many web developers at that time (including me), this was a reason why we hated Flash (and intro pages, in general). At some level, the web got more professional after the Flash times, when content was written in HTML again, and SEO was a thing. Unfortunately, today, Javascript takes over the role of Flash and at many sites, without JS there is no text appearing/embedded in HTML only.

xellisx(10000) 6 days ago [-]
nevster(4068) 6 days ago [-]

The only limit is... yourself!

notatoad(10000) 6 days ago [-]

Flash was a perfect example of 'this is why we can't have nice things'. As a platform for web mini-games or self-contained interactive animations, it remains unparalleled.

As a tool for building whole websites meant to convey information, it was always shit. If lazy web developers hadn't collectively abused flash to make annoying websites and dumb ads, perhaps we could still have fun flash games on the web today.

valtism(4201) 6 days ago [-]

It was not discontinued because people were making annoying websites with it, it was discontinued because it was a nightmare for security.

djsumdog(1073) 6 days ago [-]

Home Star Runner!!!

Go to the old homestarrunner site, grab a flash file and play it (in the browser if you still have support, or using a rendering library) on a UHD screen. Yes, you can render that ancient 2000s era video .. in 4k. Flash is dead as a door nail though, and Homestar will now only be at the resolution of when they rendered it for YouTube.

nevster(4068) 6 days ago [-]

Trogdor!!!!!! I introduced my kids to Strong Bad when they were young but just about zero of their friends know about it. :(

hoelle(10000) 6 days ago [-]

The premise of this article is just weirdly wrong and out of touch. There are monstrously huge creative spaces on the modern internet, powered by better and better tools. YouTube. Unity. Scratch. Roblox. Twitch. Itch.io. Blender. Minecraft. And so on.

It is noteworthy that professional/corporate creators haven't followed the hobbyists and kids into some of these spaces to represent their brands as much as they did into the land of Web+Flash. I don't see how that matters when measuring the 'Internet's Most creative Era.'

Consumers are more savvy, and brands are have gained some wisdom to not step on territory that makes them appear inauthentic and creepy. Second Life was an interesting experiment, but I'm happy that big brands haven't made ads in Roblox yet.

chii(3664) 6 days ago [-]

youtube is not a tool - it's a platform, and increasingly more hostile towards content creators (and increasingly biased towards commercial content or clickbait content). Twitch is same as youtube basically, but slightly more content creator friendly.

Unity, scratch, blender, these are all programming related. They are not beginner friendly at all.

itch.io isn't a tool either, but a website for distributing. The content you find on there are good, but most are not made with something easy like the flash animations would've been. Game maker comes close, but it isn't well suited for animation.

There isn't, and hasn't been, a replacement for flash animation suite, which is such a great, easy to use, integrated tool of drawing, animation and timing with a relatively simple scripting model. Even amateurs are able to make interesting animations with it. I bet you can't say the same for any of the tools named above.

buboard(3489) 5 days ago [-]

It sounds like you don't know what flash was.

omegote(4129) 6 days ago [-]

Wat. Those you're mentioning have nothing to do with the creativity that Flash provided. How are you comparing Flash with Blender?

pippy(4129) 6 days ago [-]

Removing Flash from the web was done with good intentions. It was a propitiatory system plagued with security holes. Instead of replacing it with similarly capable open standards, we've got a propitiatory Encrypted Media Extensions (EME), flaky HTML5 calls that differ between browsers, and webassembly.

It's a shame that HTML5 never really reached the ability of flash. And it's also a shame that Adobe didn't simply release the swf spec.

wavefunction(10000) 6 days ago [-]

I feel like you can do pretty much everything with HTML5 and JavaScript that you could with Flash and ActionScript. The IDE for Flash had some nice UI abstractions but there's no reason that couldn't happen in a web IDE in 2019. Distributing your website like a flash file/app as a binary is one missing feature I suppose?

TomGullen(3867) 6 days ago [-]

I wasn't a big Flash user back in the day but I think our product Construct 3 (https://www.construct.net) can do just about everything Flash did, especially with the addition of the Timeline editor and scripting we've just added. I'm a little hesitant to overtly market Construct as the "next flash" as that I think would receive a mixed reception depending on your experiences with it!

Tom Fulp from Newgrounds has nice things to say about Construct, and we're getting very positive numbers from other portals and platforms (some very big) that a lot of volume of games and good quality ones at that are being published with them using Construct 3.

I'd like to hear from more experienced Flash users if there's anything Construct 3 can't do that Flash could as it would likely be an important feature we'd be keen to add!

jbay808(10000) 6 days ago [-]

> propitiatory

You keep using that word. I do not think it means what you think it means.

https://dictionary.cambridge.org/dictionary/english/propitia...

AnIdiotOnTheNet(3914) 6 days ago [-]

To the modern dev end users are cattle, they aren't supposed to make things!

zamadatix(10000) 6 days ago [-]

Is there really a way to do DRM that doesn't involve proprietary pieces somewhere along the way?

I think the modern browser is technologically superior to what and how Flash did things (with the exception of native sockets, which is probably for the better). It's really everyone using Adobe's content creation tools (which were well integrated/designed) that made things so low friction.

It took a long time for the browser technologies to catch up but I wonder if in the end it integrates as one system better than bolting on an open flash spec would have been.

mesh(10000) 6 days ago [-]

You can find the SWF spec at: https://www.adobe.com/devnet/swf.html

Macromedia / Adobe released it quite a bit ago (while Flash was still popular).

nojvek(3909) 6 days ago [-]

Adobe did open the swf spec in later versions. Swf was compressed bytecode. Still a lot more efficient than the way we send js.

https://www.adobe.com/content/dam/acom/en/devnet/pdf/swf-fil...

There were a number of tools that compiled to swf.

It sucks that Adobe never opened the Flash Player. That was it's death nail. They had little to gain from it. It was free. The money maker was Flash Studio and other Macromedia tools.

If Adobe had opened Flash and had a sane security Sandbox it would have survived iOS. Adobe was like the old Microsoft though. They just didn't get the power of Open Source.

ehsankia(10000) 6 days ago [-]

This comment is pretty misleading. The parts you complain about have little to do with what made Flash a creative sandbox. The parts that do matter have long been replaced with Canvas/WebGL. But the web as a whole has changed. Just like we don't have as many fun quirky geocities-like websites. The issue isn't the lack of Flash, it's that the web as a whole has grown up and changed.

djsumdog(1073) 6 days ago [-]

Flash died because of Apple, and before Adboe bought Macromedia, it was an incredibly small runtime; way better than Java at very specific forms of 3D rendering. Old Flash can be upscaled to 4k. A youtube video is pre-rendered and stuck at that resolution forever.

The_rationalist(10000) 6 days ago [-]

What can flash do that modern browsers can't?

ironfootnz(10000) 6 days ago [-]

I'm part of that forgotten industry, since Flash 3. I can tell you that Steve Job letter hurt a lot the industry but I don't believe destroyed the creativity. We're more focused on clicks than interactions in the industry today.

scarface74(3825) 6 days ago [-]

Flash barely ran on Android - Adobe could never get it working correctly on mobile.





Historical Discussions: Google chief: I'd disclose smart speakers before guests enter my home (October 17, 2019: 894 points)
Google SVP Hardware: I'd disclose smart speakers before guests enter my home (October 15, 2019: 47 points)

(894) Google chief: I'd disclose smart speakers before guests enter my home

894 points 2 days ago by vezycash in 272nd position

www.bbc.com | Estimated reading time – 8 minutes | comments | anchor

Image copyright Getty Images
Image caption Google's range of camera and microphone-fitted devices include the Nest Hub Max

It's an admission that appears to have caught Google's devices chief by surprise.

After being challenged as to whether homeowners should tell guests smart devices - such as a Google Nest speaker or Amazon Echo display - are in use before they enter the building, he concludes that the answer is indeed yes.

'Gosh, I haven't thought about this before in quite this way,' Rick Osterloh begins.

'It's quite important for all these technologies to think about all users... we have to consider all stakeholders that might be in proximity.'

And then he commits.

'Does the owner of a home need to disclose to a guest? I would and do when someone enters into my home, and it's probably something that the products themselves should try to indicate.'

To be fair to Google, it hasn't completely ignored matters of 21st Century privacy etiquette until now.

As Mr Osterloh points out, its Nest cameras shine an LED light when they are in record mode, which cannot be overridden.

But the idea of having to run around a home unplugging or at least restricting the capabilities of all its voice- and camera-equipped kit if a visitor objects is quite the ask.

The concession came at the end of one-on-one interview given to BBC News to mark the launch of Google's Pixel 4 smartphones, a new Nest smart speaker and other products.

Image copyright Getty Images
Image caption Mr Osterloh first worked at Google when he headed its Motorola division, until it was sold in 2014 - he returned to the technology giant in 2016

The conversation below has been edited for clarity and brevity.

The new Pixel phones have two cameras on their rear for the first time. But is there a risk consumers say: 'The new iPhone and Galaxy S10 have three and some others have four. I'm just going to go with the ones that have more because two doesn't sound that great'?

Users are very sophisticated now about their phone purchases.

They study them. They'll read the reviews. You're going to live with them for two, three or four years.

A lot of people remember from just a couple of years ago, where all the OEMs [original equipment manufacturers] were touting things such as: 'We have eight cores in our device, so it's super-fast.'

And then they realised the actual practical mechanics of that were almost none of them was used and it was actually just sort of a marketing specification.

Image copyright Google
Image caption Google hopes to once again offer the best low-light photography of any smartphone

So, my view is in this market, people don't fall for simple numbers anymore. They look for user experiences and Pixel certainly has a brand that's known for having an absolutely terrific camera experience.

The Pixel phone is one of most leaked smartphones. You even tweeted details about its built-in radar in advance. Others go to great lengths to try to keep details of their devices under wraps until launch day. Presumably you don't think that matters?

We definitely wanted to take a bit of a different approach in how we launch and reveal elements of the products. Several months ago, we started to disclose a little bit about how the product looked and some of the core technologies. We wanted to make sure people started to get excited about it and understood a little bit about it.

Of course, there are other leaks we really would prefer did not happen. And unfortunately that shows a little bit of our adolescence in that we have to make sure we're reducing the amount of unintentional information disclosed.

At some level, we're at least happy people care and desire this information. But we do want to try to reduce leaks in the future.

There's been controversy over facial-recognition tests carried out on the phone's behalf. To make sure it worked better with dark skin tones, there's been reports a contractor targeted homeless people offering them $5 (£4) but didn't properly explain what was going on. Can you address that?

Image copyright Google
Image caption Google promoted the Pixel 4's ability to use face unlock with dark skin tones, in a teaser video

It was very important for us to make sure the face unlock system works for all different kinds of people, genders, races, et cetera.

And as a consequence, we wanted to make sure we were able to get a large number of data points that allowed us to perfect this model in a fair way. So, we went out and did a lot of research in this area.

It's come to our attention there may be some methods that were not approved, not how we would do business. So, we're investigating that. We would never find that acceptable. And so we've suspended any data collection until this is straightened out.

Just to be clear about what you think is unacceptable. Was it the targeting of homeless people? Was it not explaining exactly what people were testing? Or what?

All of those allegations would be different than what we would find acceptable.

But do you still plan to retain the data collected in this way for 18 months?

I don't know that we've discussed the length of time that we're holding data. But there have been no changes to the programme with respect to data retention.

[Note: The Verge reported being briefed about the 18-month limit in July].

But some people are going to think if the data wasn't collected with proper consent, surely you should delete it and start again.

This is all under investigation. So, I just want to be clear we do not know the full facts of what has happened.

But if the investigation concludes people didn't know what they were consenting to, are you going to delete?

The best approach here would be to discuss it once we've actually looked into the facts and understood what has happened.

There's a lot of concern about facial recognition. You're selling camera-enabled devices that sit in people's living rooms, bedrooms, and on their front doors that use the technology. Do you accept reports about what happened with the Pixel tests help undermine confidence in Google and other big technology companies' use of facial recognition data?

Image copyright Google
Image caption The Pixel 4 is the first in the series to have two cameras on the rear

There's a distinction between what's being used to train a model for face unlock and facial recognition. There are specific use cases for these different technologies and it's very important to examine each one and determine if they're being used in a way appropriate for local laws and regulations.

It is definitely important around the world for what is societally acceptable to be clearly defined in collaboration between tech companies and governments. We want to take a very cautious and thoughtful approach to these technologies.

They are very important to debate and discuss and then together we have to clarify exactly what to do.

When Amazon recently unveiled its rival Ring smart cameras many expected them to add facial recognition but it decided not to. Do you think we need politicians to act quickly to set standards because there's a growing crisis of confidence?

It would be great for regulation to be clarified quickly.

The approach we're taking is to try to keep all of this information on devices as much as possible and always keep it private and secure and encrypted.

There's certainly a key responsibility to make sure the user's information is protected.

But everyone would benefit from clarity of standards and regulation. It is a challenging space to navigate. And it's very important it's thoughtfully navigated.




All Comments: [-] | anchor

chenning(10000) 2 days ago [-]

Dumb question, but why are people so surprised by this? Can you honestly say at this point in the game that you had no idea this was happening or that this is news to you?

evross(10000) 2 days ago [-]

One simple answer is that marketing (a.k.a. thought manipulation) works. Propaganda works. What's the advertisement/marketing spending for all these surveillance companies' products in the past 10 years? And, in comparison, what about the advertising spending for consumer awareness of privacy respect and personal information security? Combine that lopsided-ness with the general person's lack of technical/surveillance education and it's more understandable why people aren't rejecting these companies' creepy behaviour.

edoo(10000) 2 days ago [-]

This happens to be a good example of how there are so many laws you likely break some all the time. California requires consent from all present who have vibrated the air while you record the air vibrations (but not with your ears and brain storage!). That means if the google phone in your pocket listens to people at a restaurant or your alexa listens to your delivery man for a second you are a criminal under California penal code.

akiselev(4111) 2 days ago [-]

I highly doubt many judges in California or the Ninth circuit will look at this situation and blame the user, especially when they aren't the ones with access to the recordings.

DJHenk(10000) 2 days ago [-]

In this case it's not the fault of the law though. Everyone with a basic level of intelligence and awareness of cultural norms, i.e. everyone working at Google developing and marketing these kind of products, is aware that this might cause an issue. But blinded by money and optimistic technobabble they do it anyway and make it near impossible or very unpractical for the user to turn it off.

I would have no problem with some legal action against those people.

Isn't here some law against tricking other people into breaking the law?

gtfratteus(10000) 2 days ago [-]

California is actively trying to make everything a crime so that they have the power to press criminal charges against anybody they dislike without violating 1A. Unfortunately, there's no Constitutional Amendment prohibiting selective enforcement of the law, otherwise these idiotic laws would have been killed as soon as they were signed.

soulofmischief(4222) 2 days ago [-]

> California requires consent from all present who have vibrated the air while you record the air vibrations

One-party recording laws enable us to catch bad actors. Otherwise, you can't do things like record Rob Ford smoking crack and being a racist piece of shit while the incumbent mayor. The fact that you don't have the right to record a conversation you are participating in is not a good thing.

jacquesm(43) 2 days ago [-]

Hahaha. Sorry, really? Your bloody thermostat requires a consent form now for visitors to your house. This madness should really stop. All these appliances with microphones phoning home your every spoken word should be burned in a large pile. Let's start over, this is not going to end well.

NoNotTheDuo(10000) 2 days ago [-]

I don't think the thermostats are included in this. The article talks about Nest Cameras

mrzimmerman(10000) 2 days ago [-]

It's not including the thermostat, but he's specifically talking about etiquette, not legally binding EUAs with your house guests.

Think of it more as 'Oh, btw, I've got cameras in the living room, kitchen, and on the back patio if that's something that bugs you.'

buboard(3489) 2 days ago [-]

Wonderful times when your thermostat cant work without spying on you having sex

kevin_thibedeau(10000) 2 days ago [-]

All's good so long as you yell out 'BB is great!'

Fnoord(3929) 2 days ago [-]

Google could have the device say that it records conversations the moment it detects an unknown voice. Google could even ask that unknown voice for consent.

netsharc(10000) 2 days ago [-]

Didn't Nest want to create devices with cameras? It should just do face recognition to detect people who hasn't consented.. /s

And just like clicking 'I'm under 18' on porn sites takes you to disney.com, or 'continuing to use this site means you accept cookies', if they don't consent, stuff connected to Google would stop working...

'Ok Google, turn on the kitchen lights'. 'Operation failed with error undefined!'

koolba(638) 2 days ago [-]

That'd get really annoying if there's a TV in the room.

lqet(4209) 2 days ago [-]

Isn't the voice recognition done on Google's servers? How can Google record the voice, send it to its servers, decide that it is someone unknown and play the consent notification if the person has not already given consent?

a3n(3313) 2 days ago [-]

Why would anyone come back to a home after such a warning?

Think back 20 years. Would you come back to a home after the resident switched on audio and video recorders and said 'you're being recorded.'

Now come back to the present, and add the word 'thermostat.' Does it make any difference?

WillPostForFood(4155) 2 days ago [-]

The Nest thermostats don't have microphones. Houses have had security cameras long before Google existed, as have a large percentage of retail businesses, banks, gas stations, convenience stores, government offices.

PeterBarrett(10000) 2 days ago [-]

Having read the short article it's pretty clear that he was simply saying that if you have an indoor nest camera it would be proper etiquette to tell your guests that they are being recorded on your device. From how I read it, he wasn't saying to let your guests know that Nest/Google might be listening.

Personally I'd never have one of these devices in my house but I would appreciate it if my friends gave me a heads up if I was in their house with one.

sb8244(4157) 2 days ago [-]

Yes, I read it this same way. I'm not really sure that's an outrageous statement....

I thought initially that Nest meant thermostats and was shocked, but it's pretty clear he's talking about Nest Cameras.

khelenek(10000) 2 days ago [-]

These comments are lunacy. He's suggesting etiquette if someone puts a Nest recording device (not a thermostat) inside their house purposefully, probably for security purposes. Security cameras have existed for decades and nothing has changed here.

pnw_hazor(10000) 2 days ago [-]

He is saying it, because recording conversations is illegal (criminal) in some states unless all parties consent - and in some states consent must be on the actual recorded content.

Video w/o sound is not an issue unless there is an expectation of privacy (e.g., in a bathroom)

tziki(10000) 2 days ago [-]

Welcome to hacker news, where facts don't matter and big tech is always out to get you.

antisthenes(10000) 2 days ago [-]

Security cameras usually monitor the exterior of the house for bad agents.

Always-on listening devices and interior cameras are absolutely a new thing and to pretend like they aren't is what's actual lunacy.

LudwigNagasena(10000) 2 days ago [-]

>'Gosh, I haven't thought about this before in quite this way,' Osterloh said. 'It's quite important for all these technologies to think about all users... we have to consider all stakeholders that might be in proximity.'

How much do they not care about privacy that a chief exec with an army of lawyers and advisors 'haven't thought about this before'?

arkitaip(3938) 2 days ago [-]

Of course they know, they just need to cover their asses sue to the litigation risk.

XJ6(10000) 2 days ago [-]

All 'big tech' are at it, I tried to go to 'My Pictures' the other day and microsoft asked me to run my pictures through facial recognition.

They said that by consenting I was asserting I had the consent of anyone who appeared in any of my photos or might appear in future which is insane to ask of anyone, no-one can really give that third party consent.

Then when I refused they showed me a nag screen which didn't have that warning.

Then just yesterday I opened google camera and it really wanted to run face recognition on my photos. I refused.

I wanted to try the google podcasts but it was a brick because I had disabled 'youtube & web activity' in google account settings. There was no way to subscribe to podcasts without web activity, the application said. And without subscribing there was nothing the app could do.

We have a company claiming that because I don't consent to have my web and youtube activity tracked that there's no way I could subscribe to a podcast that I explicitly choose to subscribe to in an application designed for subscribing to podcasts.

They don't stop trying, everyone is trying to hoover up all data, all the time.

DaniloDias(10000) 2 days ago [-]

If you are a privacy advocate, how could you work at google and sleep well?

Why sign up for that kind of pain?

To me- its more interesting that this exec is so clueless as to utter something so damning in such a cliched way. Gosh- was he really this earnestly naive? Or is it possible that this is sarcasm? How can you be a vp exposed to press and say something this unguarded or sarcastic IN FRONT OF PRESS?!? What is going on at Google that people like this are promoted/hired?

"Our customers should consider warning guests and family that they use our products" -a google exec

wmeredith(2874) 2 days ago [-]

They don't care. They're arguing in bad faith. It's also called lying or being full of shit.

zaptheimpaler(3981) 2 days ago [-]

Its a baldfaced lie. In 2019, with all the concern about privacy in the world it is inconceivable that they haven't thought about it. The transition of tech power to the sociopath overlords is complete.

michannne(10000) 2 days ago [-]

They are clearly at fault, but at this point customers need to be more cognizant of the fine print. I recently did a Google takeout of my data and found for most services they have no tracking on me after March, which is around the time I started actively hunting for options to turn off anything which reveals private information. Taking steps to learn how to reduce my online footprint also helped

csallen(1051) 2 days ago [-]

He immediately admitted that he warns guests in his own home about being recorded, and the article discusses the fact that the cameras have an LED light that turns on when recording. So it's clearly not true that he hadn't thought of this before.

UserIsUnused(10000) 2 days ago [-]

>Ask forgiveness not permission. That's all that matters to big tech, that's what brings in the $$$

mtgx(152) 2 days ago [-]

This is why I've preached this for a long time:

It's not that regular users 'don't care about privacy' as many are quick to conclude, but the vast majority simply don't understand.

If even big tech execs don't understand these issues, then how can regular people?

thisgoodlife(10000) 2 days ago [-]

I guess the whole point of hiring an army of lawyers and advisors is that you don't have to do all the things by yourself.

hanniabu(3932) 2 days ago [-]

They've thought about it but they won't say that outside of closed doors for plausible deniability.

jiveturkey(4177) 2 days ago [-]

It was not an admission, it was a poorly thought out conversational device. He would have done much better to say, 'good question'.

In the same interview, in the very next sentence of the article:

> Osterloh then acceded that warning houseguests about Nest devices' recording capabilities is proper etiquette, stating that he already does so.

Given that he already warns his guests, he absolutely 100% certainly has thought about 'all stakeholders', in quite that way.

hardwaresofton(3367) 2 days ago [-]

It's the wild west -- corporations are just doing what they want while governments are asleep at the wheel.

Also, having an army of lawyers is exactly what allows for this -- you need to know which areas are gray in order to exploit them without risk of retribution/regulation. A basic minutes-long conversation would have likely arrived at the point of how recording of the owner themselves/third parties would affect the product.

have_faith(4166) 2 days ago [-]

> stakeholders that might be in proximity

I didn't realise visiting a friends house made me a stakeholder in all of their technology choices.

tunap(10000) 2 days ago [-]

It is a scripted response tailored to convey a specific, albeit less than truthful, narrative Much like the Juul CEO telling the public not to use their product right before the next announcement they were under investigation. Hopefully, a similar investigation will be announced presently.

Edited.intentiin > narrative

saiya-jin(10000) 2 days ago [-]

C'mon, that a straight lie out there, plain and visible.

The other alternative is that he is utterly incompetent at his role, which I don't believe for a second. He is probably be incompetent in speaking with media though

hos234(4213) 2 days ago [-]

Corporate Robots are programmed with 2 rules - Everything has to be scaled up & It's easier to ask for forgiveness than permission. Then they are let loose on the world where they fuck up, junked and a new model with the same programming is wheeled in.

Can they be reprogrammed is the question.

madrox(10000) 2 days ago [-]

Every time an article like this comes up on HN it's the exact same concerns, the exact same HN comments, and the exact same arguments. I don't feel like the conversation has moved on significantly since Alexa came out. It makes me wonder why articles like these keep getting upvoted and commented on with the same stuff. Is this tabs vs spaces for the privacy world?

Sendotsh(10000) 2 days ago [-]

Tabs vs Spaces, Windows vs Apple (vs Linux to a smaller extent), Xbox vs PlayStation, Coke vs Pepsi, people are opinionated and it's as old as time itself (religion, nationality, us vs them).

People like agreeing with those that agree with them, and disagreeing with those who don't. Very rarely will any news on any of those topics actually progress anything, or change anyone's beliefs/opinions, but people still like trying.

raverbashing(3723) 2 days ago [-]

Does two-party consent even works if it's a 3rd party doing the recording and it goes who knows where?

I'm sure this will also work just fine in Germany.

throwaway13337(4026) 2 days ago [-]

It's a good question.

I hope that the law comes down on these devices such that it only makes sense to keep and analyze all recorded data locally.

If recorded data is never the property of the corporations but of the private owner of the device, a lot of the issues with them could be defanged.

On the flip side, the value proposition for these devices may then be reduced - both for the corporations that produce them and the customers that use them.

LoSboccacc(4201) 2 days ago [-]

jUst PUT a GDPR nOtice on THE DOoR

njsubedi(3880) 2 days ago [-]

You mean, a curtain [banner] that displays a notice, which the visitors need to move aside [close] to enter the house? ;)

saluki(4221) 2 days ago [-]

I was a huge Google Fan but this is becoming lunacy.

I understood trading some data and privacy for great search results, maps and free quality email.

But people willingly bringing always on listening devices in to their homes (beyond what smartphones are already capable of) I just can't comprehend it.

Why would people voluntarily do this in exchange for being able to ask for weather, play a playlist, add a todo and a few other parlor tricks.

I guess I value my privacy more than others and don't like the idea of entities compiling a record of my data that they can sell and market.

Imagine how some governments could use this data to limit freedoms, crack down on their opposition.

And what about the first data breech that includes transcripts or even audio of all your household conversations/activities over the past three years matched up to your email or even address?

It just sounds like we are heading down the wrong road.

pjmlp(298) 2 days ago [-]

Skynet apparently is somewhere under Google's cellar.

robomartin(3890) 2 days ago [-]

> people willingly bringing always on listening devices in to their homes <...> I just can't comprehend it.

The reason is 'people' lack education in technology. They can barely use a computer these days, with 'using a computer' generally means being on Facebook with a browser.

There's a very significant divide in this regard that has been growing for decades. And it isn't getting any better. This is why it is nearly impossible to convince people you know to use better passwords and adopt security and privacy-conscious behaviors. I don't know if you (plural) have experienced this, I know a few people for whom sending a link via email is a challenge. They think they sent you a link and what you get is unusable. No, it isn't about age at all.

Smart phones and tablets have changed the relationship people have with computers. They make things very easy to use and require almost no technological knowledge to operate. In computers, the browser or Chromebooks do the same.

Add it all up and the idea that someone does not even remotely think that a thermostat could be recording every word being said in home doesn't even cross their minds. Going beyond that, thinking about how and where that data is stored and how it could be used it yet another level.

All this says it is the responsibility of the tech community to make sure customers understand the important aspects of the technology we provide.

notus(10000) 2 days ago [-]

It's no different than an iPhone

naasking(10000) 2 days ago [-]

> But people willingly bringing always on listening devices in to their homes (beyond what smartphones are already capable of) I just can't comprehend it.

Always-on listening doesn't seem like a huge deal. What matters is the policy of allowing Google people to listen to captured audio, even if it's for training purposes. This is skirting uncomfortably close to a dystopian surveillance society.

If you want to gather data from people, provide an incentive to voluntarily participate. For instance, on Google maps you can become a contributor and there are benefits and bonuses if you contribute enough. Yes, this group is self-selected so it's not as good as a random sample, but it's better than the dystopian alternative.

fluffything(10000) 2 days ago [-]

> Why would people voluntarily do this in exchange for being able to ask for weather, play a playlist, add a todo and a few other parlor tricks.

I don't really understand why the device needs to store my recording and send it over the internet.

I would be willing to pay more for a device that guarantees that my recordings are deleted immediately.

snarf21(10000) 2 days ago [-]

You are right but as you say all those things apply to your smartphone and the smartphone of every person you are in range of. Also, smartphones have cameras front and back. An Alexa in my bathroom only has a mic and speakers. The future will also be powered by cameras and microphones in public and you will be tracked. Amazon is actively selling software and hardware to enable dystopia today.

I imagine that if governments start to use Alexa, et. al. to crack down on dissenters, those whole business lines disappear as people dump them all in the trash. I think we have to depend on capitalism and whistleblowers to shout if things change. If someone wants to listen in on the sounds of my bathroom, help yourself. I personally draw the line at cameras. No camera, no Ring doorbell and my phone is usually in my pocket or in a shelf with no visual access.

The other thing that I'd point out is that a lot of people publicly shout their political and other positions voluntarily. This is great for free speech but is an even easier and richer target for government crackdowns.

at-fates-hands(3659) 2 days ago [-]

> It just sounds like we are heading down the wrong road.

Humans are a lazy bunch and this feeds into these genetically coded tendencies in the worst way.

I feel like there is a growing backlash against these technologies that are getting way too invasive. Myself and a few of my friends have gone back to being as 'analog' as possible. Dumb thermostats, dumb tv's, regular old key locks (you can still get some pretty secure, non-electronic locks), dumb fridges, etc, etc. The point is to disconnect as much as possible from the internet, Google, FB and Amazon. Three of my best friends now just carry Windows phones. Internet connected, very few apps, and phone service.

We are of course, a very small minority whose seen the dark side of these technologies. I don't think enough people know or understand the consequences of having these devices in their homes.

kevindong(4221) 2 days ago [-]

> I guess I value my privacy more than others and don't like the idea of entities compiling a record of my data that they can sell and market.

Because big tech/advertising companies don't sell your individual data. They sell to other entities the ability to show people like you specific content. I wouldn't use products monetized in the first way, but I'm happy with the latter approach.

52-6F-62(3611) 2 days ago [-]

The new version of Fahrenheit 451 (film w/ Michael Jordan) does a fantastic take of this. I highly recommend it unless anyone is a Fahrenheit 451 originalist.

glitcher(2662) 2 days ago [-]

I blame Star Trek. Simply speaking out loud to ask the computer anytime something was needed, it seemed like magical tech. We just didn't anticipate the companies and extra consequences that would come along with it.

I would love to see a parody of Star Trek where all the tech is run by an advertising company exploiting all the crew members individually by capturing all their private data around the clock.

urda(4202) 2 days ago [-]

If you've carried an Android at all, you've already had a listening device in your home for years. This device also has a ton more sensors and data it collects off you than a smart speaker could.

ianmcgowan(10000) 2 days ago [-]

I have a few echos and a google home, and I love the feeling of living in the future by controlling things with my voice.

It may be naive, but I think the device vendors have way more to lose if it turns out they are actually recording all the time, not just listening for the wake word.

It will be the wrong road if we end up at 1984, but I'm hoping for the Culture, or at least the Jetsons...

bashallah(10000) 2 days ago [-]

Can't you just removed the hardware from the device? It's not like microphones are magical.

azarizotza(10000) 2 days ago [-]

This might be because I'm running on a half hour of sleep but I legit thought you said you were a Google Fan as if Google makes smart ceiling fans and you were somehow talking to me.

oppositelock(10000) 2 days ago [-]

What does having a Nest thermostat get you? I'm a tech dude, I build modern software, but I live in a dumb, old fashioned house. My programmable Honeywell thermostat, which has no internet connection, does a perfectly fine job of keeping my home the right temperature. My door locks use a key, and keys have great battery life, and my smoke detectors beep instead of speaking to me. What am I missing by not having internet connected stuff?

shadowgovt(10000) 2 days ago [-]

> Why would people voluntarily do this in exchange for being able to ask for weather, play a playlist, add a todo and a few other parlor tricks

Because it's a lot of fun and the personal risk is basically nonexistant. It's the oldest story in the book: different personal evaluations of risk of activity.

sgustard(3080) 2 days ago [-]

Are we acknowledging a difference between the Nest device he warns about (which is sold as a surveillance device) and, say, an Amazon Echo? (Which the headline mentions as equally problematic, but the body of the article doesn't mention at all.) My understanding of the Echo is that it only records after hearing a trigger word.

cmsonger(10000) 2 days ago [-]

My personal view is that the biggest issue here is the lack of user protection. I'd be a LOT less concerned about this if the appropriate legal framework protected me.

The Star Trek vision of voice as an interface is a good vision. The question is how the regulatory framework protects users in this world and keeps it from being the dystopian nightmare that our 'for profit corporations get to do whatever they like as long as you agreed in an 'all or nothing' click through' seem to be headed for.

kcdev(10000) 2 days ago [-]

You said it yourself: smartphones are already capable of it. So what difference does it make if you add an Alexa or Google Home? At least those aren't going everywhere with you. This is the defeatist attitude, but this tech is probably here to stay. We just need to do our best to make sure companies do the right thing.

raxxorrax(10000) 2 days ago [-]

People just don't know how valuable this data actually is. Sure it only gets value if a critical quantity is reached.

There is just a disconnect in understanding. With huge efforts they will create archives of 5% special price coupons, but how tech companies extract free money from them is beyond their comprehension.

The implications of having every deed and word quantified is perhaps too abstract a danger to understand.

tantalor(2757) 2 days ago [-]

Do you own a cell phone?

_davebennett(3345) 2 days ago [-]

I know I'm gonna be in the minority with this, but what if you really just don't care despite knowing the danger?

I have a Google Home in my bedroom, and it's extremely convenient for controlling my lights. I'm aware of the privacy concerns with such a device, but out of all the things in the world that worries me it's just not far up my list to disregard the convenience. I imagine a lot of consumers might feel this way. Now this would be a different story if something was forced upon me (like the government mandated some device, and I don't have an active choice in having it) but I'm making an active choice to buy this device and use it knowing that it could very well be recording the conversations, sex, whatever else goes on in my bedroom.

mr_overalls(4091) 2 days ago [-]

I couldn't agree more. Are there any good open-source projects for AI assistants? There certainly seems to be a need.

jfoster(3989) 2 days ago [-]

As you noted, smartphones are already capable of this. Smart speakers are relatively moot from a privacy perspective, unless you keep your phone in a soundproof case.

Taylor_OD(4220) 2 days ago [-]

A lot of people just don't care.

godelski(4187) 2 days ago [-]

> I guess I value my privacy more than others and don't like the idea of entities compiling a record of my data that they can sell and market.

I think a lot of people don't understand how much their data is worth. It is something I frequently hear, that 'my data is useless, so why hide it?' or 'what are they going to do, see that I like cat photos? HA!'. I think these kinds of comments are slowly diminishing as the average person is seeing 1) these tech companies are worth $1+ trillion 2) people get scared and think Facebook is always listening because they every so often hit the mark too well. But I don't think most even know the extend of the data that these companies have (e.g. they think that Google only has their email data and knows nothing about financial), let alone how that data can be used.

wnevets(10000) 2 days ago [-]

I've gotten Alexa and Google's Home for free but I refuse to set them up. Even if you believe these companies aren't going to 'spy' on you, the security of these things has been proven dubious at best.

kelnos(3842) 2 days ago [-]

> But people willingly bringing always on listening devices in to their homes (beyond what smartphones are already capable of) I just can't comprehend it.

I don't get this position: my (Android) phone can already record my voice, so how does having a Google Home materially increase the privacy risk?

The GH does have access to mains power and a wifi connection, so that makes it easier to record all the time and upload a lot of data vs. a phone, but it seems like a bit of a stretch to me to think that Google would be doing something nefarious with GH and then absolutely nothing on Android. I've also noticed that when I call out to my GH, my phone will usually respond too, so it's not like my phone can't understand me, even when it's in my pocket, so the capability is definitely there.

So if you do trust Google enough to have an Android phone, then you should probably trust them just as much to have a GH. Beyond that, then it just comes down to bugs: what if GH is too sensitive and thinks you've uttered the wake word often when you haven't? Ok, then some conversations may get recorded that you didn't intend. That's certainly a concern, but 'I just can't comprehend it' would seem a bit hyperbolic in that case.

> ... includes transcripts or even audio of all your household conversations/activities over the past three years

This seems to indicate to me that you believe that GH etc. are always recording and uploading every sound it hears, all day long, without regard for the wake word. If you believe this, I guess that is your prerogative, but I guess we're probably not going to have a productive discussion here.

Ajedi32(1903) 2 days ago [-]

> beyond what smartphones are already capable of

What do you mean by this? What exactly is it that you believe smart speakers capable of that smartphones are not? (Modern smartphones are just as capable of listening for and responding to hotwords as smart speakers are, fyi.)

Santosh83(2075) 2 days ago [-]

The problem is the illusion that these devices present of being just another utility like a toaster or a TV or even a smartphone. The always on link between these listening devices and their command and control is not evident. It gives the illusion of operating locally.

president(3836) 2 days ago [-]

The thing is, the majority viewpoint always wins, whether they are right or wrong. If we had a well-educated and well-informed populace, people might make better decisions. Unfortunately, this is not the case and the situation is worsened with marketing and advertising from companies choosing revenue over morals.

chenning(10000) 2 days ago [-]

I don't think this is just a Google issue. Osterloh just happened to be the first one to come right out and say, yeah, maybe you should tell your guests.

Personally, I thought the bigger news (insult) was when he said, 'Gosh, I haven't thought about this before in quite this way.'

Give me a break. You knew it. I knew it. We all knew it. Maybe the right thing to do is have the device recognize there's an unknown voice in the room and announce that it's listening. That hasn't happened for some reason, but I highly doubt the reason is that no one ever thought of it.

me_me_me(10000) 1 day ago [-]

> But people willingly bringing always on listening devices in to their homes

Because flying cars. The home assistant is the glimpse of future people looked forward to since first Scifi movies. Its convenient and out of way. And people don't understand nor care about privacy.

That said I am still ranting to my brother that he is using Alexa whenever I visit his home.

greggman2(10000) 2 days ago [-]

I hate not having privacy but I don't understand singling Google out here. Siri is listening (both on phones and on Apple' homepod), Alexa is listening, Cortana is listening, ...

I actually am less worried about those (since I turn them off) than I do about Netflix/Amazon knowing every movie I watch. Amazon knowing every product I order. My Credit Card and Bank company knowing every business I transact with. My Mac and PC spying on what I do with them (i think that's more Windows than Mac). My PC/iPhone/Android/Rift knowing every app I run. My ISP knowing all the websites I visit or DNS lookups I make etc... I could go on and on but I don't see how Google sticks out here.

It's a much bigger problem.

anigbrowl(71) 2 days ago [-]

'Don't be evil' as a company motto really seemed to make Google different in the early days - not just in terms of being hip and cool in contrast to their contemporary competitors, but in terms of recognizing that technology has a moral dimension and that the firm's long-term wellbeing lay not just in maximizing its technological power but distributing the benefits thereof as widely as possible, rather than conserving it for private advantage.

I find their news product a useful barometer of the extent to which they've abandoned that. Last Monday I sat down at my workstation and the top item - amid war, political instability, and everything else going on in the world - were articles about the Google Pixel 4. I know this wasn't a reflection of my browsing habits - I'm just not interested in phone technology at the product level, and and any online shopping/product research that I do these days happens on a separate account on a separate device, because I don't want to mix up my consumer preferences with my intellectual interests.

You would think that with all Google's brainpower the presentation and distribution of news would by now have seen massive advances, but instead it has simply magnified the worst aspect of print and Tv news. I'm beset with sports, entertainment, 'health' (mostly quackery), and 'technology (mostly consumer products) news every day, even though I used to be able to opt out of those things. If I make the mistake of looking at a sports article on my workstation or phone, I am swamped with sports articles for the next day or two, presumably because the sports market is so lucrative and Google's algorithms get tremendously excited at the possibility that I am adopting the consumer the consumer profile of a sports fan.

Likewise I am constantly being advanced second- and third-hand sources of news based on SEO and existing reach; every other story I read is a half-assed rewrite of original research done by a news outlet with a smaller brand footprint. If I'm lucky I can see that in the first paragraph or two and can click through to the original, if not it's because a big news outlet is just stealing or redoing stories without credit or attribution and I may find out about it later when I read Twitter.

I could rant on and on and on about Google's news product and about their other products and practices, but all my critiques of the firm boil down to this: google has become a company that takes bids (including from itself) on the order in which it provides leaves to consumers, while denying them sight of the trees they are plucked from.

squarefoot(4030) 2 days ago [-]

Then what about digital assistants, and cellphones, and connected car computers, and...?

jdgoesmarching(10000) 2 days ago [-]

Seriously, most people here probably send location data to Google several times a day and have for almost a decade. We all make security tradeoffs for convenience, people high roading on this thread are no different.

The people avoiding voice assistants and smart locks probably have 50 way more vulnerable security gaps in their personal lives

NelsonMinar(1080) 2 days ago [-]

Don't forget all the TV panels in your house listening in on you too. Me, I don't plug any 'smart' TV into the Internet. But a lot of folks do.

I think the horse has left the barn on this kind of technology. What I hate most about it is that it's impossible for an ordinary person to understand exactly what all might be spying on them at any moment.

slouch(4071) 2 days ago [-]

Sure, but TVs aren't always on like this Nest device.

RandomBacon(4084) 2 days ago [-]

Legislation we need: 'smart' devices need to come packaged with labels for people to put on their door warning guests that there is audio recording equipment inside and no expectation of privacy can be provided.

jdlyga(10000) 2 days ago [-]

Is the distinction that it's private space? There's tons of recording equipment monitoring public roads, sidewalks, streets, inside stores, etc. In Manhattan, you're on camera way more than you're off camera.

afpx(4111) 2 days ago [-]

How is this even legal if wiretapping is illegal in many places?

criddell(4162) 2 days ago [-]

I wish I could record my phone calls. It would be easy for Google to add that function to Android but they don't. I assume it's due to legal problems. On the other hand, they sell devices that record all conversations in a home.

If they want to put the responsibility on the owner then do that. Add recording to Android. Otherwise, be consistent with your other products and turn off automatic recording.

wongarsu(10000) 2 days ago [-]

The owner of the device consented either explicitly or implicitly. For everyone else it's not immediately obvious where to assign the blame: is it the owner of the device recording you without consent, as it would be with a normal microphone, or is the smart device manufacturer at fault?

Just telling people 'ask every guest for permission' ostensibly clears this up: it's the owners fault for not getting permission. Of course in many places courts are in the habit of disregarding completely unrealistic disingenuous demands.

IshKebab(10000) 2 days ago [-]

I don't think there's anywhere where it's illegal to record conversations of both parties know about it.

pmlnr(1489) 2 days ago [-]

I'm genuinely curious why only nest, alexa, google home[^1] devices get highlighted - aren't they all use the same tech as the Google App on android? Why are android phones not flagged as well?

[^1]: https://news.ycombinator.com/item?id=21262521

madez(4155) 2 days ago [-]

I wish it were socially accepted to tell people to turn their Android device off while they are in company. I wish I could tell people that visit me to turn their device off or put it into rf and sound shielded containers.

I don't mind people using computers and their powers to ease their lives. I don't want to deny them that just because they are next to me or in my home. I just don't want them to use that power to collect and track and surveil everything at all times around them, wittingly or unwittinlgy.

eloisant(10000) 2 days ago [-]

I might be wrong but I don't think Android phone are always listening by default.

davinic(10000) 2 days ago [-]

Siri is always enabled on iPhones by default.

diffeomorphism(10000) 2 days ago [-]

Two obvious reasons:

- Smart home devices are used at home, where you have a high expectation of privacy. In contrast phones go with you everywhere.

- Those devices are supposed to hear everything in the room clearly. A phone in your pocket or backpack? If they can make their mics good enough for that, I would like to have that tech.

Santosh83(2075) 2 days ago [-]

A few short years ago it was incomprehensible, even in the west, to have what is essentially a permanently bugged house. Now it has been normalised to such an extent that even people who claim to be aware of privacy use Alexa et al at home in a striking display of dissonance. The general public at large has been sold such attractive shiny toys that they don't realise that the price they pay for using them is to unconsciously disabuse themselves of any notion of assuming privacy, even in your private space and activities. This training has been so effective that they will actually defend these practices and claim that you're a 'tin foil hat wearer' and paranoid to be speaking up for reclaiming what was normal and natural as short as two generations back.

vineyardmike(10000) 2 days ago [-]

While I agree with the broader sentiment, I think that app/web tracking is worse than 'bugged houses' because it is disingenuous to claim the devices (alexa et al) are actually always listening. You specifically interact with them, and the only data lost is a voice print, and when you like to turn your lights on. Sure security cameras are the exception since they always record, but those aren't new.

irq-1(10000) 2 days ago [-]

People used to ask me why I don't use Facebook. I haven't been asked that in about two years.

I predict that three years from now the majority will share our views about voice and video recording.

jstewartmobile(3718) 2 days ago [-]

I have an ordinary thermostat that doesn't need firmware, an internet connection, or to eavesdrop on my conversations. It works great!

What exactly is the advantage here?

WillPostForFood(4155) 2 days ago [-]

They aren't talking about Nest thermostats, which don't have a mic. They are talking about the Nest Cam, and now the Google Home which has been rebranded a Nest device.

lotsofpulp(10000) 2 days ago [-]

Automatically turning HVAC systems on and off depending if the place is empty, turning it back on remotely, cycling the fan on and off to circulate air more often, removing excess humidity, using data about sunlight to adjust temps.

There must be energy savings for the utility companies to give them out for free or almost free. Although I prefer the HomeKit thermostats since I presume Apple has better privacy controls.

kareemm(388) 2 days ago [-]

Since cameras and mics are embedded in all kinds of consumer devices these days, it makes me think we need a site that recommends devices that DONT include surveillance as a "feature". Sort of a Wirecutter for privacy.

Does this exist?

neon_electro(10000) 2 days ago [-]

Mozilla Foundation has a site called *privacy not included that I think attempts to accomplish what you're asking for: https://foundation.mozilla.org/en/privacynotincluded/

woodrowbarlow(10000) 2 days ago [-]

well it really depends how far down the rabbit hole you want to go.

a laptop running OSS is better than a laptop running win10, but then there's usually still intel ME doing goodness knows what.

a phone running sailfish or f-droid is better than OEM android, but there's still your cellular service provider selling location data to goodness knows who.

anyway, check out the pinebook (laptop), librem 5 (phone), mycroft (voice assistant).

irishloop(10000) 2 days ago [-]

I know it puts me in the minority, but I honestly do not care that much. It's obfuscation by volume.

If Google wants to hear me and my roommate argue about Red Dead Redemption online or the new Bill Burr comedy special, good luck with that.

When I worked in IT, people always used to say oh you can read my emails? Nobody wants to read about your boring-ass life.

My understanding is they are using our speech to create better speech-recognition and it shows. I have much better luck with OK Google than Siri. I willingly give up my privacy for a superior product because most of my conversations are inane.

And if I ever intend to murder anyone, I will turn off Alexa.

falcolas(10000) 2 days ago [-]

This is simply a 'I have nothing to hide' argument, and the rebuttals to this have been done frequently and better than I can manage.

SlowRobotAhead(10000) 2 days ago [-]

> It's obfuscation by volume.

Not even sort of. They know EXACTLY who you are, and have the ability to determine EXACTLY what you say, or when you are home, or who is home with you, or meta data about your habits like sleep or TV, or in a very realistic possibility monitor for words beyond "Alexa"... like let's say "Sanders" or "Trump" and upload number of times those trigger words were said every 24 hours.

There is no part of this that is obfuscation by volume. Specifically you get specifically tagged.

How could it not be within reason that a door to door political volunteer is sent to your specific door with their tablet saying "Don't mention guns, Do mention tax reform"?

There are arguments I can make for personal assistants esp for elderly or disabled. But obfuscation by volume definitely isn't the case here.

eeZah7Ux(3214) 2 days ago [-]

> Nobody wants to read about your boring-ass life.

On the contrary, aggregated personal data sells for a lot of money. Entire companies exist only to collect it.

callmeal(10000) 2 days ago [-]

>It's obfuscation by volume.

I find it hilarious that you mention this and completely disregard what google is famous for: search. Volume means nothing and relying on volume to hide in a crowd is basically impossible in today's world. Even the absence of data sends a signal, so when you turn off Alexa, that's when your surveillance will increase.

>If Google wants to hear me and my roommate argue about Red Dead Redemption online or the new Bill Burr comedy special, good luck with that.

Google doesn't care about that. But you can bet that the contractors they hire are deeply interested in the 3am conversation you had with/about your significant other.

DoreenMichele(203) 2 days ago [-]

And if I ever intend to murder anyone, I will turn off Alexa.

Police cases of the future:

'He's a prime suspect because he turned off Alexa shortly before the so-called accidental death in question happened.'

augustk(10000) 2 days ago [-]

I think a device with a camera/microphone should have a physical on/off button and a LED which indicates when it's on. At least, that's how I would design it.

guyzero(3694) 2 days ago [-]

Nest changed their camera to prevent users from disabling the light that indicate the camera is recording and users complained loudly and bitterly about it.

freeflight(4194) 2 days ago [-]

Doesn't Google Glasses have an indicator LED for when they are recording?

tantalor(2757) 2 days ago [-]

I'm sure any 0-day capable of eavesdropping on your phone can also disable the LED.

FabHK(3966) 2 days ago [-]

FWIW, I'm living a Google-free live now, and it works very well:

- DuckDuckGo as default search engine

- Own domain for email, forwarded to Apple's mail

- Apple map, Yandex map, Waze, Here WeGo for maps (depending on the country)

- For videos, I search them in DDG with '<searchterm> !yt', then copy the links I want and download them with `youtube-dl`

- Firefox and Safari

- Zoho as a replacement for online docs and spreadsheets

etc.

EDIT to add: I should say that I try to live Google-free...

criddell(4162) 2 days ago [-]

You've certainly reduced your Google footprint. I don't think it's possible to reduce it to zero though.

g5pw(10000) 2 days ago [-]

Just a heads-up: Waze is owned by Google [1] now.

[1]: https://en.wikipedia.org/wiki/Waze#Google_subsidiary

justusthane(10000) 2 days ago [-]

What do you do about websites that use reCaptcha?

auiya(10000) 2 days ago [-]

You're likely not at all living 'Google free'. https://gizmodo.com/i-cut-google-out-of-my-life-it-screwed-u...





Historical Discussions: What's New in Python 3.8 (October 14, 2019: 831 points)
What's New in Python 3.8 (July 17, 2019: 13 points)
What's New in Python 3.8 (October 14, 2019: 10 points)
Python 3.8 is released! (October 14, 2019: 8 points)
Python3.8: Debugging with f-Strings (July 31, 2019: 1 points)

(832) What's New in Python 3.8

832 points 5 days ago by supakeen in 4170th position

docs.python.org | Estimated reading time – 42 minutes | comments | anchor

This article explains the new features in Python 3.8, compared to 3.7. For full details, see the changelog.

Python 3.8 was released on October 14th, 2019.

New Features

Assignment expressions

There is new syntax := that assigns values to variables as part of a larger expression. It is affectionately known as "the walrus operator" due to its resemblance to the eyes and tusks of a walrus.

In this example, the assignment expression helps avoid calling len() twice:

if (n := len(a)) > 10:
    print(f'List is too long ({n} elements, expected <= 10)')

A similar benefit arises during regular expression matching where match objects are needed twice, once to test whether a match occurred and another to extract a subgroup:

discount = 0.0
if (mo := re.search(r'(\d+)% discount', advertisement)):
    discount = float(mo.group(1)) / 100.0

The operator is also useful with while-loops that compute a value to test loop termination and then need that same value again in the body of the loop:

# Loop over fixed length blocks
while (block := f.read(256)) != '':
    process(block)

Another motivating use case arises in list comprehensions where a value computed in a filtering condition is also needed in the expression body:

[clean_name.title() for name in names
 if (clean_name := normalize('NFC', name)) in allowed_names]

Try to limit use of the walrus operator to clean cases that reduce complexity and improve readability.

See PEP 572 for a full description.

(Contributed by Emily Morehouse in bpo-35224.)

Positional-only parameters

There is a new function parameter syntax / to indicate that some function parameters must be specified positionally and cannot be used as keyword arguments. This is the same notation shown by help() for C functions annotated with Larry Hastings' Argument Clinic tool.

In the following example, parameters a and b are positional-only, while c or d can be positional or keyword, and e or f are required to be keywords:

def f(a, b, /, c, d, *, e, f):
    print(a, b, c, d, e, f)

The following is a valid call:

f(10, 20, 30, d=40, e=50, f=60)

However, these are invalid calls:

f(10, b=20, c=30, d=40, e=50, f=60)   # b cannot be a keyword argument
f(10, 20, 30, 40, 50, f=60)           # e must be a keyword argument

One use case for this notation is that it allows pure Python functions to fully emulate behaviors of existing C coded functions. For example, the built-in pow() function does not accept keyword arguments:

def pow(x, y, z=None, /):
    'Emulate the built in pow() function'
    r = x ** y
    return r if z is None else r%z

Another use case is to preclude keyword arguments when the parameter name is not helpful. For example, the builtin len() function has the signature len(obj, /). This precludes awkward calls such as:

len(obj='hello')  # The 'obj' keyword argument impairs readability

A further benefit of marking a parameter as positional-only is that it allows the parameter name to be changed in the future without risk of breaking client code. For example, in the statistics module, the parameter name dist may be changed in the future. This was made possible with the following function specification:

def quantiles(dist, /, *, n=4, method='exclusive')
    ...

Since the parameters to the left of / are not exposed as possible keywords, the parameters names remain available for use in **kwargs:

>>>
>>> def f(a, b, /, **kwargs):
...     print(a, b, kwargs)
...
>>> f(10, 20, a=1, b=2, c=3)         # a and b are used in two ways
10 20 {'a': 1, 'b': 2, 'c': 3}

This greatly simplifies the implementation of functions and methods that need to accept arbitrary keyword arguments. For example, here is an excerpt from code in the collections module:

class Counter(dict):
    def __init__(self, iterable=None, /, **kwds):
        # Note 'iterable' is a possible keyword argument

See PEP 570 for a full description.

(Contributed by Pablo Galindo in bpo-36540.)

Parallel filesystem cache for compiled bytecode files

The new PYTHONPYCACHEPREFIX setting (also available as -X pycache_prefix) configures the implicit bytecode cache to use a separate parallel filesystem tree, rather than the default __pycache__ subdirectories within each source directory.

The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ subdirectories).

(Contributed by Carl Meyer in bpo-33499.)

Debug build uses the same ABI as release build

Python now uses the same ABI whether it's built in release or debug mode. On Unix, when Python is built in debug mode, it is now possible to load C extensions built in release mode and C extensions built using the stable ABI.

Release builds and debug builds are now ABI compatible: defining the Py_DEBUG macro no longer implies the Py_TRACE_REFS macro, which introduces the only ABI incompatibility. The Py_TRACE_REFS macro, which adds the sys.getobjects() function and the PYTHONDUMPREFS environment variable, can be set using the new ./configure --with-trace-refs build option. (Contributed by Victor Stinner in bpo-36465.)

On Unix, C extensions are no longer linked to libpython except on Android and Cygwin. It is now possible for a statically linked Python to load a C extension built using a shared library Python. (Contributed by Victor Stinner in bpo-21536.)

On Unix, when Python is built in debug mode, import now also looks for C extensions compiled in release mode and for C extensions compiled with the stable ABI. (Contributed by Victor Stinner in bpo-36722.)

To embed Python into an application, a new --embed option must be passed to python3-config --libs --embed to get -lpython3.8 (link the application to libpython). To support both 3.8 and older, try python3-config --libs --embed first and fallback to python3-config --libs (without --embed) if the previous command fails.

Add a pkg-config python-3.8-embed module to embed Python into an application: pkg-config python-3.8-embed --libs includes -lpython3.8. To support both 3.8 and older, try pkg-config python-X.Y-embed --libs first and fallback to pkg-config python-X.Y --libs (without --embed) if the previous command fails (replace X.Y with the Python version).

On the other hand, pkg-config python3.8 --libs no longer contains -lpython3.8. C extensions must not be linked to libpython (except on Android and Cygwin, whose cases are handled by the script); this change is backward incompatible on purpose. (Contributed by Victor Stinner in bpo-36721.)

f-strings support = for self-documenting expressions and debugging

Added an = specifier to f-strings. An f-string such as f'{expr=}' will expand to the text of the expression, an equal sign, then the representation of the evaluated expression. For example:

>>> user = 'eric_idle'
>>> member_since = date(1975, 7, 31)
>>> f'{user=} {member_since=}'
'user='eric_idle' member_since=datetime.date(1975, 7, 31)'

The usual f-string format specifiers allow more control over how the result of the expression is displayed:

>>>
>>> delta = date.today() - member_since
>>> f'{user=!s}  {delta.days=:,d}'
'user=eric_idle  delta.days=16,075'

The = specifier will display the whole expression so that calculations can be shown:

>>>
>>> print(f'{theta=}  {cos(radians(theta))=:.3f}')
theta=30  cos(radians(theta))=0.866

(Contributed by Eric V. Smith and Larry Hastings in bpo-36817.)

PEP 578: Python Runtime Audit Hooks

The PEP adds an Audit Hook and Verified Open Hook. Both are available from Python and native code, allowing applications and frameworks written in pure Python code to take advantage of extra notifications, while also allowing embedders or system administrators to deploy builds of Python where auditing is always enabled.

See PEP 578 for full details.

Vectorcall: a fast calling protocol for CPython

The "vectorcall" protocol is added to the Python/C API. It is meant to formalize existing optimizations which were already done for various classes. Any extension type implementing a callable can use this protocol.

This is currently provisional, the aim is to make it fully public in Python 3.9.

See PEP 590 for a full description.

(Contributed by Jeroen Demeyer and Mark Shannon in bpo-36974.)

Pickle protocol 5 with out-of-band data buffers

When pickle is used to transfer large data between Python processes in order to take advantage of multi-core or multi-machine processing, it is important to optimize the transfer by reducing memory copies, and possibly by applying custom techniques such as data-dependent compression.

The pickle protocol 5 introduces support for out-of-band buffers where PEP 3118-compatible data can be transmitted separately from the main pickle stream, at the discretion of the communication layer.

See PEP 574 for a full description.

(Contributed by Antoine Pitrou in bpo-36785.)

Improved Modules

ast

AST nodes now have end_lineno and end_col_offset attributes, which give the precise location of the end of the node. (This only applies to nodes that have lineno and col_offset attributes.)

New function ast.get_source_segment() returns the source code for a specific AST node.

(Contributed by Ivan Levkivskyi in bpo-33416.)

The ast.parse() function has some new flags:

  • type_comments=True causes it to return the text of PEP 484 and PEP 526 type comments associated with certain AST nodes;

  • mode='func_type' can be used to parse PEP 484 "signature type comments" (returned for function definition AST nodes);

  • feature_version=(3, N) allows specifying an earlier Python 3 version. (For example, feature_version=(3, 4) will treat async and await as non-reserved words.)

(Contributed by Guido van Rossum in bpo-35766.)

asyncio

Running python -m asyncio launches a natively async REPL. This allows rapid experimentation with code that has a top-level await. There is no longer a need to directly call asyncio.run() which would spawn a new event loop on every invocation:

$ python -m asyncio
asyncio REPL 3.8.0
Use 'await' directly instead of 'asyncio.run()'.
Type 'help', 'copyright', 'credits' or 'license' for more information.
>>> import asyncio
>>> await asyncio.sleep(10, result='hello')
hello

(Contributed by Yury Selivanov in bpo-37028.)

On Windows, the default event loop is now ProactorEventLoop. (Contributed by Victor Stinner in bpo-34687.)

ProactorEventLoop now also supports UDP. (Contributed by Adam Meily and Andrew Svetlov in bpo-29883.)

ProactorEventLoop can now be interrupted by KeyboardInterrupt ("CTRL+C"). (Contributed by Vladimir Matveev in bpo-23057.)

builtins

The compile() built-in has been improved to accept the ast.PyCF_ALLOW_TOP_LEVEL_AWAIT flag. With this new flag passed, compile() will allow top-level await, async for and async with constructs that are usually considered invalid syntax. Asynchronous code object marked with the CO_COROUTINE flag may then be returned. (Contributed by Matthias Bussonnier in bpo-34616)

collections

The _asdict() method for collections.namedtuple() now returns a dict instead of a collections.OrderedDict. This works because regular dicts have guaranteed ordering since Python 3.7. If the extra features of OrderedDict are required, the suggested remediation is to cast the result to the desired type: OrderedDict(nt._asdict()). (Contributed by Raymond Hettinger in bpo-35864.)

curses

Added a new variable holding structured version information for the underlying ncurses library: ncurses_version. (Contributed by Serhiy Storchaka in bpo-31680.)

ctypes

On Windows, CDLL and subclasses now accept a winmode parameter to specify flags for the underlying LoadLibraryEx call. The default flags are set to only load DLL dependencies from trusted locations, including the path where the DLL is stored (if a full or partial path is used to load the initial DLL) and paths added by add_dll_directory(). (Contributed by Steve Dower in bpo-36085.)

gc

get_objects() can now receive an optional generation parameter indicating a generation to get objects from. (Contributed by Pablo Galindo in bpo-36016.)

gettext

Added pgettext() and its variants. (Contributed by Franz Glasner, Éric Araujo, and Cheryl Sabella in bpo-2504.)

gzip

Added the mtime parameter to gzip.compress() for reproducible output. (Contributed by Guo Ci Teo in bpo-34898.)

A BadGzipFile exception is now raised instead of OSError for certain types of invalid or corrupt gzip files. (Contributed by Filip Gruszczyński, Michele Orrù, and Zackery Spytz in bpo-6584.)

idlelib and IDLE

Output over N lines (50 by default) is squeezed down to a button. N can be changed in the PyShell section of the General page of the Settings dialog. Fewer, but possibly extra long, lines can be squeezed by right clicking on the output. Squeezed output can be expanded in place by double-clicking the button or into the clipboard or a separate window by right-clicking the button. (Contributed by Tal Einat in bpo-1529353.)

Add "Run Customized" to the Run menu to run a module with customized settings. Any command line arguments entered are added to sys.argv. They also re-appear in the box for the next customized run. One can also suppress the normal Shell main module restart. (Contributed by Cheryl Sabella, Terry Jan Reedy, and others in bpo-5680 and bpo-37627.)

Add optional line numbers for IDLE editor windows. Windows open without line numbers unless set otherwise in the General tab of the configuration dialog. Line numbers for an existing window are shown and hidden in the Options menu. (Contributed by Tal Einat and Saimadhav Heblikar in bpo-17535.)

The changes above have been backported to 3.7 maintenance releases.

inspect

The inspect.getdoc() function can now find docstrings for __slots__ if that attribute is a dict where the values are docstrings. This provides documentation options similar to what we already have for property(), classmethod(), and staticmethod():

class AudioClip:
    __slots__ = {'bit_rate': 'expressed in kilohertz to one decimal place',
                 'duration': 'in seconds, rounded up to an integer'}
    def __init__(self, bit_rate, duration):
        self.bit_rate = round(bit_rate / 1000.0, 1)
        self.duration = ceil(duration)

(Contributed by Raymond Hettinger in bpo-36326.)

io

In development mode (-X env) and in debug build, the io.IOBase finalizer now logs the exception if the close() method fails. The exception is ignored silently by default in release build. (Contributed by Victor Stinner in bpo-18748.)

math

Added new function math.dist() for computing Euclidean distance between two points. (Contributed by Raymond Hettinger in bpo-33089.)

Expanded the math.hypot() function to handle multiple dimensions. Formerly, it only supported the 2-D case. (Contributed by Raymond Hettinger in bpo-33089.)

Added new function, math.prod(), as analogous function to sum() that returns the product of a 'start' value (default: 1) times an iterable of numbers:

>>>
>>> prior = 0.8
>>> likelihoods = [0.625, 0.84, 0.30]
>>> math.prod(likelihoods, start=prior)
0.126

(Contributed by Pablo Galindo in bpo-35606.)

Added new function math.isqrt() for computing integer square roots. (Contributed by Mark Dickinson in bpo-36887.)

The function math.factorial() no longer accepts arguments that are not int-like. (Contributed by Pablo Galindo in bpo-33083.)

mmap

The mmap.mmap class now has an madvise() method to access the madvise() system call. (Contributed by Zackery Spytz in bpo-32941.)

os

Added new function add_dll_directory() on Windows for providing additional search paths for native dependencies when importing extension modules or loading DLLs using ctypes. (Contributed by Steve Dower in bpo-36085.)

A new os.memfd_create() function was added to wrap the memfd_create() syscall. (Contributed by Zackery Spytz and Christian Heimes in bpo-26836.)

On Windows, much of the manual logic for handling reparse points (including symlinks and directory junctions) has been delegated to the operating system. Specifically, os.stat() will now traverse anything supported by the operating system, while os.lstat() will only open reparse points that identify as "name surrogates" while others are opened as for os.stat(). In all cases, stat_result.st_mode will only have S_IFLNK set for symbolic links and not other kinds of reparse points. To identify other kinds of reparse point, check the new stat_result.st_reparse_tag attribute.

On Windows, os.readlink() is now able to read directory junctions. Note that islink() will return False for directory junctions, and so code that checks islink first will continue to treat junctions as directories, while code that handles errors from os.readlink() may now treat junctions as links.

(Contributed by Steve Dower in bpo-37834.)

pickle

Reduction methods can now include a 6th item in the tuple they return. This item should specify a custom state-setting method that's called instead of the regular __setstate__ method. (Contributed by Pierre Glaser and Olivier Grisel in bpo-35900.)

pickle extensions subclassing the C-optimized Pickler can now override the pickling logic of functions and classes by defining the special reducer_override() method. (Contributed by Pierre Glaser and Olivier Grisel in bpo-35900.)

plistlib

Added new plistlib.UID and enabled support for reading and writing NSKeyedArchiver-encoded binary plists. (Contributed by Jon Janzen in bpo-26707.)

shutil

shutil.copytree() now accepts a new dirs_exist_ok keyword argument. (Contributed by Josh Bronson in bpo-20849.)

shutil.make_archive() now defaults to the modern pax (POSIX.1-2001) format for new archives to improve portability and standards conformance, inherited from the corresponding change to the tarfile module. (Contributed by C.A.M. Gerlach in bpo-30661.)

shutil.rmtree() on Windows now removes directory junctions without recursively removing their contents first. (Contributed by Steve Dower in bpo-37834.)

statistics

Added statistics.fmean() as a faster, floating point variant of statistics.mean(). (Contributed by Raymond Hettinger and Steven D'Aprano in bpo-35904.)

Added statistics.geometric_mean() (Contributed by Raymond Hettinger in bpo-27181.)

Added statistics.multimode() that returns a list of the most common values. (Contributed by Raymond Hettinger in bpo-35892.)

Added statistics.quantiles() that divides data or a distribution in to equiprobable intervals (e.g. quartiles, deciles, or percentiles). (Contributed by Raymond Hettinger in bpo-36546.)

Added statistics.NormalDist, a tool for creating and manipulating normal distributions of a random variable. (Contributed by Raymond Hettinger in bpo-36018.)

>>>
>>> temperature_feb = NormalDist.from_samples([4, 12, -3, 2, 7, 14])
>>> temperature_feb.mean
6.0
>>> temperature_feb.stdev
6.356099432828281
>>> temperature_feb.cdf(3)            # Chance of being under 3 degrees
0.3184678262814532
>>> # Relative chance of being 7 degrees versus 10 degrees
>>> temperature_feb.pdf(7) / temperature_feb.pdf(10)
1.2039930378537762
>>> el_niño = NormalDist(4, 2.5)
>>> temperature_feb += el_niño        # Add in a climate effect
>>> temperature_feb
NormalDist(mu=10.0, sigma=6.830080526611674)
>>> temperature_feb * (9/5) + 32      # Convert to Fahrenheit
NormalDist(mu=50.0, sigma=12.294144947901014)
>>> temperature_feb.samples(3)        # Generate random samples
[7.672102882379219, 12.000027119750287, 4.647488369766392]

sys

Add new sys.unraisablehook() function which can be overridden to control how "unraisable exceptions" are handled. It is called when an exception has occurred but there is no way for Python to handle it. For example, when a destructor raises an exception or during garbage collection (gc.collect()). (Contributed by Victor Stinner in bpo-36829.)

tarfile

The tarfile module now defaults to the modern pax (POSIX.1-2001) format for new archives, instead of the previous GNU-specific one. This improves cross-platform portability with a consistent encoding (UTF-8) in a standardized and extensible format, and offers several other benefits. (Contributed by C.A.M. Gerlach in bpo-36268.)

tokenize

The tokenize module now implicitly emits a NEWLINE token when provided with input that does not have a trailing new line. This behavior now matches what the C tokenizer does internally. (Contributed by Ammar Askar in bpo-33899.)

tkinter

Added methods selection_from(), selection_present(), selection_range() and selection_to() in the tkinter.Spinbox class. (Contributed by Juliette Monsel in bpo-34829.)

Added method moveto() in the tkinter.Canvas class. (Contributed by Juliette Monsel in bpo-23831.)

The tkinter.PhotoImage class now has transparency_get() and transparency_set() methods. (Contributed by Zackery Spytz in bpo-25451.)

typing

The typing module incorporates several new features:

  • A dictionary type with per-key types. See PEP 589 and typing.TypedDict. TypedDict uses only string keys. By default, every key is required to be present. Specify "total=False" to allow keys to be optional:

    class Location(TypedDict, total=False):
        lat_long: tuple
        grid_square: str
        xy_coordinate: tuple
    
  • Literal types. See PEP 586 and typing.Literal. Literal types indicate that a parameter or return value is constrained to one or more specific literal values:

    def get_status(port: int) -> Literal['connected', 'disconnected']:
        ...
    
  • "Final" variables, functions, methods and classes. See PEP 591, typing.Final and typing.final(). The final qualifier instructs a static type checker to restrict subclassing, overriding, or reassignment:

    pi: Final[float] = 3.1415926536
    
  • Protocol definitions. See PEP 544, typing.Protocol and typing.runtime_checkable(). Simple ABCs like typing.SupportsInt are now Protocol subclasses.

  • New protocol class typing.SupportsIndex.

  • New functions typing.get_origin() and typing.get_args().

unicodedata

The unicodedata module has been upgraded to use the Unicode 12.1.0 release.

New function is_normalized() can be used to verify a string is in a specific normal form, often much faster than by actually normalizing the string. (Contributed by Max Belanger, David Euresti, and Greg Price in bpo-32285 and bpo-37966).

unittest

Added AsyncMock to support an asynchronous version of Mock. Appropriate new assert functions for testing have been added as well. (Contributed by Lisa Roach in bpo-26467).

Added addModuleCleanup() and addClassCleanup() to unittest to support cleanups for setUpModule() and setUpClass(). (Contributed by Lisa Roach in bpo-24412.)

Several mock assert functions now also print a list of actual calls upon failure. (Contributed by Petter Strandmark in bpo-35047.)

unittest module gained support for coroutines to be used as test cases with unittest.IsolatedAsyncioTestCase. (Contributed by Andrew Svetlov in bpo-32972.)

Example:

import unittest
class TestRequest(unittest.IsolatedAsyncioTestCase):
    async def asyncSetUp(self):
        self.connection = await AsyncConnection()
    async def test_get(self):
        response = await self.connection.get('https://example.com')
        self.assertEqual(response.status_code, 200)
    async def asyncTearDown(self):
        await self.connection.close()
if __name__ == '__main__':
    unittest.main()

venv

venv now includes an Activate.ps1 script on all platforms for activating virtual environments under PowerShell Core 6.1. (Contributed by Brett Cannon in bpo-32718.)

weakref

The proxy objects returned by weakref.proxy() now support the matrix multiplication operators @ and @= in addition to the other numeric operators. (Contributed by Mark Dickinson in bpo-36669.)

xml

As mitigation against DTD and external entity retrieval, the xml.dom.minidom and xml.sax modules no longer process external entities by default. (Contributed by Christian Heimes in bpo-17239.)

The .find*() methods in the xml.etree.ElementTree module support wildcard searches like {*}tag which ignores the namespace and {namespace}* which returns all tags in the given namespace. (Contributed by Stefan Behnel in bpo-28238.)

The xml.etree.ElementTree module provides a new function –xml.etree.ElementTree.canonicalize() that implements C14N 2.0. (Contributed by Stefan Behnel in bpo-13611.)

The target object of xml.etree.ElementTree.XMLParser can receive namespace declaration events through the new callback methods start_ns() and end_ns(). Additionally, the xml.etree.ElementTree.TreeBuilder target can be configured to process events about comments and processing instructions to include them in the generated tree. (Contributed by Stefan Behnel in bpo-36676 and bpo-36673.)




All Comments: [-] | anchor

omginternets(3938) 4 days ago [-]

I stepped away from Python for about a year, and now I'm coming back to it. I hardly recognize the language. I'm not happy about this at all.

I don't really have a point, except that Python 3 feels like a moving target.

probably_wrong(3630) 4 days ago [-]

I feel the same way as you do. For me, which version of an interpreter I'm using should be the kind of issue I only need to worry when solving extremely specific, deep-level problems. Python 3+ breaks this pact too often for my taste.

Considering this f-string example taken from another announcement:

   f'Diameter {(diam := 2 * r)} gives circumference {math.pi * diam:.2f}'
This is valid Python 3.8, but it's not valid in Python 3.7 (no walrus operator). And removing the walrus operator still doesn't work in Python 3.5 (no f-strings). On top of that, other comments already mention how f-strings have lots of weird corner cases anyway.

The entire point of Python in my circle of friends was that it made programming easy. Instead, I feel more and more in need of those 'It works in my machine!' stickers. And good luck solving these issues if you are not a full-time programmer...

ameliaquining(10000) 5 days ago [-]

Anyone else most excited about PYTHONCACHEPREFIX? :-P

rcfox(1774) 5 days ago [-]

Not sure about most excited, but I am looking forward to setting PYTHONCACHEPREFIX to a location that isn't in a volume mounted into my Docker container for development.

vikinghckr(10000) 5 days ago [-]

Disappointing that Python sill has no support for a sorted container in its standard library, akin to C++ `set` or `map` classes.

carapace(2884) 5 days ago [-]

https://docs.python.org/3.0/library/bisect.html

> This module provides support for maintaining a list in sorted order without having to sort the list after each insertion.

joshuamorton(10000) 5 days ago [-]

What's the value of a sorted map over an ordered map?

I don't think I've ever cared to iterate over map values based on the alphanumeric ordering of their keys.

craftyguy(3212) 5 days ago [-]

But it does have a method to sort containers in its standard library, `sorted`, that C++ doesn't have. And it's trivial to use it to sort lists, sets, and dicts...

metalliqaz(4187) 5 days ago [-]

This is the release that contains the controversial assignment expressions, which sparked the debate that convinced GvR to quit as BDFL.

They didn't select my preferred syntax, but I'm still looking forward to using assignment expressions for testing my re.match() objects.

I haven't used 3.8.0 yet but I hope its a good one because 2020 is the year that 2.7 dies and there will be a lot of people switching.

diminoten(10000) 5 days ago [-]

Because of the luddites at RedHat, Python2.7 isn't actually dead until 2024. It's infuriating.

https://access.redhat.com/solutions/4455511

Rebelgecko(4177) 5 days ago [-]

>They didn't select my preferred syntax

I don't write much python so there's probably something obvious I'm missing, but I don't see why they didn't use '='. Is there some significant difference between assignment expressions and assignment statements that makes it worth having distinct syntax?

DonaldPShimoda(10000) 4 days ago [-]

> I'm still looking forward to using assignment expressions for testing my re.match() objects.

The fact that this is the go-to example that everybody is using in justifying the introduction of the assignment expression convinces me that the real problem lies with the re module's API.

(To be clear: I will also be using assignment expressions for this case, but I don't think assignment expressions are really in line with the overall design of Python.)

kbd(3869) 5 days ago [-]

I'm stoked about the walrus operator. Ever since I heard it was being added I've grumbled when writing code that would have been clearer with it.

Of course I have to ask, what was your preferred syntax?

ohazi(3335) 5 days ago [-]

> The typing module incorporates several new features:

> A dictionary type with per-key types.

Ah, I've been waiting for this. I've been able to use Python's optional types pretty much everywhere except for dictionaries that are used as pseudo-objects, which is a fairly common pattern in Python. This should patch that hole nicely.

j88439h84(10000) 5 days ago [-]

IMO A better fix is to use http://attrs.org or dataclasses to replace the dicts entirely.

ledauphin(10000) 5 days ago [-]

i am actually pretty sad that TypedDict has made it out of typing_extensions in its current state.

It was a major missed opportunity to provide a properly duck-typed Dict type, which by definition would allow untyped key-value pairs to be added to a 'partially-typed' dictionary.

Gradual typing in general is a massive win for many kinds of real-world problem solving, but when you make it as hard as Python has to introduce partial types to a plain data object, you're leaving a lot of developers out in the cold.

I love MyPy and the static type hints since 3.6, but structural subtyping is superior and so obviously more Pythonic than nominal, yet support for structural subtyping keeps lagging behind.

Pxtl(10000) 5 days ago [-]

Ugh, I hate assignment expressions. I liked that they were missing from Python.

I've been coding in algolesque languages for 20 years and hiding assignments inside of expressions instead of putting them on the left like a statement has always tripped me up.

jtdev(4177) 5 days ago [-]

But, but, but... now you don't need to write that extra line of code! It's going to make everything sooooo much better, code will practically write itself now.

I'll just leave this here:

"There should be one—and preferably only one—obvious way to do it."

heyflyguy(10000) 5 days ago [-]

man do I wish they would make multiprocessing easier

mintplant(1669) 5 days ago [-]

Subinterpreters are coming in 3.9, which should fill most of the same roles as multiprocessing but without the complexities and edge cases of multiple real OS processes.

https://www.python.org/dev/peps/pep-0554/

svnpenn(10000) 5 days ago [-]

> In this example, the assignment expression helps avoid calling len() twice:

    if (n := len(a)) > 10:
       print(f'List is too long ({n} elements, expected <= 10)')
Um, no it doesnt?:

    a1 = [10, 20, 30]
    n1 = len(a1)
    if n1 > 2:
       print(f'{n1} is greater than two')
hermitdev(10000) 5 days ago [-]

Yes, but your example is less efficient than the new code.

Remember: every variable is an assignment into a dict, and every lookup a query into a dict. Reducing name lookups and assignments can yield good speedup in tight loops. For instance, caching os.path.join, os.path.split into local names can significantly speed up tight loops iterating over a filesystem. For example, os.path.split is potentially 5 dictionary lookups. Checking locals, nonlocals and globals for os. Then another to find path, and a final one for split. And this happens at runtime, for every invocation.

adjkant(4199) 5 days ago [-]

I'm not a huge fan of the := operator for Python for clarity and single one clear way reasons, but the draw is the saved line of code here that is the 'help'

CivBase(10000) 5 days ago [-]

As a developer who has primarily developed applications in Python for his entire professional career, I can't say I'm especially excited about any of the 'headlining' features of 3.8.

The 'walrus operator' will occasionally be useful, but I doubt I will find many effective uses for it. Same with the forced positional/keyword arguments and the 'self-documenting' f-string expressions. Even when they have a use, it's usually just to save one line of code or a few extra characters.

The labeled breaks and continues proposed in PEP-3136 [0] also wouldn't be used very frequently, but they would at least eliminate multiple lines of code and reduce complexity.

PEP-3136 was rejected because 'code so complicated to require this feature is very rare'. I can understand a stance like that. Over complicating a language with rarely-used features can definitely create problems. I just don't see why the three 'headline' features I mentioned are any different.

[0]: https://www.python.org/dev/peps/pep-3136/

thayne(4220) 5 days ago [-]

There have been many times when I've wished python had something like the walrus operator.

In particular, without the walrus operator you can't write an if-elif chain of regex comparisons. The best way to that that I've been able to find is with a bunch of nested if-else statements.

kccqzy(3070) 5 days ago [-]

PEP-3136 was proposed and discussed more than 10 years ago in the 2->3 process. The python community was very different then. As an example, Python 3 was also the language that removed the function argument tuple packing, to which my reaction is basically WTF.

thanatropism(3955) 4 days ago [-]

> Over complicating a language with rarely-used features can definitely create problems.

Agreed.

There's a number of (full) languages that compile to the Python AST [0] (I'm especially fond of http://hylang.org), but they're very different from Python. It would be interesting to see smaller variations of standard Python implemented in an interoperable way like these languages do.

[0] https://github.com/vindarel/languages-that-compile-to-python

kolinko(3286) 5 days ago [-]

I actually found an immense use for Walrus - used it to hack Python into doing pattern matching that is way more readable than without it:

https://github.com/eveem-org/panoramix

(source code for Eveem.org, which is arguably the best decompiler for Ethereum smart contracts out there. you can see a lot of pattern matching in pano/simplify.py , and I found no way to do it without extending the language/walrus while maintaining the readability)

wodenokoto(4021) 4 days ago [-]

> Same with the forced positional/keyword arguments

I also thought, why bother at first until I learned this is already a feature in python, but only for c-functions. So of course it makes sense to level the playing field.

cutler(4217) 4 days ago [-]

So Python is finally waking-up to the advantage of everything being an expression as in Ruby and Clojure.

geofft(3275) 5 days ago [-]

> The 'walrus operator' will occasionally be useful, but I doubt I will find many effective uses for it.

The primary one I want is

    if m := re.match(...):
        print(m.group(1))
and

    while s := network_service.read():
        process(s)
both of which are both clearer and less error-prone than their non-walrus variants.

The other one that I would have found useful an hour ago is in interactive exploration with comprehensions. I frequently take a look at some data with [i for i in data if i[something].something...], and being able to quickly give a name to something in my conditional, as in {i: x for i in data if x := i[something]}, helps maintain the focus on data and not syntax. Obviously it will get rewritten to have clearer variable names, at the least, when it gets committed as real code, and almost certainly rewritten to be a normal non-comprehension for loop.

Like comprehensions, I expect the walrus operator to be valuable when used occasionally, and annoying if overused. There's no real language-level solution to bad taste. In Python 3 you can now do [print(i) for i in...], and I occasionally do at the REPL, but you shouldn't do it in real code and that's not an argument against the language supporting comprehensions.

emmelaich(4028) 5 days ago [-]

As someone who has used Python for over 10 years and C, Perl, etc for over 20, I've been craving the walrus operator forever.

I love the walrus operator with a love that is unholy.

paulddraper(4026) 5 days ago [-]

> As a developer who has primarily developed applications in Python for his entire professional career, I can't say I'm especially excited about any of the 'headlining' features of 3.8.

Python is a fairly old, mature language.

What features would you have been especially excited about?

madhadron(4097) 4 days ago [-]

The audit hooks are going to allow some really interesting things, especially around security controls on executing Python code. There is hope for sandboxed Python execution!

peterwwillis(2589) 5 days ago [-]

Perl devs have been using the walrus expression for decades, but we just called it 'assigning a variable with local scope in an expression'.

  $ perl
    $a = 'foo';
    if ( my $a = 'bar' ) { print '$a\n' }
    print '$a\n'
  bar
  foo
afiori(3894) 4 days ago [-]

In my opinion it was a mistake to reject the labelled break PEP

ageofwant(3853) 5 days ago [-]

I take this as a positive sign of Python's maturity.

trymas(2536) 4 days ago [-]

IMHO walrus operator goes against the zen of python.

https://www.python.org/dev/peps/pep-0572/#differences-betwee...

https://www.python.org/dev/peps/pep-0572/#relative-precedenc...

Even examples of the spec shows how unintuitive and 'unpythonic' this is. Explicit is better than implicit.

IMHO adding features to the language to save 1 line of code for 10% of cases when you need it (I agree that there's occasional case when walrus will save you more than 1 line) is just bloat.

I am not a big proponent of Go, because it has its own flaws, though language is indeed very simple and creators of the language try to leave it simple.

IMO Python was very readable, super simple, intuitive and should stay that way, though recent releases show that Python is giving in for the feature bloat.

EDIT:

> Try to limit use of the walrus operator to clean cases that reduce complexity and improve readability.

Facepalm.

Iv(10000) 4 days ago [-]

Not all releases have to be ground breaking, especially in a mature and stable language. Cleaning up things, putting other things in place for major releases, that's commendable too.

I did not follow all the conversation around the typing system but isn't the whole point of it to propose optimizations in the future? I find it exciting

pcr910303(969) 5 days ago [-]

I like the additions of the f-strings and walrus operator, but I find myself wishing a breaking-release removing the old features that the new features covers. Python's philosophy was to have one way to do something, but the current situation of Python is very inconsistent. Python 3 has like 4~5 ways to format strings, and due to the addition of the walrus operator, we have (I understand the differences between := and = but) two different syntaxes for variable declaration/assignment.

I understand that Python can't break all kinds of code (as the 2->3 conversion is still a pain), but still I imagine a Python-esque language without all the warts that Python have with it's 'organic' growth.

jakear(4159) 5 days ago [-]

That's a good use case for a linter. Ban outdated constructs in your code, but still allow you to depend on things that use them. Beats another 2->3 split again.

eru(1532) 4 days ago [-]

Perhaps Python itself could worn you that certain constructs are deprecated and will be removed in future versions.

And you could 'import something from future' to opt-in to making those warnings errors right now.

amanzi(3360) 5 days ago [-]

Real Python have a great post out today going through most of the changes: https://realpython.com/python38-new-features/

They have a lot of code samples and examples about how and when to use the new features.Personally, I love the new debugging support in F strings.

rapfaria(10000) 5 days ago [-]

Too bad I'm losing good articles like this one because of those patronizing, click-baity newsletters that they started sending out a year or two ago.

molteanu(2901) 5 days ago [-]

Is Python now becoming the new C++?

davb(3948) 4 days ago [-]

This is fair criticism. I've worked with Python for many years now, and one of the things that attracted me was that there was always a fsirly static 'Pythonic' way to do things. The language has now grown so much that 80% of the people only use 20% of the language, but not always the same 20%, making it difficult to read other people's code. And all the syntactic changes contribute to fragmentation (not every project can have its Python interpreter upgraded regularly).

There's something to be said for a more stable, if less elegant, language.

probably_wrong(3630) 4 days ago [-]

I would argue that it's becoming the new Perl. I mean, look at this:

   f'Diameter {(diam := 2 * r)} gives circumference {math.pi * diam:.2f}'
If this is not write-only code, I don't know what is.
supakeen(4170) 5 days ago [-]

Arguably one of the hotter debated functions is assignment-as-expression through the := or walrus operator.

Quite happy with the new SyntaxWarning for identity comparison on literals and missing commas :)

Especially neat is also the `python -m asyncio` shell which allows you to run top-level awaits in your repl instead of needing to start a new event loop each time!

j88439h84(10000) 5 days ago [-]

IPython has supported top-level async for a while too, plus many other features!

airstrike(2132) 5 days ago [-]

The expansion of f-strings is a welcome addition. The more I use them, the happier I am that they exist

https://docs.python.org/3/whatsnew/3.8.html#f-strings-suppor...

heavenlyblue(10000) 5 days ago [-]

One thing I vastly dislike with them is that they are still expanded in logging, while they are not expanded when logging is using 'the old way'.

skunkworker(3972) 5 days ago [-]

This makes clean string interpolation so much easier to do, especially for print statements. It's almost hard to use python < 3.6 now because of them.

nostrademons(1712) 5 days ago [-]

I saw that and was like 'Oh, that'd be a handy feature for a lot of other programming languages too.' But the more I think about it, the more I'd rather have a feature that takes a list of expressions and converts it into a dict where the keys are the expression text and the values are the values of those expressions. Basically, syntactic sugar for this:

  { k: eval(k) for k in ('theta', 'delta.days', 'cos(radians(theta))') }
This would trivially subsume the f'{user=}' syntax for the example given: just print out the dictionary. But it'd also be useful for: filling template dictionaries; printing out status pages for HTTP webservers; returning multiple variables from a function; flattening out complex data structures; creating dispatch tables out of local functions.

You could even have a syntax like locals('theta', 'delta.days') and keep it familiar.

bhl(3982) 5 days ago [-]

Why didn't Python ship with the opposite functionality as well? Parsing instead of formatting. Given a string and a format string, return a list of variables (or a dictionary).

archie2(10000) 5 days ago [-]

I don't mind the walrus operator, but holy cow the 'positional operators' syntax is completely unintelligble...what the hell were they thinking?!

toyg(3652) 4 days ago [-]

It really is not supposed to be used unless you are wrapping a C library. Or at least that's the original reasoning, I think. Let's hope it doesn't get abused.

robocat(4203) 5 days ago [-]

  def f(a, b, /, c, d, *, e, f):
Wow: IMHO that is a very ugly syntax to define a fn with 6 parameters, although it looks to be a path dependency on previous decisions (*, and CPython ,/) and clearly there was much discussion of the need for the functionality and the compromises: https://www.python.org/dev/peps/pep-0570/

It amazes me to see how certain features make it into languages via community/committee (octal numbers in JavaScript - arrgh!).

One thing I find really difficult to deal with in languages is the overloading of different syntactic(edit:semantic) usages of different symbols, which itself is a result of limiting ourselves to the ASCII symbols that can be typed. I don't recall written mathematics having the issue badly (although I haven't had to write any for a long time!).

BurningFrog(10000) 5 days ago [-]

> which itself is a result of limiting ourselves to the ASCII symbols that can be typed

I think we're ready for programming languages using some visually good Unicode characters, instead of overloading `[]{}[email protected]#$%^&*()-_/` for everything!

smitty1e(10000) 5 days ago [-]

The point of that syntax is to offer more control for how functions, particularly in C-extensions, are invoked.

It will be a 'good problem' to have the skillz to make this useful.

jonnycomputer(4213) 5 days ago [-]

What bothers me is that the / is separated by commas, which makes it seem like the signature is one more than it actually is (even if that is not the case technically). I haven't looked too carefully here though.

bitcurious(3740) 5 days ago [-]

I'd argue that written maths is worse with its overloading of the Greek alphabet. Although in the case of maths I'd argue that it's a result of how slowly we write, with the programming equivalent usually being a variable name (autocompleteable).

chubot(3726) 5 days ago [-]

Yes, 'path dependence' is a good way to describe it. (And Python has been my favorite language for 16+ years now)

For https://www.oilshell.org/ , which has Python/JS-like functions, I chose to use Julia's function signature design, which is as expressive as Python's, but significantly simpler in both syntax and implementation:

Manual:

https://docs.julialang.org/en/v1/manual/functions/index.html...

Comparison:

https://medium.com/@Jernfrost/function-arguments-in-julia-an...

Basically they make positional vs. named and required vs. optional ORTHOGONAL dimensions. Python originally conflated the two things, and now they're teasing them apart with keyword-only and positional-only params.

A semicolon in the signature separates positional and named arguments. So you can have:

    func f(p1, p2=0, ...args ; n1, n2=0, ...kwargs)
So p2 is an optional positional argument, while n1 is a required named argument.

And then you don't need * and star star -- you can just use ... for both kinds of 'splats'. At the call site you only need ; if you're using a named arg / kwargs splat.

----

Aside from all the numeric use cases, which look great, Julia has a bunch of good ideas in dynamic language design.

The multiline strings in Julia appear to strip leading space in a way that's better than both Python multiline strings and here docs in shell.

Also Julia has had shell-like 'f-strings' since day one (or at least version 1 which was pretty recent). Python has had at least 4 versions of string interpolation before they realized that shell got it right 40 years ago :)

mintplant(1669) 5 days ago [-]

Buried in the notes for the `typing` module:

> "Final" variables, functions, methods and classes. See PEP 591, typing.Final and typing.final(). The final qualifier instructs a static type checker to restrict subclassing, overriding, or reassignment:

> pi: Final[float] = 3.1415926536

As I understand it, this means Python now has a way of marking variables as constant (though it doesn't propagate into the underlying values as in the case of C++'s `const`).

The equivalent Java:

    final float pi = 3.1415926536;
spott(10000) 4 days ago [-]

The thing is, it is only enforced in mypy... code that modifies a Final object (pi = 3 for example) later on will run fine.

xapata(10000) 5 days ago [-]

I don't see the point of `final` without an optimizing compiler. Name mangling is sufficient for stashing references to avoid accidental side-effects of overriding.

Xophmeister(3640) 4 days ago [-]

This got missed from the release announcement, but now there's `functools.singledispatchmethod`,[1] as the class method sibling to `functools.singledispatch`.[2] This allows you to overload the implementation of a function (and now a method) based on the type of its first argument. This saves you writing code like:

    def foo(bar):
        if isinstance(bar, Quux):
            # Treat bar as a Quux
        elif isinstance(bar, Xyzzy):
            # Treat bar as an Xyzzy
        # etc.
I understand runtime type checking like that is considered a bit of a Python antipattern. With `singledispatch`, you can do this instead:

    @singledispatch
    def foo(bar:Quux):
        # Quux implementation
    @foo.register
    def _(bar:Xyzzy):
        # Xyzzy implementation
With `singledispatchmethod`, you can now also do this to class methods, where the type of the first non-self/class is used by the interpreter to check the type, based on its annotation (or using the argument to its `register` method). You could mimic this behaviour using `singledispatch` in your constructor, but this syntax is much nicer.

[1] https://docs.python.org/3/library/functools.html#functools.s...

[2] https://docs.python.org/3/library/functools.html#functools.s...

hirowan(10000) 4 days ago [-]

The issue with this and `singledispatch` is that they no longer support pseudo-types from the `typing` module [1] so you can't use them with containers of type `x`, e.g. `List[str]`, or protocols like `Sequence`.

[1] https://bugs.python.org/issue34498

bouncing(10000) 4 days ago [-]

I can definitely imagine some places where this replaces type-checking, but it still seems like a bit of an unfortunate anti-pattern to me, since it's really a sort of C/C++ style function prototype match.

My immediate thought is that it's going to be hard for PyCharm to reliably point me to a function definition.

gimboland(10000) 4 days ago [-]

Nice. I missed this feature only a few weeks ago. Good to know it's landed!

xamuel(4198) 4 days ago [-]

I don't understand why anyone would want singledispatch. Instead of having the function defined in one place where you can look it up, now the function is potentially scattered all over the place. (I'm not talking hypothetically. I've had the 'pleasure' of working on a codebase where different singledispatch cases of the same function were defined in different files!)

bratao(2491) 5 days ago [-]

I do not know exactly what caused this, but my internal project test-suite got a much appreciated 5% speedup! Thank you Python team!

The_rationalist(10000) 5 days ago [-]

I you care about performance, maybe you should try pypy (the alternative python compiler)

Alex3917(431) 5 days ago [-]

Nice! My test suite got around 20% faster when going from 3.6 to 3.7. That's almost like buying a whole new computer.

santiagobasulto(4011) 5 days ago [-]

Shared Memory is the star of this release IMHO.

diminoten(10000) 5 days ago [-]

Is shared memory one of those things that I should try to access/use immediately, or wait for someone to write a wrapper library around, due to the large number of edge cases/strangeness that might occur?

ram_rar(10000) 5 days ago [-]

Does the python community care about concurrency at all? I havent seen anything new in terms of concurrency in a while. I might be wrong about this, but walrus operator seems like a gateway to writing obfuscated perlesque code.

quietbritishjim(10000) 5 days ago [-]

There were some changes to asyncio in this release but they were pulled at the last moment so they can better sign with Turio.

aeyes(10000) 5 days ago [-]

3.8 brings shared memory to multiprocessing: https://docs.python.org/3/library/multiprocessing.shared_mem...





Historical Discussions: Making the Tokio scheduler 10x faster (October 14, 2019: 808 points)

(809) Making the Tokio scheduler 10x faster

809 points 5 days ago by steveklabnik in 47th position

tokio.rs | Estimated reading time – 38 minutes | comments | anchor

Making the Tokio scheduler 10x faster

October 13, 2019

We've been hard at work on the next major revision of Tokio, Rust's asynchronous runtime. Today, a complete rewrite of the scheduler has been submitted as a pull request. The result is huge performance and latency improvements. Some benchmarks saw a 10x speed up! It is always unclear how much these kinds of improvements impact "full stack" use cases, so we've also tested how these scheduler improvements impacted use cases like Hyper and Tonic (spoiler: it's really good).

In preparation for working on the new scheduler, I spent time searching for resources on scheduler implementations. Besides existing implementations, I did not find much. I also found the source of existing implementations difficult to navigate. To remedy this, I tried to keep Tokio's new scheduler implementation as clean as possible. I also am writing this detailed article on implementing the scheduler in hope that others in similar positions find it useful.

The article starts with a high level overview of scheduler design, including work-stealing schedulers. It then gets into the details of specific optimizations made in the new Tokio scheduler.

The optimizations covered are:

The major theme is "reduce." After all, there is no code faster than no code!

The article also covers testing the new scheduler. Writing correct, concurrent, lock-free code is really hard. It is better to be slow and correct than fast and buggy, especially if those bugs relate to memory safety. The best option, however, is to be fast and correct, so we wrote loom, a tool for testing concurrency.

Before jumping in, I want to extend some gratitude.

  • @withoutboats and others who worked on Rust's async / await feature. You did a great job. It is a killer feature.
  • @cramertj and others who designed std::task. It is a huge improvement compared to what we had before. Also, a great job there.
  • Buoyant, the makers of Linkerd, and more importantly my employer. Thanks for letting me spend so much time on this work. Readers who are in need of a service mesh, check out Linkerd. It will soon include all the goodness discussed in this article.
  • Go for having such a good scheduler implementation.

Grab a cup of coffee and get yourselves comfortable. This is going to be a long article.

Schedulers, how do they work?

The role of a scheduler is to schedule work. An application is broken up into units of work, which we will call tasks. A task is runnable when it can make progress, and is no longer runnable (or idle) when it is blocked on an external resource. Tasks are independent in that any number of runnable tasks can execute concurrently. The scheduler is responsible for executing tasks in the running state until they transition back to idle. Executing a task implies assigning CPU time – a global resource – to the task.

The article discusses user-space schedulers, i.e., schedulers that run on top of operating system threads (which, in turn, are powered by a kernel land scheduler). The Tokio scheduler executes Rust futures, which can be though of as "asynchronous green threads". This is the M:N threading pattern where many user land tasks are multiplexed on a few operating system threads.

There are many different ways of modeling a scheduler, each with pros and cons. At the most basic level, the scheduler can be modeled as a run queue and a processor that drains the queue. A processor is a bit of code that runs on a thread. In pseudocode, it does:

while let Some(task) = self.queue.pop() {
    task.run();
}

When a task becomes runnable, it is inserted into the run queue.

While it is possible to design a system where resources, tasks, and the processor all exist on a single thread, Tokio chooses to use multiple threads. We live in a world where computers come with many CPUs. Designing a single-threaded scheduler would result in hardware underutilization. We want to use all of our CPUs. There are a few ways to do this:

  • One global run queue, many processors.
  • Many processors, each with their own run queue.

One queue, many processors

In this model, there is a single, global, run queue. When tasks become runnable, they are pushed onto the tail of the queue. There are multiple processors, each running on a separate thread. Each processor pops from the head of the queue, blocking the thread if no task is available.

The run queue must support both multiple producers and multiple consumers. The commonly used algorithm is an intrusive linked list. Intrusive implies that the task structure includes a pointer to the next task in the run queue instead of wrapping the task with a linked list node. This way, allocations are avoided for push and pop operations. It is possible to use a lock-free push operation but popping requires a mutex to coordinate consumers.

However, in practice the overhead needed to correctly avoid locks is greater than just using a mutex.

This approach is commonly used when implementing a general-purpose thread pool, as it has several advantages:

  • Tasks are scheduled fairly.
  • The implementation is relatively simple. An off-the-shelf queue can be paired with the processor loop sketched above.

A quick note on fairness. Scheduling fairness means that tasks are scheduled first-in, first-out. Tasks that are transitioned to running first are executed first. General purpose schedulers try to be fair, but there are use cases, like parallelizing a computation using fork-join, where the only important factor is how fast the end result is computed and not the fairness of each individual sub task.

This scheduler model has a downside. All processors contend on the head of the queue. For general-purpose thread pools, this usually is not a deal breaker. The amount of time processors spend executing the task far outweighs the amount of time spent popping the task from the run queue. When tasks execute for a long period of time, queue contention is reduced. However, Rust's asynchronous tasks are expected to take very little time executing when popped from the run queue. In this scenario, the overhead from contending on the queue becomes significant.

Concurrency and mechanical sympathy.

To get the best runtime performance out of a program, we must design it to take advantage of how the hardware operates. The term "mechanical sympathy" was applied to software by Martin Thompson (whose blog, while no longer updated, still contains much relevant knowledge).

A detailed discussion of how modern hardware handles concurrency is out of scope for this article. At a 10,000 foot view, hardware is gaining more performance not by going faster, but by making more CPU cores available to an application (my laptop has 6!). Each core can perform large amounts of computation in tiny periods of time. Relatively speaking, actions like cache and memory accesses takes much longer. Therefore, for applications to be fast, we must maximize the amount of CPU instructions per memory access. While the compiler can do a lot of work for us, as developers, we do need to think about things like struct layout and memory access patterns.

When it comes to concurrent threads, behavior is pretty similar to a single thread until concurrent mutations happen to the same cache line or sequential consistency is requested. Then, the CPU's cache coherence protocol will have to start working to ensure that each CPU's cache stays up to date.

This is all to say the obvious: avoid cross thread synchronization as much as possible because it is slow.

Many processors, each with their own run queue

Another way to model a scheduler is to use multiple single-threaded schedulers. Each processor gets its own run queue and tasks are pinned to a specific processor. This avoids the problem of synchronization entirely. As Rust's task model requires the ability to queue a task from any thread, there still needs to be a thread-safe way to inject tasks into the scheduler. Either each processor's run queue supports a thread-safe push operation (MPSC) or each processor has two run queues: an unsynchronized queue and a thread-safe queue.

This is the strategy used by Seastar. Because synchronization is almost entirely avoided, this strategy can be very fast. However, it's not a silver bullet. Unless the workload is entirely uniform, some processors will become idle while other processors are under load, resulting in resource underutilization. This happens because tasks are pinned to a specific processor. When a bunch of tasks on a single processor are scheduled in a batch, that single processor is responsible for working through the spike even if other procesors are idle.

Most "real world" workloads are not uniform. Because of this, general purpose schedulers tend to avoid this model.

Work-stealing scheduler

The work-stealing scheduler builds upon the sharded scheduler model and addresses the underutilization problem. Each processor maintains its own run queue. Tasks that become runnable are pushed onto the current processor's run queue and processors drain their local run queue. However, when a processor becomes idle, it checks sibling processor run queues and attempts to steal from them. A processor will go to sleep only once it fails to find work from sibling run queues.

At the model level, this is a "best of both worlds" approach. Under load, processors operate independently, avoiding synchronization overhead. In cases where the load is not evenly distributed across processors, the scheduler is able to redistribute. Because of this characteristic, the work-stealing scheduler is the choice of Go, Erlang, Java, and others.

The downside is this approach is far more complicated; the run queue algorithm must support the stealing operation and some cross processor synchronization is required to keep things running smoothly. If not done correctly, the overhead to implement the work-stealing model can be greater than the benefits gained.

Consider this case: Processor A is currently running a task and has an empty run queue. Processor B is idle; it tries to steal tasks and fails, so it goes to sleep. Then, the task being executed by processor A spawns 20 tasks. The goal would be for processor B to wake up and steal some of those newly spawned tasks. To achieve this, the work-stealing scheduler requires some heuristic where processors signal sleeping sibling processors when they observe new work in their queue. Of course, this introduces additional synchronization, so this operation must be minimized.

In summary:

  • Minimizing synchronization is good.
  • Work-stealing is the algorithm of choice for general purpose schedulers.
  • Each processor is mostly independent, but some synchronization is required for stealing.

The Tokio 0.1 scheduler

Tokio first shipped its work-stealing scheduler in March 2018. This was a first attempt based on some assumptions that turned out to be incorrect.

First, the Tokio 0.1 scheduler assumed that processor threads should be shut down if idle for a certain period of time. The scheduler was originally intended to be a "general purpose" thread-pool executor for Rust futures. When the scheduler was first written, Tokio was still in the "tokio-core" days. At this point, the model was that I/O-based tasks would be executed on a single thread colocated with the I/O selector (epoll, kqueue, iocp, ...). More CPU bound work could be punted to a thread-pool. In this context, the number of active threads should be flexible and shutting down idle threads makes more sense. However, the model shifted to running all async tasks on the work-stealing scheduler, in which case it makes more sense to keep a small number of threads always active.

Second, it used the crossbeam deque implementation. This implementation is based on the Chase-Lev deque, which, for reasons described below, is not a good fit for the use case of scheduling independent asynchronous tasks.

Third, the implementation was over-complicated. This is due, in part, to this being my first scheduler implementation. Also, I was over eager in using atomics in code paths where a mutex would have done just fine. An important lesson learned is that there are many cases where mutexes are the best option.

Finally, the original implementation contained many small inefficiencies. The details of Rust's asynchronous model evolved significantly in the early years, yet libraries maintained API stability over those years. This resulted in some debt being accumulated that can now be payed back.

With tokio approaching its first major breaking release, we can pay all of that debt with the lessons learned over those years. It's an exciting time!

The next generation Tokio scheduler

Now it is time to dig into what changed in the new scheduler.

The new task system

First, its important to highlight something not part of Tokio, but is critical to some of the achieved gains: the new task system included in std, designed primarily by Taylor Cramer. This system provides the hooks that a scheduler must implement to execute Rust asynchronous tasks, and it is really well done. It is both much lighter and more flexible then the previous iteration.

The Waker struct is held by resources and is used to signal that a task is runnable and be pushed into the scheduler's run queue. In the new task system, the Waker struct is two pointers wide, when it used to be much bigger. Shrinking the size is important to minimize the overhead of copying the Waker value around, as well as taking less space in structs, allowing for more critical data to fit in the cache line. The custom vtable design enables a number of optimizations which will be discussed later.

A better run queue

The run queue is at the heart of the scheduler. As such, it is probably the most critical component to get right. The original Tokio scheduler used crossbeam's deque implementation, which is single-producer, multi-consumer deque. Tasks are pushed onto one end and values are popped from the other. Most of the time, the thread that pushes the value will pop it, however, other threads will occasionally "steal" by popping the value themselves. The deque is backed by an array and a set of indices tracking the head and tail. When the deque is full, pushing onto it will result in growing the storage. A new, larger, array is allocated and values are moved into the new store.

The ability for the deque to grow comes at a complexity and overhead cost. The push / pop operations must factor in this growth. Additionally, on growth, freeing the original array comes with additional difficulties. In a garbage collected language, the old array would go out of scope, and eventually the GC would free it. However, Rust does not come with a GC. This means we are responsible for freeing the array, but threads may be in the process of accessing the memory concurrently. Crossbeam's answer to this is to use an epoch based reclamation strategy. While not terribly expensive, it does add non-trivial overhead in the hot path. Every operation must now issue atomic RMW (read-modify-write) operations when entering and exiting the critical sections to signal to other threads that the memory is in use and to avoid freeing.

Because of the cost associated with growing the run queue, it's worth investigating if growing the queue is even a requirement. This question is what ended up spurring the rewrite of the scheduler. The new scheduler's strategy is to use a fixed size per-process queue. When the queue is full, instead of growing the local queue, the task is pushed into a global, multi-consumer, multi-producer, queue. Processors will need to occasionally check this global queue, but at a much less frequent rate than the local queue.

An early experiment replaced the crossbeam queue with a bounded mpmc queue. This did not result in much improvement due to the amount of synchronization performed by both push and pop. A key thing to remember about the work-stealing use case is that, under load, there is almost no contention on the queues since each processor only accesses its own queue.

At this point, I opted to read the Go source closer, and discovered they used a fixed size single-producer, multi-consumer queue. The impressive thing about this queue is how little synchronization is needed for it to function. I ended up adapting the algorithm for use in the Tokio scheduler, making a few changes. Notably, the Go implementation uses sequential consistency for its atomic operations (based on my limited knowledge of Go). The version implemented as part of the Tokio scheduler also reduces some copying in less frequent code paths.

The queue implementation is a circular buffer, using an array to store values. Atomic integers are used to track the head and tail positions.

struct Queue {
    
    head: AtomicU32,
    
    tail: AtomicU32,
    
    mask: usize,
    
    buffer: Box<[MaybeUninit<Task>]>,
}

Pushing into the queue is done by a single thread:

loop {
    let head = self.head.load(Acquire);
    
    let tail = self.tail.unsync_load();
    if tail.wrapping_sub(head) < self.buffer.len() as u32 {
        
        let idx = tail as usize & self.mask;
        
        
        self.buffer[idx].as_mut_ptr().write(task);
        
        self.tail.store(tail.wrapping_add(1), Release);
        return;
    }
    
    
    match self.push_overflow(task, head, tail, global) {
        Ok(_) => return,
        
        Err(v) => task = v,
    }
}

Note that, in this push function, the only atomic operations are a load with Acquire ordering and a store with Release ordering. There are no read-modify-write operations (compare_and_swap, fetch_and, ...) or sequential consistency. This is important because, on x86 chips, all loads / stores are already "atomic". So, at the CPU level, this function has no synchronization. Using atomic operations will impact the compiler in that it prevents certain optimization, but that's it. The first load can most likely be done safely with Relaxed ordering, but there were no measurable gains in switching.

When the queue is full, push_overflow is called. This function moves half of the tasks in the local queue into the global queue. The global queue is an intrusive linked list guarded by a mutex. The tasks being moved to the global queue are first linked together, then the mutex is acquired and all the tasks are inserted by updating the global queue's tail pointer. This keeps the critical section small.

If you are familiar with the details of atomic memory orderings, you might notice a potential "issue" with the push function as shown above. An atomic load with Acquire ordering is pretty weak. It may return stale values, i.e. a concurrent steal operation may have already incremented the value of self.head but the thread performing the push had an old value in the cache so it didn't notice the steal operation. This is not a problem for the correctness of the algorithm. In the fast-path of push, we only care if the local run queue is full or not. Given that the current thread is the only thread that can push into the run queue, a stale load will result in seeing the run queue as more full than it actually is. It may incorrectly determine that the queue is full and enter the push_overflow function, but this function includes a stronger atomic operation. If push_overflow determines the queue is not actually full, it returns w/ Err and the push operation tries again. This is another reason why push_overflow will move half of the run queue to the global queue. By moving half of the queue, the "run queue is empty" false positive is hit a lot less often.

The local pop (from the processor that owns the queue) is also light:

loop {
    let head = self.head.load(Acquire);
    
    let tail = self.tail.unsync_load();
    if head == tail {
        
        return None;
    }