Rendered at 12:55:08 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
outworlder 21 hours ago [-]
At my company, we migrated basically all CI/CD to Github Actions. Also our Gitops processes. I have warned that putting all the eggs in one basket and they key to the fridge in the same basket was a bad idea. People didn't care. So I don't care anymore.
Github seems to have degraded measurably after they got acquired. As is tradition.
nrmitchi 20 hours ago [-]
> Github seems to have degraded measurably after they got acquired. As is tradition.
I'm not disagreeing with you that it has degraded, however it was acquired almost 8 years ago at this point, and the software world has been turned on it's head in the last... 3?
Software generally degrades post-acquisition, I'm not denying that. Github has definitely degraded. I'm not sure I would agree that the core reason in this case is "the acquisition" and not "AI making software and free-CI-load demand 3-5 orders-of-magnitude higher than it was 3 years ago"
JohnMakin 20 hours ago [-]
The latest problems correlate pretty closely when they tried to migrate the entire platform onto azure in an insane timeline, something like ~1 year (for a project of that size, that's borderline insane).
doomroot13 19 hours ago [-]
Yes, but also in combination with the not-borderline-and-actually insane growth in commits and load on the platform due to AI.
sleepybrett 19 hours ago [-]
this is often the argument that is trotted out, but i don't see any evidece of that.
JohnMakin 18 hours ago [-]
It has been reported new user volume has nearly doubled, and commit volume 10x+ - that could certainly have an impact, but "Azure" and "scale" in my career are words that have never really gone together. Azure Cloud is the only 'hard no, will not interview' of any cloud provider that's listed on a job posting, and I've worked with just about all of them by now I think. Nothing even comes close to such a miserable experience.
The thought of migrating something at the scale of github to azure gives me the cold sweats.
mday404 19 hours ago [-]
Agreed. It's not at all clear that GitHub as an independent company would be doing much better at dealing with this new world than GitHub within Microsoft is doing.
tyromaniac 19 hours ago [-]
Would they have migrated to azure on such a timeframe?
simoncion 16 hours ago [-]
They would not have moved to Azure... so, no.
Even taking into consideration the fact that Google never wants to talk to you for any reason, it has been my professional experience that Azure is -by far- the worst of the Big Three "cloud" providers.
franktankbank 16 hours ago [-]
> Would they have migrated to azure
No
nvme0n1p1 19 hours ago [-]
> I'm not sure I would agree that the core reason in this case is "the acquisition"
Playing devil's advocate: from the graph alone, it could just be that the status page became more accurate, so same amount of down time, but reported more now.
On the other hand, I don't remember things being this bad before the acquisition.
nonethewiser 19 hours ago [-]
That is quite stark.
Whats the deal with the colors? Red and yellow don’t strictly seem to mean higher/lower. Some yellow dots are lower than red.
zimpenfish 18 hours ago [-]
As best I can tell, the red dots are "major outages", the yellow are "partial outages".
19 hours ago [-]
bromuro 15 hours ago [-]
Wasn’t that page debunked already?
soraminazuki 11 hours ago [-]
It doesn't matter when GitHub has been acquired. The only thing that matters is whether it's part of a broader pattern, which it is. Remember Skype?
shimman 20 hours ago [-]
These are incredibly complex and massive systems, they do not fail instantly. The failures happen incrementally until they become insurmountable and the system collapse. Incompetence doesn't show its head immediately, it can take quite a while.
Either way, it's clear the acquisition has not helped github in any meaningful way. GitHub is owned by a multi trillion dollar transnational corporation. They have enough money, resources, and human labor to achieve whatever they want, but this is what they want and this is what they've achieved.
neya 20 hours ago [-]
It's Microsoft's touch. Anything they touch gets fucked up. It's because of their shitty infrastructure aka Azure. There was an employee here who actually gave insight into how bad the situation was a while back. Just reading that should make you re-evaluate anything owned by Microsoft.
I feel sorry for the teams at this point. These kind of incidents at this scare and frequency are bound to create an acrimonious environment where everyone would be looking for scapegoats on the pretext of finding root causes.
sleepybrett 19 hours ago [-]
it's almost like they should have a test environment with proper load testing...
sandeepkd 16 hours ago [-]
Its infeasible to have a test environment with 1:1 mapping to production for everything in any big enough company. Most of the times people practically aim only for certain critical components to ensure coverage.
coldpie 20 hours ago [-]
> People didn't care. So I don't care anymore.
Yeah. Enjoy the rest of your day off. Winter's coming, maybe go hit a park for a few hours or play a new video game or something. Or take some time to learn a new skill or something. Not your fault if the company can't provide the tools to do the job, go find something better to do.
hack1312 20 hours ago [-]
Great ideas if working remote, can’t imagine busting out my Steam library in the office though.
I was working in the Techstop (40) @ Google, 2011-ish, when a nice person from PeopleOps came through and politely requested my admin powers to help them install Steam on their corp-issued Windows laptop.
I'm sorry, what?
Well see, PeopleOps was going to have an onsite intending to help recruiters better understand Engineering types, and part of the whole business was going to involve everyone firing up a copy of Portal and playing a puzzle-solving video game.
So I immediately got to it. Never heard back as to the success of the endeavor, but turns out there are circumstances where you can break out your Steam library at work.
sleepybrett 18 hours ago [-]
we had a member of my team at a company about ~15years ago that they tried to fire for installing steam on their corporate laptop when they had sent him to another city for for what was supposed to be a week but ended up being three weeks. this was against our wishes (he didn't need to be there, there was no good reason for it, he could do all the work remotely from our office and had been doing it successfully for three months, the project was 'at risk' due to schedules but none of that was our fault at all).
I had a knockdown drag out fight with both HR and Security. I agree that it was probably better for him to have packed his own personal laptop or similar gaming device but security had shown absolutely no risk and it was clear that HR was just 'clucking to cluck'. Eventually they relented with a mealy mouthed 'warning'. The fact that he had installed steam after his stay had been extended and they couldn't show any evidence that he was playing during working hours was a hillarious, and lucky, footnote.
esseph 20 hours ago [-]
Coward! /s
999900000999 20 hours ago [-]
That’s a good way to get blamed and let go.
But sometimes it’s a catch 22. I’ve had numerous times where systems simply didn’t work.
I brought it to my managers attention and they didn’t care.
Then later I leave in disgust or get fired. All my fault. I should have used my magical powers to convince my manager he’s wrong while not making him feel stupid.
Ultimately if you’re on the titanic it’s easier to grab a life boat vs trying to avoid the ice berg.
(Man, unbelievable the mileage we're getting out of this comic.)
I mean this really seriously. Maybe not a game on your computer, that's unprofessional (unless your TL/manager does it first). But
> Not your fault if the company can't provide the tools to do the job, go find something better to do.
is just Truth. What should I do in this case to best deliver value to the company, fly to SF and hand Github a ringbound volume of my changeset? Patch our corporate network so we can interface to Github via RFC 1149?
More "professional" ideas:
- read a book; a novel or TAOCP for nerdmaxxing
- play a game on your phone
- learn to draw. Draw hands!
jbs789 21 hours ago [-]
Sounds like you now have your audience for the previously proposed solution.
dewey 20 hours ago [-]
The attention for ideas like that usually fades as quickly as it came after the downtime is over.
outworlder 20 hours ago [-]
After we spent a couple of years ripping and replacing the previous solution? You may have a point.
I guess I can embrace the corporate way, repackage the old solution with a few improvements as something brand new, and pitch to execs.
jbs789 3 hours ago [-]
Are you willing to propose and implement the solutions? If so, there are lots of companies that would love to have you!
gateauxd 20 hours ago [-]
Curious what even is the alternative. The stickiness is quite high and risk is scary to swap providers.
x86a 20 hours ago [-]
Claude/codex has made the stickiness much less of a problem. We just say "no new repos in GitHub" and then put agents on refactoring the existing repos for GitLab compatibility.
InsideOutSanta 19 hours ago [-]
This exactly. Moving from GitHub to Forgejo was much less of a problem than we thought when we just told Claude to figure everything out. Kinda crazy, but it worked.
Bnjoroge 20 hours ago [-]
Forgejo, and Gitea are decent alternatives if you want to fully move off Github! If you are hoping to move away from only Github Actions, I've been working on a ~drop-in replacement for their control-plane + runner for the past few months. Each each job runs in a cross-platform vm so you can run your ci locally and update your checks. It's still beta, so I certainly wouldnt use it in prod just yet, but it's great if you want to run some of your CI locally. it allows me to extend it where possible to introduce some new features like JIT pipelines, improve performance or security. https://github.com/preloopdev/preloop if anyone's curious!
msdz 20 hours ago [-]
The alternative is, I’m guessing, probably any other hosted git + CI/CD service that doesn’t offer a free tier to zillions of users (human and LLM alike) so that their computing capacity isn’t in a constant arms race with growth.
sssilver 20 hours ago [-]
What makes it sticky? I just don't see it. The declarative Github Actions manifests?
I imagine very few people use the wiki, because it's so subpar.
The issues are a problem, but this is why I always thought issues should be a core part of the Git repo, checked into it, sort of how Fossil does it. And so should the wiki, for that matter.
ionetan 20 hours ago [-]
Built an issue tracker like that, that persists state as an event graph in your repo in a dedicated branch. It is vendor agnostic and works offline:
Absolutely anything? GitHub slowly became a standard through market share. Concourse CI is much better, in fact nearly anything is better than GitHub as it doesn't have first class container support.
embedding-shape 20 hours ago [-]
Self-hosted $whatever, managed your way. CI/CD doesn't have to be dog-slow or ever go down when you need it, only when you do your upgrades yourself, as god intended it.
I personally use Forgejo + their built-in Actions stuff (mostly GitHub compatible), have one old Linux desktop as a server which runs Linux + Windows runners, one Mac Mini for macOS/iOS builds, using NixOS so I don't have to remember how things work, and it just keeps running 24/7, always there when I need it.
tracker1 20 hours ago [-]
There's something to be said for the social availability that github brings. I'm far more likely to submit a pr for a bugfix to a project on gh than a random self-host site.
You can do your own ci/cd runner system, but you still rely on webhooks or triggers, when you aren't hosting the code.
embedding-shape 20 hours ago [-]
> There's something to be said for the social availability that github brings. I'm far more likely to submit a pr for a bugfix to a project on gh than a random self-host site.
Yeah, that used to be the single favorable argument for GitHub, nowadays I think it might be the opposite, at least personally. If the contributor can't be arsed to join the community proper (and maybe get a sign-in code to their email, or whatever login you use), I think I prefer life without their drive-by PRs.
sleepybrett 18 hours ago [-]
Here is the thing, we have self-hosted runners running behind our firewall and even those go down for these kinds of outages. All the jobs just sit in the queue and are never sent to our self-hosted setup at all.
esseph 14 hours ago [-]
1. Why isn't observability setup to monitor the queue depth and alert?
2. Why aren't the CI runners self-healing?
embedding-shape 17 hours ago [-]
> we have self-hosted runners running behind our firewall and even those go down for these kinds of outages
Sounds like it's not as self-hosted as you think, or you have some other unnecessary dependency on something here? As long as GitHub isn't the sole place for your SCM, why would your self-hosted CI runners go down just because GitHub goes down?
outworlder 20 hours ago [-]
See, my main issue is with Actions. If that breaks, there's little alternative for the more complex flows. Minor infrastructure changes could be done manually through AWS, but complex deployments would have to be replaced. That's a big task even with AI.
The pull request system and related machinery is useful, but Gitlab and the ilk can do the same. Even self-hosted solutions like Gitea. But I don't particularly have a problem with that living in Github. If there's an outage, break glass, force push to main.
The actual Git repository hosting part is trivial. If that breaks, Git is distributed anyways, push and pull to any random remote storage. All you need is SSH. The main headache is that automations would break as they are usually hard-coded to a given origin. That is solvable in a pinch, and we have setup mirrors in the past.
It really depends on your use-case and how much you are tied to them (Github authentication?)
EngineeringStuf 20 hours ago [-]
swarmfile was built to deal with this type of problem
boxed 20 hours ago [-]
> Github seems to have degraded measurably after they got acquired. As is tradition.
Their explosive growth caused by LLMs seems like it must be a much better indicator than the acquisition. I bet Microsoft are losing a shit ton of money on GitHub right now and seriously questioning the decision to buy them.
maccard 20 hours ago [-]
No, their decision to focus on a move to azure is the indicator, as the reliability issues started shortly after that, but before their boom in agentic commits
lenerdenator 19 hours ago [-]
So they're giving you random free coffee breaks of varying length? Quite the perk.
shadowgovt 20 hours ago [-]
It's correlated and possibly semi-causal, but IIUC the real issue GitHub is suffering from is they don't have a clean mechanism for filtering AI-generated activity, which is growing exponentially and putting significant strain on their backend in new ways that didn't show up before. Many AI solutions use PRs, for example, as a plan-of-record journal, which creates very different access patterns from how humans use GitHub.
They could, hypothetically, filter bot traffic; I believe they are probably avoiding that solution because they don't want to be, as a market position choice, "anti-AI" (and you could blame the acquisition on this; I'm not sure pre-acquire GH wouldn't have responded to the AI craze with "LOL, no" and instituted a "no agent-driven PRs" policy).
esseph 14 hours ago [-]
> which creates very different access patterns from how humans use GitHub.
it can
> They could, hypothetically, filter bot traffic;
Nobody seems to be able to do this reliably for a whole huge list of reasons, including: the LLMs themselves can simply mimic human access patterns, timing, and behavior.
rfgplk 20 hours ago [-]
They haven't degraded, their service is being hit by an enormous amount of traffic. PRs/commits are what? Doubling every quarter?
FLeXMurphy 21 hours ago [-]
>Github seems to have degraded measurably after they got acquired. As is tradition.
This old chestnut.
sleepybrett 19 hours ago [-]
Same, they seemed to have a moritorium after there last flurry of debacles, but clearly they are letting their engineers deploy slop again.
I will never trust github at any other company ever again.
cwyers 20 hours ago [-]
There is a screamingly obvious answer here and I don't understand why they aren't taking it -- raise prices, make the free tier less generous. Either slow down the growth or up the revenue pipe so you can throw more compute at it.
Normal_gaussian 20 hours ago [-]
As it stands, customers aren't leaving because of the degraded performance. Free is generating paying customers.
You can free tier but still sensible rate limits and QoS policy. You can have separate region or cluster for your paying customers. Finally you can spend time re-architecting GitHub Actions or just aquehire one of many competitors that do better co/cd.
torben-friis 20 hours ago [-]
What if the load isn't fully the cause, and they end up with a significantly smaller userbase still getting downtime?
consumer451 19 hours ago [-]
I thought the same thing at first. But that only works if there were no sharks in the water who would love them to do that.
cwyers 19 hours ago [-]
Where are the sharks, though? I just see minnows. GitLab has a lot of outages/brown-outs on a much smaller load and Gitea is so confident in their ability to handle load that they host their own development on GitHub.
pksunkara 1 hours ago [-]
I am trying to approach this problem from a different perspective at https://juju.bi
esseph 14 hours ago [-]
> GitLab has a lot of outages/brown-outs
Not remotely compared to github, and you can run GitLab OSS internally anyway.
consumer451 19 hours ago [-]
Musk via Cursor is the biggest example I can think of.
edoceo 19 hours ago [-]
I got a notice about some incoming rate-limits.
bakugo 19 hours ago [-]
Trying to slow down the growth would be an admission that everyone having their AI agents pushing 500 commits per hour autonomously may not be an objectively good thing for everyone involved. They won't do that.
7458db0d0637c04 20 hours ago [-]
[flagged]
cwyers 19 hours ago [-]
Because East River Source Control hasn't gotten back to me yet.
haute_cuisine 20 hours ago [-]
Their infra is under constant DDoS + they provide a lot of free stuff for open source projects. There's also a constant churn of dependencies, patches, abandoned libraries, etc. We should give them some appreciation, dealing with this stuff is hard.
ianm218 20 hours ago [-]
I think you are giving them way too much leeway. For the service they provide there’s no technical reason it can’t be infinitely scalable. For example their “pull request” page constantly goes down. Serving that kind of workload at high burst load is a fully solved problem.
If they are genuinely compute constrained on CI they should obviously reduce the free offerings rather than take down all their paying customers constantly.
dewey 19 hours ago [-]
You are basically saying that you, from the outside, have solved the problem perfectly and the whole team of engineers at GitHub just don't know about this one simple solution you are proposing.
Have you considered that you don't have the full context of the problem and it's not "just" a few small tweaks?
ianm218 18 hours ago [-]
My guess is that the problem is internal politics and incentives. Yes I think it is a fully solved problem for a company worth trillions to have an infinitely scalable "pull requests" page.
For example, Microsoft is known to pay significantly less than their competitors. I am guessing they just don't have the systems internally to say "This problem is worth literally billions to us in revenue per year, so we will pay millions to IC's to get it fixed and clear internal roadblocks".
It's not "just" small tweaks it is likely that people who are entrenched need to be moved and they need people with the right incentives in place to get it done.
dewey 18 hours ago [-]
> and they need people with the right incentives in place to get it done
Their c-levels wrote blog posts about the current state and why it happens, pretty sure they know what's going on.
ianm218 16 hours ago [-]
I think we just fundamentally disagree here. I'm sure they know the current state and that it doesn't work. I think it's likely that they need entirely different people knowing about these problems and working on them in a totally different reporting structure to get the job done. It's a hard job and they're not up for the task.
I think the reason they can't make that change is that Microsoft is a slow moving rigid culture and doesn't have the kind of org that can adapt to a 10x or 100x increase in usage compared to companies like Google or the AI Labs that handled much more difficult scaling problems.
> Their c-levels wrote blog posts about the current state and why it happens, pretty sure they know what's going on.
Having read many of the formal statements from the C*Os in the BigCos I've worked for, I can say with confidence that that is not at all a safe bet to be making.
privacyishuman 20 hours ago [-]
Agreed. I don't think people realize how much more people are using their free services. It's staggering.
Chyzwar 20 hours ago [-]
Then just reduce free tier if they cannot handle. I paying for a service but I get best effort freeware quality.
awakeasleep 19 hours ago [-]
Even without out the free stuff, no one in this thread seems to have experience managing a self-hosted git environment in even a medium sized company.
It is also constant outages, and then it's your job to diagnose and fix it- and the technical work of that is the easiest part.
It's broken because a team figured out a new way to abuse the company's git infra to get around some process or limitation, and now you have to argue with their manager and then their director about why their team needs to stop using your VCS as a docker registry or whatever.
boredatoms 20 hours ago [-]
They need to cut back on the free tiers if it’s causing outages for paying customers
esseph 14 hours ago [-]
What if it's the paying customers causing the outages...
somehowadev 21 hours ago [-]
It's almost a running joke at my company that around 15:00 UTC we're going to get the weekly Github actions outage.
I'm assuming it's something to do with the USA users getting their first work in?
serial_dev 21 hours ago [-]
I find it baffling that I knew about the incident 20 minutes before the official status update…
amw-zero 21 hours ago [-]
Why is that baffling? You usually have to diagnose what's going on before summarizing it into a status update.
Or do you mean you want to see: "Something is wrong, we're looking into it."
serial_dev 21 hours ago [-]
Yes. An acknowledgement of the issue should be fine, even if they don’t know why it happened. If a random dev with one PR can see that no jobs are picked up, GitHub should see it as well and add an entry to their status “hey, actions aren’t getting picked up, we are looking into it”. I wasted 20 minutes looking into out actions config looking got an issue, thought maybe we messed something up, but in turned out it was GitHub.
pasc1878 20 hours ago [-]
Yes then you know the problem is not in your set up and so do not have to waste time looking into that possibility.
99954bb63ccc 19 hours ago [-]
The status page should detect the thing they are looking into. Kinda like a Check Engine light or gauge reading notifies you to investigate a problem with your engine.
jacquesm 21 hours ago [-]
20 minute rule: if you fix it before 20 minutes pass it never happened.
khalidsaidi 6 hours ago [-]
[flagged]
icameron 21 hours ago [-]
I’m going to hope they can figure things out and improve their uptime. It’s so engrained in my teams workflow at this point, tearing it out and switching to anything else would be extremely disruptive.
I did work for a group using Atlassian bitbucket once, and that had its own problems at the time. I’m still pretty sure even a degraded GitHub beats it. That is to say I don’t know what alternatives for GitHub even are
OlivOnTech 20 hours ago [-]
We're using gitlab.com with self hosted runners (that can access lan).
Outages only come from our infra.
Migrating means migrating your ci workflows, but a decent llm can do that easily.
gateauxd 20 hours ago [-]
That's the thing what is the alternative feels like we are just at mercy to their outages.
macrofriendly 20 hours ago [-]
swarmfile supports git, self-hosted runners, is self-hosted swarmfile.com/public/swarmfile/swarmfile-server
cwyers 20 hours ago [-]
Did a swarmfile write this message
macrofriendly 20 hours ago [-]
A real human did :)
hack1312 20 hours ago [-]
You could self-host GHE. Or Gitlab. Or Forejo.
Rantenki 15 hours ago [-]
I am surprised that this is still newsworthy. It seems like _something_ is down at Github nearly every day, and their status history backs up that assessment: https://www.githubstatus.com/history
If you are doing CI/CD and want to be able to build and deploy on demand, you pretty much have to treat Github as a mirror and not the source of truth for your applications. If you have an outage, need to deploy a fix, and Github is down, you're on their recovery schedule, not your own.
efficax 19 hours ago [-]
github has basically become a joke at this point. everyone engineering org that i know of is at least asking themselves if there's a migration path off there if they're not in the process of building one. Every slack has a custom emoji of github on fire. Github is so joever
I wonder if this will ever end (meaning they've solved scaling for AI), and if not, is that due to the problem being unsolvable at this scale or bad decisions on GitHub's part? For example, they always bring up AI as a cause, but it's also apparent the move to Azure didn't help.
EduardoBautista 21 hours ago [-]
I am going to try out Cursor Origin, GitHub is just too much of a bottleneck these days. The only thing that is not readily available is integration with claude code, but with the new claude code mods feature I think it's something I could vibe code quickly.
rosslh 20 hours ago [-]
Isn't it still just essentially a UI wrapper around GitHub?
ocamoss 8 hours ago [-]
No, it's a separate forge. You can choose to mirror repos between Github and Origin, but you can also have repos on just Origin and w/ no dependency on Github.
I'm wondering if they can just build an entirely new platform, onboard new users to it, and then slowly migrate old ones. Even if they don't migrate, the load would stabilize. What are the cons here?
JohnMakin 20 hours ago [-]
They should just call it Monday now
throwaway613746 20 hours ago [-]
[dead]
defenestration 21 hours ago [-]
GitHub actions staying 'Queued', started 16 minutes ago for me
nix0n 20 hours ago [-]
I'm seeing at least one job showing up that way on the Github website that has already completed (on a self hosted runner, which is how I can tell).
kingcauchy 20 hours ago [-]
I know there's a lot of GitHub annoyed people out there but after investigating a lot of the instances of outages it does seems like A) They're abusing ElasticSearch in a way is was never meant to be used B) They've seen an unprecedented increase in scale and now have to provide agent slop a lot of free action minutes until they figure out how to separate commercial support from free support C) Who knows how many other systems were never meant to see this scale there
Still grateful for a lot of the free services and the semblance of community that GitHub still offers. It does seem like it's still the yellowpages of tech.
minraws 19 hours ago [-]
Can we split Github, paid/pro users, my orgs are payings north of 25k per year to Github, can we get better service please, even AWS assigns a proper support performance once you hit few tens of thousands in spend. This is just insane, that all of github starts failing at once, I would like to have better experience for when I am working compared to when I want to spam commits with ai to slop something.
easton 18 hours ago [-]
They have this with the ghe.com hosting option, it cuts you off from the GitHub.com instance so you can’t share projects there though, if that matters (also $$$).
minraws 18 hours ago [-]
So you are saying paying 25K isn't enough we need to pay more.. oof I got it, I am too broke for this world.
FLeXMurphy 21 hours ago [-]
It would not surprise me if they are going to shift to Plan B sometime soon, because this isn't working. They can't keep provisioning hardware resources in perpetuity to keep up with volume that the AI is pummeling them at. I think most other repo providers have either experienced downtime and/or have locked themselves up to prevent the onslaught.
It is impressive they've been able to keep going, but this is a regular denial-of-service attack at this point, nonstop and relentless.
dannyw 20 hours ago [-]
They have plenty of levers to pull.
For example, GitHub can reduce/eliminate free GitHub Action minutes.
Or just cap concurrent jobs for free users from 20 to 2 or 1… lots of levers, their inaction is sad to see.
tonyhart7 20 hours ago [-]
maybe just charge money if you use more compute ????
esafak 21 hours ago [-]
To all those in the middle of conjuring a Github-killer into existence, more power to you, and please ship it already; I can't take this any longer.
I think most enterprises that make the switch aren't moving to another cloud product. They're self hosting it. Otherwise, yeah, you are subject to the same types of outages.
We've had very few noteworthy incidents self-hosting GitLab. We can size our deployments as we need, and perform upgrades during maintenance windows we control.
esseph 14 hours ago [-]
> GitLab is often recommended when GitHub issues occur, but GitLab Cloud
Github seems to have degraded measurably after they got acquired. As is tradition.
I'm not disagreeing with you that it has degraded, however it was acquired almost 8 years ago at this point, and the software world has been turned on it's head in the last... 3?
Software generally degrades post-acquisition, I'm not denying that. Github has definitely degraded. I'm not sure I would agree that the core reason in this case is "the acquisition" and not "AI making software and free-CI-load demand 3-5 orders-of-magnitude higher than it was 3 years ago"
The thought of migrating something at the scale of github to azure gives me the cold sweats.
Even taking into consideration the fact that Google never wants to talk to you for any reason, it has been my professional experience that Azure is -by far- the worst of the Big Three "cloud" providers.
No
Would data change your mind?
https://damrnelson.github.io/github-historical-uptime/
On the other hand, I don't remember things being this bad before the acquisition.
Whats the deal with the colors? Red and yellow don’t strictly seem to mean higher/lower. Some yellow dots are lower than red.
Either way, it's clear the acquisition has not helped github in any meaningful way. GitHub is owned by a multi trillion dollar transnational corporation. They have enough money, resources, and human labor to achieve whatever they want, but this is what they want and this is what they've achieved.
Edit: here's the link: https://isolveproblems.substack.com/p/how-microsoft-vaporize...
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
Yeah. Enjoy the rest of your day off. Winter's coming, maybe go hit a park for a few hours or play a new video game or something. Or take some time to learn a new skill or something. Not your fault if the company can't provide the tools to do the job, go find something better to do.
I was working in the Techstop (40) @ Google, 2011-ish, when a nice person from PeopleOps came through and politely requested my admin powers to help them install Steam on their corp-issued Windows laptop.
I'm sorry, what?
Well see, PeopleOps was going to have an onsite intending to help recruiters better understand Engineering types, and part of the whole business was going to involve everyone firing up a copy of Portal and playing a puzzle-solving video game.
So I immediately got to it. Never heard back as to the success of the endeavor, but turns out there are circumstances where you can break out your Steam library at work.
I had a knockdown drag out fight with both HR and Security. I agree that it was probably better for him to have packed his own personal laptop or similar gaming device but security had shown absolutely no risk and it was clear that HR was just 'clucking to cluck'. Eventually they relented with a mealy mouthed 'warning'. The fact that he had installed steam after his stay had been extended and they couldn't show any evidence that he was playing during working hours was a hillarious, and lucky, footnote.
But sometimes it’s a catch 22. I’ve had numerous times where systems simply didn’t work.
I brought it to my managers attention and they didn’t care.
Then later I leave in disgust or get fired. All my fault. I should have used my magical powers to convince my manager he’s wrong while not making him feel stupid.
Ultimately if you’re on the titanic it’s easier to grab a life boat vs trying to avoid the ice berg.
(Man, unbelievable the mileage we're getting out of this comic.)
I mean this really seriously. Maybe not a game on your computer, that's unprofessional (unless your TL/manager does it first). But
> Not your fault if the company can't provide the tools to do the job, go find something better to do.
is just Truth. What should I do in this case to best deliver value to the company, fly to SF and hand Github a ringbound volume of my changeset? Patch our corporate network so we can interface to Github via RFC 1149?
More "professional" ideas:
- read a book; a novel or TAOCP for nerdmaxxing
- play a game on your phone
- learn to draw. Draw hands!
I guess I can embrace the corporate way, repackage the old solution with a few improvements as something brand new, and pitch to execs.
I imagine very few people use the wiki, because it's so subpar.
The issues are a problem, but this is why I always thought issues should be a core part of the Git repo, checked into it, sort of how Fossil does it. And so should the wiki, for that matter.
https://ljtn.github.io/epiq
I personally use Forgejo + their built-in Actions stuff (mostly GitHub compatible), have one old Linux desktop as a server which runs Linux + Windows runners, one Mac Mini for macOS/iOS builds, using NixOS so I don't have to remember how things work, and it just keeps running 24/7, always there when I need it.
You can do your own ci/cd runner system, but you still rely on webhooks or triggers, when you aren't hosting the code.
Yeah, that used to be the single favorable argument for GitHub, nowadays I think it might be the opposite, at least personally. If the contributor can't be arsed to join the community proper (and maybe get a sign-in code to their email, or whatever login you use), I think I prefer life without their drive-by PRs.
2. Why aren't the CI runners self-healing?
Sounds like it's not as self-hosted as you think, or you have some other unnecessary dependency on something here? As long as GitHub isn't the sole place for your SCM, why would your self-hosted CI runners go down just because GitHub goes down?
The pull request system and related machinery is useful, but Gitlab and the ilk can do the same. Even self-hosted solutions like Gitea. But I don't particularly have a problem with that living in Github. If there's an outage, break glass, force push to main.
The actual Git repository hosting part is trivial. If that breaks, Git is distributed anyways, push and pull to any random remote storage. All you need is SSH. The main headache is that automations would break as they are usually hard-coded to a given origin. That is solvable in a pinch, and we have setup mirrors in the past.
It really depends on your use-case and how much you are tied to them (Github authentication?)
Their explosive growth caused by LLMs seems like it must be a much better indicator than the acquisition. I bet Microsoft are losing a shit ton of money on GitHub right now and seriously questioning the decision to buy them.
They could, hypothetically, filter bot traffic; I believe they are probably avoiding that solution because they don't want to be, as a market position choice, "anti-AI" (and you could blame the acquisition on this; I'm not sure pre-acquire GH wouldn't have responded to the AI craze with "LOL, no" and instituted a "no agent-driven PRs" policy).
it can
> They could, hypothetically, filter bot traffic;
Nobody seems to be able to do this reliably for a whole huge list of reasons, including: the LLMs themselves can simply mimic human access patterns, timing, and behavior.
This old chestnut.
I will never trust github at any other company ever again.
Raising prices risks churn, reducing free tier risks lowering customer acquisition rate.
You can free tier but still sensible rate limits and QoS policy. You can have separate region or cluster for your paying customers. Finally you can spend time re-architecting GitHub Actions or just aquehire one of many competitors that do better co/cd.
Not remotely compared to github, and you can run GitLab OSS internally anyway.
If they are genuinely compute constrained on CI they should obviously reduce the free offerings rather than take down all their paying customers constantly.
Have you considered that you don't have the full context of the problem and it's not "just" a few small tweaks?
For example, Microsoft is known to pay significantly less than their competitors. I am guessing they just don't have the systems internally to say "This problem is worth literally billions to us in revenue per year, so we will pay millions to IC's to get it fixed and clear internal roadblocks".
It's not "just" small tweaks it is likely that people who are entrenched need to be moved and they need people with the right incentives in place to get it done.
Their c-levels wrote blog posts about the current state and why it happens, pretty sure they know what's going on.
I think the reason they can't make that change is that Microsoft is a slow moving rigid culture and doesn't have the kind of org that can adapt to a 10x or 100x increase in usage compared to companies like Google or the AI Labs that handled much more difficult scaling problems.
Have you seen this legendary email thread from 2003 https://www.osnews.com/story/19921/full-text-an-epic-bill-ga...? This isn't a new problem at Microsoft.
Having read many of the formal statements from the C*Os in the BigCos I've worked for, I can say with confidence that that is not at all a safe bet to be making.
It is also constant outages, and then it's your job to diagnose and fix it- and the technical work of that is the easiest part.
It's broken because a team figured out a new way to abuse the company's git infra to get around some process or limitation, and now you have to argue with their manager and then their director about why their team needs to stop using your VCS as a docker registry or whatever.
I'm assuming it's something to do with the USA users getting their first work in?
Or do you mean you want to see: "Something is wrong, we're looking into it."
I did work for a group using Atlassian bitbucket once, and that had its own problems at the time. I’m still pretty sure even a degraded GitHub beats it. That is to say I don’t know what alternatives for GitHub even are
Outages only come from our infra.
Migrating means migrating your ci workflows, but a decent llm can do that easily.
If you are doing CI/CD and want to be able to build and deploy on demand, you pretty much have to treat Github as a mirror and not the source of truth for your applications. If you have an outage, need to deploy a fix, and Github is down, you're on their recovery schedule, not your own.
Still grateful for a lot of the free services and the semblance of community that GitHub still offers. It does seem like it's still the yellowpages of tech.
It is impressive they've been able to keep going, but this is a regular denial-of-service attack at this point, nonstop and relentless.
For example, GitHub can reduce/eliminate free GitHub Action minutes.
Or just cap concurrent jobs for free users from 20 to 2 or 1… lots of levers, their inaction is sad to see.
GitHub also has more traffic, compare to GitLab
We've had very few noteworthy incidents self-hosting GitLab. We can size our deployments as we need, and perform upgrades during maintenance windows we control.
Stop.
> Gitlab Cloud
No.
https://gitlab.com/gitlab-org/gitlab
There’s no CEO of GitHub to contact and Microsoft does not care as Copilot, and Tay continue to wreck the platform as the incidents continue.
No better time to self-host, since as this point your self hosted setup would be far most stable than GitHub as predicted 6 years ago. [1]
[0] https://news.ycombinator.com/item?id=50057012
[1] https://news.ycombinator.com/item?id=22868406