reading about cluster computing makes me think less of google in some ways.
specifically people talking about how things like slurm and mesos "assume the work load is infinite but the compute is not." which matches HPC clusters, and in turn matches how mainframes and such *actually work*.
the comparison was kube "assumes there is always more compute available" and i guess is actually not nearly as good at the job because of it. which does reflect google's mentality that efficiency is irrelevant when you're a billionaire and can just buy another datacenter and boil another ocean.
opinions might change with experience. mesos has basically no future and kube currently still can be used without systemd. it seems the whole mesos and DC/OS thing is a case of "this was better designed, but it's not what we got to use."
Q.U.I.N.N.
in reply to Q.U.I.N.N. • • •that being said the last time i tried to cluster compute was using afanasy and that software has not aged super well nor does it particularly like you trying to do things that aren't just render farming.
it really wants you to render farm with a set catalogue of commercial tools. its POSSIBLE to make custom batches but you are seriously fighting it.
plan-A
in reply to Q.U.I.N.N. • •@Q.U.I.N.N.
Interesting, it got me deepen in this subject, although I've read about it @ CompTia + I never went further in the subject.
So you can set multiple pc to work as 1 using on the network. Which could empower your capabilities on GPU and CPU.
Now I see there is some Golem Network as open source render management, it supports Cloud and advanced features as 1 bundle.
Never tried all of this
So sorry if I miss the point here.
Q.U.I.N.N.
in reply to plan-A • • •@zer0unplanned nah you're close.
a mainframe is basically one humongous computer which was designed to never go down. they're very rare things anymore.
a "beowulf cluster" is a pile of computers networked to do a similar job. instead of being one massive thing, its a network of much cheaper smaller things.
scientists often end up having to share the big computer so a lot of cluster computing stuff came out of or was for scientists.
google came along and started doing it for serious corporate use, which everyone else followed and mainframes-proper kind of died.
so the scientific software is still based on a fixed-size compute center (they can't just go buy another one, need grants etc) and optimizing scheduling on it. google and meta just have huge warehouses of boxes and buy them by the thousand. they're not really concerned with things being done optimally just that stuff is happening.
that being said i have seen vary
... show more@zer0unplanned nah you're close.
a mainframe is basically one humongous computer which was designed to never go down. they're very rare things anymore.
a "beowulf cluster" is a pile of computers networked to do a similar job. instead of being one massive thing, its a network of much cheaper smaller things.
scientists often end up having to share the big computer so a lot of cluster computing stuff came out of or was for scientists.
google came along and started doing it for serious corporate use, which everyone else followed and mainframes-proper kind of died.
so the scientific software is still based on a fixed-size compute center (they can't just go buy another one, need grants etc) and optimizing scheduling on it. google and meta just have huge warehouses of boxes and buy them by the thousand. they're not really concerned with things being done optimally just that stuff is happening.
that being said i have seen varying reports of how much of a cluster is operating at a given time. labs tend to be using all they can get. companies its more like 30-40%. some people have reported their kube installs have gotten to 90%.
i think you can replace the scheduler in kube but most of kube's users are not people who are trying to maximize scavenged hardware.
plan-A likes this.
plan-A
in reply to Q.U.I.N.N. • •@Q.U.I.N.N.
You are good!