My latest toy: One box, eight Xeon Phi 7220 PCI cards, and a total (sustained!) hash rate of about 24kH/s for monero/sumo/etn/etc … Yeehaw!
This box is actually already up and running for over a week, but since I realized that a lot of people might actually like hearing about this I decided to finally share this build … after all, I think this is the highest-performing single-box build (for cryptonight/monero) out there right now….
Okay, where did this start? When I started writing lukMiner earlier this (wait – now “last”) year Intel had just announced some PCI version of the x200 Xeon Phis, the 7220 cards. While googling around on where to get one I found a page from Exxact Corp, which offered a system with 8 such cards for around $20k or so. Got me curious ;-).
Unfortunately, when I tried to place an order I was told that those cards had been un-announced, and would not be released after all…. However; I could never forget this setup, and eventually found some cards on ebay. So after eventually scrounging some of those cards together (and initially having quite some trouble finding some motherboards that they would actually run in#@!!@), I finally got Exxact to sell me a no-cards-included version of that box they had listed earlier this summer. Once it arrived I popped in the cards, installed the matching software stack, and here we are – eight 7220 cards, each doing 2800-2850 H/s (running at about 80-85 degrees C), plus two low-end Xeons in the host …. a bit of linux magic to start it all automatically upon boot, and la-voila – we have a machine that DwarfPool and NiceHash say makes an average of 24kH/s, at a power draw of only about 2.4kW… (And all in all, I paid less than $12k for the parts!).
The pic above is from when I assembled it in my basement… had to actually pull power cords from two different rooms to not pop my circuit breakers, and had to eventually move it to a co-hosting data center due to “somewhat excessive noise” (think “jet engine”) … but still, that was one heck of a fun project!
Happy Mining!
The lukMiner blog 
Hey, can i get the name of the motherboard you ended up using ?
It should be good to go as long as the cards have designated slots and the main cpu is xeon right ?
Greets 🙂
LikeLike
Unfortunately it’s not that easy. First of all you can’t even get those cards all that easily – I was just lucky to see some on ebay, but they’re very rare. If you ever get them, they only work in certain motherboards – I have one x99 board that runs them, but at least 5 others that don’t (don’t ask – no clue).
That particular machine I have them in is a SuperMicro SYS4028GR-TR (https://www.supermicro.com/products/system/4u/4028/sys-4028gr-tr.cfm). Follow that link to get the exact specs.
LikeLiked by 1 person
Yeah, well I can get the phi 5220 for an affordable price.
Did you try any of these ?
https://streamhpc.com/blog/2015-08-01/xeon-phi-knights-corner-compatible-motherboards/
LikeLike
Which one is running well?
LikeLike
I have confirmed working:
a) the server I bought from Exxact, the 8-card one that is on the blog. Costs abouto $6k, though. Dont’ have the detailed specs here right now, but can provide.
b) a old supermicro sys2027gr-trf that i bought used off ebay (seller ‘unixsurplus’), for order $900. Is rated for four cards, but does actually have space for six. Have only ever tried with two cards yet, but those two now have 24/7 uptime for weeks now.
c) a 1u supermicro system with two cards; with e5-2643v4, 3.4GHz; would have to look up exact supermicro system number, though. Bought it used off ebay.
Am curretnly expecting some more cards, will post more info as soon as I get them (right now I’d have to disassemble the 8-card machine to run those tests, which I’m not going to do :-/).
LikeLike
BTW: If that was you contacting me on ebay then yes, I did get it 😉
LikeLike
And which not?
LikeLike
I have tried several Xeon E5-v3 and v4 workstations and desktops, all of which but one did _not_ work. I also have one v3 supermicro server that is “flaky” with the phi cards (DMA errors when I put in more than one), but not sure if that’s a problem with this particular cpu or board version, or simply a flaky system.
LikeLike
I’ll be going for the ASUS X99-A II Intel X99 LGA 2011-v3 ATX
Thanks for the reply and great work on the miner !
I’ll keep you posted on whether or not i’ll get them hashing 🙂
LikeLike
The 5110 has about 25% of the hashrate of the x200s and uses the same amount of power. In other words, unless you have free/very cheap electricity it’s likely to not be worth it
LikeLike
I kind of agree, but not entirely – in fact, it depends very much on how much you pay for those cards. In terms of revenue and power draw, the x100s are still comparable to a GTX1060 or 1070, so using them for mining isn’t all _that_ outlanding of a thought: at order 200-250 Watts and order 600-650H/s they’ll certainly pay for the power, even at full residential prices.
Now if you want to purchase something _new_, then most likely the x200s are the better deal (five times the revenue at same power draw and only about twice the cost) – but if you already have some of the x100s – or a good line to get them cheaply – then using them still makes total sense.
LikeLike
Its kind of ridiculous how much these Xeon Phis are selling for. In some cases they are double or triple msrp. This is a cool project but I can’t imagine this being worthwhile for anyone unless they can get these well under a $1000. The AMD Vegas are the best bang for the buck right now. I still browse ebay occasionally though in hopes I can get a halfway cheap Phi 🙂
LikeLike
For the x100s I agree – I bought some for development, but would not buy them for actual mining… at least not at what they’re listed for on ebay
For the x200s, the story is different, though: If you talk to a distributor such as Exxact, you can get a 4-node 7250 system (e.g., https://www.exxactcorp.com/Exxact-TS2-210339-IPS-E210339) for order $5.5k right now (and that includes memory and OPA cards that you wouldn’t need for mining, and might be able to sell on ebay). At that price, you’re getting a 7250 (which should actually be even faster than the 7220s in this blog) for less than $1500 – and without needing a host machine to put them in, without having to fight with drivers (I already spent two days getting my Vega to work, still no luck!@#!@), and in a form factor you can move to any co-hosting facicility without any issues ….
I have no doubt that the Vegas are good (I’ll know more when I ever get mine to work!@#!@)… but at least for myself, I’m now going with the system listed above 😉
LikeLike
I was initially curious if the new ASUS B250 19-slot mobo would work for these cards, but after reading this, I have my doubts. I currently have 8x of the 7220’s as well – just need to finish out the setup.
LikeLike
If you have no other use for them: for the right price I’d take them ;-).
No, seriously – they definitely don’t work in all motherboards. I can confirm this machine works, and can confirm a smaller 1U 2card machine works (can find the exact specs); I even have three working in a regular workstation – but I have at least 10 machines I tried them in which they didn’t work at all.
LikeLike
Hello, lukMiner.
What precision do you use mostly in the miner?
I know that the Phi 72×5 (Knights Mill) doubles the performance in Single precision (FP32) and is 4 times higher with Half precision (VNNI). But in Double precision it’s 2 times lower.
LikeLike
Generally speaking mining doesn’t use floating point, so half/single/double doesn’t make a difference.
LikeLike
Also, are your 7220’s the active or passive cooling versions?
LikeLike
Passive – which is why I invested so much into that monster machine that they’re in – it has (barely) enough airflow to keep them cool.
LikeLike
I’m looking through some different options to help with cooling. Obviously lowering the ambient temp of the room will help with concerns to airflow, but is there anything else that can be utilized to help with cooling in your system?
LikeLike
There’s basically two options – buy a professional rackable server that’s optimized for a lot of airflow going through the “GPU” slots; or add some custom cooler fans to the back fo the card (for one of my builds I took these: https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2380057.m570.l1313.TR0.TRC0.H0.Xcustom+cooler+xeon+phi.TRS0&_nkw=custom+cooler+xeon+phi&_sacat=0 … the blue stuff didn’t actually fit (was intended for x100s, the 7220s have different power connectors :-/), but the blower itself worked. I’ll send another post on that second system later.
LikeLiked by 1 person
If you don’t mind, how much did you have to give up for the box they’re in now?
LikeLike
About 6k, including everything. See latest update on blog…
LikeLike
I know these things are fussy about what motherboard they’ll run in, but does anyone know if they’ll run in a x1 slot? This application doesn’t need high performance IO anyway, so it would be nice to use every slot available (assuming it works with the motherboard).
What were the symptoms of the systems they wouldn’t run in? They wouldn’t show up at all, or was it something else?
LikeLike
No clue. I have one in a x8 slot, but that’s all I know.
As to symptoms of incompatible motherboards: I’ve seen many different ones. Some don’t even boot with cards plugged in. Others boot, but immediately reboot. Some boot, and don’t show up in lscpi. And yet others boot, have the cards show up in lspci and micctrl, have them properly start in the mpss service (ie, you can even log in), but then see the cards “hang” (with DMA errors in dmesg) after some while. The latter is also what happens if the cards overheat (say, going over 90), so maybe that’s actually not a MOBO issue, but simply my inadequate cooling when I played with it.
That 8-card system can actually take 10 cards (it has 10 pci slots), but with 10 cards also starts showing the DMA errors – no idea if that’s because there’s then too many PCI agents, or if it’s simply overheating.
LikeLike
This is quite interesting. Have u ever try Intel VCA / VCA2.
LikeLike
I have not, for the simple reason that I don’t own any. If you know anybody at Intel that wants to donate or loan one to me I’d be happy to test 🙂
LikeLike
I did notice the miner seems to handle a lot of algo’s, does it also do Lyra2rev2? If not what would a bounty do to get it going?
LikeLike
I have never even looked at Lyra, yet. Right now it only does two algorithms – cryptonight, and cyrptonight light, both of which are nearly identical – and all the coins I mention are just variations of that same algorithm.
I do have a ether verion “mostly” working, but do have a day job, too, so progress is a bit slow (I can only work on that over a weekend).
If you want to have a conversation on lyra(?), please send me an email …
LikeLike
Is there something different motherboards or server which works Xeon Phi coprocessor as a cheaper?
LikeLike
I’m not exactly sure what you mean, but at least right now, the only options to buy Phis are in 2u 4node rack mountable server units. You can get a 4×7250 unit for about $5.5-$6k (say, from Exxact corp), and apparently, you can also get a 4×7210 unit from asrock rack for somewhat less than that (though in a different config).
LikeLike
Is there still a way to buy Xeon Phi 7220 coprocessor? I tried to find out it for a lot of countries’ shops, but I could not find them. I know how difficult to get it that discontinued series computer parts. But I just want to buy to get them. DO you know something able to buy 7220?
LikeLike
At least not right now. I bought mine on ebay, but that of course is a matter of luck…
LikeLike
Unfortunately, my current understanding is that they’ve been discontinued, and that the only way of getting any is being lucky on ebay…
LikeLike
Hi Luk, I’m planning on purchasing 2 servers from exxactcorp.com. Which will give me basically 2x 2U servers with 8x 7250’s. I’m hoping this will get me somewhere in terms of profitbility monero wise. Was planning to run them in my garage and hoping power bill wont be through the roof vs profit. Josh
LikeLike
Have a look at the latest article – there’s something cheaper than exxact 🙂
LikeLike
If this machine stopped mining it would be just a mass of money?
LikeLike
Not sure I understand the question?
LikeLike
The case looks very similar to Supermicro CSE-418GTS-R3200B. Is that by any chance a SuperServer 4028GR-TRT? I guess 4U case. What power supplies are in your box exactly?
LikeLike
Correct; that’s exactly it, with some e5-2609v4s put in. Built a second one from parts recently; exactly same config, works like a charm!
And yes, 4u case; with four PSUs, though I’d have to look up the wattage. I only have two plugged in, though, so you definitely don’t need all four.
LikeLike
Hello lukMiner, do you know that mining is now legal in Mexico? I am going to start mining myself and pay my taxes!!, I have never done that. Please quote your lukMiner under Linux to put the mirror (or your program) into a 5 dollar 16GB USB for AION and for ZEC Criptocurrencies BOTH, please send the quote to my email. I have 8 used Intel Xeon Phi Coprocessor 5110p PCI Blades. I know they are going to be 4-6 times slower than the 7220 new ones. But I have them already. I too need to know what NEW motherboard should I buy for them? Please advise me the exact brand and model number, and I understand (from reading your blog) that I can put any Intel processor on my motherboard? And NO DDR memory of any kind? Is this correct?
LikeLike
Re motherboard: I’d actually try to find an old surplus server that can take them, then you’ll also have the psus, the right airflow, etc (keep in mind your cards are ‘p’ for passive!).
Re “legal”: I’m actually surprised you mention that – was it _not_ legal there in the past???
Re memory: If x200s have integrated MCDRAM, and don’t need memory; the x100s that you have are PCI cards, so of course you will still need “some” host memory for the machine you plug them in (not much, though).
Re the “quote” for the USB ISO image – it doesn’t cost anything, feel free to download and burn.
Re ZEC: lukminer doesn’t support ZEC, only crytponight coins.
LikeLike
ReRe “legal”: What I mean “legal” is that the government is NOW buying from you, your Bitcoins and Ethereums (a lot of them), and controlling you pay the due taxes.
ReRe motherboard: Does a “passive card” will undermine or slower my Crypto Mining??
ReRe ZEC: AEON and AION is the same Crypto?
Thanks and congratulations in your work.
LikeLike