Good news – based on some recent suggestion by “Mike” I ran some tests with the Xeon Phi SC7220A active PCI cards in various different Z170A motherboards (with Celeron CPUs), and that seems to work like a charm! (with about 3000-3100H/s for Sumo):
For those that don’t need the background, feel free to jump right down to the “The Results” section …. For those that want the full info, keep on reading ;-).
Now, before I say anything else, I first want to give a bit shout-out to “Mike”, which last week made a comment on the blog that pretty much said “I just put one in a Z170A Krait board, and it worked”… When I first read that I could have kicked myself that I didn’t get this idea myself: all the time I’m writing about the issues I’ve seen with getting those PCI cards to work in Xeon v3/v4 systems, and about the options with either newer or older Xeons … but not once did I ever think of simply trying a non-Xeon system. Duh – the trees obscuring the forest, I guess.
Anyway – I did follow up on his suggestions, and did try out some Z170s (looking good!), which is what this article is about.
7120As in Z170s? What’s the big deal?
Okay; one step back. In one of my recent articles I had mentioned that I had finally gotten my hands onto some of the Xeon Phi PCI cards – both passively cooled ones (7220P and 7240P) as well as some of the even rarer actively-cooled ones (7220A – extra credit if you can guess what the ‘A’ and ‘P’ are for!). Since then, I have of course spent my evenings and weekends trying to figure out how to actually best use those, in the most reliable way. One thing I can say up front: You should not even try to put a passive on into a open rig or a workstation (it’ll overheat within seconds, unless you install an extra fan), and neither should you put an active one into a server (the active obscures “passive” back-to-front airflow, and requires air from the side – if you put them flat into a server, back to back, then they won’t get enough air). Here the two cards side by side – as you can clearly see the ‘A’ variant wouldn’t do well in a chassis that assumes air flowing through the card …
That technical thing aside, what gave me the most headaches was the 7120A cards: For the passive ones, I’ve already posted about options of using old (v2-based) surplus servers; and of course there was the original 8-“GPU” SYS-7048GR-TR system that seems to work well with those cards (100% uptime since I bought it 🙂 ); and at least what the grapevine tells me the latest Skylake based server systems are also supposed to work with those cards without a hitch …. but not everybody is comfortable with server systems, and many of the crypto “enthusiasts” that read this blog would arguably prefer the active cards, to be put into a workstation, desktop, gamer PC, or even an “open rig”.
So when I got some, I was pretty happy – but unfortunately, most of my existing mining machines I had (almost all X99 boards with v3 Xeons) didn’t like them all too much: I had pointed out one such machines working in one my articles, but why the heck this one worked and the others didn’t, I still don’t know. I did find some more machines that worked with those cards (e.g., my Supermicro SYS-7048GR-TR reliably runs with two such cards), but most of the x99 boards didn’t. I tried putting them into some super-old Dell with a v1 Xeon, but no luck; and even the x99 based machine I used for writing the x100/KNC code doesn’t take them. That would leave some server boards with “v5” “Skylake Scalable” processors in them – but those aren’t exactly cheap, and probably not too widely used at home. So that did indeed give me something to think about.
With that, Mike’s idea of simply trying for a low-cost Z170 board was simply brilliant: The Celerons that go into those are wimpy little things, but all newest generation (v7 “Kaby Lake” and v8 “Coffee Lake”). Sure, they’re not Xeons, but nobody said they’d have to be Xeons … I was just so focussed on Xeons that I never even thought of it, but Mike did. Oh well, getting old, I guess.
The Experiment: SC7220As in Z170A Motherboards
Anyway – as soon as Mike mentioned his Z170 experiment I realized how stupid I’d been, and realized I’d have to sit down and run some more tests myself – yes it made total sense (after he mentioned it :-/), but it could still have been a lucky shot. Only way to figure out: Go on ebay, buy a bunch of different Z170 boards, get a stack of Celerons to put in (memory and PSUs I already had), and just try it out.
So, got five different boards with this chipset – both the Krait that Mike had reported about and four others – got the matching Celerons (only about 35 bucks each – really? That cheap??), and waited for them to arrive. For good measure, I also picked two other Celeron based boards (one H110 based Asrock Rack BTC mining board, and one Z370 based one, with a v8 “Coffee Lake” Celeron). To be super-sure I also got some really good CPU fans – they cost more than the CPUs themselves, but hey, just in case…. And finally, just out of pure curiosity, I also got some PCI riser cables (the 1x to 16x ones often used for GPU mining rigs).
Quite fittingly all that stuff arrived for last weekend – boxes over boxes (not everybody in my household happy about it) – and I of course had fun like a kid in a toy store. Took some DIMMs and PSUs I had lying around, unpacked it all, and plugged it all together – really simple; I guess taking the pictures took longer than racking it all up.
First surprise: The $35-Celerons actually already came with their own CPU fans, so not only did I spend more on the fans than on the CPUs, I don’t even need them at all. Oh well.
Other than that: Assemble the stuff, connect a PSU, google a bit to figure out which two Mobo pins to short to make it boot, and ready to go (for those that don’t have a case and are curious, it’s the two back ones on the four-pin row of JFP1 – differently located on every board, but always labeled JFP1):
7220A’s in Z170’s: The results!
As to the outcome of these experiments:They’re pretty good – a few downsides (I’ll talk about those in a minute), but bottom line seems to be that the Z170s take exactly one 7220A each. I haven’t gotten them to boot with two such cards (likely not enough PCI lanes in the CPU); but all the ones that booted without the card also booted and ran with one (one of the boards didn’t boot at all – DoA – but that has nothing to do with the card).
Overall here’s the four boards I successfully ran:
- ASUS Z170 Pro Gaming Aura, with a G3930 “Kaby Lake” Celeron (2 cores, 2.9GHz, 4MB L3 cache)
- MSI Z170A PCMate, with a G3930.
- MSI Z170A Gaming M5, with a G3920 Kaby Lake (only 3MB cache)
- MSI Z170A Krait Gaming, again with a 3930.
For each of these four systems, the very first thing you should do – before plugging in a card – is to go into the BIOS, enable “advanced mode” (press F7), then look for the PCI settings, and enable the “above 4GB decoding” setting (in one of those boards also – quite fittingly – called the “cypto mining support” – go figure!).
Once done, all four of these systems worked perfectly with the 7120As; I now have two of them running continuously since last weekend, without any issues – perfect! The one issue I’ve seen was that the 3920 Celeron with only 3MB cache led lukMiner to crash when trying to mine sumo: Though I only want to use the MPSS miner it still tried to also initialize the CPU miner, and since 3MB cache isn’t enough to run even a single thread it bailed – oh well, guess I’ll have to make a 0.10.8 soon that fixes that. Other than that sumo-related problem, this board and CPU still run fine.
For each of these four systems I also tried running two cards, but then they don’t even boot up. I also tried putting in a GPU, and that worked without any problems – tried both a “full” Zotac GTX 1070, as well as a wimpy, cheapo, PCIe-1x Zotax GT 710, and both worked. You can also run completely without a GPU – the Kaby Lakes have built-in graphics, and that worked just as well.
What did not work …
Though the above is pretty good news, a few things I tried did not work out. For example, the risers I bought didn’t work. Even with only one card – no luck.
As stated above I also tried a H110 based board, and a Z390 one. The H110 didn’t work at all (no clue, actually looks like DoA, not KNL related at all). The Z390 (with a v8 “Coffee Lake”) is a bit more interesting: It “kind of” seemed to work: it booted, it recognized the cards, it brought up the MPSS stack, and everything ….. but unfortunately, it seems to use a newer Ethernet chip than what the CentOS 7.3 on my lukStick recognizes, so I couldn’t get the miner to connect to the network, and to actually mine. As such, at least until I fix this driver issue I call this a failure, too, even though it seems like the board and chip are working fine with the card.
The Bottom Line…
In summary, I’m actually pretty happy about this outcome. Yes, getting the Z390 to work would have been nice; and of course, it’d have been great if I could actually have gotten two cards to work in each board (still don’t see a reason why it wouldn’t work). But even so – every one of the Z170s I tested (ex the one that was DoA) worked in a reproducible manner, and in particular, reliably. Also great: the cards do seem to automatically adjust the fan speed based on heat, so I’ve not once seen any one of the active cards overheat, despite the fact that with order 15 cards in that one room (4.5kW!) it’s getting pretty warm at times.
Also good: These Z170s seem to be widely used, are easy to get, and are dirt cheap, as are the CPUs they use (pretty sure I paid less than $150 on average, including the CPUs). Add in a single stick of cheap DRAM, and a cheapo PSU (all you need for one board with one card is a 450W PSU, max), and there’s a very cheap way of getting those cards to work. And since each of those cards makes about 3100H/s for sumo right now, that’s a very interesting option – all without building a server, without having to find a professional colo service, etc.
So with all that said: Mike, you’re a genius!
With that – Happy Mining!