Good news – based on some recent suggestion by “Mike” I ran some tests with the Xeon Phi SC7220A active PCI cards in various different Z170A motherboards (with Celeron CPUs), and that seems to work like a charm! (with about 3000-3100H/s for Sumo):
For those that don’t need the background, feel free to jump right down to the “The Results” section …. For those that want the full info, keep on reading ;-).
Background
Now, before I say anything else, I first want to give a bit shout-out to “Mike”, which last week made a comment on the blog that pretty much said “I just put one in a Z170A Krait board, and it worked”… When I first read that I could have kicked myself that I didn’t get this idea myself: all the time I’m writing about the issues I’ve seen with getting those PCI cards to work in Xeon v3/v4 systems, and about the options with either newer or older Xeons … but not once did I ever think of simply trying a non-Xeon system. Duh – the trees obscuring the forest, I guess.
Anyway – I did follow up on his suggestions, and did try out some Z170s (looking good!), which is what this article is about.
7120As in Z170s? What’s the big deal?
Okay; one step back. In one of my recent articles I had mentioned that I had finally gotten my hands onto some of the Xeon Phi PCI cards – both passively cooled ones (7220P and 7240P) as well as some of the even rarer actively-cooled ones (7220A – extra credit if you can guess what the ‘A’ and ‘P’ are for!). Since then, I have of course spent my evenings and weekends trying to figure out how to actually best use those, in the most reliable way. One thing I can say up front: You should not even try to put a passive on into a open rig or a workstation (it’ll overheat within seconds, unless you install an extra fan), and neither should you put an active one into a server (the active obscures “passive” back-to-front airflow, and requires air from the side – if you put them flat into a server, back to back, then they won’t get enough air). Here the two cards side by side – as you can clearly see the ‘A’ variant wouldn’t do well in a chassis that assumes air flowing through the card …
That technical thing aside, what gave me the most headaches was the 7120A cards: For the passive ones, I’ve already posted about options of using old (v2-based) surplus servers; and of course there was the original 8-“GPU” SYS-7048GR-TR system that seems to work well with those cards (100% uptime since I bought it 🙂 ); and at least what the grapevine tells me the latest Skylake based server systems are also supposed to work with those cards without a hitch …. but not everybody is comfortable with server systems, and many of the crypto “enthusiasts” that read this blog would arguably prefer the active cards, to be put into a workstation, desktop, gamer PC, or even an “open rig”.
So when I got some, I was pretty happy – but unfortunately, most of my existing mining machines I had (almost all X99 boards with v3 Xeons) didn’t like them all too much: I had pointed out one such machines working in one my articles, but why the heck this one worked and the others didn’t, I still don’t know. I did find some more machines that worked with those cards (e.g., my Supermicro SYS-7048GR-TR reliably runs with two such cards), but most of the x99 boards didn’t. I tried putting them into some super-old Dell with a v1 Xeon, but no luck; and even the x99 based machine I used for writing the x100/KNC code doesn’t take them. That would leave some server boards with “v5” “Skylake Scalable” processors in them – but those aren’t exactly cheap, and probably not too widely used at home. So that did indeed give me something to think about.
With that, Mike’s idea of simply trying for a low-cost Z170 board was simply brilliant: The Celerons that go into those are wimpy little things, but all newest generation (v7 “Kaby Lake” and v8 “Coffee Lake”). Sure, they’re not Xeons, but nobody said they’d have to be Xeons … I was just so focussed on Xeons that I never even thought of it, but Mike did. Oh well, getting old, I guess.
The Experiment: SC7220As in Z170A Motherboards
Anyway – as soon as Mike mentioned his Z170 experiment I realized how stupid I’d been, and realized I’d have to sit down and run some more tests myself – yes it made total sense (after he mentioned it :-/), but it could still have been a lucky shot. Only way to figure out: Go on ebay, buy a bunch of different Z170 boards, get a stack of Celerons to put in (memory and PSUs I already had), and just try it out.
So, got five different boards with this chipset – both the Krait that Mike had reported about and four others – got the matching Celerons (only about 35 bucks each – really? That cheap??), and waited for them to arrive. For good measure, I also picked two other Celeron based boards (one H110 based Asrock Rack BTC mining board, and one Z370 based one, with a v8 “Coffee Lake” Celeron). To be super-sure I also got some really good CPU fans – they cost more than the CPUs themselves, but hey, just in case…. And finally, just out of pure curiosity, I also got some PCI riser cables (the 1x to 16x ones often used for GPU mining rigs).
Quite fittingly all that stuff arrived for last weekend – boxes over boxes (not everybody in my household happy about it) – and I of course had fun like a kid in a toy store. Took some DIMMs and PSUs I had lying around, unpacked it all, and plugged it all together – really simple; I guess taking the pictures took longer than racking it all up.
First surprise: The $35-Celerons actually already came with their own CPU fans, so not only did I spend more on the fans than on the CPUs, I don’t even need them at all. Oh well.
Other than that: Assemble the stuff, connect a PSU, google a bit to figure out which two Mobo pins to short to make it boot, and ready to go (for those that don’t have a case and are curious, it’s the two back ones on the four-pin row of JFP1 – differently located on every board, but always labeled JFP1):
7220A’s in Z170’s: The results!
As to the outcome of these experiments:They’re pretty good – a few downsides (I’ll talk about those in a minute), but bottom line seems to be that the Z170s take exactly one 7220A each. I haven’t gotten them to boot with two such cards (likely not enough PCI lanes in the CPU); but all the ones that booted without the card also booted and ran with one (one of the boards didn’t boot at all – DoA – but that has nothing to do with the card).
Overall here’s the four boards I successfully ran:
- ASUS Z170 Pro Gaming Aura, with a G3930 “Kaby Lake” Celeron (2 cores, 2.9GHz, 4MB L3 cache)
- MSI Z170A PCMate, with a G3930.
- MSI Z170A Gaming M5, with a G3920 Kaby Lake (only 3MB cache)
- MSI Z170A Krait Gaming, again with a 3930.
For each of these four systems, the very first thing you should do – before plugging in a card – is to go into the BIOS, enable “advanced mode” (press F7), then look for the PCI settings, and enable the “above 4GB decoding” setting (in one of those boards also – quite fittingly – called the “cypto mining support” – go figure!).
Once done, all four of these systems worked perfectly with the 7120As; I now have two of them running continuously since last weekend, without any issues – perfect! The one issue I’ve seen was that the 3920 Celeron with only 3MB cache led lukMiner to crash when trying to mine sumo: Though I only want to use the MPSS miner it still tried to also initialize the CPU miner, and since 3MB cache isn’t enough to run even a single thread it bailed – oh well, guess I’ll have to make a 0.10.8 soon that fixes that. Other than that sumo-related problem, this board and CPU still run fine.
For each of these four systems I also tried running two cards, but then they don’t even boot up. I also tried putting in a GPU, and that worked without any problems – tried both a “full” Zotac GTX 1070, as well as a wimpy, cheapo, PCIe-1x Zotax GT 710, and both worked. You can also run completely without a GPU – the Kaby Lakes have built-in graphics, and that worked just as well.
What did not work …
Though the above is pretty good news, a few things I tried did not work out. For example, the risers I bought didn’t work. Even with only one card – no luck.
As stated above I also tried a H110 based board, and a Z390 one. The H110 didn’t work at all (no clue, actually looks like DoA, not KNL related at all). The Z390 (with a v8 “Coffee Lake”) is a bit more interesting: It “kind of” seemed to work: it booted, it recognized the cards, it brought up the MPSS stack, and everything ….. but unfortunately, it seems to use a newer Ethernet chip than what the CentOS 7.3 on my lukStick recognizes, so I couldn’t get the miner to connect to the network, and to actually mine. As such, at least until I fix this driver issue I call this a failure, too, even though it seems like the board and chip are working fine with the card.
The Bottom Line…
In summary, I’m actually pretty happy about this outcome. Yes, getting the Z390 to work would have been nice; and of course, it’d have been great if I could actually have gotten two cards to work in each board (still don’t see a reason why it wouldn’t work). But even so – every one of the Z170s I tested (ex the one that was DoA) worked in a reproducible manner, and in particular, reliably. Also great: the cards do seem to automatically adjust the fan speed based on heat, so I’ve not once seen any one of the active cards overheat, despite the fact that with order 15 cards in that one room (4.5kW!) it’s getting pretty warm at times.
Also good: These Z170s seem to be widely used, are easy to get, and are dirt cheap, as are the CPUs they use (pretty sure I paid less than $150 on average, including the CPUs). Add in a single stick of cheap DRAM, and a cheapo PSU (all you need for one board with one card is a 450W PSU, max), and there’s a very cheap way of getting those cards to work. And since each of those cards makes about 3100H/s for sumo right now, that’s a very interesting option – all without building a server, without having to find a professional colo service, etc.
So with all that said: Mike, you’re a genius!
With that – Happy Mining!
The lukMiner blog 
Well, I guess I just picked the wrong board before I knew what I wanted lol. My H110 Mobo is recognizing the 7120, but it just doesn’t want to go. You suggested to get the thing flashed first and I will be doing so tonight. I will let everyone know how it turns out. Crossing my fingers I can actually get it to mine tonight. I think I am close! Wish me Luk.
LikeLike
I’m flattered that you called me a “genius” when I was really just too impatient to wait for my Supermicro rackmount server and just threw it in some stuff I had laying around. Thank you for doing a more comprehensive survey of the consumer landscape. I think this will really help a lot of people as the server grade stuff (even the older stuff) is still cost prohibitive for many. Most people will probably have this combo lying around as it’s popular for GPU mining so there isn’t a need to buy new hardware in addition to the Phi.
I wish I had clarified earlier that I didn’t get 2 units to work on 1 mobo in case I lead you on there, but was also able to get 1 GPU and 1 7220A/P to work.
LikeLike
Quick question, just curious as to not supporting Sumokoin on the x100s. Is it a limitation of the cards? obviously these mobos have limitations on being able to, so is it the same issue?
LikeLike
Ladies and Gentlemen, you may add the Gigabyte H110-D3A to the list of working Mobos… and boy do those x100 Ps run hot. Have two high CFM fans going on it. If you hear a house burning down in Coral Gables, it might be mine.. . Good luck out there
LikeLike
Anyone have a solution for the noise of a blower fan I need for a passive one? The wife is complaining about the 65db one I got. Anyone have something in the 45 db range I can use for these things? Please help or I may have to just sell and move on. I am begging you guys because I spent a lot of time getting this up and it was mining and I don’t want to just sell it now because of noise…
LikeLike
I had/ve* the same noise problem. I tried to use it by separating power and ground cables and connecting directly to PSU and connecting remaining rpm and pwm cables to a case fan header. Then using software i tried to reduce the load, which helped reducing the noise. However, with enough load, I guess %70, to cool off, it was still noisy.
Then I jumped on buying larger blower fan from delta, BFB1212HH (Actually I wanted buy BFB1212EH, but bought HH version by mistake instead). That one helped reducing noise even though when it is running at full load, as it is capped at a lower rpm, however, that one seems to cool the card barely enough. It works so far, though.
Then I bought another one, BFB1212GH, same size but has similar specs with EH version, and hence more powerful badass one, but didn’t have a chance to try it out as it comes with a different fan header (mini pci 6 pin, actually), so I need to figure out which cable is which before powering it on. But that one will need pwm control like the first one I mentioned, it is almost rated at 4amps and will probably be quiet noisy at full load as well, and adding pwm software to the mix something I don’t want to do to avoid further complications, so, I am not touching it yet. Because, as you can imagine, I spent enough time as well.
*: Using 120mm HH fan still makes noise, but not as much as 90mm fans. So I couldn’t remove noise problem altogether, just reduced it to be somewhat more manageable.
PS: I actually also tried creating a wind tunnel out of cardboard and use hi static pressure case fans but no, that wasn’t enough and therefore won’t recommend it.
LikeLiked by 1 person
Nice! Care to share an image? 😉
LikeLike
Sure. Here are the two pictures showing my add-on blower fan with duct tape magic 🙂
And the badass one that waits be tested 🙂
Actually using duct tape is not a good idea for this setting, as it is not heat resistant and can peel off (which actually happened on the twin brother of this one before I sold it) . Therefore I bought a high temp teflon tape as well. But like testing other fan, I didn’t have the chance, nor the will, to test it yet.
LikeLike
Seriously cool, man! 😉
Though hate to that – an active card would have saved you a ton of trouble 🙂
LikeLike
Indeed 🙂 But like you, I have been looking for them like forever and by the time they are available, I already spent time figuring out using passive cards.
I am still considering to buy one, but based on your findings, I may be out of luck as I have a H110 MB, AsRock btc one, and I prefer using risers, which also seems to be a problem with Active cards. The one at the picture works without any issues with risers for instance.
LikeLike
Well – _I_ now have them :-). Send me an email, let’s chat about this offline – I’m seriously interested in this riser thing, and if s passive one worked the active one should too (there’s literally no difference whatsoever other than the fan!), so rather curious on that – in particular why it works for you but didn’t for me…. send me an email!
LikeLike
Hi,
It’s funny that I have just read your reply after you figured out already, sorry for not returning earlier but I didn’t see your reply before. I have actually been looking for reference temp values for 7220p, specifically for VCCCLR one, as it runs around 100-101 Celsius at the moment. Then I decided to check your blog again and saw that you were able to successfully run 7220A using H110 chipset. That’s great news.
When I was reading your update I realized that you were using SATA powered risers and let me warn you about those risers, as even though I believe you are very experienced with higher spec machines, you seem to use risers and regular motherboards only recently. It is IMPORTANT to not to use SATA powered risers or in an event of other power sources available in the same riser, use others (MOLEX or 6 PIN PCIE). Reason being why SATA power cables are not suitable for powering risers is because of the following two reasons, in order of importance;
1) They are not designed to deliver more than 54 watts of power at 12V Rail and 22.5 at 5V rail (and AFAIK 5V is not used by riser cards). Because PHI cards designed to draw power proportionally by each power interface, PCIE, 6 pin and 8pin respectively, it consumes 60+ watts from PCIE interface, as it is rated to allow 75W. Hence, continuous use can cause meltdowns, and fires eventually. So be careful about that. A quality MOLEX cable, with thick cables, should be normally be capable of delivering enough juice, as the connector is designed to allow up to 13A current (vs 4.5A SATA) and 6 pin PCIE at least 75W (but I think it can be more, even though that’s not the standard).
People usually use those risers with GPU’s, and sometimes using undervolting as well, and GPUs prefer to draw power using PCIE power cables primarily, and PCIE socket power is secondarily used AFAIK, as opposed PHIs’ proportionate approach, which allows them to getaway using SATA power cables, as they are not loaded as much, specifically if they are undervolting.
You can see my current values below for power draw, for instance:
micsmc-cli show-data –freq
mic0:
Observed Frequency and Power Usage:
Core Frequency : 1.2990 GHz
PCIe : 68.0 Watts
2×3 : 64.0 Watts
2×4 : 113.0 Watts
Average 0 : 245.0 Watts
Current : 245.0 Watts
Maximum : 248.0 Watts
VCCP : 74.0 Watts
VCCU : 6.0 Watts
VCCCLR : 49.0 Watts
VCCMLB : 46.0 Watts
VCCMP : 16.0 Watts
NTB1 : 3.0 Watts
For more information, I strongly advice visiting this site, which I have used many times when I started designing my mining rig: http://www.playtool.com/pages/psuconnectors/connectors.html.
2) Many SATA connectors, if not all, are made using “molds”, which, based on manufacturer quality, could cause short circuits, specifically when they are under stress. That is not something I would risk, given the nature of 24/7 mining business, because it may be a fire hazard. For more info, please check this video as well: https://www.youtube.com/watch?v=TataDaUNEFc. IMHO, it is not either isolated to a few bad examples.
Also, do not connect multiple power cables to same riser if multiple power inputs are present. Specifically if power sources are different, i.e. different PSUs, accidentally did it once and both riser and PSU burned, likely saved the GPU, and I did it again same mistake last week,(facepalm), but, thanks GOD, nothing happened.
Anyways, I was going to ask your reference temp values, at least for VCCCLR value, for a 7220p card, or 7220A, if you no longer use Passive ones. Mine are like below at the moment:
micsmc-cli show-data –temp
mic0:
Thermal Information:
Die : 72 C
Fan Exhaust : 64 C
VCCP : 74 C
VCCCLR : 101 C
VCCMP : 85 C
West : 55 C
East : 60 C
I will appreciate it, if you can check it out and let me know about it. Thanks.
LikeLike
Wow. Lots of info – in particular this “micsmc-cli” tool sounds very handy … kind of ridiculous that I didn’t even now about it, but either way: a bit “thank you” for pointing this out! (and I’m sure other readers will like to read about that, too!)
LikeLike
Oops. I wrote a rather long response, and realized that I misunderstood what you said about SATA power cables, in the first place, LOL. You were talking about powering the MOBO, not risers. Actually you are using 6pin PCIE powered SATA risers, as I saw in one of the pictures. Anyways, at least i provided some reference info, to you and may be others as well :). I think I need to sleep.
LikeLike
Hey guys,
I have two 7210s CPUs
Would like to run them in Dual Mode.
I cannot find a Board with two balls sockets. I only found ASUS C621E WS SAGE. But still waiting for Asus confirming. The Sockets are p sockets and lga3647 with Chipsets 612. What makes me nervous is the fact that each socket provides 205 Watt max. So not enough right?
LikeLike
Though you’re not the first one to ask I’m reasonably sure that the 7210s are _not_ going to work in dual-CPU setups. Though in the course of writing this blog I’ve seen and heard a lot of things from users that I was sure wouldn’t run (e.g., a 7250 desktop system booting and running windows!?), I’m still reasonably sure that dual-mode won’t work. If anybody does get it to work after alll please let me know (in which case I’d post about it right away), but I’ve never seen or even heard about a dual-socket system in either my mining or non-mining professional life… and since lots of poeple have asked about it, I’m reaonably sure that by now somebody would have told me :-/. As such – your best option of “high density” for the Phis is probably one of the 4n2u servers, or, if in a workstation/open rig scenario, a single-socket phi board with 7220A PCI cards…
LikeLike
Oops. I wrote a rather long response, and realized that I misunderstood what you said about SATA power cables, in the first place, LOL. You were talking about powering the MOBO, not risers. Actually you are using 6pin PCIE powered SATA risers, as I saw in one of the pictures. Anyways, at least i provided some reference info, to you and may be others as well :). I think I need to sleep.
LikeLike
Hi Chapo,
did you receive a answer on the motherboard ASUS C621E?
I’m in the same case like you.
Thanks
Carlos
LikeLike
This question has come up a lot; I still haven’t received conclusive answer from anybody, but I’m _almost_ sure phis will _not_ work in that. If anybody has data to contradict that, please let me/us know….
LikeLike
I confirm that asus has no compatible motherboard for this king of cpu.
I have contacted the asus support.
LikeLike
Guys, is there any way to control or set the fan of the 7220A card? Mine switch between 50% RPM and 100% RPM every few seconds, it is very annoying as the 50% noise level is acceptable but 100% is way too much. And every few seconds you hear it switches between these 2 levels is irritating. I rather keep it like constant 75% but doesn’t look like it can run at 75%. Hope someone can help! thx!
LikeLike
_I_ have never found out how …. pretty sure there is, but no clue how. Anybody else knows?
LikeLike