Phew – that took a while. And got me some heartburn ….. But here we are, finally: Asrock Rack has finally confirmed that their 7210 and 7250 based systems reach the performance I had been “expecting” (well: at least in the last two weeks, “hoping for” would have better described it)….
So before I say anything else, let me make the two key points of this article:
- you can get a 4-node 7210 Xeon Phi system from Asrock Rack for order $3k (full system, with four CPUs!)
- Asrock Rack has run some experiments of their own on these systems, and confirm 2800H/s per 7210 CPU, and 3kH per 7250 CPU on those system. Ie. roughly 11kH/s for order $3k (in the 4×7210 system)
Background
When I did my first measurements with 7210s I first used some home-grown system that I had built from parts (to which I literally had to take a steel saw at one point in time!), as well as – indirectly – systems from others that shared their achieved hash rate (to all those that did: Very grateful indeed!). Even back then I realized that I could get 2700-2750H/s on my 7210, but most users only got 2600-ish; and we could never figure out what the problem was. Then I found those 7220s, and got the 2800-2850H/s that I had expected, so everything looked good – until I finally got my first 4×7250 system two weeks ago, on which I still haven’t figured out why I only get 20-30% less performance than one would expect (drives me nuts!).
Note I did get independent confirmation that the 7250s can do 3kH/s – I measured that in my own K1SPE board, and another user confirmed – so I know the CPUs can get that performance …. but at least with this 4-node system I have so far I still haven’t gotten the performance I expected, so eventually started to get worried that maybe those server boards would not ever reach that performance … who knows?
Anyway – those fears are now finally laid to rest, because Asrock Rack has finally confirmed that at least for their system – with some tuning they’ve done to them – they can finally confirm roughly 2800H/s per Phi 7210, and roughly 3kH per Phi 7250 (Yeehaw!). In fact, since I like to share: here’s a little screenshot of my mailbox, showing the relevant part of email that just made my day:
About the Asrock Rack 2U4N Phi Systems
I will actually write a more detailed blog on their exact system later on, but at least for now: The Asrock Rack systems are 2U rackmount servers, with four Phi nodes each (you can chose whether to take 7210 or 7250s) – i.e., four full 7210s, or four full 7250s.
The particularly nice thing regarding the Asrock systems (apart from the fact that they’re now confirmed performant for lukMiner! 🙂 ) is that they come at a very compelling price: only around $3k per 4×7210 node, which is only around $750 for a complete node that can do 2800H/s. (in comparison, a Vega can do order 2000H/s, and costs way more even without adding the cost for the machine to mount it in!).
The reason these machines are so good on price is that Asrock has agreed to strip everything else from those systems that you wouldn’t need for mining: No memory (not required – the above screenshots are from nodes without any DRAM whatsoever), no OPA network cards (you have on-board ethernet, that’s more than enough), etcpp.
Again, I hope I’ll eventually find some more time to write a bit more in detail about the different options (asrock vs exxact, 7210 vs 7250, DRAM vs no DRAM, etc) – but at least for now, the two key take-aways are:
- confirmed 2800H/s (7210) and 3000H/s (7250) on the Asrock Rack systems
- you can get such a system with four CPUs at a pretty good price (order $3k for a 4×7210 system that makes order 4x2800H/s=~11kH/s).
Asrock Rack inquiries: Oh, almost forgot: if anybody that reads this is interested in talking to Asrock Rack to learn more about those systems, please use mining_phi@asrockrack.com … whoever gets this email knows what this is about, and what you’re looking for.
Until then: Happy mining!
PS: I don’t think I required to do any financial disclaimers a la “…we are long … in stock xyz … ” that financial writers have to do: but just in case (and just in case it’s not obvious): I do have a few of those systems coming myself … obviously… 🙂
The lukMiner blog 
Great article – thanks for sharing your experiences
LikeLike
Mate you are a legend, Will go Asrock direction now. Josh
LikeLike
Which algo do they achieve these hashrates on?
LikeLike
Lukminer currently ( 🙂 ) only supports crytponight coins. So monero, sumo, karbo, etn, bcn, aeon, …
LikeLike
Dang, pretty nice! CPU mining with these is more profitable than mining with GPU’s rn!
LikeLiked by 1 person
Dude you are the man! The amount of work you have done for Cryptonight mining is legendary. Selfishly I am glad that you aren’t well known on bitcointalk hahaha. I just emailed the address, hopefully I can pick up one of these. Is this the original model that you are referencing? http://www.asrockrack.com/general/productdetail.asp?Model=2U4N-F/X202#Specifications Looks like there is a free PCIE port in there, could probably shove in a Vega to take it to the next level hahahah.
PS. Thank you for making your miner download wget compatible!
LikeLike
Thanks Luke, will be ordering 2x 7210 systems from asrock rack. Too bad can’t get them till April/may. Josh
LikeLike
Luke, thanks for the write up. Can you comment on whether your setup requires 220-240 lines or are the supplies supported on 120V supplies. Do you know what your overall power draw is? What miner were you using to achieve the 2800-3000 H/s. What size drives were you running per node? Great article very exciting to read.
LikeLike
Re “what miner” – well, lukMiner of course 🙂 (https://sites.google.com/site/lukxmrminer/getting-lukminer)
Re power: Those are server PSUs, and therefore of course can take anything from 110 to 240. I have miner running at 110 at home, and at 210 in the data center where I move them after the initial build.
Re drives: The exxact system comes with drives (4x150GB SSD); the asrock rack system doesn’t include drives – what I use for mine is a linux installed on a 16GB. $5-a-piece, USB stick, that has the miner readily installed and run it on boot time (I’ll make that iso image available soon, just didn’t get to it, yet). With that, all I have to do when I get a new machine is pop four of those blank sticks into a workstation, copy the image onto them, five minutes later move those sticks to the knl machine, pop ’em in, and press the power button …. 🙂
LikeLike
Does anyone have a real world measurement of roughly what power draw we can expect at the PDU for 4x Phi system under load?
LikeLike
If anybody help can measure please feel free to post here; I’d be happy to approve it!
Actually just wanted to measure it myself, but apparently I fried my kill-a-watt meter when I tried it on the big 8×7220 machine last time :-/. Until then, here’s upper and lower bound:
– upper bound is obviously the power supply – it has 1600W PSU, so certainly wont’ draw more than that (probably far less, since we use no drives, memory, OPA cards, or other stuff that this is supposed to power). Note the system comes with two such PSUs, but the second is purely for redundancy, I dont’ even plug it in.
– more realistic: Each 7210 has a TDP of 215 W (https://ark.intel.com/products/94033/Intel-Xeon-Phi-Processor-7210-16GB-1_30-GHz-64-core), so assuming we’re maxing the chip that’d be 860W – plus ‘a bit’ for the boards and fans, and let’s say something between 1K and 1.2K would sound realistic to me.
I can also say I have two on one 110V 20-amp circuit without the breakers popping, which also supports my estimate. Of course, will actually measure and post as soon as I have a new watt-meter ….
LikeLike
Thanks for the response Luke. If I can get this machine I would be more than happy to buy the ISO image. I had one question left around buying from ASROCK. Did you or has anyone else tried to use an escrow company to protect your funds while waiting for the units to be shipped from Taiwan. A $3k payment and we promise to ship it to you in April is a little unnerving.
LikeLike
Escrow? No, I have not…. but I can confirm that my machine arrived.
I do admit I was initially a bit antsy, too, but more for fear of either the money or the machines might get “lost” somewhere on the way. Note asrock rack is not a random garage dealer; they’re the HPC subsidiary of Asrock (the motherboard manufacturer), and – at least as far as I know – a official Intel distributor. Again – my first machine arrived without problems, and I know at least one other that’s already gotten a shipment, too.
Maybe a thought: You could ask them if they’d sell oyu one through ebay – then you should have the ebay protection … but of course, ebay takes quite a cut, so you’d probably have to pay quite a bit more
LikeLike
I’m getting 2x Asrock servers with 4×7210’s each. Will cost me just under 10k landed to Australia. I will also use my power metre to measure the power and post results. I will be running these at home in the garage. Our house new house will have 20 solar panels using a 6Kw invertor so Hopefully during the day most of the power is provided by the panels. That should somewhat keep the costs down per day and keep the earnings on a positive run. Luke thanks for providing the asrockrack people.
I will either purchase an 2x ssd for the boot drives or as you say install centos straight to a fast usb drive then make copies of it so if usb ever dies I can just swap. And of course usb sticks are cheap as chips.
LikeLike
Can anyone comment on what the costs for import/duty/customs and shipping to the USA is? Also is there special paperwork that has to be filled out prior to the shipment hitting customs on US soil. I am trying to understand all the costs associated with this purchase.
LikeLike
Hi Michael, In Australia we have duty, customs, GST fees but my estimate of just under 10k is including all of this. 2 servers for this price is great. My previous quote of the exxact system peoples for 2 servers were just under 20k so a huge saving. Josh
LikeLike
Since asrock paid my customs fees I can’t say what those might be, but almost certainly less than the ones Josh was reporting from Australia!
In terms of paperwork: In my case DHL called me asking for my tax ID number because the shipment was over $2500, but other than that, nothing. (and yes, that’s normal – had this before when I ordered antminers from bitmain).
Shipping was order 200 or so in my case.
Of course, all that would be easier if there was a similarly good price to be had right in the US… if I find one, I’ll post it!
LikeLike
I can’t believe how nice and helpful everyone on this blog is. Josh if crypto completely went to zero what would you do with your hardware to recoup your investment. What is the resale value on machines like this stripped down for mining efficiency? I am super interested as well and I am one click from buying two servers myself. I just need to figure out the details. I was estimated $400 shipping and route fees. Not sure what the import fees would be but a broker quoted $186 for the paperwork to clear customs with no duty paid. Amazing the adventure one can have just following some links on the web and stumbling onto a mining Blog. here I was just happy with my Ryzen 7 1700 PC with a RX 580 card mining 1.2Kh/s
LikeLike
Hi Michael, if you read about Monero its protected from going below a certain limit. It’s not like bitcoin. I can see me getting back my investment just over a year. These machines are worth quite abit and if things don’t go my way then my gamble i’m taking will just be at a loss. it’s a risk i’m willing to take to try and venture into trying to make a little on the side.
LikeLike
Josh, could you explain more details about how Monero is protected from going below a certain limit? Curious to learn more.
Thanks!
LikeLike
lukas – interested to hear more about your colo arrangements. I’m waiting on a system and trying to get colo space – most local options dont support the kind of power densities these 2U machines require. I got responses like you can get 10A only if you buy 42U space. for a 2U server LOL.
Interested to hear about the economics of colo with these machines.
LikeLike
Well – yes, power density seems to be a bit higher with these systems than what many colos are used to :-). For my place, when I first talked to them they sounded like they thought I was a bit muddled, telling me that they usually have two _redundant_ 110V@10A rails for an entire rack (while I had asked or two _full_ rails, at _220_V, for _half_ a rack 🙂 ). But at the end it was merely a question of price – I could use their advertised special for the space, but had to pay for the extra power extra. A bit more than I’d have liked to (13ish cents or so), but I just _had_ to get them out of my house, so right now pretty happy with it.
Note there are also several readers of this blog that have contacted me saying “we have colo space, would that be interesting”? Maybe I should talk to some to set up some hosting service? Question is whether people would really want to sent their servers to somebody they dont know (I have the same scruples, BTW 🙂 ).
On the upside: Try finding a co-lo space that accepts do-it-yourself GPU mining rigs… I tried 🙂
LikeLike
Luk, why dont you run them in your home? Have you run out of space? it would be more profitable if you ran from your garage etc wouldn’t it?
LikeLike
Well, let me see: I have two phi workstations, a total of two such quad-units (three more coming), one system with 8 cards, another with 3, another with 2; one or two KNC testing machines; about 8 or so AMD multi-GPU workstations; about 4 or 5 NVidia multi-GPU ones; and a handful of other workstations etc thrown in (not counting my “regular” machines for gaming etc) …… so unless I want my house to appear as a glowing dot on the early-warning sattellites I had to move some of that hardware out of the house ;-).
No, seriously – there’s only so much power you can pull in an american residential buiding before your wires melt; I’m already pulling extension cords all across my basement to power all the machines I have in there; and that’s not nearly enough for everything; heat and noise not even considered….
So yes, for two or three machines you can always power them at home – but IMHO the fact that you _can_ easily move those machines off site to co-hosting is a big pro…
LikeLike
Wow man you have a lot of gear that would of cost an arm and a leg over a period of time. I will be looking forward to my servers coming can’t wait to kick start and get into it. Thanks for your blog and info regarding asrock servers.
LikeLike
Hi Josh,
I am also from Australia and interested in ordering 2 of these servers. Can you let me know how you are arranging for the shipment? Which shipping company? Is it by air or by boat? How do you estimate the tax and customs fees? I am new to this kind of thing. Thanks.
LikeLike
Josh can write more, but in my case it’s asrock that organizes the shipping (DHL in my case). They should be able to tell you how much it costs. Customs I don’t know (yet :-/)
LikeLike
Hi Matt, 2 servers costing me roughly $400 usd ($504 AUD) for shipping, You can use a customs calculator to get a rough estimate. I’m estimating roughly $1456 for Customs fees including GST etc. I’m have gone ahead and purchased the 7230 systems roughly $9000+ AUD. So just over $10k AUD landed.
LikeLike
Normally customs will contact you when the courier informs them of a package’s arrival. You send them their GST payment via internet banking and the package is delivered. We don’t often have this issue since shipments below $1000 – such as eBay purchases – are exempt.
LikeLike
Lol, yes. I do have a fair amount of hardware by now – having a wide range of different systems and OSes helps a _lot_ in hardening the software, and since all that hardware actually pais for itself I don’t mind buying it (_and_ you can deduct it from your taxes on crypto-earnings, too, so you kind of _want_ to spend it!). The only downside is that my basement is getting a bit loud, hot, and looks like a blade runner trash yard now. (Which is exactly why I’m starting to look more at rackacable machines that you can farm out!)
LikeLike
How is it possible for them to sell systems at this price when the cost of a single 7210 is over $1800? By the time they stuff just 2 processors in each system, they’re over the price. So I don’t understand how they can sell 4 on a Mobo with a power supply.
LikeLike
There’s a pricing special out right now, where intel distributors (such as asrock, exxact, colfax, etc) can get the “older” KNLs at a discount (and I meant “old” in relative terms – in that they’re older than the Knights Mills (KNMs) that are now out, too)
LikeLike
No response from the asrock email after two weeks. Have I missed out?
LikeLike
I don’t think so; I still get frequent emails from them. If you send me an email I’ll forward to my contact there.
LikeLike
It’s OK, I heard from Weishi today. Seems as if the Chinese New Year dragged on a little longer than expected in Taiwan. 🙂
LikeLike
Would love for you to forward an email for me. Let me know
LikeLike
Sure; send me an email and I’ll forward!
LikeLike
Hi Luke, thanks for the great work you done! I also sent the request to asrock rack but haven’t received any response so far. May I ask you to sent my email to you contact person as well. Sorry for my pushiness, just can’t miss a chance to buy one of those systems….
LikeLike
Asrock is working on it; at least that’s what they told me ….
LikeLiked by 1 person
Could you help me too please? I haven’t got any response from them :((
LikeLike
According to Asrock they had to first get a new supply of phis; but as far as I understood they already got it ordered. So I’d expect them to _eventually_ reply to all those requests ….
LikeLike
Josh:
I just visited whattomine.com and calculated the profitability for mining Monero at US$8.29 a day using Australian electricity rates of US$0.22/kw.h so these things will take a long time to pay for themselves. If you’ve committed to two machines and paid for them already, and are having any second thoughts, I’m willing to take one of them at your cost (inc. shipping and GST) when it arrives. I’m not wanting it for mining, I have a completely different use.
I’m in Brisbane in case that matters.
LikeLike
Never mind, I ordered one with 4 x 7230s in it. Too good a deal to pass up.
LikeLike
This is insane! Thanks for sharing. I guess Asrock must be backlogged or sold out. I’ve sent several emails over the past few days with no response.
LikeLike
Hey,
would you guys say that a rig with an ASUS C621E WS SAGE and 2 Xeon Phi 7250 work with Linux/Windows to do the 5000-6000h/s ?
LikeLiked by 1 person
Probably not. The Phi 7250s “probably” won’t work in that board: though I can’t say for sure I only know of three boards that take them so far: The supermicro K1SPE workstation board, the Asrock Rack board in the Asrock Rack servers, and whatever motherboards are in those Intel/Supermicro 2u4n “AP” servers. So unless you can get somebody to confirm to you that this board _can_ definitely take two phis (which I personally would find rather surprising) I wouldn’t hold me breath. (Too sad, though – would be great if it did!)
LikeLike
Great, thanks for the info. Though, 1 would work right ? Since its especially for that socket it should.
1 Box with 3000h/s would be okay too 🙂
LikeLike
I would double check first…. I really don’t know, but just because it will physically fit doesn’t mean the board will recognize it if it wasn’t designed for it…..
LikeLike
Hi Tim, does the Asus Board work? I have asked Asus, but no answer yet. I another forum i got message that x200 Series cannot run dual sockets. So just 1 one cpu per board. However would be cool to know if you have tested the board with one Phi..
LikeLike
Hi Luk,
Just wonder if there is any motherboard that is commercially available would take Phi 7210?
Thanks,
Ian
LikeLike
Other than those server machines there is only one I know of: The Supermicro K1SPE takes them (I actually built one this way, so know for sure it works). Problem is those are hard to get (they don’t usually sell individually), and even if they do it’s very hard to find a fitting cpu cooler/fan for it (I got a watercooler through CoolIT, but only because I knew somebody there :-/) ; and even if you get that there’s plenty of hurdles. For example, I have to start mine by shorting some pins on that mobo – fun, but maybe not everybody’s cuppa tea …
LikeLike
I was wondering, how many threads per core or CPU do you run on these Phi CPUs? I.e. 7230 has 64 cores and 4 threads per core. But there is only 32MB of L3 cache, so not that much to feed quickly all 256 threads with 1MB for each thread. Or do you relay on MCDRAM and use all 256 threads? Thanks!
PS. Ordering in progress…
LikeLike
MCDRAM, of course; it’s basically the Vegas’ equivalent of HBM.
LikeLike
I was wondering, how you can get such a high hashrate at 2800H/s. I try to use the lukminer latest version(0.11.0) on my Xeon Phi 7210 server,but only to get the hashrate at 2100+H/s. I have set the MCDRAM to the cache mode and set the nr_hugepages to 5000. My hashrate is too low compared from you showed in this blog. So, I want to ask you what problems may cause this difference and if there are any else settings to configure ?
LikeLike
If you’ll read through the various blog posts you’ll find four different things that will influence hash rate:
– mcdram settings: CACHE/QUADRANT mode is best (you at least have cache, i don’t know if you have quadrant)
– large pages – you seem to have that
– OS/kernel version: Generally ubuntu seems to hash higher than centos – shouldn’t, but apparently it does.
– lastly, BIOS version: the asrock systems should update to the latest bios that the asrock guys can provide – though for the non-asrock systems nobody has yet provided the proper BIOS with the same internal settings that the asrock bios has.
LikeLike
Thanks for your explains. Actually, My KNL server is not the asrock’s server but still has the bios settings which you mentioned. So I tried the different bios settings (OS is centos and use large pages,for some reason I can’t change OS yet):
1. CACHE,SNC4. This case got the hash rate at 2500H/s.
2. CACHE,QUADRANT. Hash rate is 2100H/s
3. FLAT,ALL2ALL. 2300H/s
4. FLAT,QUADRANT. 2600H/s
All cases were running with the numa memory bind and cpu affinity to get better performance. As you mentioned, centos did get a lower hash rate. Finally, I got the expected hashrate at 2600H/s in case 4.
But the funning thing is that CACHE/QUADRANT got the lowest hashrate while FLAT/QUADRANT got the highest. I think this problem may cause by centos’s memory management.
LikeLike
Whats the situation on this now? With Asics , xeon phi dead?
LikeLike
The cryptonight coins have generally done very well in changing their algorithms from time to time, so ASICs shouldn’t do much damage…. at least for the “modified” coins (monero, loki, ryo, etc). Non-modified coins like etn or sumo by now are asic coins, yes, but the modified ones will probably (or so I’d expect) change again as soon as anybody comes out with a new ‘v7’ asic.
LikeLike
Is ASRockRack still selling this unit? I emailed them, so just (hopefully) waiting for a response.
Regarding your “wheels within wheels” post, due to the low hash rate on the CPU you said you “Had to build a special version that ported the MPSS offload also into the luk-phi miner.” Is that version available for download?
Also, I was asking in my email how much one would expect to pay for a x200 pcie card.
Thanks
LikeLike
Hey – sorry for the late answer …. since I dont’ think anybody else ever even had a phi-within-phi machine I never bothered to put that version up on the web, but I certainly do still have it. As to the x200 cards – you can only get them on ebay, and I have no idea how much they are going for right now. I sold mine when crypto prices were _way_ higher; the way
LikeLike
If anyone is interested I have one of these ASRock Rack 4x Xeon Phi servers. I can sell it for $2700. I can send pictures and specs. I wanted it for scientific computing but after a long time searching I figured out it only has 2 channel memory. I need 6 channel memory. Don’t think that matters for mining. It will be on eBay very soon so reply here or look there or email me.
LikeLike
Hi John, I am interested. eBay account lukymusic
Thanks
LikeLike