Wheels within wheels… or better: Phis within Phis?

Another weekend, another article – this time about a special from-scratch “build” for a cute little desktop machine that contains a bootable Xeon Phi 7250 as main processor, and – to top it off – also some Phi 7220A KNL cards… so “wheel within wheels”, indeed!

This article is actually kind-of long overdue – part of it starts as early as early 2017(!), and even when it comes to writing I had initially planned an article for as early as January (as the time stamp on some of the photos indicated), but then forgot about it …. though truth be told, the second half of the article about adding the cards (which likely is being the more fun one) didn’t happen until more recently, anyway. So, better late than never.

Now before I write anything else, let me be clear that at least for this article it’s not exactly clear how “practical” it will be for the average reader. There was a ton of work in that one, most of the pieces are insanely hard to get, and in combination I don’t think that was the most profitable use of my time …. but it’s also been a huge amount of fun, so I’m not ruing it, either.

How did this all start …

The base motivation for this particular build was exactly the same as what several others have since experienced: I found some cheap KNL “ES/engineering sample” processors on Ebay (7250s, I think), thought that this would be a very cheap and easy way of building my own Phi box, and just bought them. After all, $300 for a $2k list price processor? And all one needs is a motherboard, plug it in, and done, right? How hard could this be? In particular, that was in early 2017, before all the pricing specials came out (and when coin prices where sky high), so this sounded like a really good deal.

Now once I had these Phis the “get a motherboard and plug it in” turned out to be a little bit harder than I had thought – back then, there were only two boards available that could take those chips: The “Adams Pass” server boards for the 2u4n Adams Pass servers, and the Supermicro “K1SPE” desktop boards that are also in those Supermicro SYS5031 KNL Desktop systems (and in the Colfax “Ninja” DAP systems – which were super-cool, by the way). Of those, the AP systems were near impossible to get in barebone form, and though I finally found one single node (for a horrendous price!) from some canadian reseller I never ever managed to get a chassis for it, and never figured out how to power up that board without said chassis (which has the power connectors for those nodes 😦 ).

So, option 2 it is: Build a desktop machine (which was the preferred option, anyway, since you can actually put that under your desk without going deaf). Now for that I needed a Supermicro K1SPE board, which turned out to be equally hard to find – apparently Supermicro tried to enforce some policy that those could be sold only in fully assembled systems (far too expensive), so finding one took forever – but eventually, I did.

OK, got a board, got a CPU, plug it in, power it up…. except that you need a fan, and try find one that fits this board…. sigh. Tried leaving off the fans and just blowing lots of cool air on it, and that made it at least boot – but only for a few seconds, then it shut off (surprise). After lots and lots of googline I eventually found a distributor in China that sold a heat sing that claimed to be compatible with this socket, except it wasn’t. The combination of “steel saw”, “file”, and “lots of sweat” eventually did convince it to fit the board’s screw holes, but by that time I had taken the CPU in and out so often (it gets stuck to the heat sink via the cooling paste :-/) that I seem to have bent some pins, and the board wouldn’t boot any more. Oh shucks.

Now, end of 2017 I went to the SC17 (“Supercomputing 2017”) conference in Denver, and while there, also strolled past the “CoolIT Systems” booth: CoolIT was the company that built the water-coolers for the Colfax DAP systems (as the big sticker on those coolers very clearly states), and though they didn’t sell those coolers individually, they were – upon asking – helpful enough to help me get one (well, maybe it didn’t hurt that I had worked with them in the past, in another capacity). Maybe not the most practical way for the average person to get hold of one, but either way – I turned lucky, and got one.

So, now having a cooler, it was time to reactivate this build. Sent the board in for repair (another $80, plus shipping, plus hours on Supermicro’s support web page), and finally got it back. And this is where the story starts.

Part 1: Building a Phi Desktop System

OK, at this point – Jan 2018 – I had the following pieces: A Xeon Phi 7250 bought off ebay; a Supermicro K1SPE board; and a CoolIT water-cooler that fit this board.

So, step 1: Take out my brand-new(ly repaired) K1SPE board, and plop in the 7250:

Step 2: Temporarily attach the water-cooler, and screw it on (tip: it might be easier to do this after you put it into a case :-/) :

Now, to test it all out, hook up some PSU (I used an EVGA Supernova 850). The board connects as usual, but for the cooler I couldn’t find a good way to connect so used a fan extension able that plugs into a SATA molex:

Now, to make it boot, I first had to find and read the manual: Since the board is not a regular ATX board it also doesn’t fit the regular power button etc connectors from a regular ATX case …. so had to read the manual to find the pins that the power button would go to if I only had the right proprietary SuperMicro case (you can see which two pins to short from the image below). Note it actually takes a while to boot – it doesn’t just boot up upon the first touching of those pins, you actually have to hold them shorted for a few seconds. Weird, but that’s what it is. Either way, eventually it booted.

Note I did add in a GPU as well – just for testing – but of course that’s not required, since the board does come with a VGA on-board graphics port (and in fact, for the BIOS config it’s easier if you use the VGA port, since the GPU “may” not yet be active to show it by that time. Also don’t get worried if it takes forever to boot – that is normal…. can be up to a few minutes until it beeps and enters BIOS, at least in my board. Either way, eventually it did boot, and I could enter the BIOS to change MCDRAM to cache/quadrant mode:

Interestingly my BIOS complained about something with KNM incompatibility – which doesn’t make the slightest sense because I have a Knights Landing CPU – not a Knights Mill one (at least as far as I know!?), so that was funny. Either way, it eventually did allow me to make the changes, anyway (just showing that for the fun of it 🙂 ), so everything turned out well.

Rebooted, added a harddisk, installed, and used this as a development machine for quite a while (initially even without a case – who needs a case, anyway!?).

Step 2: Putting it into a case

OK, though one doesn’t need a case, I thought that for the article it’d be way nicer to actually put it into one … so took a Phanteks Enthoo case I had from another (GPU) build, and put that in. Now why an actual section on that? Because as with everything “special built” it turned out it wan’t all that easy, of course.

The problem with fitting it into a case is that the board is not an ATX form factor board. It has about the same dimensions – so it physically fits – but …. but…. the screws to screw the board into the case are in different positions (at least some of them), so they don’t exactly fit. Now some do fit, so my first reaction was simply “I’ll just use the ones that fit” – after all, why would a board need more than two or three screws, anyway – it’s not going to fall out, is it? Well, let me give a clear warning here: The problem is not which board holes do not have any screws fitting them, but which “expected” screws have no holes on the motherboard: turns out there’s a few of those little thingys where the screws are supposed to go into in places where the board doesn’t have holes, and these do scratch the board from the bottom – which I only realized because it no longer wanted to boot (probably shorted something). Turns out I was lucky, and no lasting damage was done – but I did definitely have to take the board out again, find all the “offending” screws, and take them out.

Second problem was that after taking out all the wrong screws there’s so few screws that the board in some areas touches the case (in particular when you press on it to insert PCI cards, etc), so after the previous experience I decided to first cut out some insulation (the foam that comes in most motherboard shipping boxes), and put that in between board and case metal. Screwed on the remaining three or so screws that did have a fitting hole, and perfect. Squeezing in the water cooler (I put it at the top, so it can readiate out on the case’s top) was a bit tricky, but eventually – with some gentle nudges – it actually did fit. So here’s the complete system:

IMG_0486

Part 3: Wheels within Wheels

Now those first two parts were actually done in January 2018 (actually, over the Christmas/New Years’ holidays), and initially, I just put in some GPUs to use the machine as a plain old desktop box. But of course, if you already have some Phi-powered motherboard, wouldn’t it be fun to also use some Phi PCI cards in that? Phis within Phis?

Well, when I first built this machine I had already had some 7220P passive Phi cards from ebay – but those turned out to be horribly hard to cool in a desktop setup (see related article), and though I did think of moving the turbo-fans from that desktop build over into the KNL box it simply didn’t fit. So at least initially, I abandoned this idea – I was doubtful it’d even work, and I wasn’t going to spent all the work trying to rig something up that wouldn’t work.

Now later this year, as I wrote in another article I finally got my hands on some actively cooled Phi PCI cards, and ever since then wanted to play around with trying those in this machine – if I could get Z170 board to take one, maybe the phi board would take one, too?

Well, some time last week somebody that also reads this blog eventually asked me exactly this question: Do the K1SPE boards take Phi cards? And with that question, I finally decided to finally just try it out. And turns out, it does: First tried only one card, removed the old harddisk (which didn’t have MPSS on it), and tried with a regular lukStick … and turns out, it worked like a charm. Now getting bold, I also plugged in the second card, and lo and behold, that one still works. Amazed – all kind of boards have issues with those cards, and the one board that takes two cards without the slightest hitch probably wasn’t even ever intended to work with them.

Here a picture, of the full system: Note I used an old Celeron plastic box as a “spacer” to improve airflow into the cards (else they are back on back).

img_0492.jpg

With this experiment successful, only one thing had to be fixed: The existing luk-mpss-knl miner for mpss offload actually expected a regular (ie, non-KNL) host CPU, and thus got only low hash rate on the host CPU. Had to build a special version that ported the MPSS offload also into the luk-phi miner, but with that, lo and behold, we not have ca 9 kH in that one box: ca 2.6k from the host CPU (it’s centos, after all, and runs the MPSS offload), and two times 3.2kH from the cards…. which is nigh on perfect:

OK, that’s it for today – really have to get some food now…. I do want to stress again that this is probably not the most practical build to try and replicate – but I sure do hope you’ll enjoy it, anyway… it’s been super fun to build, so hope you’ll have some fun just by reading.

With that: Happy Mining!

PS: One of the fun things of this build is that it’s still using the EVGA Supernova 850G: A 850W power supply for a total of three KNLS (of 300W TDP each), plus board, case, fans, etc…..

 

Making lukSticks under Windows …

In the last few days I’ve received several questions from users that had issues with the lukSticks…. in particular, while most found them rather useful from the linux side of things, for the main user group for which they were intended – Windows users that aren’t comfortable with linux – there were some issues. And as I myself am not all too much of a windows person, I never ran into these issues myself, and just assumed that whoever is a windows person will likely figure out how to do all the iso image burning etc. Well, seems out that wasn’t the case, so here a few quick instructions.

First: Update to lukSticks v3.1

Before I go into the actual burning instructions: If you still have a lukStick image pre the images I uploaded last night (yes, I still have to write about those …): Please update to those “v3.1” image (they’re at http://www.lukminer.net/releases/lukSticks-v3.1). These v3.1 lukSticks not only have the latest version of the miner you’ll need for the post-sumo times, they also have several improvements in particular for windows users:

  • reduced size: To adjust for USB sticks’ slight variations in “usable” size the old lukSticks already had some “unused space” at the end, so when burnt onto a sligthly too small USB stick no real harm would be done, because it was only ‘unused’ space that was lost. That in theory worked, but apparently some burning programs simply threw an error if they couldn’t write the whole thing, so the new sticks are only as big as they really need to be (Duh, should have done that in the first place). The cpu-phi one should be about 7.5GBs in size, the mpss-knl one is about 15 GBs in size (it’s so much bigger because it basically contains 8 live images for the up to 8 KNLs).
  • DOS/Windows newline/line feeds: In the old lukSticks, the config files were on a DOS/Windows readable FAT file system, but unfortunately were written under linux, so contained only unix-style newlines … meaning that whenever one opened those under Windows one would see all the individual config file lines squashed together into a single line; and if one did succeed in editing those, the editor almost invariable saved with windows-newlines, which then confused the mining script, leading to errors such as “unknown algorithm ‘-a sumo<newline>'” … The new sticks fix that: the lukMiner.cfg file is in DOS newlines, and the miner script converts those on the fly when running.

Of course, make sure to download the right stick. Currently, there’s two versions (the KNC one is still missing):

  • mpss-knl: That’s the one to use only if you have a system with 7220A or 7240 PCI cards (which usually have regular Xeon/Core/Celeron host CPUs). Do not use those on the bootable phi asrock/exxact etc machines. Note on terminology: the “mpss-knl” stands for “mpss offload for knl cards”.
  • cpu-phi: This is the one to use if you have an Asrock, Exxact, Neon Miner, etc based system (the cpu-phi stands for “cpu or phi”)

Unzip

The iso images I’ve uploaded are gzipped (of course), so unzip them first until you have the full ‘.iso’ images. I used 7zip (https://www.7-zip.org/), but any other tool should do.

Burn to Disk (“imageUSB”)

Now this is the tricky stage, because “apparently” the default “right-click then “burn to disk”‘ option under windows doesn’t always do what one expects it to do (at least one user had issues with that).

The tool used is called “imageUSB” (https://www.osforensics.com/tools/write-usb-images.html), and it seems to work just fine. I’m not much of a windows user, so I don’t have any opinions on this vs probably many other tools – it’s just the first one that google came up with (except one other one that wanted me to register for something), and it seems to do the trick.

Here a screenshot of using this tool, on a brand new SanDisk Cruzer 16GB stick:capture-lukstick-begin

Once it’s done burning, it should show something like that:capture-lukstick-burned

Starting to configure: Un-plug, and Re-plug

Now after you’re done burning (and just as the README says you should): First unplug your USB stick, and re-insert it – else Windows won’t realize that there’s now different file systems on that stick.

When you do so, what should happen is that windows eventually (after possibly a few seconds) find two new drives (D: and E: in my case), it’ll likely open the file explorer (or whoever this thingy is called under windows), and – and this is perfectly OK – it’ll open a window saying it can’t read one of those drives, and suggests to format it … something like the following:

capture-lukstick-reinserted

Note this “error” is perfectly OK, so do not format this drive! What happens here is that the USB stick contains two partitions (what windows calls “drives”) – one windows one with the config files (D:, which is OK), and of course the linux partition with the linux OS and miner – and this of course windows can’t read. Do not format it, though, or you won’t have a linux OS (or miner) to run from ….

Last: Edit the lukMiner.cfg config file

Once you click away the annoying “format” windows, just open the windows-readable drive (if it didn’t open automatically), and start editing the lukMiner.cfg file, which on my machine looks roughly like that:

capture-lukstick-editing-lukMiner-cfg

Note how in the v3.1 version this config file is nicely editable under windows, with proper newlines and everything, so from that point on you should be safe.

With that: Happy Mining!

PS: And yes, I did go through that – two times already – and the resulting USB stick seems to work just fine!

 

New release (v0.11.0) – please update!

Dear Users – it’s been a while since I last posted (been rather busy, both in day job and in mining) – and even for this one I’ll keep it short (tons of things to write about, but not enough time right now). For now, I’ll only post this little update and request.

New Release – v0.11.0

After a ton of work I’ve finally put live the latest release (v0.11.0), that contains an equal ton of updates, bugfixes, improvements, etc – I’ll write a bit more about that later, when I’ve also updated the new lukSticks (that will have an equally long list of fixes and improvements…); but the latter will take a little bit longer because I want to run just a few more burn-in tests.

Though the lukSticks are still copying, the miner release itself is already live, at the usual place (http://www.lukminer.net/releases).

To everybody that mines Heavy : Please update ASAP

If you’re mining cryptonight heavy, chances are you’ve seen – or at least heard about – the blow-up of sumokoin. I still don’t completely get who are the good guys and who are the bad guys; but at the end of the day, that’s neither here nor there – the bottom line is that after a long of what looks like a civil war in this coin, the Sumo community has decided to go back from cryptonight heavy to the original, old, cryptonight classic.

Now if you like this (I certainly don’t!) you can still mine sumo with the “-a xnclassic” flag – but since Sumo will now be ASIC-mineable whatever you’re going to put to it with lukMiner probably won’t make a dent. Maybe at least for Sumo that change is for the better – who knows – and if it does prosper this way then I’ll wish it all the best (I still have a lot of currently locked coins, so won’t complain) …. but bottom line, that coin is now no longer of interest to me.

All that said, the real problem is that v0.10.7 actually diverted all developer shares to a sumo pool – but with sumo having just changed back to xnclassic those hashes will no longer be valid, so chances are your miner will stop working. If you were mining sumo to start with you’ll already have found that out; but if you’re mining another ‘heavy’ coin (I’m currently going Loki and Saronite) then you’ll probably run into this trap: The miner itself will still be able to mine Loki and Saronite – but since the devShares won’t work it’ll likely die, anyway. As such: If you are mining any heavy coins, please update to 0.11.0 immediately.

I’ll send a few more updates later, but at least for now that’ll have to be it.

With that…. Happy Mining!

 

Phis, H110 Pro BTC+ boards, and risers …

Aaaand – another big shout-out for one of my readers: “somebody” (and I can’t for the life of me find the comment to figure out who it was…) recently mentioned he’d gotten his Phi card to work in an Asrock H110 Pro BTC+ board, and with a riser to boot … and since in my last post re the Z170A desktop boards I had explicitly mentioned that I had unsuccessfully tried both this board and some PCIe riser cables (also in the Z170 boards) I at first found this a bit incredible. However, he also sent me a picture, so apparently I must have done something wrong last week, and decided to re-do that experiment… and lo and behold, it does indeed work…. Hm. Oh well.

First, I looked up what I had actually written about the H110BTC board – and turns out I had never actually said that it didn’t work with the cards, I had only said it didn’t boot, and that I suspected that it was DoA – but after re-assembling it again it actually booted up just fine. Maybe I didn’t have the board’s external power connectors in correctly last week (the H110’s have these weird external SATA power cable connectors to power the PCIe slots), or maybe I had simply forgotten to put in the memory (not the first time – I keep forgetting that non-Phi CPUs do need memory :-/), but either way, the board without any cards booted up just fine.

Second, I dug out my PCI1 1x-to-16x riser cables I had recently gotten on ebay, hooked it up to a 7220A card, plugged in a lukStick, and indeed, it booted up just fine. First time the system didn’t properly boot the card (0xff mic status in the dmesg, and “error” status in “micctrl -s”), but by that time the card had already shown up on the PCI bus (“lspci | grep Device”), the kernel module has already loaded, and “micctrl” already showed that there was a card in the system – so that was a good sign. Rebooted again, went into the BIOS, and changed a few settings – in particular, I set the PCIe speeds from “auto” to the highest the boards offerened (Gen2 for some, Gen3 for others!?), and forced primary graphics to “on board” (above 4GB decoding was already on), and rebooted … and la-voila, it booted up like a charm, with riser cable that I had claimed wouldn’t work with those cards, and everything. Duh. Either way, here two pics of the “final” rig – maybe not the most professional rig, but all I could do between “getting up” and “getting my coffee”….

Bottom line: Unlike what I may have said in previous articles and replies to comments: The H110 Pro BTC+ board apparently does work with the 7220A card (at least, with one – I haven’t tried more, yet), and the 7220A works with PCIe 1x-to-16x risers (at least in this board – haven’t tried in other boards, yet).

With that – happy mining…

 

PS: Of course, the main reason for using riser cables is to use more than one card, which I haven’t tested yet. I’ll post info on that as soon as I find the time to run that.

SC7220A (active) cards confirmed working in Z170A Motherboards

Good news – based on some recent suggestion by “Mike” I ran some tests with the Xeon Phi SC7220A active PCI cards in various different Z170A motherboards (with Celeron CPUs), and that seems to work like a charm! (with about 3000-3100H/s for Sumo):

img_0413.jpg

For those that don’t need the background, feel free to jump right down to the “The Results” section …. For those that want the full info, keep on reading ;-).

Background

Now, before I say anything else, I first want to give a bit shout-out to “Mike”, which last week made a comment on the blog that pretty much said “I just put one in a Z170A Krait board, and it worked”… When I first read that I could have kicked myself that I didn’t get this idea myself: all the time I’m writing about the issues I’ve seen with getting those PCI cards to work in Xeon v3/v4 systems, and about the options with either newer or older Xeons … but not once did I ever think of simply trying a non-Xeon system. Duh – the trees obscuring the forest, I guess.

Anyway – I did follow up on his suggestions, and did try out some Z170s (looking good!), which is what this article is about.

7120As in Z170s? What’s the big deal?

Okay; one step back. In one of my recent articles I had mentioned that I had finally gotten my hands onto some of the Xeon Phi PCI cards – both passively cooled ones (7220P and 7240P) as well as some of the even rarer actively-cooled ones (7220A – extra credit if you can guess what the ‘A’ and ‘P’ are for!). Since then, I have of course spent my evenings and weekends trying to figure out how to actually best use those, in the most reliable way. One thing I can say up front: You should not even try to put a passive on into a open rig or a workstation (it’ll overheat within seconds, unless you install an extra fan), and neither should you put an active one into a server (the active obscures “passive” back-to-front airflow, and requires air from the side – if you put them flat into a server, back to back, then they won’t get enough air). Here the two cards side by side – as you can clearly see the ‘A’ variant wouldn’t do well in a chassis that assumes air flowing through the card …

That technical thing aside, what gave me the most headaches was the 7120A cards: For the passive ones, I’ve already posted about options of using old (v2-based) surplus servers; and of course there was the original 8-“GPU” SYS-7048GR-TR system that seems to work well with those cards (100% uptime since I bought it 🙂 ); and at least what the grapevine tells me the latest Skylake based server systems are also supposed to work with those cards without a hitch …. but not everybody is comfortable with server systems, and many of the crypto “enthusiasts” that read this blog would arguably prefer the active cards, to be put into a workstation, desktop, gamer PC, or even an “open rig”.

So when I got some, I was pretty happy – but unfortunately, most of my existing mining machines I had (almost all X99 boards with v3 Xeons) didn’t like them all too much: I had pointed out one such machines working in one my articles, but why the heck this one worked and the others didn’t, I still don’t know. I did find some more machines that worked with those cards (e.g., my Supermicro SYS-7048GR-TR reliably runs with two such cards), but most of the x99 boards didn’t.  I tried putting them into some super-old Dell with a v1 Xeon, but no luck; and even the x99 based machine I used for writing the x100/KNC code doesn’t take them. That would leave some server boards with “v5” “Skylake Scalable” processors in them – but those aren’t exactly cheap, and probably not too widely used at home. So that did indeed give me something to think about.

With that, Mike’s idea of simply trying for a low-cost Z170 board was simply brilliant: The Celerons that go into those are wimpy little things, but all newest generation (v7 “Kaby Lake” and v8 “Coffee Lake”). Sure, they’re not Xeons, but nobody said they’d have to be Xeons … I was just so focussed on Xeons that I never even thought of it, but Mike did. Oh well, getting old, I guess.

The Experiment: SC7220As in Z170A Motherboards

Anyway – as soon as Mike mentioned his Z170 experiment I realized how stupid I’d been, and realized I’d have to sit down and run some more tests myself – yes it made total sense (after he mentioned it :-/), but it could still have been a lucky shot. Only way to figure out: Go on ebay, buy a bunch of different Z170 boards, get a stack of Celerons to put in (memory and PSUs I already had), and just try it out.

So, got five different boards with this chipset – both the Krait that Mike had reported about and four others – got the matching Celerons (only about 35 bucks each – really? That cheap??), and waited for them to arrive. For good measure, I also picked two other Celeron based boards (one H110 based Asrock Rack BTC mining board, and one Z370 based one, with a v8 “Coffee Lake” Celeron). To be super-sure I also got some really good CPU fans – they cost more than the CPUs themselves, but hey, just in case…. And finally, just out of pure curiosity, I also got some PCI riser cables (the 1x to 16x ones often used for GPU mining rigs).

Quite fittingly all that stuff arrived for last weekend – boxes over boxes (not everybody in my household happy about it) – and I of course had fun like a kid in a toy store. Took some DIMMs and PSUs I had lying around, unpacked it all, and plugged it all together – really simple; I guess taking the pictures took longer than racking it all up.

First surprise: The $35-Celerons actually already came with their own CPU fans, so not only did I spend more on the fans than on the CPUs, I don’t even need them at all. Oh well.

Other than that: Assemble the stuff, connect a PSU, google a bit to figure out which two Mobo pins to short to make it boot, and ready to go (for those that don’t have a case and are curious, it’s the two back ones on the four-pin row of JFP1 – differently located on every board, but always labeled JFP1):

IMG_3099

7220A’s in Z170’s: The results!

As to the outcome of these experiments:They’re pretty good – a few downsides (I’ll talk about those in a minute), but bottom line seems to be that the Z170s take exactly one 7220A each. I haven’t gotten them to boot with two such cards (likely not enough PCI lanes in the CPU); but all the ones that booted without the card also booted and ran with one (one of the boards didn’t boot at all – DoA – but that has nothing to do with the card).

Overall here’s the four boards I successfully ran:

  • ASUS Z170 Pro Gaming Aura, with a G3930 “Kaby Lake” Celeron (2 cores, 2.9GHz, 4MB L3 cache)
  • MSI Z170A PCMate, with a G3930.
  • MSI Z170A Gaming M5, with a G3920 Kaby Lake (only 3MB cache)
  • MSI Z170A Krait Gaming, again with a 3930.

For each of these four systems, the very first thing you should do before plugging in a card – is to go into the BIOS, enable “advanced mode” (press F7), then look for the PCI settings, and enable the “above 4GB decoding” setting (in one of those boards also – quite fittingly – called the “cypto mining support” – go figure!).

Once done, all four of these systems worked perfectly with the 7120As; I now have two of them running continuously since last weekend, without any issues – perfect! The one issue I’ve seen was that the 3920 Celeron with only 3MB cache led lukMiner to crash when trying to mine sumo: Though I only want to use the MPSS miner it still tried to also initialize the CPU miner, and since 3MB cache isn’t enough to run even a single thread it bailed – oh well, guess I’ll have to make a 0.10.8 soon that fixes that. Other than that sumo-related problem, this board and CPU still run fine.

For each of these four systems I also tried running two cards, but then they don’t even boot up. I also tried putting in a GPU, and that worked without any problems – tried both a “full” Zotac GTX 1070, as well as a wimpy, cheapo, PCIe-1x Zotax GT 710, and both worked. You can also run completely without a GPU – the Kaby Lakes have built-in graphics, and that worked just as well.

What did not work …

Though the above is pretty good news, a few things I tried did not work out. For example, the risers I bought didn’t work. Even with only one card – no luck.

As stated above I also tried a H110 based board, and a Z390 one. The H110 didn’t work at all (no clue, actually looks like DoA, not KNL related at all). The Z390 (with a v8 “Coffee Lake”) is a bit more interesting: It “kind of” seemed to work: it booted, it recognized the cards, it brought up the MPSS stack, and everything ….. but unfortunately, it seems to use a newer Ethernet chip than what the CentOS 7.3 on my lukStick recognizes, so I couldn’t get the miner to connect to the network, and to actually mine. As such, at least until I fix this driver issue I call this a failure, too, even though it seems like the board and chip are working fine with the card.

The Bottom Line…

In summary, I’m actually pretty happy about this outcome. Yes, getting the Z390 to work would have been nice; and of course, it’d have been great if I could actually have gotten two cards to work in each board (still don’t see a reason why it wouldn’t work). But even so – every one of the Z170s I tested (ex the one that was DoA) worked in a reproducible manner, and in particular, reliably. Also great: the cards do seem to automatically adjust the fan speed based on heat, so I’ve not once seen any one of the active cards overheat, despite the fact that with order 15 cards in that one room (4.5kW!) it’s getting pretty warm at times.

Also good: These Z170s seem to be widely used, are easy to get, and are dirt cheap, as are the CPUs they use (pretty sure I paid less than $150 on average, including the CPUs). Add in a single stick of cheap DRAM, and a cheapo PSU (all you need for one board with one card is a 450W PSU, max), and there’s a very cheap way of getting those cards to work. And since each of those cards makes about 3100H/s for sumo right now, that’s a very interesting option – all without building a server, without having to find a professional colo service, etc.

So with all that said: Mike, you’re a genius!

With that – Happy Mining!

Phi-based “NeonMiner” v3 system now uses lukMiner (and currently has stock!)

Bottom line: “NeonMiner.com” now has lukMiner-powered four-node Xeon Phi systems, in stock, in the US…

Background

My latest article on the Xeon Phi PCI cards has stirred quite some feedback; however, though these cards are undoubtedly more interesting than a rack-mount server from a “rigging”/”modding” perspective, I can’t say often enough that those cards are discontinued products, do not come with any warranty, and may or may not work in whatever system one plugs them in (I’ll write some more on that soon – but yes, it’s a 50-50 thing right now).

As such, I’d like to point out that at least for those interested in more “production” mining (in the sense of “buy it, plug it in, and let it make some money”) I would overall still suggest to not have those fancy cards distract you from the much more ready-to-go Xeon Phi four-node servers I had written about before.

In terms of those servers, the – by a wide margin – biggest concerns people have brought up over the last few months are two-fold: First, the worry about buying machines on the other side of the planet (with questions like shipping cost and time, customs fees, etc); and second, the not inconsiderable delays of actually getting the machines … because they are built to order, in high demand, often sold out, and in short, typically come with up to a few months of delivery times.

The “NeonMiner”

For those that need something quicker: Have a look at the latest version of the “NeonMiner” (www.neonminer.com). The guys there now offer a phi-based system (running lukMiner), and as far as I understand they have machines directly available.

A bit of background: I first stumbled over the NeonMiner name by chance, when I saw one of those systems listed for sale on Amazon, several months back — but then at a pretty steep price (before the Phi “specials” kicked in), and without much mention of what software it would come with. At about the same time, the guys behind this system apparently heard about lukMiner, and contacted me through the blog. We’ve since loosely worked together, in the sense that I helped hook them up with the right contacts to get the pricing specials used by the other Phi suppliers (meaning those NeonMiner systems are now way cheaper!), and at the same time suggesting that lukMiner be preinstalled to make the systems easier to use (which they immediately did). As a result, the latest version of the NeonMiner now comes with lukMiner pre-installed, ready-to go.

Though we did exchange quite some emails, I myself am not part of the NeonMiner team, nor do I even have a system at hand to “take apart” and have a closer look at. As such, I do not know what exact hardware components that system is comprised of, nor how the software setup works, etcpp; however, judging from the pictures and numbers on the website,  the system appears to be a 4-node 2U “Adams Pass” system similar to what you’d get if you order from Intel, Exxact, or Colfax; probably with four Phi 7210s or 7250s (I honestly do not know which, yet, though I’d expect the latter). This 4×7210/7250 assumption would also be consistent with the hash rate of about 11+kH/s claimed on their web site, so this number is perfectly plausible to me (and though the web site doesn’t say that, since it runs lukMiner it should also run cryptonight heavy, light, etc).

What other components (memory, disks, …?) the system may or may not come with I cannot say, nor do I know what exactly they’d cost. However, one thing the web site is very clear on (and which they’ve confirmed to me when I double-checked!) is that they have systems “in hand” right now, in the US (and apparently, it even comes with a 6-month warranty), which should address at least some of the issues I’ve mentioend above:

neonminer-instock

As such, if anybody here wants to get their hands on some 11+kH of Cryptonight/V7 hashing power in a ready-to-rack and ready-to-run system then feel free to contact them. If or when you do, make sure you mention that you found them through this blog, which – as I was told – should give you a discount over the official price. Full disclosure: If you do mention the blog I, too, will get a (small) referral fee (though I do hereby promise that by the end of the year I will donate any such referral fee to a charity!).

If any of you guys are getting any of those systems, please let me (us?) know; as said above I haven’t gotten my hands on any of those systems yet (sure, I could buy one, but have no spare power to run it right now, anyway :-/), so if anybody is willing to share their experiences on what hardware exactly is included, shipping time and cost, how to configure/update the miner, measured performance and wall-draw wattage, etc, please share!

With that,

Happy mining!

 

SC7220A (active) cards: They exist! They do well!! And I got some!!!

I’ll write a bit more on this topic soon, but wanted to at least drop a brief note before I find the time for a real article: I’m talking about the “active-cooled” Xeon Phi SC7220A PCI cards – they really do exist! They do pretty well (around 3000H/s on crytponight heavy/sumo). And yes, I finally got my hands on some!

In past articles I’ve called them “rare”, “rumored”, and in one instance even “as elusive as a two horned unicorn” when somebody mentioned he might have one (turned out he didn’t, it was something else). I had heard that there’s three models of the Phi cards – the passive 7220P and 7240s, and the active 7220As – but so far had never ever gotten at anything other than the 7220Ps. Not the 40’s, and certainly not the active ones. In fact, until somebody listed an “A” on ebay the day before yesterday I had never even seen an image of one. Yet finally, I got my hands on some. Wow. That took a while.

Why is this such a big deal? Until today, all I ever had for phis was either the 7220Ps, or the bootable 7210/7250s (Asrock, Colfax, etc). The latter ones are great if you’re building a farm or want to go co-location to a data center – I have some, they’re great, and I recommend them to everybody that goes that route. But – they’re somewhat less interesting for the “home miner” that is used to building his own mining rig “GPU style”, with a cheap motherboard, CPU, PSU, minimum RAM, and just plugging in a PCI card.

The 7220Ps I got a while back on ebay are “kind of” going in this direction, but as described in my two past posts on this topics (e.g., the original one on building a desktop mining rig and last week-end’s one on putting 7220Ps in surplus servers) I had already pointed out that those passive cards need a lot of really strong airflow to stay cool (well, at least once you start the miner :-/). And that, for regular desktops or mining rigs, is simply not the case – and once they overheat, they’ll turn off, you’ll simply see “DMA error”s in the dmesg, and can power-cycle.

Mounting additional high-powered fans as I had done in my desktop rig is a solution for a “proof of concept”, but only borderline practicable. With an “A” (ie. “active”) card, though, the card itself has a fan – just like a GPU – so cools itself … meaning you can actually mount it in a desktop, workstation, or free-standing rig – I now had my first machine running about 48 hours, and still going strong (2 cards in one box, together about 6500H/s on sumo). Of course you still need the right mobo that it’ll work in, but that’s something for another post – I now have two machines where I have A’s working in, and at least two other users seem to have that, too, so “we’ll figure it out”. Luckily this experimenting will be relatively easy, because the mpss-knl lustick I did for the servers seems to work just as fine with desktops… and that saves a ton of time.

Anyway – for now I got a stack of those two-horned unicorns (and a few 7240P’s, too, for good measure), and will start experimenting with them. Already ordered a range of different mobos and CPUs to test with, and will certainly keep everybody updated on that through this blog. Also, I won’t be able to power all those cards myself, so will soon start to sell some off on ebay (also get my investment back – I did have to pay for those cards :-/). If you see some appear in the ‘lukMiner’ ebay account – then yes, that’s I (and of course, you can also drop me an email if you want to save the ebay fees).

At least so far that’s it – will soon update you guys on which rigs I get them to work in and which ones not. ASAP, I promise (weekend, probably – I do have a day job).

Until then – happy mining!