电子技术论坛_中国专业的电子工程师学习交流社区-中电网技术论坛

标题: 基于Pine A64+的一款Cluster [打印本页]

作者: module-eccn 时间: 2016-12-14 16:15 标题: 基于Pine A64+的一款Cluster

在这个应用中，作者用Pine A64+做了一款cluster。基于64位四核ARM A53芯片，运行在1.2GHz，千兆以太网和1GB DDR3内存。

Bargain 5 Node Cluster of PINE A64+Building a 5-node cluster for under ￡75 (￡175 including five quadcore PINE A64+s)
After the interest in my cluster of Raspberry Pi 3s in April, I was keen to try building a cluster with some of the other excellent SBCs now on the market. PINE 64 in the States generously sent me 5 of their latest PINE A64+ ARM boards, each with a 64-bit quad core ARM A53 SoC running at 1.2GHz, gigabit Ethernet and 1GB of DDR3 memory. Launched as a Kickstarter-funded project it was hugely successful, raising over $1.7 million dollars from tens of thousands of enthusiasts.

The completed cluster measures 146.4 (w) x 151 (h) x 216mm (d) and weighs 1.5kg. (5.6 x 5.9 x 8.3", 52oz)
PINE A64+ vs Raspberry Pi 3The PINE A64+ board has similar specifications to the Raspberry Pi 3, but has faster memory, much faster networking and yet is quite a bit cheaper even when you include international shipping to the UK:

Model	PINE A64+	Raspberry Pi 3
SoC	4-core ARM A53 1.2GHz	4-core ARM A53 1.2GHz
Memory	1GB DDR3 (2GB option)	1GB DDR2
Network	1000Mbps	100Mbps
WiFi	option	802.11bgn
Bluetooth	option	4.1 + BLE
Infrared	yes (header)	no
Storage	microSD card	microSD card
USB spare	2 fitted	4 fitted
Video	HDMI 1.4 CEC, DSI-LCD	HDMI, DSI
Camera ports	CSI	CSI
Audio	3.5mm, SPDIF (header)	3.5mm
Microphone	yes (headphone+mic jack)	no
Size	127 x 79mm	85 x 56mm
Power	1.0 → 2.9W 2A max, microUSB	1.2 → 2.1W 2.5A max, microUSB
Launched	Q2 2016	Q1 2016
Price (UK)	￡25.661	￡33.59
1US$19 A64+ + $12 shipping + 20% VAT + 0% import duty = ￡25.66

3D design in SketchUp
I modified my original Raspberry Pi cluster design using the free version of SketchUp and built rough 3D templates of the PINE A64+s, network switch, USB hub, sockets, etc. I didn't bother to include ventilation slots/grills in the 3D model.

Download SKP file for SketchUp 2013

2D design in InkscapeI used the free Inkscape application for 2D design, ready for exporting to the laser cutter. Each colour is a different pass of the laser, at different power/speed levels, so the green lines are cut first to make holes for ports/screws/ventilation, pink are extra cuts to help extract delicate parts, orange is text/lines that are etched and finally blue cuts the outside of each panel.

Download files for laser cutting on one 600x400x3mm sheet:

Read more about the laser cutting and the screwless case-clipping system in my original article. I probably went a little overboard with so many ventilation holes, which meant the laser-cutting took 45 minutes to finish.Design Changes from the Raspberry Pi 3 clusterAlthough similar to my cluster of Raspberry Pi 3s, I made a number of changes and improvements:

I kept the horizontal mounting rail design but the A64+ has M3 holes which are easier to find parts for, rather than M2.5 on the Pi. And the holes are further apart because the overall board size ismuch bigger than the Pi. Screwing the plastic nuts onto horizontal rails is very tedious, and I'd like to find some sort of removable plastic ‘clip’ that holds the boards in place along each rail.
Internal Case Fan vs External – I was sure that the A64+ boards would benefit from active cooling, and so I stuck with an ultraquiet 92mm fan, but moved it inside the case, and to the back rather than on top.
Case USB ports – while perfectly functional, I never really liked the look of the twin USB ports on my Pi cluster, so for the A64+ cluster I sourced a more compact 2-port USB mount.
Internal USB hub on case side vs Internal shelf – attaching the USB hub to the case side panel meant I could simplify the design (it can now be cut from a single sheet of 600x400mm acrylic), and so reduce the costs. Removing the shelf would have reduced the rigidity of the case, but screwing the horizontal mounting rails to the side panels keep it secure. I also moved the USB hub right up to the rear case panel, so it could be plugged in without needing a case-mounted socket & lead, which again saves costs, and avoids the need for soldering a custom cable.
Right-angle USB extenders – I didn’t realise until everything else was finished that the USB cables would stick out too far at the back and block the fan so I had a last-minute purchase of two right-angle extenders so I could connect up the USB. A neater solution would be making/buying a single cable.
Anear 60W 6 Port USB vs SAVFY 50W 6 Port USB – I was hoping to find a 6-port USB hub that could be screwed directly onto a case panel (without needing any glue), but the cheap SAVFY 50W proved unsuitable, as it was almost impossible to remove the outer casing and had no mounting holes in the PCB. So I stuck with the Anear 60W hub.
Green microUSB cables vs Boring black – bit of a stretch to call them pine-coloured... ? Could do with being slightly longer to more easily reach the furthest two boards.
Flat 25cm LAN cables vs Round 15cm – I loved the rainbow LAN cables in my RPi3 build, but it was a tight fit bending them in the case. These flat cables bend far more easily, but 25cm is too long really and 15cm would be neater.
White LAN cables vs Boring grey – goes nicely with Gelid fan?
Gelid Solutions Silent 9 fan vs Nanoxia Deep Silence fan – I was very happy with the performance of the Nanoxia fan (and their excellent customer service) but wanted to try out a cheaper option. The Gelid rubber grommets are thicker than the Nanoxia ones, so I increased the case mounting hole diameters by 0.5mm.
Gigabit vs 10Gigabit-switch – the A64+ network ports are each 1000Mbps (10x faster than the Pi), so using at least a 1000Mbps switch is a no-brainer really. A 10Gbps switch would eliminate any bottleneck (e.g., if the A64+s were simultaneously saturating their links talking to the outside network), however these are still very expensive at ￡600+, and too large to fit inside this case. The new Asus XG-D2008 switch looks promising.
Ceramic heatsinks vs Aluminium – rather than the small 14x14x5mm aluminium heatsinks I've used for most of these SBCs, this time I tried some innovative 25x25x5mm ceramic heatsinks that extend over the edge of the SoC to give a greater cooling area, and feature a “micro porous ceramic finned” structure. They seem to work well, and I'm going to write a separate article that compares some of the different heatsinks available.
Universal cluster case – the A64+ boards are much bigger than the Pi, which made squeezing them into the cluster case a challenge, especially with the internal USB hub. The obvious answer would be to have made the case larger, but I'm trying to design a cluster case that will fit any of the ARM SBCs, simply by swapping the 2 side panels. Work in progress!

Read about some of the design choices on the original Pi cluster.Power, temperature & coolingAt idle, the entire system of five A64+s, network switch & 5V fan sips a mere 9W, and at 100% load it still only uses 19W in total. There is the possibility of further reducing the power requirements by disabling HDMI?Is a fan needed? The PINE A64+ boards dissipate their heat better than any of the other SBCs I've tried (perhaps helped by the much larger surface area of the boards?), including the Raspberry Pi 3. You could certainly use the cluster without any fan, but I would fit a 5V fan if you want maximum performance or are going to run the cluster at high load for extended periods.

The current release of Debian Linux (3.10.102-2-pine64-longsleep kernel) is unusual in that it ships with the CPU performance governer locked to 1152MHz. I think this will be changed in a future release, and I've configured my boards to use the ondemand governer that dynamically adjusts the CPU speed between 480MHz and 1152MHz as required. This reduces electricity used as well as keeping the boards cooler:echo ondemand | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/nullThe USB hub can supply up to 60W (2.4A per A64+), which is more than enough for a couple of power-hungry external devices to be plugged into the USB ports. Using:
cat /sys/devices/virtual/thermal/thermal_zone0/tempto measure the SoC core temperature, the cluster idles at 33.4°C (92°F) with cooling from the 5V fan.At 100% load and the 5V fan, using:
sysbench --test=cpu --cpu-max-prime=200000 --num-threads=4 run &the SoC core temperatures reached a stable 63.2°C (146°F). If you don't provide at least some active cooling, the SoCs will get as high as 80°C and automatically throttle down their clock speed, to avoid overheating. They can safely run long-term at that temperature, but you don’t get maximum performance.Exactly the same case design should work with the A64 model.
Silent coolingTo cool down the cluster I fitted a single 92mm fan inside the rear of the case. I used a (effectively) silent fan recommended by Quietpc.com, the Gelid Silent 9 (￡5.53).At 5V I have to get my ear within 50-75mm (2-3") to hear even the slightest whisper from the fan, and the supplied rubber gromets definitely do a good job of isolating the case from any small vibrations. However at 12V the fan is quite audible in an otherwise silent room, so I was looking for the voltage that would provide enough cooling, while keeping the fan silent.
For a single A64+ you might not need a fan at all, or perhaps a small 40-60mm one.

description	heatsinks?	idle	100% load	performance
Case, 12V fan, 1500 rpm	yes	30.4°C	53.8°C	OK
Case, 7V fan, ? rpm	yes	31.4°C	57.6°C	OK
Case, 5V fan, ? rpm	yes	33.4°C	63.2°C	OK
Case, no fan	yes	40.8°C	75.6°C	throttles

Temperatures between the different A64+s was spread over 13 degrees, so an average of 75.6°C (168°F) actually had three of the boards throttling.
Powering the fanUnlike PC motherboards, a A64+ doesn't have a standard 3 or 4-pin fan header, but it is quite easy to power the fan using the GPIO 5V power rail instead. WARNING you can't run anything that needs a lot of current this way – however a low-power fan should be fine. You do need to make sure you have a quality power supply, and not too many power-hungry USB devices plugged into your A64+, otherwise you'll get crashes & rebooting. The main GPIO pins are for data, but using the VDD_5V power rail on pin 2 or 4 should work in theory.This should be enough to run the fan with 5V, or if you need more cooling, splice a step-up/boost converter in the middle to adjust the speed of the fan, by controlling its voltage between 5V and 12V.
Building the A64+ Cluster

Remove network switch case (2 small screws)
Attach the network switch to case base, using 4x 6mm brass spacers + 4x 14mm bolts + nuts. This only fits one way around. (The external LAN cables route neatly underneath the switch PCB)
Remove USB hub case (tricky – needs to be carefully prised open, there are no screws). Fit 4x 6mm brass standoffs to the case side, using screws
Superglue the metal top of the USB hub to the brass standoffs. This needs to be correctly positioned so that the USB hub power socket sticks out of the back of the case when it is fitted together
Screw 2 external LAN ports to inside of the case back (has “AC100-240V” etched on the outside)
Fit the 92mm case fan to inside of the case back using the supplied rubber grommets. The fan logo should face inwards if you want the fan to blow cold air into the case. The Gelid rubber grommets are a VERY tight fit and might need to be pushed into place with a small screwdriver/pen as well as pulling quite hard!
Clip case back to case base (marked C+D)
Plug external LAN into network switch ports 1+3 (no room to use 1+2)
Place a heatsink on the SoC chip of each A64+, and hold in place for a few seconds to let the adhesive stick
Slide 5 A64+s onto 4 threaded rods using 48 nuts to secure. This might be faster with the rod in an electric screwdriver? Leave 9mm space at left end, space each A64+ 22mm apart. The LAN+HDMI side of the A64+s point towards the front of the case
Attach case sides to A64+ rods (EF+AB), secure with 8 metal nuts.
Fit the 3-pin fan plug into the GPIO pins of one of the A64+s, so that black (ground) connects to pin 6, red (power 5-12V) to pin 2, and yellow (fan speed) is unconnected. To remove the individual wires from the plug, depress the tiny metal tabs (e.g., with a small screwdriver), and the wires will slide out. I cut off the yellow wire which I wasn't interested in, and slid the wires back into the plug so that red was on the left, the middle was empty, and black was on the right:
Plug A64+ LAN cables into network switch ports 4-8 (be very careful if removing these later – it is easy to break tiny plastic clips on the switch ports)
Clip case sides to case bottom (E+F, A+B)
Plug USB hub to network switch DC power
Plug green USB cables into USB hub
Route HDMI cable through case
Screw external twin USB to front case
Bolt external HDMI to front case, 2 nuts
Plug 2 external USBs + right-angle adaptors into any A64+ (or 2 separate A64+s)
Plug A64+ LAN cables into A64+s
Plug A64+ micro USB cables into A64+s
Plug HDMI cable + right-angle adaptor into any A64+
Clip case front to case bottom (G+H)
Clip case lid to case sides, front & back (M+N+O+P+Q+R+S+T)
Attach self-adhesive rubber feet to underside of base

Bill of materialsMost of these parts were sourced from individual sellers on Amazon or eBay, which of course racks up the postage charges. If there were enough demand, it would be cheaper to bulk buy the parts and have a kit with everything you need to build the cluster.

Edimax ES-5800G V3 Gigabit Ethernet Switch	￡13.49
White flat 25cm Cat6 LAN cables (5 pack)	￡4.45
Green micro USB cables (5 pack)	￡9.45
USB Charger Anear 60W 6 Port USB	￡14.99
M3 steel screws 8mm (10 from a 20 pack)	￡1.70
RJ45 male to female screw mount (2 pack)	￡1.98
M3 steel hex nuts (4 from a 5 pack)	￡1.10
M3 steel screws 14mm (4 from a 5 pack)	￡1.35
M3 brass female standoff 6mm (8 from a 10 pack)	￡0.99
M3 steel threaded bar 150mm + nuts (4 from a 5 pack)	￡5.50
0.5m HDMI male to female panel mount (inc. bolts)	￡2.39
Twin USB female socket to male cable	￡2.19
USB right-angle male to female socket extender (2 pack)	￡1.98
Polyurethane rubber feet (4 from a 12 pack)	￡2.99
M3 nylon hex nuts (48 from a 100 pack)	￡1.29
3mm extruded clear perspex 600x400mm	￡5.32
Laser cutting charge	n/a
HDMI 270 degree adaptor	￡0.99
Gelid Silent 9 92mm case fan	￡5.53
Subtotal inc P&P	￡77.68
PINE A64+ at US$19/each (5 pack)1	￡93.56
Ceramic heat sink 25mm x 5mm (5 pack)	￡4.74
Total inc P&P	￡175.98

1The PINE A64+ is duty free to import into the UK, costs US$18.00 to ship 5 boards from the USA, and there is UK VAT at 20% bringing the total to ￡93.56.
The cluster of A64+s (excluding the boards themselves) is cheaper to build than the Pi3s: this is mainly because I'd found better deals on some of the components, as well as simplifying the design to remove the case shelf + AC socket.
Show me the Benchmarks!!I'm writing a separate article benchmarking this PINE A64+ cluster against my original Raspberry Pi 3 cluster, as well as detailed benchmarks comparing the individual boards. Watch this space...Clusters of other Single Board ComputersSo far I’ve built clusters using the following ARM boards:

The NanoPC-T3, Orange Pi Plus 2e and PINE A64+ clusters mostly share the same components, including the acrylic case panels – only the 2 side panels are unique because the boards are different sizes. I'm planning to revise my original Raspberry Pi cluster design so it will share most of these same case/components.I’d like to build a small cluster of all the current crop of sub-$100 ARM SBCs, comparing the different features, and with detailed benchmarks. e.g., Odroid C2/XU4 and the Banana Pi M3. Please email me if you'd like to send boards for review.
Software to run on a cluster?or... What is it for??
Education, training, blah, blah... well personally I’m just running a Debian Linux provided by Pine64.Pro on each A64+ for now, and I’m going to experiment with things like load-balanced web/database servers.
Running Docker on ARM on each node looks like an excellent way of controlling the cluster.

Nick Smith, August 2016.

欢迎光临电子技术论坛_中国专业的电子工程师学习交流社区-中电网技术论坛 (http://bbs.eccn.com/)