Category: Technology

Programming, gadgets (reviews thereof) and computers

Aug 30 2011

Photovoltaic solar power – one year on

This time last year we had a photovoltaic solar power system installed, a few years after we had a thermal solar system installed. This post is an update on how we’ve got on with the photovoltaic solar system. This is facilitated by my weekly collection of electric, solar and gas meter readings – I suspect this data collection is a bit of a minority sport. A couple of months ago the company that the installed the system did offer me a fancy monitoring system… which cost about £1500!

As my earlier blog post highlights the calculated output for our system, given the latitude and orientation of our roof and the peak capacity of our system is 1393kwh per year. The first meter reading was on 28th August 2010, as of 25th August 2011 the reading is 1362.9kwh. So the measured output is 98.5% of the predicted output – not bad at all! Our import from the grid for the same period was 1896.2kwh compared with an annual consumption of 2397.5kwh in the previous year. So a 20% reduction in consumption and generation of 57% of our pre-installation level. The difference here is because our peak generation (mid-late morning) does not match our peak usage (early evening), since we can’t store any electricity the excess goes onto the grid. These figures are illustrated graphically below:

consumption

We receive the feed-in tariff and payments for exported electricity from our supplier, npower. This seems to be a small scale operation since to set this up you interact with named individuals who don’t change! So far we’ve received two cheques from them for £105.29 for the period 16/9/10-04/03/11 and £174.38 for the period 4/3/11-21/5/11. I should probably go prod them about whether they want another meter reading.

We can dig a little deeper into the data, below I show our monthly electricity consumption (red/green lines) from the grid for the last few years along with the last year of solar generation (the blue line).

monthlyconsumptionOur electricity consumption has been fairly constant through the year with a hint of an increase during the deep winter months due to the shortening days and an increased use of electric lighting. Since about March this year our electricity consumption has been significantly reduced compared to previous years, offset by our solar generation. This pattern wasn’t maintained in the September / October / November last year – I think because I was at home (using electricity) for an extended period following an operation.

The amount of solar generation varies smoothly through the winter months but seems to plateau during the summer. We did have a week of zero generation when the panels were covered in snow. I suspect in principle that the average power generation will vary sinusoidally through the year, since the variation in day length is sinusoidal, but that this can disturbed by “weather”, in particular the figure for May includes the very sunny Easter period without this boost the curve would have varied smoothly through the summer rather than plateauing.

In summary: we’ve generated almost exactly the amount of solar electricity we anticipated which amounts to nearly 60% of our annual consumption of electricity.

Aug 27 2011

Living in code

Eric Schmidt, chairman of Google is in the news with his comments at the MacTaggart Lecture at the Edinburgh International Television Festival. The headline is a general criticism of the UK education system but what he actually said was more focussed on technology and in particular IT education: bemoaning the fact that computer science was not compulsory and what of it that there was about the use of software packages rather than how to code.

I was born in 1970, and learnt to program sometime in the early 80s. I can’t remember exactly where but I suspect it was in part at the after-school computer club my school ran. A clear memory I have is of an odd man who’d brought in a TRS-80 explaining that a FOR-NEXT loop was an instruction for a computer to “go look up its bottom” – this was at a time before CRB checks. My first computer was a Commodore VIC-20, Clive Sinclair having failed to deliver a ZX81 and the BBC Micros being rather more expensive proposition than my parents were willing to afford.

Many children of the early 80s cut their teeth programming by typing in programs from computer magazines; a tedious exercise which trained you in accurate transcription and debugging. Even at that time the focus of Computer Studies lessons was on using applications rather than teaching us to program although I do remember watching the BBC programmes on programming which went alongside the BBC Micro. As I have mentioned before, programming is in my blood – both my parents were programmers in the 60s.

About 10 years ago I was teaching programming to undergraduate physicists, from a class of 50 only 2 had any previous programming experience. The same is true in my workplace, a research lab where only a small minority of us can code.

Knowing how to code gives you a different mindset when approaching computer systems. Recently I have been experimenting with my company reports database. The reports are stored as PDF files; I was told the text inside them was not accessible – now to me that sounds like a challenge! After a bit of hacking I’d worked out how to extract the full text of reports out of the PDF files but then code that once worked stopped working. This puzzled me, so I checked the text that my program was pulling from the database and instead of being a PDF file, it was a message saying “Please don’t do that"!

At the moment I’m writing a program that takes an address list file, checks to see if the addressees have a mobile phone number and if they do uploads it to an SMS service, spitting out into a separate file those that do not have a mobile phone number. To me this is a problem that has an obvious programming solution, for the people who generate the address list it’s a bit like black magic.

These days we are surrounding by technology bearing code, just about every piece of electrical equipment in my house has code in it, but it seems that ever fewer of us have been inducted into the magic of writing our own code. These days there’s just so much more fun to be had from programming: there are endless online data sources and our phones and computers have so many programmable facilities built into them.

At what age can I teach my child Python?

Jul 16 2011

Lead mining in the Yorkshire Dales

Bunting level: building, level entrance and hush

Bunting level: building, level entrance and hush

On a recent trip to the Yorkshire Dales we came across the remnants of lead mining; as with many things in the field these are blank discoveries with no indication of what they mean at the site. A very long time ago I did an OA level in geology, and I seem to have inherited an interest in industrial archaeology, so I resolved to find out more…

The majority of the mine workings we saw were in Gunnerside Gill, which extends north from Swaledale shown on the map below. On the ground we found rock strewn gullies and heaps of gravel, still devoid of vegetation. There were also ruined mine buildings, the occasional opening into the hillside and the odd bit of rusting ironwork. The most distinctive elements of the remains are the “bouse teams”, these are sets of bunkers – just the dividing walls remain – which were used to store the ore (or bouse). Quite a few of the stone surfaces are covered in a reddish deposit; elsewhere this has been identified as a lichen, but I believe it’s lead oxide. The workings are named by the “level” or horizontal tunnel which they served, the photos here are taken around the Bunting (or Bunton) level.

 

Lead mining, bouse teams

Lead mining: bouse teams at Bunting Level

Further down the Gill, closer to Gunnerside, are the Sir Francis Level workings; although these are less extensive at ground level, they turn out to have quite fantastic remains underground. Intrepid cavers have been down and taken some impressive photos, and you can read more about the Sir Francis Level workings in this report , here and here too. These were the most recent workings in Gunnerside Gill and were in operation from 1864 until 1891.

Lead mining has been carried out in the Yorkshire Dales since Roman times, the workings at Gunnerside Gill started in the 16th century but the majority of the visible surface remains are from the 19th century. The Bunting (or Bunton) level where the photos were taken was opened in 1802 and closed in 1898.

Lead mining: entrance to Bunting level

Lead mining: entrance to Bunting level

The Yorkshire Dales are substantially formed from limestone and gritstone outcrops laid down in the Carboniferous Period some 320 million years ago1. The lead ore occurs in veins which were deposited hydrothermally: hot pressurised liquids containing a range of minerals were forced through cracks in the native rock and deposited what they carried when the pressure and/or temperature dropped. The minerals are not conveniently deposited by type but mixed together. The veins are quite narrow – typically only a metre wide – but stretch deep underground.

This is reflected in the manner in which they are mined: initially vertical shafts were mined down to follow the veins. Later horizontal “levels” were mined to meet up with these vertical shafts which made the extraction of both ore and water easier – water will drain out of a level but puddle in the bottom of a shaft. In addition to this type of mining “hushing” was also used: this involves sluicing large quantities of water down the line of the vein to erode out the rock. This process was used in the Gunnerside Gill area from the 1780s to 1820s, and is what has left a large part of the surface remains. These are marked on the map as “hushes”. The extent of the workings can be seen quite clearly in Google Earth images (here).

The particular mineral in which the lead occurred is galena (lead sulphide), it is relatively dense and this property was used to separate out the galena from other minerals using a process akin to panning for gold. The ore containing rock is broken up into pieces and as far as possible the galena rich chunks are picked out and chipped off using hammers – this is known as dressing. This ore is then further broken up into smaller pieces. Earlier, this would have been done manually but later it was done using a water-powered mills, and still later it was done by steam-powered machines. These smaller pieces of ore are then buddled in troughs or hotching tubs – basically this simply means add water then shake, this separates out the material by density.

Lead mining, mill culvert

Lead mining: mill culvert at Bunting Level

Stonebreaker, with Sharon in background

Stonebreaker above Bunting Level with Sharon in background

Once reasonably well-separated ore is obtained it is converted to metal in two chemical steps. First the ore is roasted to drive off the sulphur and convert it to lead oxide, then the oxide is smelted with carbon which removes the oxygen from the lead to produce the metal. There were smelting mills in Gunnerside Gill but ore was also transported underground to the next valley and smelted in the Old Gang Smelting Mills. We didn’t visit these but they had flues extending up the hillside a distance of hundreds of metres. The reason for this was two-fold: firstly, the gases driven off the ore during smelting were pretty noxious and secondly, the long flue allows soot to condense back out as it cools and this soot can be contain high concentrations of minerals.

Life for the miners sounds pretty grim; they were expected to work for 6 hours a day underground and the mine workings were often inaccessible without accommodation so their days may well have started with an hour or so walking simply to get to work. The Swaledale Museum, in Reeth, as well has having a great deal of other information on lead mining has some mining company rules, dated 1872. One of them states: “Children aged 12-16years only allowed to work 54 hours per week, or 10 hours per day.”

Footnotes

1. See p13-22 of this report by the North Yorkshire Geodiversity Partnership for more geology.

2. Related is this report on the Peak District Lead Mining Industry

3. English Heritage scheduled monument report

Jul 03 2011

Computational Photography

Lightfielddemo

Lytro, Inc, a technology spin-off company founded by Ren Ng, have been in the news recently with their announcement of a re-focusable camera: take one “image”, and change where the focal plane lies after the fact. This is illustrated in the images above, generated from a single shot from the prototype camera. As you move from left to right across this sequence you can see the focus shifting from the front left of image to back right.I saw this work a few years ago at the mighty SIGGRAPH conference, it comes out of a relatively new field of “computational photography”.

All photography is computational to a degree. In the past the computation was done using lenses and chemicals, different chemical mixes and processing times led to different colour effects in the final image. Nowadays we can do things digitally, or in new combinations of physical and digital.

These days your digital camera will already be doing significant computation on any image. The CCD sensor in a camera is fundamentally a photon-counting device – it doesn’t know anything about colour. Colour is obtained by putting a Bayer mask over the sensor, a cunning array of red, green and blue filters. It requires computation to unravel the effect of this filter array to make a colour image. Your camera will also make a white balance correction to take account of lighting colour. Finally, the manufacturer may apply image sharpening and colour enhancement, since colour is a remarkably complex thing there are a range of choices about how to present measured colours. These days compact cameras often come with face recognition, a further level of computation.

The Lytro system works by placing a microlens array in the optical train, the prototype device (described here) used a 296×296 array of lenses focusing onto a 16 million pixel medium format CCD chip, just short 40mmx40mm in size. The array of microlenses means means that for each pixel on the sensor you can work out the direction in which it was travelling, rather than just where it landed. For this reason this type of photography is sometimes called 4D or light-field photography. The 4 dimensions are the 2 dimensions locating where on the sensor the photon lands, and the direction in which it travels, described by another two dimensions. Once you have this truckload of data you can start doing neat tricks, such as changing the aperture and focal position of the displayed image, you can even shift the image viewpoint.

As well as refocusing there are also potentially benefits in being able to take images before accurate autofocus is achieved and then using computation to recover a focused image.

The work leading to Lytro was done by Ren Ng in Marc Levoy’s group at Stanford, home of the Stanford Multi-Camera Array: dispense with all that fiddly microlens stuff: just strap together 100 separate digital video cameras! This area can also result in terrible things being done to innocent cameras, for example in this work on deblurring images by fluttering the shutter, half a camera has been hacked off! Those involved have recognized this propensity and created the FrankenCamera.

Another example of computational photography is in high dynamic range imaging, normal digital images are acquired in a limited dynamic range: the ratio of the brightest thing they can show to the darkest thing they can show in a single image. The way around this is to take multiple images with different exposures and then combine together. This seems to lead, rather often, to some rather “over cooked” shots. However, this is a function of taste, fundamentally there is nothing wrong with this technique. The reason that such processing occurs is that although we can capture very high dynamic range images, displaying them is tricky so we have to look for techniques to squish the range down for viewing. There’s more on high dynamic range imaging here on the Cambridge in Colour website, which I recommend for good descriptions of all manner of things relating to photography.

I’m not sure whether the Lytro camera will be a commercial success. Users of mass market cameras are not typically using the type of depth-of-field effect shown at the top of the post (and repeated ad nauseum on the Lytro website). However, the system does offer other benefits, and it may be that ultimately it ends up in cameras without us really being aware of it. It’s possible Lytro will never make a camera, but instead license the technology to the big players like Canon, Panasonic or Nikon. As it stands we are part way through the journey from research demo to product.

Jun 19 2011

Book Review: The Visual Display of Quantitative Information by Edward R. Tufte

 

tufte“The Visual Display of Quantitative Information” by Edward R. Tufte is a classic in the field of data graphics which I’ve been meaning to read for a while, largely because the useful presentation of data in graphic form is a core requirement for a scientist who works with experimental data. This is both for ones own edification, helping to explore data, and also to communicate with an audience.

There’s been something of a resurgence in quantitative data graphics recently with the Gapminder project led by Hans Gosling, and the work of David McCandless and Nathan Yau at FlowingData.

 

The book itself is quite short but beautifully produced. It starts with a little history on the “data graphic”, by “data graphic” Tufte specifically means a drawing that is intended to transmit data about quantitative information in contrast to a diagram which might be used to illustrate a method or facilitate a calculation. On this definition data graphics developed surprisingly late, during the 18th century. Tufte cites in particular work by William Playfair, who was an engineer and political economist who is credited with the invention of line chart, bar chart and pie chart which he used to illustrate economic data. There appears to have been a fitful appearance of what might have been a data graphic in the 10th century but to be honest it more has the air of a schematic diagram.

Also referenced are the data maps of Charles Joseph Minard, the example below shows the losses suffered by Napoleon’s army in it’s 1812 Russian campaign. The tan line shows the army’s advance on Moscow, it’s width proportional to the number of men remaining. The black line shows their retreat from Moscow. Along the bottom is a graph showing the temperature of the cold Russian winter at dates along their return.

800px-MinardInterestingly adding data to maps happened before the advent of the more conventional x-y plot, for example in Edmund Halley’s map of 1686 showing trade winds and monsoons.

Next up is “graphic integrity”: how graphics can be deceptive, this effect is measured using a Lie Factor: the size of the effect shown in graphic divided by the size of the effect in data. Particularly heroic diagrams achieve Lie Factors as large as 59.4. Tufte attributes much of this not to malice but to the division of labour in a news office where graphic designers rather than the owners and explainers of the data are responsible for the design of graphics and tend to go for the aesthetically pleasing designs rather than quantitatively accurate design.

 

Tufte then introduces his core rules, based around the idea of data-ink – that proportion of the ink on a page which is concerned directly with showing quantitative data:

  • Above all else show the data
  • Maximize the data-ink ratio
  • Erase non-data-ink
  • Erase redundant date-ink
  • Revise and edit.

A result of this is that some of the elements of graph which you might consider essential, such as the plot axes, are cast aside and replaced by alternatives. For example the dash-dot plot where instead of solid axes dashes are used which show a 1-D projection of the data:

ddp

Or the range-frame plot where the axes are truncated at the limits of the data, actually to be fully Tufte the axes labels would be made at the ends of the data range, not to some rounded figure:

range

Both of these are examples are from Adam Hupp’s etframe library for Python. Another route to making Tufte-approved data graphics is by using the Protovis library which was designed very specifically with Tufte’s ideas in mind.

Tufte describes non-data-ink as “chartjunk”, several things attract his ire – in particular the moiré effect achieved by patterns of closely spaced lines used for filling areas, neither is he fond of gridlines except of the lightest sort. He doesn’t hold with colour or patterning in graphics, preferring shades of grey throughout. His argument against colour is that there is no “natural” sequence of colours which link to quantitative values.

What’s striking is that the styles recommended by Tufte are difficult to achieve with standard Office software, and even for the more advanced graphing software I use the results he seeks are not the out-of-the-box defaults and take a fair bit of arcane fiddling to reach.  Not only this, some of his advice contradicts the instructions of learned journals on the production of graphics.

Two further introductions I liked were Chernoff faces which use the human ability to discriminate faces to load a graph with meaning, and sparklines – tiny inline graphics showing how a variable varies in time without any of the usual graphing accoutrements: – in this case one I borrowed from Joe Gregorio’s BitWorking.

In the end Tufte has given me some interesting ideas on how to present data, in practice I fear his style is a little too austere for my taste.There’s a quote attributed to Blaise Pascal:

I would have written a shorter letter, but I did not have the time.

I suspect the same is true of data graphics.

Footnote

Mrs SomeBeans has been referring to Tufte as Tufty, who UK readers of a certain age will remember well.