Oct 24 2017

Scala – installation behind a workplace web proxy

I’ve been learning Scala as part of my continuing professional development. Scala is a functional language which runs primarily on the Java Runtime Environment. It is a first class citizen for working with Apache Spark – an important platform for data science. My intention in learning Scala is to get myself thinking in a more functional programming style and to gain easy access to Java-based libraries and ecosystems, typically I program in Python.

In this post I describe how to get Scala installed and functioning on a workplace laptop, along with its dependency manager, sbt. The core issue here is that my laptop at work puts me behind a web proxy so that sbt does not Just Work™. I figure this is a common problem so I thought I’d write my experience down for the benefit of others, including my future self.

The test system in this case was a relatively recent (circa 2015) Windows 7 laptop, I like using bash as my shell on Windows rather than the Windows Command Prompt – I install this using the Git for Windows SDK.

Scala can be installed from the Scala website https://www.scala-lang.org/download/. For our purposes we will use the  Windows binaries since the sbt build tool requires additional configuration to work. Scala needs the Java JDK version 1.8 to install and the JAVA_HOME needs to point to the appropriate place. On my laptop this is:

JAVA_HOME=C:\Program Files (x86)\Java\jdk1.8.0_131

The Java version can be established using the command:

javac –version

My Scala version is 2.12.2, obtained using:

scala -version

Sbt is the dependency manager and build tool for Scala, it is a separate install from:

http://www.scala-sbt.org/0.13/docs/Setup.html

It is possible the PATH environment variable will need to be updated manually to include the sbt executables (:/c/Program Files (x86)/sbt/bin).

I am a big fan of Visual Studio Code, so I installed the Scala helper for Visual Studio Code:

https://marketplace.visualstudio.com/items?itemName=dragos.scala-lsp

This requires a modification to the sbt config file which is described here:

http://ensime.org/build_tools/sbt/

Then we can write a trivial Scala program like:

object HelloWorld {

 

def main(args: Array[String]): Unit = {

 

    println(“Hello, world!”)

 

  }

 

}

And run it at the commandline with:

scala first.scala

To use sbt in my workplace requires proxies to be configured. The symptom of a failure to do this is that the sbt compile command fails to download the appropriate dependencies on first run, as defined in a build.sbt file, producing a line in the log like this:

[error] Server access Error: Connection reset url=https://repo1.maven.org/maven2/net/
sourceforge/htmlcleaner/htmlcleaner/2.4/htmlcleaner-2.4.pom

In my case I established the appropriate proxy configuration from the Google Chrome browser:

chrome://net-internals/#proxy

This shows a link to the pacfile, something like:

http://pac.madeupbit.com/proxy.pac?p=somecode

The PAC file can be inspected to identify the required proxy, in my this case there is a statement towards the end of the pacfile which contains the URL and port required for the proxy:

if (url.substring(0, 5) == ‘http:’ || url.substring(0, 6) == ‘https:’ || url.substring(0, 3) == ‘ws:’ || url.substring(0, 4) == ‘wss:’)

{

return ‘PROXY longproxyhosturl.com :80’;

}

 

These are added to a SBT_OPTS environment variable which can either be set in a bash-like .profile file or using the Windows environment variable setup.

export SBT_OPTS=”-Dhttps.proxyHost=longproxyhosturl.com -Dhttps.proxyPort=80 -Dhttps.proxySet=true”

As a bonus, if you want to use Java’s Maven dependency management tool you can use the same proxy settings but put them in a MAVEN_OPTS environment variable.

Typically to start a new project in Scala one uses the sbt new command with a pointer to a g8 template, in my workplace this does not work as normally stated because it uses the github protocol which is blocked by default (it runs on port 9418). The normal new command in sbt looks like:

sbt new scala/scala-seed.g8

The workaround for this is to specify the g8 repo in full including the https prefix:

sbt new https://github.com/scala/scala-seed.g8

This should initialise a new project, creating a whole bunch of standard directories.

So far I’ve completed one small project in Scala. Having worked mainly in dynamically typed languages it was nice that, once I had properly defined my types and got my program to compile, it ran without obvious error. I was a bit surprised to find no standard CSV reading / writing library as there is for Python. My Python has become a little more functional as a result of my Scala programming, I’m now a bit more likely to map a function over a list rather than loop over the list explicitly.

I’ve been developing intensively in Python over the last couple of years, and this seems to have helped me in configuring my Scala environment in terms of getting to grips with module/packaging, dependency managers, automated doocumentation building and also in finding my test library (http://www.scalatest.org/) at an early stage.

Sep 30 2017

Book review: Pandora’s Breeches by Patricia Fara

pandoraInspired by Claire Brock’s biography of Caroline Herschel I found Pandora’s Breeches by Patricia Fara which is a broader survey of women in science during the enlightenment – from around 1500 to 1800.

Fara is interested not only in the people but also the methodology of history. Early on in the book she lays out a manifesto for a better history that doesn’t seek lonely heroes, as is often the case in history of science books. That’s to say her aim is not to simply replace the men in a normal scientific biography with women. As inspiration she cites books like Jenny Uglow’s Lunar Men which is an ensemble of biographies covering several people – I approve of this approach!

The chapter headings are pairings of woman and man, for example, “Anne Conway / Gottfried Leibniz”, at first sight this seems wrong. Surely this is a book about women in science, why tie each of them to a man? But actually it fits with the logic of the book, these women did not operate in isolation but neither did their male counterparts. Their male counterparts benefited from the more or less formal community of “scientists”, and those that had gone before them. But those male counterparts also benefitted from the practical support of their wives, daughters, sisters, other family members and friends. This book shows that practical support was not simply “she made him dinner so he didn’t have to”, it was in correspondence and the exchange of ideas, it was in the practicalities of running a laboratory at home, it was in the translation and explanation of scientific ideas and in the salon. To this group of women should also be added the invisible horde of male helpers, workmen and assistants who also go largely unmentioned.

The book starts by considering how nature has often been represented as a woman, whose intimate parts are accessed, or unveiled, or probed by scientists (usually men). In engravings from the Enlightenment period nature is often represented by a female form. This is not a framing that has disappeared, this quote by a geologist is from 1980: “Her flanks are shuddering… we don’t know of her intentions. Scientists haven’t been able to probe her deeply enough with their instruments”.

This may seem a harmless piece of flowery prose with more than a hint of sexual innuendo but it should be read in a context of a stream of scandals, at the very least in the US, where senior male scientists have acted inappropriately towards women at universities. Francis Bacon, very much the father of the modern scientific method, explicitly rejected women from his new science. A lead followed by the Royal Society who accepted men regardless of nationality and religion but could not abide women.

The book is divided thematically, the first few chapters are on aristocratic women and how they corresponded with and nurtured men who are now far more widely known. This was part of a system of scientific endeavour which was very different from that found today. There was no profession, only the sponsorship of monarchs and the wealthy. Fara discusses Elisabeth of Bohemia, and how she pushed Descartes to explain his ideas fully and Émilie du Châtelet who lived with Voltaire, conducting her own experiments and translating Newton’s Principia, although “translate” underplays greatly her work. This network was known as the Republic of Letters, and Fara highlights how women played a part in it.

The next theme is on women and science in domestics settings. Prior to the 19th century, science took place in the home which was typically managed by the women of the house. Science was an all consuming passion which inevitably brought in other members of the household. Marie Paulze Lavoisier was the wife of Antoine Lavoisier and was clearly deeply involved in his chemical experimentation, she is shown recording the results of experiments in a drawing of the time and was also responsible for highly detailed diagrams of the equipment used in their laboratory. As well as this she arranging for the publication of his work after he was executed during the French Revolution.

The women in Pandora’s Breeches were, in general, heavily engaged in the scientific endeavour. That is to say they did the things they did because they wanted to not because they had been dragged in by their men folk. This struck me particularly in the case of Elisabetha Hevelius who went out of her way to marry the much older, widowed Johannes a merchant and brewer with a substantial rooftop observatory, driven by her passion for astronomy. Priscilla Wakefield, who wrote Introduction to Botany along with 16 other textbooks, also falls into this class. She wrote, quite deliberately, for a large audience with a view to earning money from her writing.

The book finishes with Mary Shelley and Victor Frankenstein. Frankenstein is about how science fits into the wider world. Here Fara highlights that these women of 200 years and more ago did not have the same aims as feminists today, education for women was not generally promoted as a route to equality rather a way by which women could become more useful and pleasing to their families.

Throughout the book Fara highlights that these women are just those for which some written record remains, because of the prevailing culture of the time discoveries which were in truth joint efforts were written down solely to the “great man of science”.

This book is definitely worth reading, it brings to light different facets of the development of science and it is highly readable.

Sep 03 2017

Book review: Chester AD400-1066 by David Mason

chester400-1066adI couldn’t resist more Chester history, so now I am reading Chester AD400-1066: From Roman Fortress to English Town by David Mason. I’ve just read Jane Laughton’s book on late medieval Chester covering the period 1275-1520 (review here), and read David Mason’s book on Roman Chester (review here) towards the end of last year. This book fills most of the gap between those two books, but not quite.

Evidence for the early part of the period is sparse, particularly in the earlier years between about 400AD and 600AD – it isn’t known as the “Dark Ages” for nothing. During this time, after the Roman withdrawal, no durable mass-produced items such as coins or pottery were being produced. Elsewhere, in nearby Wroxeter, archaeological evidence suggests that the early Britons built wattle-and-daub huts within substantial Roman buildings. In Chester there is little such evidence. The various Roman buildings in Chester would have decayed at different rates. The baths under the now Grosvenor Shopping Centre had metre thick walls and would have only fallen down slowly, whilst the barracks in the north east quarter of the city were less substantial. At the barracks there are black deposits, possibly pigeon droppings, deposited between Roman and later date-able layers. It would seem that for most of the period from 400AD to 900AD Chester was a Roman ruin with some desultory farming and living taking place within its walls.

A little after this earliest period to 600AD there are are some written records, such as the Anglo-Saxon Chronicles, and Bede’s History which were written some time – dozens, even hundreds of years – after the event. There are songs and poems from Wales, transmitted orally, which refer to the period.

Chester enters the historical record of the period with the Battle of Chester in AD616, for which there is archaeological evidence in the form of a mass burial at Heronbridge, just south of the city. Here 120 skeletons were found dating to the same period as documentary evidence for the battle and with wounds consistent with dying in battle.

St John’s church, founded in 689AD sits next to the old Roman amphitheatre from whose remains it was built, although there is some suggestion that Christians were martyred at the amphitheatre, so it wasn’t simply a case of being unwilling to carry heavy stones further than necessary!

In common with the Roman period, and the later period covered by Laughton’s book Chester holds a key strategic position between Wales and Ireland on the one hand and the rest of England, variously inhabited by Dane’s and Anglo-Saxons on the other. I found some of the discussion here confusing with what I would refer to as “Celts” from Wales, the West Country, Scotland and Ireland described as “British” and those from Mercia and Wessex as “English”.

The final third or so of the book is devoted to Chester as a burh one of the network of fortified towns set up by King Alfred – Chester was established in 907AD. It’s at this point that Chester appears to turn into a proper town, and a moderately important one at that. In the years after its establishment as a burh Chester had “moneyers” – effectively a mint, twice the size of London. This is reflected in the discovery of coins minted in Chester being found in Ireland and around the Irish Sea. The establishment of the burh mandated taxation and physical labour to build its defences and bridges – it’s likely in this period that the modern circuit of the walls was constructed. There is more archaeological evidence from this period and the start of some form of systematic written records, rather than the non-contemporaneous writing alluding to previous periods.

It is during the period after the establishment of the burh that King Edgar holds his coronation in the city, in 973AD. By the time of the Norman conquest and subsequent Domesday census Chester had 431 houses and a further 56 belonging to the Bishop (presumably of St. Werburgh’s church which was founded by in Edgar in 758AD). It also had a system of laws and taxation detailed in the Domesday book.

The book ends with Chester suffering a setback in the years after the Norman conquest as a result of its part in rebellion against the incoming King William.

Chester AD400-1066 is a fairly slender volume but more readable than Roman Chester. As a result of the sparsity of the archaeological and written records for the period it is wider in its scope than books set before and after this period.

Aug 28 2017

Book review: Life in a late medieval city by Jane Laughton

medieval_chesterI’m back to local history with my next review. A while back I read Roman Chester by David J.P. Mason. I have his book Chester 400AD –1066AD on my “to read” pile but instead I am jumping forward to Life in a late medieval city: Chester 1275-1520 by Jane Laughton.

The book starts with some definitions and background. How do we define a city? What was the hierarchy of settlements in Cheshire and indeed the rest of England and Wales? This information can be inferred from various charters, and the like.

The book is laid out thematically, so having covered definitions of towns it then goes on to provide an overview of the historical background to the period. This is generally revolves around what kings were doing (invading Wales, crushing rebellions) but also mentions the rise and fall of Chester with famine and the Black Death.

Chester was an important location in Roman times, acting as a forward base for the Roman invasion of northern England and a potential jumping off point for Ireland. So it was in the late medieval period. The start of the time span of the book coincides with the time Edward I’s invasion of Wales when Chester was used as a garrison for the invasion force.

The next chapters cover the topography of the city and the built environment. The central streets of Chester, Eastgate, Bridge Street, Northgate and Watergate play a key part here – as they do to this day. In the period covered by the book these streets provided the key administrative divisions of the city, when citizens interacted with the bureaucracy they were labelled with their name and home street.

I am intrigued by the Pentice, which was a lean-to structure built against St Peter’s church at the Cross in the centre of the city which served as the base for administration for many years (you can see a picture of the Pentice on this page) – it was finally demolished in the early years of the 19th cenutry. The “Rows” are a key feature of the built environment even now, as they were in the late medieval period. For those that haven’t visited Chester the Rows are an arrangement whereby walkways runs through the first storey of the shops on Eastgate, Bridge Street, Watergate and Northgate (to a limited extent). Towards the street from the Row there is a flat, slightly sloped “stall” which was used to lay out goods in the past, beyond which is open to the street at first storey. Opposite the stalls are shops, and beneath the “Rows” are undercrofts. You can see pictures of the rows here, they haven’t changed a great deal over the years.

Laughton makes it clear that the book is based on the records of courts and taxation within the city. So we know, for example, that the tanners in the city worked outside the Eastgate because there are court records of them being charged with blocking the city ditch with their cess pits. We know of the types of trade taking place in the city through taxation, rental and customs records.

Sometimes the relationship with the underlying records feels a bit direct. We are introduced to messuage with no explanation, a messuage is a dwelling and its land. And also pavage, murage and even pontage – these are taxes raised for the purpose of building and maintaining roads, walls and bridges respectively. The portmote, crownmote and piepowder courts are similarly introduced with little ceremony. Portmote are essentially courts in port town, crownmote are courts which cover the most serious offences and piepowder courts cover justice surrounding traders coming in from outside the city. Similarly a range of now obscure occupations such as corviser, souter, barker are introduced with little explanation. On the plus side I have learnt a host of new words for which to find application!

The book goes on with chapters on the hierarchical society and urban government. There is some overlap here with men moving through positions in the government of Chester, more rapidly if they are nobility.

This history of Chester is quite distinct from my readings in the history of science, the period it covers lies before the main developments in Western science. Copernicus (1473-1543), Mercator (1512-1594) and Galileo (1564-1642) are the earliest I’ve read about, most of the history of science I’ve read is post English Civil War. Science does have an equivalent to the administrative records in this book but they are impersonal records of the locations of stars and planets, and the like. Reading Life it strikes me that Chester (and undoubtedly the rest of England) had quite complex systems of law, ownership, trade and so forth from a very early time – science is something of a latecomer.

It feels like a book a bit more for the specialist than the general reader but I found it pretty readable and enjoyed the link it gave me to the medieval inhabitants of the city I live in.

Aug 13 2017

Book review: The Comet Sweeper by Claire Brock

thecometsweeperA return to women in science in this post where I review The Comet Sweeper: Caroline Herschel’s Astronomical Ambition by Claire Brock, a biography of a woman who discovered comets and nebulae and published a catalogue of astronomical objects in the later years of the 18th century. For scientists the name “Herschel” will not be unknown. Caroline Herschel’s brother William discovered Uranus, and was paid as an astronomer by King George III. Her nephew, John was also well known as a scientist. However, relatively little has been written about Caroline.

The Comet Sweeper is based substantially on the autobiographical writing of Herschel. However, she was sufficiently well-known at the time to be referenced elsewhere, and indeed later in her life was bestowed with various honours and medals for her astronomical work.

Herschel was born in Hanover in 1750, her father Isaac was a musician and very much a self-taught man – something he passed on to Caroline. Anna, her mother, gets a less than sympathetic treatment from her daughter and consequently this book. For her early years Anna treated Caroline as a servant, and stopped her education as soon as it appeared it would help her leave the Herschel household in Hanover. She was finally given a means of escape when her brother, William, invited her to Bath to work in music with him in 1771. She had no previous training in music and put herself assiduously to learning what she needed to know. William Herschel was earning up to £400 per year from music lessons and the like when he invited his sister to join him. It seems that Caroline became a significant musician in her own right, at least until her brother dragged her into astronomy.

This is something of a theme through the book, Caroline Herschel is clearly very capable and when given the opportunity can excel in whatever she turns her hand to. But the choices she has are limited. In the first instance her mother controls what she can do, then her brother – switching her from music to astronomy with little regard for her own wishes.

In astronomy Herschel started by assisting her brother in the workshop – at the time, to get the best telescope, you built them from scratch yourself. She supported him in his observations but she also carried out observations on her own. The “sweeping” of the title is the systematic scanning of the night sky with a telescope to identify static features such as stars and nebulae but more specifically to find comets. To a degree the discovery of nebulae was incidental to the main task of finding comets, nebulae were easily confused with comets so recording their locations was an essential part of finding comets. The Herschel’s work followed, but only by a few years, the publication of Charles Messier’s first catalogue of diffuse celestial objects in 1774.

As well as discovering comets and nebulae Herschel was also responsible for publishing Catalogue of Nebulae and Clusters of Stars in 1798, which built on the earlier work of Flamsteed. Ultimately this became the New General Catalogue of stars. Amateur astronomers will know this work, Messier’s catalogue provides information on the 100 or so most prominent objects whose identifying numbers are prefixed with an M- beyond this are the NGC objects – from the New General Catalogue which is the descendant of Herschel and Flamsteed’s catalogue.

Herschel was honoured in her own lifetime with a gold medal from the Royal Astronomical Society, as well as honorary membership and medal from the King of Prussia, at the age of 96. She was the first woman to be published in Philosophical Transactions the journal of the Royal Society. These awards did come until quite late in her life although she was paid £50 per annum by King George III as an assistant to her brother. He was paid rather more, £200, but notably rather less than he earned as a musician.

I found the broader insight that The Comet Sweeper gave into the lives of Georgian women was interesting. Women did not have formal positions within the scientific community of the time but they contributed as wives, sisters, daughters. At the time there was little in the way of formal, paid, scientific community – it was very much a gentleman’s club but there was a place for women in it although not necessarily of equal status.

This was to change later in the 19th century when science became institutionalised, as a result women were excluded by, for example, not being able to receive degrees or even attend lectures at university.

The Comet Sweeper is not a long book, it is readable and casts an interesting light on women in science in Georgian England and the specific contributions of Caroline Herschel.