Here's a random collection of things I've learnt this month, mostly written to jog my memory.

About race

I've learnt some names: Trayvon Martin, George Floyd, and lists of others, the length of which shows how much effort white people have put in to protecting whiteness - we're more comfortable with allowing oppression and even killing of people of colour than we are in accepting this reality and our complicity with it. Because we know how fragile whiteness is - it has no just grounds for existence and is rotten through, yet we protect it by closing our eyes and ears, explaining things away as isolated incidents, or silencing voices through claiming colourblindness. And we know how much privilege comes from whiteness so we feel threatened and defensive - even to the point of defending things that are plainly wrong. e.g. there have been campaigns about slave-trader and philanthropist Colston's statue since the ’90s, pointing out how horrible it is to be celebrating a man for the riches he gave to the local area (and the Church of England) while completely ignoring the fact that he made most of that money by selling human beings. Since the 90s, people have expressed how atrocious this is and have asked councils and committees: could the statue moved to a museum where it could be put in a bigger context? could a plaque be placed to explain that this man was a slave trader? No, it couldn't, apparently, this smug pillar of white-supremacy must be left in place. So during to the recent rise of Black Lives Matter campaigning from the brutal murder of George Floyd,  it was thrown in the sea to cries from those defending whiteness including politicians and mainstream media of "terrible behaviour, you could have asked nicely".

I'm on a journey myself to understand race. I've come to understand the binary choice we're faced with: you can do things that are either racist or anti-racist; you can't be non-racist / neutral because that means you're content with the way things are. As Andre Henry said, ignorance doesn't cause racism, it's the other way around. As a white person, and (some sort of) Christian I've got a lot to deconstruct, but I'm committed to doing that and to being an ally as best I can. I'm not always going to get it right but I'm going to keep listening and trying. I've been grateful to and for Reni Eddo-Lodge for Why I'm no longer talking to white people about race, Andre Henry who I first heard on a Liturgists podcast, and more recently, Anthony Reddie & Ravelle-Sadé Fairman on Nomad podcast. I found the first episode of Channel 4's The school that tried to end racism a helpful way to start talking to my children about it.

Why am I putting this here, on my business' blog instead of separating it off as 'personal'? Because that's not how I do business. Artful Robot exists as a way to improve things, using my skills (and my privilege) to support people campaigning for social and environmental justice. The separation of business and personal is often used to protect businesses. All those flights for work while recycling at home? Nonsense, it's the same world. "Technology is neutral" so we'll use Nation Builder (who boasted about getting Trump elected), Amazon (who skip taxes and dehumanise workers to such an extent that they're paying for adverts now trying to show "real" happy workers), Facebook (don't get me started, but hey, delete facebook is coming to Europe) ... No thanks. Business exists in the same world as the rest of life and so should be involved with the same challenges.

Server monitoring

I've had need to understand load on my servers better recently. I've found a command line tool called stresss which you can use to 100% utilise a CPU which has helped me understand load averages, as shown by uptime, top etc. The three figures are averages over on minute, 5 mins and 15 mins, and a load average of 1 means that one CPU has been fully used during the whole of that time period. So on a multi-core system, say with 2 cores, a load average of 1 would mean 50% total available cpu use. I've started monitoring these load averages ÷ the number of processors, obtained by the nproc command. I'm still a little confused as to the meaningful difference between a CPU and a processor core, especially as most of my servers are virtual/"cloud" servers, but the output of nproc seems to be the right thing to use.

I spent too long trying to understand memory use, too. It's phenominally complex thanks to efficiency gains from shared memory etc. In the end I've decided to use sar's % memory used figure, which is the amount of memory needed by all the current processes. This means that %age can rise above 100 (i.e. when swap is used), but it's a good enough metric.

Looking into the amount of memory used by PHP was hard. I use FPM, and the status page offers a memory use by the last worker figure, but this does not include the memory used by the worker itself, so is usually around 8MB lower than the figures you can get from the /proc/ system. Getting an understanding of this is important for deciding a suitable number of max_children to configure, given an available amount of RAM. Too low and a surge of traffic will quickly fail, too high and you'll start swapping, which begins a sluggish progression towards death and/or worse eventually the kernel kills something, which could cause data loss.

For MySQL (MariaDB) I built a script to figure out what config settings to use to limit RAM use to a given level, given the number of max connections (which needs to exceed the php max worker children). It outputs config like the following:

# Generated by: /home/rich/bin/mysql-memory-config-suggestions -c 150 -r 1500
# At          : 2020-07-04T10:08+01:00
# On          : meowko
# Note        : These values depend on the data itself at the
#               time of running the script.
# {
#   "conns": 150,
#   "ramMB": 1500,
#   "ramLeftMB": 1294,
#   "innoDBDataSizeMB": 17.1,
#   "innoDBIndexSizeMB": 22.9,
#   "innoDBTotalSizeMB": 40,
#   "suggestedKeySizeMB": 10,
#   "suggestedInnoDBBufferPoolSizeMB": 1284
# }
max_connections = 150
key_buffer_size = 10485760
innodb_buffer_pool_size = 1346371584

If you're interested, drop me a line.

jq (no, NOT jQuery!) and JSON nginx logs

This is an amazing command line tool for handling JSON data. It's really hard to search for tutorials and help on because searches tend to get it confused with jQuery - and there's a zillion pages about that!

Recently I've learned to efficiently parse stream using the -n flag and then a reduce inputs as $line  which is more efficient (memory, and speed) at processing large files then using slurp -s . I've now got my nginx logs stored as json entries, which has enabled lots of easy querying abilities, and I've got jq scripts to publish hourly counts of 2xx (success) 3xx (redirects), 4xx (request errors, like 404s), 5xx (server errors) for each of my sites. Also I have it drawing bar charts per minute/ten minutes/hour of things like how long the php process took to run.

I wrote a runner script to take some command line arguments and adjust the jq script before running it because I was finding I needed to create almost-identical scripts in many cases.

$ jq-run -l /var/log/nginx/access.log{.1,} -- requests.jq \
    inputFilter='.rsTime > "2020-07-04T04" and .rsTime < "2020-07-04T07"' 
    period=ten

"2020-07-04T04:0━━━━━━━━━━━━━━━━━━━━━◀                                   75 total "
"2020-07-04T04:1━━━━━━━━━━━━━━━━━━━━◀◀◀◀◀                                85 total "
"2020-07-04T04:2━━━━━━━━━━━━━━━━━━━━━◀◀                                  80 total "
"2020-07-04T04:3━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀                             96 total "
"2020-07-04T04:4━━━━━━━━━━━━━━━━━━━━━━◀◁                                 77 total "
"2020-07-04T04:5━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀                            99 total "
"2020-07-04T05:0━━━━━━━━━━━━━━━━━━━━━━━━━◀◀◀◀                            98 total "
"2020-07-04T05:1━━━━━━━━━━━━━━━━━━━━━◀◀◀                                 80 total "
"2020-07-04T05:2━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀◀◀                        107 total "
"2020-07-04T05:3━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀                  127 total "
"2020-07-04T05:4━━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀◀◁                       108 total "
"2020-07-04T05:5━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀◀                       111 total "
"2020-07-04T06:0━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀◀◀         157 total "
"2020-07-04T06:1━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀                          100 total "
"2020-07-04T06:2━━━━━━━━━━━━━━━━━━━━━━━━◀◀◁                              87 total "
"2020-07-04T06:3━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀◀◁         154 total "
"2020-07-04T06:4━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◁◆                  123 total    1 errors"
"2020-07-04T06:5━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━◀◀◀       163 total "

CiviCRM

I've done quite a bit of development in between childcare. It's been quite scattergun - I tend to fix things as I find them, sometimes little sometimes bigger. And when asked for functionality by a client, I usually try to abstract it into an extension that someone else could benefit from, since I think we all need to be working together.

Extensions: I've released an extension called Refine Mailing which helps you take a draft mailing to a group of people and then segment it for important groups that might need slightly different varients. It complements my Resend Mailing extension I released a few years ago that allows you to resend a sent mailing to a new set of contacts. Several people have contributed to my GoCardless extension, which is great. I've also been working on Keenies which is an experiemental search extension that aims to provide some specfic search capabilities that are currently lacking, e.g. find people who have opened mailings A, B, C but not clicked them, or find people in this group or that group and also in some other group, but only those without a signed petition activity.

Core contributions: I contribute to the core/main CiviCRM project/"product" as much as I can afford to. It's a difficult thing because some of the issues get very big and deep and it can easily hoover up days of time. It's hard to maintain a commitment that's enough to help the project move on, but without meaning I'm dropping/failing on commitments to my clients. The financial stuff is particularly hard because it's so important to everyone, it's phenomenally complex, and difficult to move forward at the same time as not breaking the older systems. It's an area that we need more developers involved in, but progress seems to steepen the learning curve. I'm trying (as are others) to improve documentation as a way to make it more accessible, too.

Zsh autocomplete, CUBE CSS, RSS for podcasts

• zsh: I'm always tweaking. This month it's been adding Marlon's aync autocomplete which brings fuzzy search (type abc and it's like a glob on a*b*c*) and instantly interactive matches.

• CUBE (Composition, Utility, Block, Exception) CSS looks like a good way to go. I've been using BEM (Block, Element, Modifier) a bit, but I've found it a bit clunky. My recent projects have used my home-spun utility/functional CSS classes, which has been quite liberating and more efficient. The efficiency comes at the cost of tracking which of the classes are in use, and only generating those. I do this manually. PostCSS can, I've heard, be used to do a static analysis of a static site and figure out which classes are needed, but this content is not static, so that's not possible. Anyway, CUBE looks sensible - I like splitting the composition layer out, I've been doing this for a while anyway (where the functional classes weren't enough). I think the challenge will come when the 'exceptions' are really numerate, e.g. if you have a block that needs to present differently according to the background of the container, and you end up with mulitple dimensions of variations (mobile - desktop, colours, styling options...)

• RSS! I've used RSS for, well, ever, but only just realised that it's a great way to keep up with podcasts. I feel silly for not noticing this, but I think it came from the Nextcloud News reader mobile app supporting audio content now. Still loving Nextcloud.

• Also been thinking about the rule of Big Evil Tech Giants and what can be done about them; when we feel they're the only option; how we can use them and still maintain/build resistance; and how many good organisations help these libertarian, anti-democratic companies by choosing to use them - and how I often feel the Googles get more out of that relationship than the good orgs do.

Feel free to comment if you'd like to.