I was playing with ChatGPT and Klavier.ai the other day. It purports to import documentation so that you can ask questions about it. I let it read in the PRQL Language Book, and you can see the rest of my experimentation at the PRQL site.
Author Archives: Rich Brown
Save money while keeping latency low
I often hear people saying they have bad responsiveness / high latency on their high-speed internet connection. Even though they have a contract for 500 mbps+ service from their ISP, a speedtest shows high ping times when the link is loaded.
The answer is well-known – install a router with SQM that knows how to control latency (“bufferbloat”). But… they counter, a router to handle my high-speed network is expensive! And that blows out my budget!
I posted a contrarian viewpoint in a post on the OpenWrt forum:
Save money and purchase a 250mbps to 350mbps connection and use good SQM (for example, the IQrouter v3, or any reasonable performance OpenWrt compatible router.)
Unless you’re unusual, and the transfer speeds of your bulk up/downloads are unacceptable, it’s likely that a lower speed to your ISP with a modestly-priced router that controls latency will make you just as happy.
Not only will you save money with a less-expensive router, you save every month with a lower ISP bill.
uBlock Origin and other blockers
Many people are surprised to hear that their web browsing doesn’t have to be crowded out by intrusive ads. There is an entire industry of “ad blockers” and “content blockers” that claim to fix this.
uBlock Origin is the one I use. Its author, “gorhill” states that it’s Free. Open-source. For users by users. No donations sought.
Accept no substitutes! The other products carry names that combine the words “ad”, “block”, “guard”, “plus” and a whole host of synonyms. And I wouldn’t trust any of them – they all come with with strings (or fees) that make them less than perfect.
For Chrome or Firefox:
- To install uBlock Origin, Google “ublock origin” (use that exact phrase) and then click the link for “ublockorigin.com“
- Follow the instructions
For Safari:
- uBlock Origin is no longer available for Safari
- Use AdGuard for Safari. It’s also free, and seems to follow the same ethos as uBlock Origin. (Tested with current Safari on 12Feb2023).
- For a state of uBlock Origin for Safari, and the recommendations that lead me to AdGuard for Safari, see https://github.com/el1t/uBlock-Safari/issues/158
For Other Browsers:
- Go to the ublockorigin.com site and follow their instructions.
Format pretty reports in PRQL
I can no longer bring myself to write bare SQL – PRQL makes building queries so easy.
I have become enamored with PRQL – Pipelined Relational Query Language, pronounced “Prequel”. It’s a simple, powerful, pipelined SQL replacement.
From the PRQL Github page: Like SQL, it’s readable, explicit and declarative. Unlike SQL, it forms a logical pipeline of transformations, and supports abstractions such as variables and functions. It can be used with any database that uses SQL, since it compiles to SQL.
I wrote a blog post over on the prql-lang.org site that tells how to make SQL queries “look good” using PRQL. See more »
Stable sorting for DB Browser for SQLite
I had some data in SQLite, and wanted to sort on the columns (and on a pair of columns). I really like DB Browser for SQLite because it’s free, powerful, has a nice GUI. It turns out that a stable sort on multiple columns is easy, but not well-documented. Here’s the how you do it:
- Open a SQLite file in DB Browser for SQLite
- In the Browse Data tab, click a column head to sort by that column
- Click it again to reverse-sort
- To sort by multiple columns, Control-Click (Command-click in macOS) on a second column heading. You’ll see small numbers next to the field names to indicate their order.
- Ctl/Cmd-click a second time to reverse-sort that column
- Click another column head to clear the previous selection(s)
- Here’s a video that shows how it works: https://youtu.be/KMowPpzZ2Vc
Docker vs. Ansible
I recently gave a talk to the local Dartmouth-Lake Sunapee Linux User Group describing some similarities between Docker and Ansible.
Both give you a repeatable configuration by executing some sort of script of actions. Here is a link to the slides I used for the talk. https://docs.google.com/presentation/d/1W_vEPSZ-rhareK5q56pd10m2Y7JT2mc-LxrHAE6yaSI/edit#slide=id.g14247b9f80b_2_75
Update: March 2023 – I recently learned about Dev Containers that provide the ability to bundle up all the tools required for a development environment. The PRQL project is using it to set up a complicated Rust/Javascript/Hugo/mdbook/Python environment.
INCLUDEPICTURE in Microsoft Word Mail Merge
What a crock! How badly documented can a facility be? There are lots of tutorials on the basics of Microsoft Mail Merge, but there is a dearth of information for the INCLUDEPICTURE facility to include images, especially if the images are retrieved from a link/url.
That said, I needed to use it for a project to create a document that plucks up variables from a spreadsheet and drops them into a pretty template for each page. I wrote this note so that the next time I need to revise the document (and stumble over this process), I’ll have these notes. I’m not claiming this is the only way/best way, just that it worked for me. I think I described everything, but please let me know in the comments if I’ve missed something.
I had the following hassles:
- Word uses voodoo magic to accomplish this. [The documentation for how to include images in a mailmerge document is terrible (at least, my google-fu wasn’t strong enough) and it’s worse for images specified by URLs.]
- You need to use (poorly-documented) keyboard commands to type the commands
- Once I had images inserted into the document, I needed to figure out how to adjust their size.
I succeeded – here’s what I did…
- Create an Excel spreadsheet with links (URLs) to the desired images.
- The Excel document should have headers in the first row that describe the data. If you’re retrieving images from a website, you’ll need a field with the links.
- Create a mailmerge document in Word. To do this:
- New Document, then from the Ribbon:
- Click the Mailings tab
- Click Start Mail Merge and choose Letters
- Click Select Recipients – you’ll need to choose the Excel document (and proper tab) that has the data to be merged
- Ignore the “Filter recipients…” stuff
- Click “Insert Merge Field” in the Ribbon and select your fields. The field name appears within «…».
- They’re voodoo magic. They’re kind of cool. When you click the Preview Results button (in the Ribbon), you’ll see your mail merge results, with substitutions of the live data for your
«variables»
- Click Preview Results again to un-preview results and continue editing the document.
More voodoo magic
- If you type
Alt-F9Ctl-F9 (Option-Fn-F9 on Mac), the«variables»
in the document turn into their underlying representation:«Field1»
turns into{ MERGEFIELD Field1 }
It’s editable text. - Don’t do it. It’ll leads to tears. But at least you can see how these are formatted.
Adding an image…
- If you want to insert an image (say, using a URL from that Excel file) you need to do even more voodoo magic. The format is straightforward, but a bit gnarly:
{ INCLUDEPICTURE { MERGEFIELD Name-of-URL-Field } * MERGEFORMAT \d }
- BUT… You can’t just type that string: remember, the
{ ... }
are voodoo magic. - Type Alt-F9 (Cmd-F9 on a Mac) to insert an empty set of braces in the document. Then fill in the text between. If you need a new set of braces inside, just type another Alt-F9 (Cmd-F9) in the right place, and keep typing ’til you enter the correct command.
- Preview Results to see if it’s working.
Resizing an image
- This is Microsoft Word. So there are no rules. Actually, there are, but they’re not known by mere mortals. (Sorry for the snark.)
- The images will be any size they want to be.
- To constrain the images, I found it useful to insert a one-by-one table, and let the content fill the cell. To do this, insert a 1×1 table, and adjust the size of the cell with Table Properties
- I set the Preferred Width (Table tab)
- I set “Specify Height” (Row tab)
- I set “Automatically resize to fit contents” (Options button)
- I then copy/pasted the voodoo magic for the image into the cell.
Finishing up…
- OK – You think you’re almost done. Maybe.
- The Finish and Merge button (in the Ribbon) actually performs the mail merge.
- Edit Individual Documents seems to create a new Word document containing a concatenation of all the mail-merged pages. You can save this (in Word or PDF format) or print it.
- Print Documents will print, or let you save to a PDF.
- You may need to force-load the images, to replace the (single) placeholder image with the right ones. To do this, Select All, then hit F9. You may see a spinner while Word retrieves all the images, but they should all be there.
Additional Thoughts
- I was successful with Microsoft for Mac 2019 using a
.docx
file, despite the fact that Google brings up a lot of (old) advice about using.doc
format, not the (new-fangled).docx
format. - Keymaps: This article gave me a breakthrough, showing how to insert the voodoo magic { } in a document https://wordmvp.com/Mac/FieldsInMacWord.html
- It’s easiest if the images are all the same size, to minimize resizing effort.
- I think the
\d
option of the INCLUDEPICTURE command causes Word not to save the images in the document, making the source file considerably smaller. - I could not find documentation about what the
\* MERGEFORMAT
string does. It works for me. - Let me know in the comments if you find find out more. I’ll update this article.
More Fun with Docker Tooling
I continue to enjoy using Docker to encapsulate developer tooling so that it doesn’t pollute my laptop with varying versions of software I don’t use regularly. (See Jonathan Bergknoff’s Run More Stuff in Docker and Andrew Welch’s Docker for all the things for further justification.)
In addition to using Andrew Welch’s vitejs-docker-dev project for Javascript development, I converted a couple of my personal projects to use a Dockerfile. I have submitted PRs to the upstream repo’s to incorporate the Dockerfile – we’ll see if they are accepted.
- RPM Test – responsiveness test tool for network latency
- Wireguard Vanity Address – creates a Wireguard public key with an easily-recognizable prefix.
- TrackR-Web-Bluetooth-API – reverse engineering the API for the TrackR Pixel gizmo that helps you find your lost keys
- Unifi-Controller – Instructions for a pre-built Docker container that runs on a Raspberry Pi
TL;DR Using a Docker container for these tools doesn’t really get in the way. Startup may be slightly slower (adding a second or so), but otherwise these tools run plenty fast. Plus, Docker eliminates a whole raft of hassles getting the software installed, and maintaining it across OS upgrades (say, on my laptop). I’m content.
Update: I am now contributing to the PRQL project. They have a highly-developed Docker container that encapsulates the build tools which mean that all the versions of Rust, Javascript, and other tools are isolated from my host operating system.
Migrating Snowpack to Vite (and Docker!)
About eighteen months ago I migrated a small Javascript app to use Snowpack development tooling. (This was mostly for fun, I had already had it working with Webpack.) Snowpack claimed simple dev tooling with nearly instant updates, using the power of ES Modules. It worked pretty well.
About six months ago, the team that developed Snowpack realized that their efforts had paralleled those of the Vite.js tooling. Vite.js also used ES Modules, and provides a mature code base and strong community support. Since the Snowpack team wanted to work on other projects (Astro), they switched their underlying tooling to use Vite.js.
So… I decided to see what it would take to migrate my Snowpack project to use Vite.js. Everything I had read said that it was easy. Here’s a field report from what was required.
A Side Project – Using Docker: Since this Javascript app was a side project, I have also been using it as a learning environment. I had read Jonathan Bergknoff’s Run More Stuff in Docker that made a lot of sense to me. (The magic of using Docker is that once you’ve created the instance, all the tool and dependency versions remain the same. It’s easy then to hand the Dockerfile to a colleague who can build an identical development environment in a few minutes. It also avoids cluttering my daily-driver laptop with multiple versions of Node, npm, Go, Python, rust, and any number of little-used tooling – they’re all encapsulated in the Docker container.)
So I decided to investigate whether using Docker to create the Javascript tooling would make my life better. Googling around led to Andrew Welch’s vitejs-docker-dev project. It builds a Docker instance with full development tooling (Vite.js, pnpm, hot reloading, etc.) that watches the source files in your local directory for updates. You develop the code using your favorite editing tools. Changes are immediately reflected in your browser/test environment. This is very slick. The vitejs-docker-dev
repo has good documentation. It describes a lot of background of how the Docker machine gets built and how to use it.
Update: Andrew Welch (who created the vitejs-docker-dev project) sent me a link to his article about using “Docker for all the things.” It’s a good adjunct to the original “Run More Stuff in Docker” post.
Back to the main story – here are the steps I followed to get my Snowpack project on the air with Vite.js:
- Create the Docker instance. Clone the
vitejs-docker-dev
repo. Then runmake docker
to put the set of development tools into a new Docker instance. This one-time step takes a few minutes. You may see several warnings (as described in a Github issue) but these don’t seem to be important. - Check the default Vite.js app builds on Docker. Run
make vite-pnpm run dev
in one terminal window, then runmake app-pnpm run dev
in a second window (waiting between the commands as described in the README). Click the http://localhost:3000 URL to see if the test Vite app starts up. Edit theindex.html
file to see if the change is reflected in the web browser. (It should be…) - Customize the
/app
directory for your app. Copy all your app’s files into theapp
directory. (I renamed the original toapp-old
and created a newapp
directory.) This required a bit of jiggering to adjust between Snowpack and Vite.js, such as:- My
index.html
file for Snowpack was in thepublic
directory; for Vite.js, I moved it to the top-levelapp
directory - Copy the
package.json
and other important directories toapp
directory - Adjust the paths for
index.html
: Snowpack bundles source files from/src
into the/dist
directory; Vite.js processes the files directly from/src
- I can’t remember whether I made other tweaks; it certainly wasn’t odious.
- My
- Remove references to the Snowpack modules from the
package.json
file. Optionally, you could take the opportunity to update the versions of dependencies. - Rebuild the Docker instance (#1 above) and restart the development process (#2 above). Click the link to http://localhost:3000 and your app should begin to run. You’ll probably need to make adjustments, but you’ll be substantially on the air.
TL;DR The process of converting to Vite.js wasn’t very hard (at least, not on my small project). It required a little farbling around, but nothing terrible. The jury’s still out on whether vitejs-docker-dev
will make my life better – but I think it just might.
Using Apple’s RPM tool
macOS Monterey ships with a tool that measures the responsiveness of your network connection. It saturates the network with traffic for 20 seconds, then measures the rate of short transactions to compute “Responses Per Minute.” Big numbers (above 2000) mean your network remains responsive when the network is heavily loaded. Small numbers (under 800 or so) mean your network isn’t responsive – potentially caused by bufferbloat.
There’s an iOS version described at https://support.apple.com/en-gb/HT212313
Stuart Cheshire and Vidhi Goel talked about the RPM tool at WWDC 2021. Apple also published an Internet-Draft that describes the RPM technique
Here’s a sample run from my Mac. The RPM tool displays my download and upload speeds (nominally 25mbps), and the number of simultaneous flows required to saturate the link (12, in this case). It shows the responsiveness as 1995 round-trips per minute. That’s really good: the average latency – even during heavy load – only increases a bit above the baseline (idle) 21 msec.
% /usr/bin/networkQuality -v ==== SUMMARY ==== Upload capacity: 22.657 Mbps Download capacity: 23.755 Mbps Upload flows: 12 Download flows: 12 Responsiveness: High (1995 RPM) Base RTT: 21 Start: 11/7/21, 7:18:37 AM End: 11/7/21, 7:18:47 AM OS Version: Version 12.1 (Build 21C5021h) %
Here’s a video that shows the tool in operation: https://youtu.be/e9DUTB9okMA