Apple made system calls horribly slow in the 10.14.6 update. Thank you, Apple!

After installing the Mac OS 10.14.6 update from the end of September 2020, I noticed something wasn't right. A simple Perl script that scanned a directory for all files and that invoked the stat system command for every file, had become way slower than before. I'm not sure if the same would have happened if it had relied on Perl's own stat() call, but this script needed the shell command because it can offer information that Perl's own stat couldn't provide.

Also when running a homebrew update, it seemed way slower than usual. It looks like Apple had messed something up w.r.t. system calls from within scripts, and perhaps other programs as well, although I didn't notice anything in regular apps. I knew this was likely to be reported by a gazillion developers and fixed soon, so I didn't bother. That horribly slow script of mine was annoying though, so I started seeing if it couldn't be improved.

The problem here was that the Perl script did the obvious thing of invoking a new instance of stat through backticks, for every single file it encountered. That's a lot of overhead. Until now the overhead wasn't bad enough for me to be sufficiently annoyed, but Apple's “update” had now pushed this way into the zone of bad words and gnashing teeth. The solution was pretty obvious: reduce the overhead by reducing the number of system commands. Luckily this could be done: the stat command does accept multiple files as argument, and returns the results as multiple lines in the same order as the arguments.

So, all I had to do was postpone the invocation of `stat` and build a queue of file paths, and execute the aggregated system command whenever it reached a certain size. In theory I could wait until the command line was about to exceed 262144 bytes (minus some margin to allow environment handling), which is the maximum system command length as reported by “getconf ARG_MAX.” In practice, I used a lower limit of 16 kBytes because above a certain threshold, the gain becomes pretty negligible anyway.

The result was that for a particular run, the script went from a 120 second runtime to 3 seconds. After Apple fixed the performance issue in a supplemental 10.14.6 update, the runtime became sub-second. So in the end I guess I should thank Apple for forcing me to refactor my script and making it way more efficient. Maybe every operating system should now and then introduce a temporary penalty on certain operations, or a limit on resources, just to force developers to be less lazy and actually improve their software instead of writing sloppy code and letting the machine just brute force it…

Cosmic Frontier: Override, a remaster of Escape Velocity: Override

A year ago I posted how it is still possible to enter your registration code in EV Nova, by jumping through some hoops. If you want to play the previous instalment of Escape Velocity however, Escape Velocity: Override, maybe this will soon be possible again without any need for special magic, in a brand new reincarnation of the game.

A Kickstarter campaign has been launched to rebuild the EV: Override game engine for current operating systems. Due to legal reasons, the game will have a different name: Cosmic Frontier: Override. One of the persons on the team for this Kickstarter is Peter Cartwright, who was the scenario designer for the original EV Override game.

The remaster will feature the original scenario, augmented with new content like new ship types and new mission strings. Depending on your backer level, you can influence this extra content, like having your own starport bar or ship variant. At the time of this writing, the campaign is just beyond its 50% funding mark, with 19 days to go. I'd really like to see this succeed, so if you are a fan of games like Escape Velocity and you have about $10 (or more) to spare, head over to the Kickstarter page and make your contribution.

The Dreaded Butterfly Keyboard: trying to postpone the inevitable

In a nutshell: If you have an Apple MacBook Pro or other machine with the dreaded ‘butterfly’ keyboard and it starts to exhibit problems, you options are:

• If you experience the problem of a certain key sometimes not reacting, try detaching the keycap (important: do not try this without first reading the “How to” section below for instructions). After looking—with a magnifying glass—for obvious dust, hairs, or crumbs and carefully removing any with precise tweezers, blast the mechanism with air from multiple directions, and reassemble the key.
• If you experience the problem of a key producing multiple keystrokes per press, first do the above, and after reassembling the key, hit it many times as hard as you're comfortable with. Just type a piece of text like a madman, and there's a good chance that the key will behave afterwards.
• If the above doesn't help, or problems occur too often, or you are not comfortable with taking the keys apart, and you have bought the machine less than 4 years ago, your better option is to go for the Keyboard Service Program.

The full story

A few years ago I was kind of forced to buy a new MacBook Pro, because my previous one, a 2011 model, had the dreaded GPU failure—another design flaw by Apple albeit lesser known than the keyboard debacle. Given the high price and the rumours about the bad keyboard, I seriously considered buying a Thinkpad or other PC laptop, but it looks like I am still too much locked into the Apple ecosystem to take that step. I hoped that the newer design with the extra silicone ‘flaps’ built into the keyboard, would make it reliable enough that it wouldn't develop problems in the situation where the MacBook Pro would be sitting 95% of its lifetime. But that was too much wishful thinking.

Even despite the fact that this laptop sits on a Griffin Elevator in a normal room with regular levels of dust, some keys did develop problems within less than one year. The down arrow key was first, it would sometimes miss a press. Then the same started happening with the Return key. I found a very useful page on iFixit Answers that showed how both thes keys were assembled, and this allowed me to detach them without destroying them. I then blasted the insides of the mechanism with air like crazy, and that has effectively remedied this problem.

A while later however, another problem started showing up: some keys, especially the ‘E,’ started producing double and sometimes even triple presses. The extra press would often come late enough that typing ‘bed’ could result in ‘bede.’ I again took off the ‘E’ key and inspected the internals, but all I found was a tiny hair that was only visible with a 10× magnifying glass. I gave the key the same treatment as the others, but it kept on exhibiting the multiple keystrokes problem.

It seems to me that this key repeating problem is not caused by dust particles. It rather looks like the contacts become unreliable, maybe due to an oxide layer forming on them? The keyboard can become pretty hot when the machine is under heavy load during an extended period, like when running a modern game. Maybe this degrades the contact surfaces? This theory is further corroborated by the fact that I eventually did manage to make this problem go away, by simply punching the key much harder than usual. Percussive maintenance does work, even on delicate modern machines…

The good news is, after a class-action lawsuit, Apple has started a repair program that allows to bring in your machine if it obviously exhibits these problems with the keyboard. They will then replace the keyboard, which usually means replacing the whole top assembly. As explained in this video by Louis Rossmann, theoretically it is possible to replace only the keyboard, but only with very specialised tools because the thing is actually riveted into the case and in many machines you need to detach the glued-in batteries without rupturing them and setting your workshop on fire.

And by the way, I fully agree with Rossman that I expected a machine this expensive to at least be as reliable as a much cheaper one with similar specs, and also much more serviceable. Where are the days when Macs were modular machines that could be entirely taken apart without even needing a screwdriver?

Even though the repair program is free, the downside of course is that you will have to do without your computer for a few weeks. So if you are only having mild problems with the keyboard, it may be a better option to first try to remedy it yourself, but only if you feel confident enough that you won't make things worse.

How to detach the keycaps without destroying them

If you want to have an attempt at detaching the keycaps yourself, BE VERY CAREFUL. Do not simply yank on the caps from any direction, or you will break little hooks and the cap will be forever loose. The trick is to slide a thin piece of plastic in between the cap and the butterfly mechanism (see SHOVE in the pictures below). Do not go too deep, try to keep the plastic as horizontal as possible to ensure it does not go under the butterfly hinges themselves. Then shift the plastic towards the clips (called slide in the pictures) to detach them. For most of the regular keys, the cap is attached with two clips at its top end, and two hooks at the bottom. For these keycaps, start the SHOVE in the middle of the left and/or right edge, as indicated with the arrows in the photo below, then slide upwards. Once one of the clips is detached, you can usually keep sliding across the upper zone of the cap to detach the other clip as well.

A piece of cut-up blister packaging works well for this purpose, but a thin guitar pick (plectrum) should also work well. The longer keys like backspace will typically have 3 clips instead of 2.

For other keys, the procedure differs only in the place where you have to shove the pick and in which direction you should then slide it. For the ‘up’ arrow key, the clips are at the right. For the ‘down’ key, the clips are at the left. For the tall European style of Return key, the clips are at the left, and there are 3 of them. I have borrowed the photos from the iFixit page to illustrate this. Note how the whole mechanism of the return key has been detached in this photo, this is not how it is supposed to be.

To reassemble, slide the hooks back in their place, then re-engage the clips by pressing on the zone where they reside.

The space bar is notoriously difficult. If you feel you need to take it apart, consider going for the Apple repair program instead. If you break things, you won't be able to benefit from the free repair.

And of course, the usefulness of the repair program is very debatable. The replaced keyboard will suffer from the same flaws. I wish there would be a replacement program where I could exchange my inherently flawed MacBook Pro for a considerable rebate on a late 2019 MacBook Pro with its reliable scissor keyboard.

A final piece of advice

If it isn't already obvious: never do anything in the vicinity of your butterfly keyboard equipped machine, that could cause any small particles to fly around. Or, cover the keyboard with a sheet of paper if there is any such risk.

Recently, the backspace key started acting up. Something had obviously got inside the mechanism, but with the naked eye I couldn't see anything suspicious. Even after repeated sessions of removing the key cap and blowing air, it kept feeling strange and sometimes did not respond to presses. I then took a flashlight and magnifying glass, and finally saw that some tiny piece of something had got under the mechanism. It was a pain to remove because Apple's silicone flaps that were supposed to keep junk from getting inside, now made it hard to get the junk out. I eventually managed it with extra fine tweezers, and it proved to be a tiny piece of clipped fingernail any other keyboard wouldn't care about. So, don't clip your fingernails in the vicinity of the keyboard…

True Tone display abruptly changing tint? Try not covering the sensor!

On various forums, one can find complaints about the True Tone display of the more recent MacBook Pro models suddenly changing tint, or the intensity of the screen flickering. Some of those forums will tell you to disable the True Tone or Night Shift feature. If you don't care about those features, it's an effective workaround, but it is kind of stupid to buy a machine with all these advanced features and then disabling them.

I have also experienced this, and found no rhyme or reason behind it—until now.
I'm pretty sure there are some other causes for this problem, including hardware or software faults, but before you go bringing your MacBook Pro to a service center, you might want to check the following first.

Many people cover the camera built into laptop screens with a sticker or something else, for reasons of privacy and/or security. The problem is that the True Tone sensor is very close to the camera. If your sticker is large enough, it will partially or fully cover the sensor. Even if that's not the case, maybe you often do use the camera, and then have to park the sticker somewhere, and a convenient place is right next to the camera—possibly right on top of the sensor.

When covered, the sensor will receive much less light than it should, possibly filtered in some strange way, and this can cause the True Tone system to react wildly to tiny changes in ambient light. I still don't understand why the adaptation algorithm allows such quick changes at all, but one can't really blame the system for starting to act weirdly in a very poorly conditioned situation.

The sensor is very hard to see, which is why most people are unaware that it exists at all, let alone avoid covering it. On my MacBook Pro model (2018 15 inch), the sensor proves to be at 17 mm to the right of the (more visible) camera, as can be seen by shining a strong flashlight across it. You may want to verify on you particular machine where the sensor really is situated.

So the morale of this story is: if you have any thingamajig that covers the camera, make sure it stays well away from the sensor. If you then still have strange variations in the tint of the display, then you may start worrying about a deeper problem. As a matter of fact, in my case the problem still occurs occasionally. It looks like the sensor gets confused in situations where there is little light, or rather weak light coming from a direction mostly parallel with the screen surface. What usually helps in such cases, is to provide a stable source of light closer to the computer, so just turn on a lamp.

Calibrating a cheap digital Chinese hygrometer

This article describes an method that could be used to get reasonably correct read-outs on a digital hygrometer even if it has no provisions for calibration at all. This was mostly an experiment but it has proven to work for the few cases where I tried it. Don't expect miracles from this method, but it may keep the hygrometer readout pretty reliable around the point where it will normally be used.

Introduction

Very cheap combined digital thermometer and hygrometer units can be bought from the usual Chinese marketplaces. The thermometer part of these things is usually reasonably accurate, but the hygrometer part is a game of roulette. Especially if you allowed the sensor to become truly wet by placing the meter in a room with 100% humidity, the risk is high that the calibration will be way off. A proper hygrometer should have a calibration button or button combo that has to be pressed after leaving the unit in an enclosed space with a saturated salt solution, which will stabilise at 75% RH after many hours. These cheap things however rarely have such button.

In the hopes that the device shown in the above photo would have some secret button combo to reset the measurement to 75%, I have tried inserting the battery while holding down buttons in various combinations, or mashing the buttons while the unit was already on, but without any success. I doubt whether devices like this contain any EEPROM at all, the measurement tables are probably hard-coded in ROM, and it is assumed that the humidity sensor will be accurate enough.

Hence the only way to correct the read-out, is to modify the signal coming from the sensor itself. There are two possible approaches:

1. Add resistors to either increase or reduce the electrical impedance of the sensor. This is probably a bad idea however, because it will change the slope of the humidity response curve. The meter will only be correct around a very limited set point, and deviate increasingly when moving away from this set point. It would take some fancy op-amp circuitry to truly shift the impedance curve without altering its slope, and this is way too much effort for a cheap device like this: just buy a better unit for the money that would go into buying parts and spending time on building this circuit!
2. Physically alter the sensor itself. These sensors are simple: it is a substrate sandwiched between two metal plates, or with two intertwined grids of metal ‘fingers’ at one side. The substrate contains some mildly hygroscopic material that will absorb moisture from the air, and the more humid the substrate, the better it conducts electricity and the lower the impedance. Therefore if we can alter the degree to which the substrate wants to absorb water, we can shift the impedance value. I'm not sure whether this won't also alter the slope of the impedance curve, but from my initial tests this seems pretty OK when using the right substances.

I have tried approach 2 with reasonable success. While in theory you could use only a saturated salt solution to check the correctness of your adjustments at 75% RH, it is better to also rely on a true calibrated hygrometer to verify that the cheap meter gives more or less correct readings around the humidity level where you will use it the most.

The method (updated 2020/11/25)

A warning in advance: to get good results, you will need a ton of patience. This is not something you can get right in a few hours unless you are extremely lucky. This takes multiple iterations of a cycle that takes at least 2 days. If time is money, buying a good and reliable meter is much more economical.

I have encountered two styles of sensors and they react somewhat differently to manipulation, but in the end the following strategy seems to work best. We'll start out by applying too much of a substance that is hygroscopic, in other words that likes to absorb water. Then we'll gradually wipe away some of this substance until the read-out is correct. The tricky thing is that wiping the sensor will somehow disturb it in such a way that one needs to wait several days until it stabilises.

Initially I used only hand sanitiser gel as the hygroscopic substance. Such gels will always contain something that attracts water, because otherwise the alcohol in the gel would dehydrate your skin. Typically it will be glycerine (glycerol) but it doesn't really matter what exactly it is. The concentration of this additive is also usually quite low in these gels, which makes it easier to apply small adjustments. The hand gel worked OK for the first type of sensor I tried to tweak. For the second type of sensor however, I had to use a calcium chloride (CaCl₂) solution, mixed with a bit of the sanitiser gel to make it more sticky. I obtained the CaCl₂ from the typical passive humidity absorbers that use a bag or brick of CaCl₂. This absorbs humidity from the air and gradually turns into a brine solution that drips down into a container. These devices are mostly useless as far as dehumidification goes, but they are a good source of CaCl₂. I simply took a drop of the brine solution, mixed it with a bit of the gel, and applied it to the sensor, wiping off the excess such that it wasn't entirely soaked and gave a read-out below 99%.

Next to the hygroscopic substance, you'll also need a classic hygrometer calibration box. This consists of a watertight sealed box, with inside it a small container filled with a saturated salt solution. Take a small cup like a shot glass, fill it with salt, then add water until the salt has become a wet sludge with the water just not spontaneously flowing out of it. This kind of salt sludge has the interesting property that when placing it in a sealed box, it will make the relative humidity inside that box gradually become almost exactly 75%.

So to recap, the things you need are:

• Some hygroscopic substance that sticks to things. What seems to work best is a mix of hand sanitiser and calcium chloride brine.
• A sealed box with inside it a small cup of saturated salt solution,
• a reasonably well-calibrated hygrometer,
• a lot of time.

Steps

The general strategy is to start with too high a read-out and then gradually wipe the sensor to bring it closer and closer to the correct value. The most tricky thing is not to wipe too much, because then you have to start all over again. Wiping the sensor disturbs it only a little. Re-applying the hygroscopic stuff brings it totally out of whack and it takes many iterations to make it stable after that. Therefore you will want to avoid bringing the sensor to a state of too low read-out. Wipe carefully!

1. Start by ensuring the sensor is clean. If there is any residue on it from a previous failed experiment, remove it with water or rubbing alcohol, whatever works best.
2. Cover the sensor uniformly with the hygroscopic stuff. Most likely this will cause the read-out to jump to 99%. Wipe off the excess until it drops below 99%.
3. Now rely on the calibrated hygrometer to continue wiping the sensor until it is near the actual humidity value. Needless to say, take care not to breathe humid air towards the general direction of either meter. This is the most tricky step. In my case I had to wipe until the read-out was 5 to 10% below the actual value. I can't really explain why, and it is possible you may need to aim for a different offset. This is one of the things that makes this process tedious. It seems the required offset became smaller with each cycle, so you should become more careful as the error on the read-out decreases with each cycle of this procedure.
4. Now put the hygrometer in the calibration box with the salt sludge and leave it alone for at least 36 hours. After that, see what value it settled at and remember this value, then take the hygrometer out of the box and let it settle again for at least 12 hours.
5. Now look at the final value and compare to the known humidity. Ideally you will want to do this at a humidity level where you want the hygrometer to be the most accurate.
6. If the read-out is very close to correct, resist the temptation to get it right to the last percent: most likely you will mess it up and will have to start all over. Just stop and be content with what you have. Remember, it is much harder to correct too low a read-out than too high.
7. Otherwise, if the value is still too high and it also was well above 75% in the box, go back to step 3 and wipe, but take care to make increadingly small corrections.
8. If on the other hand the value is too low, it depends. If it is way too low and the value in the box was also well below 75%, you have over-done the wiping and you will have to go back to step 2 and prepare yourself for several more days of going through the same cycle over and over. If it is just a bit too low but the read-out in the sludge box was very close to 75%, then most likely you will never get a perfect reading from this meter across the whole range. You could try to apply a different mix of hand gel and CaCl₂ in the hopes of getting a better overall response curve, but in the end you might just have to be content with slightly too low readings at lower humidities.

The 75% calibration box has a double purpose. First, it offers an initial test to see if the sensor has any chance of being close to correct. Second, it speeds up the stabilising of the sensor after it has been disturbed by the wiping. Again, I cannot really explain why it behaves like this, but in the end it does provide a good result.

As you can see, this method is tedious and involves a lot of trial and error. This is what makes it mostly useless, unless you're in the middle of a virus pandemic and are looking for things to kill your time and gain some usable measurement devices during this process…

Even if you can perfectly nail the calibration to 1% accuracy, of course you still shouldn't trust these things to be reliable. Also never use these simple hygrometers in environments where relative humidity get near to 100%, because in extreme cases, a drop of water may form on the sensor and disturb the hygroscopic layer. Just buy a good hygrometer that can be calibrated if accuracy and reliability is important.

Get rid of the bottom toolbar in Google Chrome on Android

In a recent update, Chrome on Android introduced a change that caused the previously experimental ‘duet’ feature to be enabled by default. This places an additional toolbar at the bottom of the screen with buttons for new tab, search, and share. This bar is supposed to disappear at the same time when the top toolbar disappears (typically when scrolling down), but on my phone this was often not the case, especially on pages that are not long enough to allow scrolling. Worse, the bottom part of such pages could then become permanently obscured by the toolbar, pretty damn annoying.

It does make sense to place controls at the bottom because this makes them usable with one's thumbs, but then everything should be put at the bottom, not both at the top and bottom! Because the latter is currently impossible in Chrome however (as far as I know), the best you can do to avoid the possible nuisances of this bottom toolbar, is to disable it.

This used to be controlled by a single setting called “Chrome Duet” in the chrome flags. Now however, there seems to be a new one called “Duet TabStrip Integration” that also enables this feature. It is unclear how these interact.

To completely disable the bar, enter “chrome://flags” in the address bar, and then enter “Duet” in the Search flags box. Set both “Chrome Duet” and “Duet TabStrip Integration” to “Disabled”. Then restart Chrome. If the bar is still present, toggle the flags back to “Enabled” and then “Disabled,” and restart Chrome yet again. Keep repeating this dance and eventually it will work.

I'm inclined to try another browser but unfortunately Google managed to lock me into their ecosystem because I use Chrome on other devices and it is pretty handy to have everything synchronised, like bookmarks. If they keep annoying me with unexpected changes like these however, I may just become motivated enough to migrate all devices to another browser.