zondag 3 februari 2019

Enhancing headphones or earphones with Audio Hijack 3

Long ago there was an awesome audio enhancer for Mac OS X that could improve system-wide audio in various ways. It started out as ‘OSS 3D’ by Subband Technologies, later on it was bought by Joesoft and rebranded to ‘Hear,’ and it eventually migrated to the Prosoft brand name. This program offered various effects like 3D audio, a straightforward equalizer, a virtual subwoofer, and various other things that were both useful and gimmicky. Everything was fine until the program became unable to enhance audio coming from iTunes, arguably its most important target. Prosoft was never able to fix this and eventually the program stopped working altogether with the latest major OS X updates. Rest in peace, Hear.

I have been looking for a way to simulate at least part of Hear's functionality. I especially relied on it to obtain a more natural listening experience with headphones. If you have ever listened to e.g. Beatles songs on headphones, you will have noticed first-hand that headphones do not offer the same listening experience as loudspeakers. With loudspeakers, sound from the left speaker will not only reach your left ear but your right ear as well and vice versa, both directly and indirectly through reflections. With head- or earphones however, the left and right channels are fed strictly to the respective ear only. If the music was mixed with sound coming only from either left or right (as was often the case with the first stereo recordings like from the Beatles), this offers an unnatural experience. Even today a lot of music is mixed with loudspeakers in mind and will not sound ‘right’ when listening with headphones.

A good solution is to preprocess the music before sending it to the headphones, simulating it being played on loudspeakers. The most basic way to do this is to mix a bit of the right channel into the left and vice versa. This already makes Beatles songs less awkward to listen to but it's not quite it. To add more realism, the tiny delay caused by the longer audio path from the left loudspeaker to the right ear and vice versa must also be taken into account. From my DSP courses I remember that a realistic delay is 390µs (this can be derived from the geometry of a typical stereo setup and the shape of a human head). To further increase realism, there must be some low-pass filtering as well because higher frequencies will have a harder time reaching the opposite ear. And finally to get the sensation of listening in a real room, some reverb can be added.

This idea can be implemented with Audio Hijack from Rogue Amoeba Software. It is not for free but is well worth its price for anyone who needs to do more with audio than merely playing some music. So let's see how we can implement our fancy headphone enhancer with Audio Hijack.

The above image shows the setup. The upper part is the one that simulates a room. A slight reduction in volume is applied to compensate for the fact that we will be adding a cross-mix of the channels later on. A reverb effect makes the sound less ‘dry’ by simulating it bouncing off walls, floors and ceiling. The configuration of the reverb unit is a matter of personal taste. I started out with the defaults of the Matrix Reverb and reduced the ‘large’ parameters to have a faster drop-off.

The lower part is the most important one, it represents the cross-mix between loudspeakers and ears, the extra delay due to the longer audio path, and the attenuation of high frequencies. The 390µs delay cannot be obtained with the standard effects available in Audio Hijack, a custom plug-in is needed. I've found this free sound delay audio unit from Voxengo to be adequate. You need to reboot after installing the AU plugin to make it available in Audio Hijack. Enter a value of 0.39 msec in the effect configuration. People with really fat heads might try a larger value but it should never deviate much from this.
The exact value for the low-pass filter isn't crucial. This filter has two purposes: first, simulate the actual attenuation of higher frequencies that would occur in the situation depicted above, which is difficult to quantify anyway. Second, avoid interference between trebles in the original audio and the delayed copy. Without the low-pass filtering, there would be a comb filter effect that can make some trebles sound weird. If you still notice this, you may want to use a lower frequency than the 6 kHz I used here.

After mixing everything together, there is another volume control and an equalizer. These are optional but unless you have extremely high-end headphones, a little extra EQ can get that bit of extra fidelity out of them. The reason why you may need to precede the EQ with a volume reduction is that if you need a considerable bass boost, you'll need to attenuate the input to avoid sending the audio into overdrive. You may add other effects to your liking, e.g. a dynamic range compressor in case you're listening in a noisy environment (although a lot of modern music has no dynamic range to begin with…)

zaterdag 19 januari 2019

Entering your EV Nova registration code is still possible

In a nutshell: if you have a registration key for EV Nova and still know the moment when Ambrosia sent you this key (within an accuracy of a few days), you can still register new installs of the game if you set your computer's clock to a time just after that moment. You only need to do this for the registration, after that you can set your clock back to normal. It may be necessary to disconnect all networks to disable additional attempts of the registration app to check the time.

The long story:
Escape Velocity Nova, or EV Nova in short, was one of the most iconic games from the now defunct Ambrosia Software. Their website is still online at the time of this writing but buying anything would be a bad idea because their registration service is dead. Not that you would want to buy much of their catalogue because most of it is too outdated to run on modern systems.

Recently the GPU in my early 2011 MacBook Pro had self-destructed and I decided to buy a new MacBook Pro. While waiting for it to arrive, I could still use my old MBP by forcibly disabling the discrete GPU. Of course this meant I couldn't play any games needing the GPU, hence I started playing EV Nova again. Everything was fine until I finally got my new MacBook Pro and wanted to continue the storyline I was in. Two problems:

  1. EV Nova simply does not work anymore in OS X Mojave. It seems some are able to run it, but for me neither copying the binary from my old MBP to the new one, nor reinstalling it and re-applying the RyuKorijo patch helped.
  2. Entering my registration code in any newly installed copies was impossible because the code has expired and there is no way to obtain a new one, moreover Ambrosia's licensing server is dead anyway.
Problem 2 proved relatively easy to solve. The registration system will accept codes if they are not older than a certain number of days. This means you can still register the game if you turn back your computer's clock to e.g. 1 day after the buying date. Not sure how many days of margin one gets, but luckily I keep all my mails (and I really mean all) so I knew exactly when I had been issued my old registration code and I set the clock to 1 day after. Normally the game also does an online check of the code but that can of course be easily sabotaged by disconnecting all networks. With the Ambrosia servers offline however, even that is now unnecessary. Still, when trying it in Mac OS 9, the app seemed to somehow check the time online so I had to disconnect the network after all.

I still haven't solved problem 1 however. Others have managed to get EVN running in OS X Mojave but for some reason it keeps failing on my system. I have opened a new issue on the RyuKorijo GitHub in the hopes of getting some help. Given that there are rumours 32-bit application support will be entirely dropped in one of the upcoming major OS X updates though, trying to keep EVN running in OS X feels like a futile exercise anyway. I've tried running the game on two different platforms instead.

First in SheepShaver, because that would be the most convenient. I got the game to launch but the menu buttons lack their labels and do not work. Luckily the functions of the buttons can also be accessed by pressing certain keys: N for new pilot, O to open one, E to enter the game, P for preferences, Q to quit, and A for about (there's also an easter egg key by the way). There are some other minor problems like EVN sometimes crashing at startup and the target cursor not being visible which makes it difficult to select planets or ships. A plus however is that pilot files are interchangeable with the OS X version.

Next, in Windows 10 under Boot camp. Slightly annoying but I reboot for other games anyhow. I used all the tweaks found here and the game works, but it is only really playable in windowed mode. In full-screen the graphics are extremely jittery. I had to use these values in the resolution patcher to get the window take up almost the entire screen of my 15" MacBook Pro: 1600, 992, 1600, 800, 1280 (only the first two matter probably). A big drawback is that I can't port any pilot files from the Mac to the Windows version and there are some things off about the Windows version, like sound quality, readability of text, and the graphics are still jittery at times. And obviously it feels wrong to have to resort to Windows in order to play one of the most iconic Mac games!

In the end I prefer SheepShaver in spite of the issues it has, and I now play EVN inside it because the game just feels much smoother than in Windows and I can continue with my existing pilots. If the remaining issues in SheepShaver can be solved, I think it will end up the best long-term solution to keep playing EVN.

vrijdag 11 januari 2019

Is ETCV (Ethereum Classic Vision) a scam or not?

Logo shamelessly stolen from ETCV website
There is quite a bit of FUD about the upcoming Ethereum fork, called Ethereum Classic Vision (exchange token ETCV) being a scam like Ethereum Nowa. This Dutch website for instance tries to give some evidence of this claim.

If ETCV is a scam indeed, the authors sure have done a lot of effort to make it less obvious than with Nowa, whose site has spelling errors and makes ridiculous claims about the forked cryptocurrency. The biggest warning light for Nowa is the procedure to supposedly obtain free ‘ETN’ (which by the way is an already existing token for Electroneum, another fact that makes Nowa extremely suspect). They outright ask you to transfer your ETH to their address and those who are even more gullible than that, can even make it worse by sending them their private keys as well. First rule of cryptocurrencies: never give anyone your private key. Second rule… never give anyone your private key!

The Classic Vision website on the other hand looks a lot more legit. It seems the ‘scam’ claims for ETCV are mainly based on two things:
  1. Confusion between the supposed Nowa fork and the ETCV fork. The aforementioned article mentions some facts that appear to originate from the news about Nowa, like the photo models story. This confusion is understandable because the (supposed) dates for these two forks are only 1 day apart.
  2. The idea that you are supposed to enter your ETH wallet private key to claim ETCV. There is some truth about this because there are only two ways to view an ETCV wallet on their website: either through a keystore or by directly entering the private key. Even though the latter seems to violate the First Rule, in this embryonic stage of the new currency with no third-party support yet, it makes some sense that these are the only options. Obviously the smarter option is to use the keystore but when following proper crypto hygiene, there is no risk with directly using the PK either as I will explain below.
The other suspect facts mentioned are that the website lacks certain information like the exact block number for the fork (which is true and pretty annoying), or obvious contact information for the authors. I don't know about you but in this day and age no sane person would spread all their personal details on a website. There seem to be enough contact methods albeit not very direct.


Proper Forking Hygiene

This leaves the private key issue as the only big source for concern and until I have actually seen the fork happen, I cannot tell whether this concern is warranted. It is perfectly possible they will indeed use any wallet credentials you enter on their website to plunder your ETH wallet instead of handing you free ETCV. However if you treat the fork as follows, there is zero risk of your ETH being stolen:

  1. Create an entirely new ETH wallet using MyEtherWallet or a similar service. You will never use this wallet for anything else than the ETCV claim.
  2. Transfer as much ETH to this wallet as you want to be replicated into 3 times the amount of ETCV (as they promise).
  3. Leave the ETH on this wallet until you're certain the ETCV fork has happened.
  4. Before doing anything else, withdraw all your ETH from the wallet to another wallet.
  5. Now claim the ETCV using the wallet address from step 1.

If this is a true fork, what will happen is that the entire ETH chain state is duplicated into the new ETCV branch. This means any existing wallet addresses that exist for ETH at that point will also be valid wallet addresses for ETCV. Anything that happens with ETH after the fork will have no effect on the ETCV branch and vice versa because they each go entirely their own ways. Hence if you withdraw the ETH from your wallet with private key abc123, the ETCV wallet with the same private key will still contain the amount of coin that existed at the time of the fork (which in this case will be multiplied by 3). And of course if the whole thing was a scam after all and someone obtains your private key after you have emptied your ETH wallet, they cannot steal anything.

Update: It's (Most Likely) a Scam Alright!

Now the fork is supposed to have been performed, how do things look? Pretty bad:

  • Their website has been suspended
  • Before the site disappeared, a highly suspect prompt was added to it to ‘buy’ ETCV by sending ETH directly to a certain address. This prompt contained conflicting information that the ratio of obtained ETCV would be both 50:1 and 1:1. This looks just like a desperate attempt to grab some more ETH before their whole façade collapses.
  • Their wallet site is suspended
  • Their twitter account is suspended
  • News articles about ETCV being a scam are popping up

Although all this is not 100% irrefutable evidence that it's a scam, let's say it is 99.9% evidence. I have to hand it to them, they were a lot more crafty than the Nowa people. They did a lot more effort to make their scheme appear legit. On BitcoinTalk they managed to gather a considerable following and it seems there still are some believers at this moment.

However, I have lost nothing because I have followed the above procedure even though I haven't executed the last step. I'll keep the wallet address just in case we are in that 0.1% and ETCV exists after all, but I don't hold my hopes up high.

woensdag 23 mei 2018

3D Eksperten Silky Gold and Copper PLA review

I have been looking for a gold-like filament for a while, and it proves difficult to find one that has a convincing metallic appearance. Many filaments with ‘gold’ in their name are merely a yellow or orange colour with perhaps a hint of metallic additives. For instance I bought a spool of Polaroid ‘gold’ PLA and even though it is overall a good material, it proved to be merely a deep yellow with only under ideal lighting conditions a very slight metallic sheen.

A while ago I saw some makes on Thingiverse that looked quite promising, and the author was friendly enough to tell me that the filament was ‘Silky Gold’ from a Danish manufacturer called 3D Eksperten (3DE). At that time 3DE didn't ship to my home country and only offered expensive express shipping. I checked out the store again after a few months and they did ship to my country with an option of more reasonably-priced shipping, hence I ordered both a spool of the Silky Gold as well as Silky Copper. I added the latter not only to reduce the relative shipping cost, but also because it looks stunning on photos. The total cost was about €60, which means €30 per 1 kg spool which is reasonable for a specialty filament. In hindsight I should also have added a spool of Silver to have a full set of shiny metals, so I know what to buy the next time.

Despite the cheaper shipping it still arrived within a mere 3 days with tracking information. The filament comes in stylish boxes and is packaged in resealable plastic bags with of course a packet of silica gel contained within. This is nice but I have learnt that all plain plastic bags are only weak barriers against moisture: they will only slow down the rate at which moisture can reach the spool. For this reason one should refresh the desiccant in such bags at regular intervals, and for longer-term storage it is recommended to use mylar bags instead. Those have a metallic layer that is much better at stopping moisture (this is why all foodstuffs sensitive to humidity are packaged in metallic bags). The desiccant inside these particular bags proved to have absorbed near their saturation level of moisture, so I dried the filament in an oven to be absolutely certain it was in optimal condition. A weight comparison however showed that it hadn't taken up any noticeable amount of moisture.

The spools themselves sure do look promising out of the box, especially the copper one which looks like a big transformer coil. I have never seen such a convincing plastic simulation of a metal. The gold is very nice as well, still obvious it is merely plastic but it comes closer than any other attempt I have seen so far. It wouldn't be the first time that printed objects lose the appearance of the raw spooled material however, so the proof is in the pudding and it's time to load up this stuff and get printing.

The tolerances on the filament are within the typical ±0.05 mm range. On the gold I measured an average diameter of 1.77 mm, on the copper 1.72 mm. These diameters remained consistent over the length of material I have used so far. The filament also appears well wrapped without risk of tangling, something that cannot always be said from the cheaper brands. I used my typical PLA settings of 200°C extrusion and 60°C bed temperature, on a glass bed with 3DLac and moderate cooling fan speed. Adhesion is excellent even with only a thin layer of hairspray. There is no obvious odour during prints, not even the typical popcorn-like smell of regular PLA.

I started out with my traditional test print, a variation on the classic 20 mm calibration cube printed at 0.2 mm layers. This already looked very promising: the surfaces are shiny enough to reflect light like actual metal albeit with a matte finish (a true mirror it ain't). I then printed the 3DHubs Marvin (or rather my ‘improved crotch’ remix, which is a good test for overhangs and rounded surfaces. These looked even better, especially the copper one which from a distance could easily fool someone into believing it is made from actual metal.

The Marvin prints do show it is even more essential to avoid visible seams with these filaments than with most other materials. The glossy effect is visibly disturbed at the places where an outer contour starts. To understand why, we need to make at least a good guess at how the metallic effect is achieved.

Left: piece of free-air extrusion, right: normal printed extrusion
Close-up view
A close-up view reveals the usual trick of mixing in little metallic specks is not being used. Instead, there is a shiny effect that varies with extrusion direction. How does it work? There are two important observations: one, both the spools and the printed results reflect light only at angles perpendicular to the direction of extrusion. Second, the filament tends to contract very quickly in the direction of extrusion when extruding in free air, and the end result loses its shiny appearance altogether and becomes a plain dull yellow or orange strand. I can only explain this by assuming that the plastic contains a kind of elastic fibres that become glossy when stretched (like silk indeed). During normal extrusion the material is not allowed to contract, which ensures that the fibres remain stretched. For the same reason you should not print this filament too slowly: if the extruder keeps heating the same part for too long, the shiny material gets too much time to lose its ordering and gloss.

This also explains why seams are especially visible: the extrusion changes direction abruptly at the start of the contour and it takes about a millimetre until the fibres are fully aligned to the new direction.
Vase mode print of thing:415360 by Alphie

There is unfortunately no magic solution to hide seams. If you're lucky, the object has enough concave corners to allow your slicing program to hide the seams inside. If not, a typical strategy is to randomise the starting point of the contours but this will leave the surface littered with quite visible specks due to the effects described above. The only way to eliminate seams is to print in ‘spiral vase’ mode when possible. I have done a few vase prints and they look impressive.

Objects with many different surface orientations are the most rewarding hence I made a vase model with faceted hexagonal outer surface specifically to showcase this filament. Also when printing coin- or medal-like models the best results are obtained when keeping the special properties of the filament in mind. Top surfaces of coins should be printed as one continuous spiral in order to get a finish that has no seams and reflects light in interesting ways.

Left coin was printed with spiral top infill, right with plain rectilinear. Model by FLOWALISTIK. (Too bad these aren't worth actual Bitcoins…)
The material proves quite tough: extruded material doesn't break easily and is not easy to cut up. If my fibres theory is correct, perhaps they also contribute to strength. While I consistently have stringing problems with almost all other PLA filaments in my Micro Swiss all-metal hot-ends, these silky filaments are much less prone to stringing, perhaps again thanks to their tendency to quickly contract in free air.

There is one problem with ordering this filament and it is that the 3DE store is mostly in Danish only. The language is similar enough to my Dutch mother-tongue that I managed to get through the ordering process with the occasional help of Google Translate, but it may prove challenging for people from other countries. Also, no shipping outside the immediate surroundings of Europe. On Amazon one can find similar filaments which might come from the same source (judging from the similar hexagonal pattern on the spools), so it is worth trying if you are not able to order from the 3DE store.

To conclude, this is a great filament for decorative pieces and medaillons or coins. The copper is obviously my favourite because it comes very close to looking like the real thing. The gold may not be able to fool anyone but still it looks better than other attempts at a ‘gold’ filament I have seen.

zondag 15 april 2018

Hacking the 4th generation IXO: white instead of yellow LED

I have a 4th generation Bosch IXO screwdriver and it is pretty good, aside from the measly LED that is supposed to help light up the thing you're working on. The LED is a yellow-orange color and is not very bright. It also has an uneven circular intensity pattern. In other words, it is pretty useless. I have been planning to replace it with a white LED for a while, and it seems I am not the only one with this idea. A bag of Nichia NSPW300DS 3 mm LEDs has been sitting on my desk for more than a year and I finally decided to have a go at it. These are probably not the brightest 3 mm LEDs currently available, but they were available at the shop where I usually buy components, and 15000mcd isn't bad at all.

However, as Joken mentions on his IXO teardown post, one cannot just replace the LED: the yellow LED is only supplied with 2 V through a 68 Ω resistor. Dropping the resistor only gains about 0.7 V, and 2.7 V is still way too low for a typical white LED that requires about 3.2 V.

I have a solution though: I bought some booster circuits from AliExpress a while ago, in fact these are buck/boost circuits that provide a steady 3.3 V output from any input between 1.8 V and 5 V. They aren't terribly efficient and I don't need the buck capability here, but the PCB is tiny which makes them a good candidate. They look like the following photo:

I did some tests by connecting the booster instead of the LED+resistor, but it is unable to reach 3.3 V with the white LED on its output. This is because the circuit that drives the LED seems to be a current source, limiting the current to either 11 mA (when pressing the switch without activating the motor) or 20 mA (when the motor is active). Due to P = U⋅I and some loss due to inefficiency of the booster, it can only push either 4.5 mA or 10 mA through the white LED. However, there is a way around this: by reducing the value of the 4.7 kΩ resistor that comes before the transistor, we can make it supply more current.

I simply soldered another 4.7 kΩ on top to reduce the resistance down to 2.35 kΩ:

This shifts the current source to 17 and 36 mA respectively. The booster turns this into 7.5 mA and 17 mA, which is much closer to the nominal current for the white LED, so this is the final configuration I used. There is no series resistor for the LED, I purely rely on the limited current coming from the transistor. I removed the 68 Ω resistor and connected the booster circuit. Instead of trying to solder onto the tiny resistor pads, I picked the connected through-hole pad. The little PCB (with the 3-pin header removed) fits nicely under the battery.

The result is pretty nice, finally the LED is useful! The lighting is also a nice even area instead of ugly concentric circles. It does draw more current now, but compared to the motor, 36 mA is still pretty negligible.

vrijdag 19 mei 2017

Making Prosoft Hear work again with iTunes 12.6

The only reason why I stick with iTunes is that I have been suckered into it from the start when it still was a pretty good application for managing an offline collection of music files. Lately, the app has shifted towards cloud-based stuff and artificial stupidity algorithms that try to suck all the joy out of being your own virtual DJ. It is still usable for its original intent however so I haven't tried to move to an alternative… yet.

Lately, Apple has done two more stabs at endlessly annoying me and making me regret that even my alarm clock is tightly coupled to iTunes by means of Koingo's Alarm Clock Pro. The first thing they did was break the visualizer plug-ins, in the typical Apple fashion that has since many years made me shy away from developing anything specific for OS X. This fashion is of course doing it in total silence, without warning anyone or even mentioning it in the release notes. Starting with iTunes 12.6 the visualizer menu simply only shows the two built-in visualisers, ignoring any installed plugins. I am pretty glad I never took the effort to polish up my now obsolete Spectrograph plug-in. Maybe I unconsciously saw it coming.

The second thing they did, and what this post really is about, is that they also broke the Hear audio enhancer from Prosoft. I heavily rely on Hear to make listening with headphones more enjoyable. When properly set up, Hear can make even the cheapest headphones sound like expensive Sennheisers. Or in my case, make mid-priced Sennheisers sound like an explosion of aural bliss. It can also squeeze extra fidelity out of simple Bluetooth speakers or other sound systems. Great was my frustration when I noticed the controls had no effect anymore after iTunes had sneakily upgraded itself to 12.6.1, and it was even greater when I saw on the Hear product page that Prosoft officially does not support iTunes anymore. That's right, they admit that their product primarily geared towards enhancing your music experience, does not work with the de facto standard for music playback in Mac OS. I commend them for that, but it doesn't make the fact less annoying.

My guess is that iTunes has become yet another application in the row of ‘sandboxed’ apps like Safari, which are nailed shut to avoid exploits from malware and the like. After all, iTunes is linked to a store and the store is linked to credit card details. Why this also has to break the ability to manipulate the audio stream, beats me. I contacted Prosoft and they are looking if the problem can be fixed, but I'm not sure if they'll be able to circumvent the sandbox restriction.

However, I noticed that applications like Nicecast still work fine with iTunes. This made me pretty confident that I can still make the audio go through Hear after all, albeit by means of a pretty clumsy roundabout. And indeed, after the necessary cursing and kludging I made it work as follows.

Making it work

The strategy is quite simple: re-route iTunes' audio output through another non-sandboxed program that plays the stream on standard output. We can do this by means of two different free, open source projects: SoundFlower and Audacity.

  1. First install SoundFlower, the current 2.0b2 release from mattingalls' GitHub seems to work fine, at least in El Capitan. Don't bother with trying to get the Soundflowerbed application to work, you don't need it.
  2. In your Sound system preferences, set the ‘Soundflower (2ch)’ device as output device.
  3. Open Audacity and select the ‘Soundflower (2ch)’ device as the microphone device. Select the ‘Built-in Output’ as your output device.
  4. In Audacity's preferences, Recording, enable ‘Software Playthrough’.
  5. Drag both the input and output volume sliders in the top-right corner of Audacity's main window to maximum. Now press ‘Click to Start Monitoring’ and start playing some music. You should see Audacity's VU meter start to move, and hear the sound as usual. You may need to give the Hear control panel a kick by toggling the on/off button or nudging a slider to make it hook itself to the audio output, but it should work.

Of course this has some disadvantages, like needing to keep Audacity open, and a noticeable extra delay on the audio as well. You will have no sound at all if you break any component in the whole chain. This can be pretty annoying if like me, you use your Mac as an alarm clock (it will be trying to wake you with silence which results in you being late for work). Moreover, when you switch back to direct playback through internal speakers, two things require attention. First, your volume will be at 100%, which can lead to unpleasant surprises if you forget to turn it down. Next, when you have switched back to internal speakers in the Sound control panel and plug or unplug something from the headphone jack, OS X will once more switch back to Soundflower output, and you must again select internal speakers.
In other words, not only do you need a whole ritual to set up this workaround, you also need to do a little dance to tear it down. For all these reasons I hope Prosoft manages to find a solution. I wouldn't mind if it would simply implement this workaround under the hood and have some extra delay, if only it is reliable.

This hack does have some extra advantages however, for instance Hear will also work with Safari or any other sandboxed application's audio. And, if at any time you would want to record what's playing, all you have to do is hit the record button in Audacity.

maandag 23 januari 2017

Dark Sky / Forecast.io: avoid using the temperature, humidity, or pressure values of the first hourly block

Here is yet another post about something highly specific that is totally unrelated to everything else on this blog. Enjoy!

If you rely on Forecast.io to obtain weather prediction data, you may also have implemented fancy things like a graph of how the temperature will evolve over the next few hours. Or maybe just a rising/falling trend indicator. This seems very easy to do: just take the “hourly” data block and take the “temperature” values of each entry, and either plot them or compare them. This will work, but you may notice that as the time gets nearer to the next hour change (eg. 09:57), the first point in your graph starts wobbling around wildly every time you refresh the data. Why is this?

A plot made at 16:55, current reported temperature was 0.1°C.

The first hourly data block represents the current ongoing hour. For instance if you retrieve predictions at 09:57, the first block will represent 09:00. One would expect that the folks at Dark Sky would rely on their historical data to fill in the values for that block, given that it represents a moment in the past. For some reason however they merely take the current observations, in this example from 09:57, and extrapolate them into the past using the data for the next predicted hourly block. For instance if it is currently 15.12 degrees and the temperature for 10:00 is predicted to be 15.79 degrees, then the value for 09:00 will be calculated as (15.12-15.79*57/60)/(1-57/60) = 2.39 degrees.

Extrapolation works fine for data that is known to be relatively stable if the extrapolated point is near the interval of known data. In this case only one point is an actual observation though, the other one is a prediction: also an extrapolation in its own right. This makes it doubly dangerous to extrapolate another value out of it, especially if it lies so far away from the two given points.

The morale of the story is: do not rely on the temperature, humidity, or pressure values from the very first hourly data point, unless the hour has only just started. If you want to make a temperature graph, it makes more sense anyway to start the plot at the current time and not at the start of the current hour. Even if at some point the people at Forecast.io update their system to really put historical data in that block, if you would merely dump the values from all blocks in a graph, visitors would still be seeing a curve that starts almost an hour ago when they load your page 09:57, which is not very intuitive.

The same plot as above, but now starting at the actual observation.

After figuring this out, I now start my plot with the current observation and I calculate the trends by comparing the current observation to a linear interpolation at the current time plus one hour. Interpolating also is not ideal but a lot more defendable here than a wild extrapolation.