Today I finished integrating the rate of change calculator in Typica such that you can go into your roaster configuration, set up an indicator, and have that show up when you roast coffee. Well, at least if you’re using Typica built from the development branch, which you aren’t so maybe it’s better to say that I can do that, and I did, and now that I have the feature available, I better understand why people wanted it. This is, in fact, a pretty useful thing to have. There are a couple of rough edges that I need to smooth off before I consider that feature worth releasing, but this is definitely going into the next release build and I want to hurry up and get that finished because this feature needs to become available for more people to use.

Some observations:

A cache time of 5 seconds and a scale time of 30 seconds works pretty well for me. The value changes are nice and smooth and the updates are fast enough to be useful. I should probably make these the default values.

This needs to be persisted with target profiles and put on the graph. I have some thoughts on how I can get that into Typica faster than initially planned.

Other updates:

The first steps in getting rid of the Measurement class have been taken. The current development build is using a compatibility shim, but everything that was once a Measurement is now effectively a QVariantMap. This makes it a lot easier to intercept and manipulate measurements in script code and makes it much easier to do things like replace parts of the UI with things that are written in QML. It also makes it much easier to have measurements carry additional information which makes a ton of potential features a lot easier to implement. This also helps to move forward on addressing some maintainability issues.

As anybody who has asked about the technical details of implementing it themselves in Typica knows, I’ve put a fair amount of thought into how one might implement a rate of change (also known as rate of rise or sloppily as just Δ) indicator in Typica. Today I’m putting that into practice so the feature will be ready to go for the next release of Typica. The code itself is trivial and it follows the same design pattern as half a dozen other objects in Typica. Connect to a source of measurements, when new measurements come in do the required calculations and emit the updated value for the derived series. This is very easy stuff.

There are a couple of issues when it comes to the practical use of this sort of feature. The way this sort of indicator is usually implemented (or the way the value is calculated in the head of the person roasting the coffee in manual logging situations) is you take two measurements that have the desired time separation and do a simple subtraction on the measured values. Graphically it’s the slope of the secant line defined by the points of interest multiplied by a time scale. That last description is interesting because it decouples the two key tuning parameters.

It’s not uncommon to see arrangements with multiple rate of rise indicators set up for different time periods against the same primary data series. That’s because there’s a trade off here. You can get this data with just the most recent two measurements, but if you’re scaling that value out to 30 or 60 seconds, random measurement error will be amplified and you’ll have an indicator that flails about too wildly to be useful. On the other hand, if you consider data points that are a minute apart, you won’t see that sort of fluctuation but there will be substantial delay before a change in the rate of change is fully reflected in the indicated value and that’s also not ideal. Thus we have two key variables here. What time period do we use to calculate this value (cache time in Typica parlance) and how much time is in the denominator of the rate of temperature change value that we display (scale time, since this is really a fraction, change in temperature over change in time)? These will be independently adjustable in Typica so if you want to see the rate of temperature change in terms of degrees per 30 seconds but you want that to be updated based on values from the most recent 5 seconds, you would set the cache time to 5, the scale time to 30, and Typica will calculate the difference between the most recent measurement and a measurement 5 seconds prior and then multiply that value by 6 (this is a slight simplification). Naturally, if you want multiple rate indicators with different tuning parameters Typica will let you have as many as you want, but I’m hoping that allowing these parameters to be set independently will reduce the need for more indicators taking up space and competing for your attention at the roaster.

The alternative to this approach involves some basic calculus but I’m going to see if the fast and stupid way produces something useful before investing time into a more complicated approach.

Yesterday I made a modification to Typica that allows scripts to be informed when certain links in web views were clicked. Today I’ve been messing around with using that feature. Early on I decided that this would be more useful if I could modify the content of the web view without having to regenerate all of it. That meant allowing access to instances of QWebElement from scripts so I added that. The basic use model here is that with any element I want access to later I set its id attribute to some some meaningful value. The link modifying that element should contain the same value. When the link is clicked, the script can just request the desired element and perform whatever modifications are desired.

For example, I rotated the Coffee Purchase Previous Years Comparison report and each row has an id of yxxxx where xxxx is the year. The year itself is an element of the row, but this is now a link with an href of typica://script/yxxxx where xxxx is again replaced with the year. Clicking the link emits a signal that carries this yxxxx value which we can use to query the row element. In this particular test case I added a row (only if that particular row had not previously been added) containing a table detailing all of the green coffee purchase invoices for the given year. The invoice ID is also a link with an href of typica://script/ix where x is the invoice id. Clicking this opens up a new window that allows you to view and edit the invoice just as if you had pulled it up from the invoice list.

I haven’t pushed that example report yet because I don’t think this is exactly what I want to do with that report. I’d like to try a few other things first, see what’s useful. I’ll probably at least commit the rotation without the new links tonight as this fixes an open issue.

Today’s big update to Typica’s development branch replaces QWebView with a new TypicaWebView subclass. This removes all of the script functions that were previously added as properties when accessing a web view from the host environment and makes them proper Q_INVOKABLE class methods that the meta-object system just makes available to scripts automatically. It also adds a new signal that is emitted whenever someone clicks a link that starts:

typica://script/

What that means is that it is now possible for reports to generate links of this form that someone can then click on and the script will be informed of what was clicked (through whatever comes after that in the URL) and perform whatever action is appropriate. That might be updating the view to drill down into more detailed information. It might be opening up a different window. There are a lot of possibilities there. There are some new reports that I would like to write that will take advantage of this and some of the older reports can sensibly be updated to take advantage of this as well.

It comes at the end of a paragraph (literate programming) explaining the issue completely. At issue is type information getting lost at the boundary between languages with different type models.

Qt 5.0 was released today. That might leave some wondering what’s going on with Typica, which uses Qt extensively. For now the answer is, not much at all. Typica will continue to be built against Qt 4.8.x in the near term, but the porting impact is expected to be minimal. There are some known toolchain issues at present that need to get shaken out in a later point release, but if that doesn’t happen Typica could be built with different toolchains. Portions of the framework that have been dropped in the 5.0 release were all things that were known to be depreciated and Typica wasn’t using any previously depreciated stuff. Qt 5.0 does depreciate QtXml which Typica used rather extensively. That’s annoying, but the Typica 2.0 roadmap involves substantial redesign of the portion of Typica that used that so moving away from that is something that has been planned for some time. Probably the best approach at this point is to stick with Qt 4.8 for Typica 1.x and move to Qt 5.x (there will have been another point release by then I’m sure) for Typica 2.0 and later.

The other project that I mentioned is where I’ll be spending most of my development time in the near future so I’ll have quite a bit of experience working with Qt 5 by the time I go back to regularly working on Typica and that experience should also greatly simplify any porting issues.

324 pages of documentation for a 1 line program. Suddenly the 399 pages of source code documentation for Typica doesn’t look nearly so excessive.

Typica version 1.4 is now out. There was a lot that I wanted to get into this release that I just couldn’t get polished to an acceptable point. At some point you have to say, “no, it’s time to release and let people start using what is ready.” That said, I wouldn’t be surprised to see a revision point release in December or at least an updated example configuration. There are currently 8 known bugs but none of them are show stoppers and most people will probably never run into them so while I intend to fix all of them I’m not going to delay the release to do it.

There are several behind the scenes changes that will make future maintenance of the program easier for me and help with future planned features.

The following is a high level non-exhaustive overview of what’s changed since the 1.3.11 release.

Application Icon:

Typica now has an application icon. It looks like this:

Arithmetic Support in Numeric Table Columns:

Tables in which numeric values are expected now allow entering mathematical expressions. I’d like to extend that feature to numeric fields that are not part of a table in the future where it would be even more useful, but there’s a planned feature that makes it potentially useful in the tables, too. (previously reported here)

Automatic Database Setup:

The Setup Database button has been removed from the example configuration. This is now handled automatically.

Bugs Fixed:

Some SQL queries used the same positional placeholder multiple times in a query to hold the same value. I think this worked at one time but it no longer does. The provided example configuration has been audited to remove all such duplicate placeholders (I hope). There were some other examples of SQL statements that were not quite right which have also been corrected.

Some typographical errors in example configurations prevented proper operation. Where noticed, these have been corrected.

A missing initializer in the MeasurementModel class has been corrected leading to faster performance when opening windows containing a table view of roasting data and eliminating the possibility of this causing the program to become unresponsive.

A bug involving periodic unresponsiveness when logging data with certain versions of NI DAQmx on Windows has been fixed.

Data Export:

Exporting roasting data as CSV now only exports the data currently shown in the table view (often one measurement every several seconds instead of multiple measurements per second). To export every measurement as CSV it is necessary to set the view to show all measurements first. In general this means that exported data is much smaller, more regular, and easier to work with.

Derived Measurement Series by Linear Spline Interpolation:

It is now possible to produce a measurement series based on linear spline interpolation using measured values as input. I can’t guarantee that the technique is generally applicable but I’ve been using it to generate a data series at my lab roaster that matches the measurements I would be collecting at the production roaster. This allows me to use half as much coffee to develop new roast profiles and reliably match these profiles at the production roaster.

Expanded Hardware Support:

In addition to the previously supported National Instruments USB 9211, USB 9211A, and USB TC01, it is now possible to use the 9211A with either NI DAQmx Base or NI DAQmx on Windows (on Mac OSX NI DAQmx is unavailable but hardware that is supported through NI DAQmx Base will continue to work). This allows the use of the 9211 module with newer USB carriers which do not work with NI DAQmx Base. Additionally, there is limited support for devices which communicate using the Modbus RTU protocol through a serial port on all platforms. Such devices are available in a number of standard panel mount sizes from several different manufacturers and can generally replace the panel displays provided with the roaster. Note that you will need documentation on any Modbus RTU device you would like to use with Typica indicating the addresses of the data you intend to log for that particular device and instructions on how to check or set the communications settings for your particular device.

Geometry Management:

All windows now save their size and position when closed and restore themselves to the same state when re-opened if possible. Configurations no longer need to handle this explicitly. Some other widgets with size and position state also preserve that information which previously have not.

Historical Profile Viewing:

When viewing a previously roasted batch from the batch log, all of the view transformation options that are available in the logging view are also available here. Saving to a file, exporting to CSV or XHTML+SVG, and printing are also now available in this view. This should improve usability for those who insist on working in Celsius and for those who wish to share roasting data with outside firms.

Internal Device and Logging Window Configuration:

There is now a graphical interface within Typica for configuring the devices and annotation controls for one or more coffee roasters. This has allowed the number of example configurations provided with Typica to be reduced to 1. This configuration window can be used to set up logging windows equivalent to all of the previously provided example configurations and all of the most common customizations that were performed with the previously provided examples including changing the thermocouple type for a given channel and customizing the annotation controls.

Multiple Device Configuration Support:

Through the above and with some other changes behind the scenes, it is now easy to configure Typica to collect measurements from multiple devices simultaneously. Note that it is recommended at least one channel in any given configuration collects data at a rate faster than 1 measurement per second. This is related to one of the aforementioned bugs. You pretty much don’t need to worry about this unless you’re attempting to use a NI 9211 module with all four channels in use and no other devices or if attempting to use a Modbus RTU device on a serial port operating less than 1/10th the slowest rate I’ve seen for such devices.

Profile Translation:

It is possible to configure Typica to perform roast profile translation as seen in this video (but with the column to match and the temperature at which to perform that match configurable).

Promptless Configuration:

The new -c command line argument can be used to specify which configuration you want Typica to use instead of prompting for that when the application is started. This is most useful when specified in a shortcut.

Reports:

Reports no longer need to be referenced explicitly in configurations. Placing a file describing a report into the Reports directory will add that report to the Reports menu. That file will be read every time you instantiate the report so fine tuning new reports can be done much faster as Typica need not be restarted between edits. (previously reported here)

It is now possible to organize reports into a menu hierarchy of arbitrary depth (please try to keep it shallow). (previously reported here)

Some included reports now allow weight and weight derived data to be reported in either pounds or kilograms. Changing that setting in one report will propagate that preference to all reports opened after that which currently support this feature. I intend to extend this to all reports.

Roasting Data:

Table views on roasting data no longer pretend to be editable. The background on this is too long for this post. The short story is that this view never should have been editable. I might revisit this during the 2.0 release series.

Unified Driver Build:

There is no longer a need to be careful in selecting which build of Typica to download. All hardware that is supported on a given platform is now supported in the single build for that platform. All supported hardware can now be used simultaneously.

Upgraded Tooling:

The pre-compiled distribution of Typica on Windows now uses Qt 4.8.3, libpq 9.0.3, and is compiled with g++ 4.6.1. Typica on Mac OS X also uses Qt 4.8.3.

Version Info:

Version info is now displayed in the executable file properties on Windows.

Web Site:

Not a change in Typica per se, but the project web site has an updated design. There’s still some work to do on this (it needs more pictures and documentation updated to the current release).

Today I tackled the issue of multi-device configurations in Typica. Technically there was nothing stopping a person from configuring Typica to collect data from multiple data acquisition devices simultaneously before, but the way certain things were being done made the assumption that every channel would deliver measurements with the same time stamp. This is obviously untrue if different devices are running at different sample rates and even in the case where the devices are all running at the same sample rate, they are managed independently and are unlikely to all have measurements ready in the same millisecond. With the addition of support for devices that communicate with Modbus RTU, the problem might even exist with a single device if data from non-consecutive addresses are requested.

The solution that I decided to go with was to fake it. Basically, we define a current column set in the log that represents all of the channels in the current data acquisition task. Whenever a new measurement comes in at a time later than the previous measurement time, the previous measurement from all other columns is duplicated at the new time. If multiple channels are reporting measurements taken at the same time, the new measurement will overwrite the synthetic measurement. If no data was collected on that channel at that time, there’s a fake measurement there. The sample rates should always be high enough that nobody ever notices this in day to day operations, but this makes showing all of the data associated with an annotation more reliable and allows me to avoid adding extra logic to the log serialization code.

What this means is that if you really want to collect measurements on dozens of different channels across several data acquisition devices that now works reasonably. After I get this next release out I’m thinking of putting together a new video demonstrating how to configure Typica to use some stupidly large number of devices. With device configuration moved to a new graphical interface within the program rather than involving editing a rats nest of XML configuration files it would be a good excuse to show that off as well. While it’s now possible to go beyond the realm of practicality, there are some reasonable use cases for this. Maybe you connected one thermocouple with a NI USB-TC01 (because the hardware is relatively cheap compared with options using the 9211 modules and at the time you only needed one channel) and you want to add a second thermocouple connected to a second TC01 rather than upgrade to a 9211 and have a spare data acquisition device gathering dust. Your mileage may vary, but so far I haven’t been able to push the channel count high enough that there are performance issues with this. Someone trying on recent hardware would run into the problem of not having enough screen space to make sense of all of that data long before hitting channel limits.