Archive for the Programming Category

Posted on Programming

Google docs: formatted for reading

I never like sharing google docs with people—they’re great when you’re editing a document together, but the reading experience could certainly be better. I don’t like the default styles, and even when specifically formatting a document for reading, there’s a lot going on to distract from that experience. And there’s no dark-mode for the web.

In the past, I’ve gone so far as to design PDFs or even webpages for viewing, but keeping two instances of what is essentially the same document up-to-date is a pain. Unfortunately, one of the great joys and terrors of being a programmer is that, in most cases, when you want to, you can just make things yourself.

Which is all to say that I made a Google Doc viewer. Google docs, formatted for viewing. It automatically styles the document—we’re talking typeface, colours, line-height and text indent; titles, headings, sections, and styles for the first line and letter of each section.

It recognizes Google’s headings, titles and section breaks, but it also supports markdown style headings.

# Heading 1
## Heading 2
### Heading 3
#### Heading 4

It recognizes Google’s bold, italics, underline, and strikethrough, and it also supports some markdown styling.

_italic_
*bold*

And it automatically generates a table of contents (when applicable), which it to show your current position in the document.

2. Mercy is bold because that’s the section it’s currently scrolled to.

There are two progress bars along the bottom that reflect how far you are into the current section as well as the document as a whole.

And the best part is that if I change the Google document it automatically comes through to the viewer. I never have to share a raw Google doc again. I can link to read.gigglingcorpse.com instead.

Here’s a sample document—no table of contents though, since there’s only one heading in this document.

Posted on Programming

Alert system details

So…. I designed my own query and templating languages (along with associated editors and complete with syntax highlighting). It was definitely overkill, but it was fun and now I can define alerts any way I like: easy ones, complicated ones, whatever alerts I want. With integrated documentation and helper functions to assist with defining your queries. As I’m sure you can imagine, this post expands on TWIG Irrigation alerts.

Let’s take a look at the interface with an example query:

An alert for when an RTU is low on power

The cursor (which you can’t see) is at the end of the line—right after rtu.mc. On the right, there is a description of the available data objects. RTU is opaque (but its properties are not, though you can see its mc property is highlighted). The editor detects where you are and organizes the objects on the right, highlighting the relevant information. It’s why the entirety of MC is opaque. So if you want to deal with a field particular to an MC, you know what they are and their datatypes. You can either type it in, or click on the field there and it will insert itself.

You can do a lot that way—just by clicking. The buttons above the trigger code block insert the MC IDs; each one represents a farm. The buttons below the trigger block insert the various conditionals you can apply.

And of course there is error checking:

rtu.mc.eventswitch_index expects a number, not a string.
olderthan expects to be used on a datetime, and to be used with a time offset, not a string.
rt isn’t a valid primitive.
fake field isn’t a real property of an RTU.

To learn about the trigger definitions in even more detail, you can click the little i in a circle which will reveal the documentation in-place.

But what about the query that got us here? [rtu.power < 2200] and rtu.mc

The first section is pretty obvious (it will return true if an RTU’s power field is less than 2200) but the rest looks a bit weird. but let’s ignore that for now and get into testing a query.

If you test a query and the query has no results it means that there is no error, and so the alert won’t be triggered yet. That’s great, but it’s not very useful when you’re working on an alert. You won’t see anything in the results tray, so you won’t have any handy examples to show you the shape of the data you’ll be able to reference in the alert itself.

That’s what the square brackets are about. [rtu.power < 2200] means that when testing the query, the system will ignore this condition. So if there are no RTUs with low power, you’ll still see results in the tray. They’ll just have power higher than 2200.

But why and rtu.mc? That has to do with enriching the results.

In the returned data, sub-primitives are only filled in as necessary, but you may find yourself in a situation where you want deeper access. If the query were simply rtu.power < 2200 you’d get back all the fields of the RTU, but the MC object wouldn’t be filled in. Which is fine, unless we want to reference one of the MC’s fields in the alert message. So an easy way to fill it in is just to include it in the query without an associated condition.

This is where you define the alert message and what happens when the alert is triggered.

The message block is where you define what the alert will say when it is triggered. As you can see it allows HTML formatting, and you can insert values from the results. This message will be repeated, one for each result, all in a single message so you’re not bombarded with emails.

You can see that it references the MC’s name (the farm name), as well as the RTU’s power level. But it also includes a property not present in the results: rtu.zone_names.

Various primitives include helper properties (which you can learn more about by clicking on the i in the circle). rtu.zone_names returns the valve names associated with an RTU as a comma-delimited string, which can help you determine which RTU you’re looking for if you haven’t named it. In our case, the valves are often named for specific fields, but the RTUs are not.

The TEST TEMPLATE button will show you how the message will be rendered based on the same results TEST QUERY would generate.

And that’s pretty much it! Except that things can get much more complicated.

You may remember that in my previous post on the topic I mentioned that I analyzed the data and programmatically generated meter alerts for each farm? Let’s take a look at one of those!

The data contains pressure and flow metres, and depending on the field we may expect different value ranges.

Oh, yeah, and the templating language allows conditional inclusion, where the conditions are evaluated using the same syntax as the query language.

Alerts aren’t re-sent until the system detects that they’ve been corrected. This means that if things continue to go wrong you won’t know, but also means that you wont’ have a bunch of cascading messages obfuscating what was most likely the root cause (the first error). That makes sense for something like meter range errors and it’s why I did all the metres for each farm as a single alert. That way you’ll only get one metre error per farm at a time.

It doesn’t make as much sense for errors that are unrelated (like the RTU power alert), but I can address that with an alert setting when and if it becomes an issue. It’s not overly important at the moment; since it can be currently handled by defining individual alerts rather than grouping them together.

Posted on Programming

TWIG Irrigation alerts

As you know, my brother is a farmer (via the professional poker player to farmer pipeline), but he doesn’t deal with just a single farm. No, he deals with many farms. Five, I think; and each one needs watering.

He uses the TWIG irrigation control system from Nelson. It’s great. He can set up irrigation plans and run them. He can even see what’s happening using their app… But only one farm at a time.

He wanted to see them all at once, so he contacted the company. They were very helpful. Travis got involved, suggesting (and vetting) various potential solutions, before mentioning something interesting…that they have an API, which they kindly gave us access to.

So I made it. A dashboard he can keep open on the office computer showing the current state of all things irrigation, across all the farms.

The map view of the farms

These screenshots aren’t the best example as they’re not actually watering right now, but those white boxes turn blue if a field is being irrigated, and they slowly fill as the watering progresses.

The valve view for one of the farms

He can see which valves are open, their signal strength and how their batteries are doing. There’s a chart view as well (when a irrigation plan is running) where he can see how far through the plan it currently is, what’s been watered and what remains to be watered.

Problem solved.

But, since I already had API access, he mentioned something else that would be useful: alerts.

So I built an alert system.

Green means everything is good. They change colour depending on their state.

It’s pretty cool. You can define arbitrary alerts—if a battery is low, if a pressure or flow meter is out of a certain range, if water is flowing when a irrigation plan isn’t running, etc. And since I was collecting all this data anyway, I recorded the meter values over a week or so and then ran an analysis to determine and auto-generate pressure and flow alerts for the different fields, at the farm level.

I might go into the alert system’s mechanics in more detail later, but for now let’s focus on things more interesting to people who aren’t me. Suffice to say that it’s perhaps a bit overpowered.

One of the neat things about these alerts is that, in many cases, information we’ve seen previously can hint at what or where something went wrong, and we can take advantage of that. If a pressure metre reads lower than expected, it’s probable that a valve didn’t close in the previous set of active zones. So it makes sense to check those fields first. And if you know that, maybe you don’t have to drive around to all of the fields checking that none of their valves are stuck open.

So when you get an alert like this—

Heeringa‘s Flow meter looks high (65.1 GPM – it should be between 33 and 53). It’s likely there are too many zones open. Did Zone 11 actually turn off?

—You immediately know you should check out Zone 11.

So, I mean, go check out Zone 11 or whatever. The valve’s probably stuck open.

Posted on Programming

Imposition

Sometimes when all you know is cold plastic against your fingertips, you get curious about other things. Things that aren’t programming. Your hands go clickity-clack against the keys all day and you wonder, is it the same for everyone? Do they suffer the same joys and frustrations I do? How do they plan and what are their processes like?

So you watch some Youtube videos.

And you start to wonder what knowledge they have built up, through years and experience, that they would gloss over when describing what it is they do because it’s just so ingrained. Things you would traditionally pick up piecemeal, and wouldn’t necessarily think of as a novice who doesn’t understand the questions that want asking. Or maybe they don’t even know why they do certain things past the unsatisfying answer tradition.

You find yourself wondering: if you watch enough Youtube videos, will you see the process from enough angles to pick out some of those tripping points experience runs smooth?

These are the sort of puzzles that leave your monitor a tangled nest of queued videos and Google queries, and a small voice in the back of your mind starts to whisper at you: wait, is this just programming again?

For me, most recently, this has resulted in what any reasonable individual would objectively deem as too many book binding videos.

What the heck, 8.5×11″

I thought I was starting to build up a high level understanding of it—at least enough to recognize words like “signature” and “tail”, and the existence of something called paper grain—and then I found a gap I just didn’t understand how people navigated.

The only paper that seems readily available is 8.5×11″. The type of paper an everyday printer uses. There must be other papers, but how do people find them? Moreover, how do they print onto them? Do they do it themselves or have specialized printing companies do it for them? It certainly seemed like many videos I watched ended up constructing books of blank pages. But surely that wasn’t always the case. I mean, I’ve read a book before. Quite famously, those things have words.

In real life, I imagine this is the sort of question you could just ask someone. But real life is a dangerous place, full of sentence fragments, curly brackets, and semicolons. I am a child of the internet.

No, best not to venture there.

Maybe it was all 8.5×11″ printer paper. And if it wasn’t (it almost certainly wasn’t), I imagine it theoretically could be. It’s not like I was actually doing any book binding. Problem solved.

Which, as solutions often to do, led to a new problem of its own. Sure, you could fold an 8.5×11″ along the long edge and end up with a roughly page-shaped rectangle, but as far as I could tell most 8.5×11″ sheets’ grain went along the long edge. Which would result, after the fold, in the grain going along the short edge of of your page.

From my meticulous research (watching some Youtube videos), it seemed that this was far from ideal. Paper it seems, bends easier in the direction perpendicular to the grain (so the grain fibres aren’t being bent). Apparently folding it the opposite way can even crack some cheaper papers, and having the grain in the right direction (vertically) can impact the longevity and feel of the book—this is my possibly incorrect understanding, anyway. If the grain is aligned correctly, the expansion and contraction of the paper and the tendency to bend a certain way don’t work against the integrity of the spine quite as much, and the pages turn more easily since that is the way the paper likes to bend.

So what the heck, 8.5×11″.

The internet suggested to me that one possible solution is buying a ream of 11×17″ and cutting it in half to a short-grained 8.5×11″. But I was intrigued instead by the thought of cutting 8.5×11″ down to 8.5×5.5″ and, after the fold, ending up with pages that are 4.25×5.5″.

This would mean that on each sheet of 8.5×11″ you’d have 8 pages when printed double-sided.

As a potential solution it seemed reasonable enough, and for a moment I thought maybe I could move past the whole thought experiment, but then I started to wonder how one might actually accomplish that… How would you lay out the pages appropriately so everything printed correctly?

It turns out this process is called imposition (I think), and that both InDesign and Acrobat didn’t solve my particular problem. They could do it, sure, just apparently not in strangely specific the layout I wanted.

So I wrote a script for it, and discovered that maybe this was just programming after all.

I can make that

And as things so often do, everything eventually circled back to the ol’ trap of – I can make that. So I made an imposition script in Python which takes a PDF and generates a PDF, and it seemed to work. I also came to realize that I very much did not like Python’s PDF libraries.

At the same time something else occurred to me. Often in videos I would see an angled line drawn across the back of the book block to provide a visual indicator that all the signatures were in the right order. It left a little mark on each signature (possibly why they’re called signatures). But if I was assuming double-sided printing anyway, I could just generate those marks as well and print them along with the text.

So I rewrote the whole thing in PHP, and added that to it.

Maybe this is common practice. I don’t know, I’m not a book binder! Besides, I don’t actually have a printer, so who knows if it works. Maybe it does, maybe it doesn’t. There’s no way to find out, and maybe we’ll never know.

Coincidentally, writing it in PHP made it easy to put a web UI over it, and maybe it’ll be useful to someone.

If you want to give it a try, head on over to https://gigglingcorpse.com/dev/imposing or just scroll a bit further. If you do print, fold the long edge and then the short edge. With a signature size of 2, and rotate the bottom half set to true everything should align like pages in a book. You’ll still have to cut the page in half, but that seems like a problem easy enough to get around.

This was actually from May, 2020, but I never posted it...
Posted on Business

Hourible: the time tracking tool I use

I made a new online tool!

Okay so new is a bit of a stretch. It’s based on an internal tool I’ve used at altered effect for years now, but I’ve cleaned that up and now it’s available for public use:

Time tracking is the horrible. (see what I did there?) But does it have to be? I don’t know.

It’s one of the great mysteries of life, and something I worry we’ll never know the answer to.

Long story long, I wasn’t happy with any of the time tracking solutions I found so I settled on something quick and easy – using Google calendar. I was already comfortable using it, and I it was surprisingly intuitive to just drag events to the appropriate size in order to keep track of how much time I spend on projects.

It actually works wonderfully, but when it comes for invoicing you need to add up the hours.

Or do you? Trick quesion – you do, but Hourible does it for you!

Over whatever time frame you like. Plus, you can easily filter the events. Since I use Logipar any chance I get, Hourible supports logical operators in the filter string. It sums up your hours for you, and you can take a deeper look to see how they’re broken down.

Of course, you still need to make the invoices then. Which you can do manually if you prefer, or Hourible can generate them automatically with the click of a button. Which… is a lot less work for me than making them manually. It’s quite convenient actually.

All that and Hourible is free as you want it to be!

Give it a try if you want, especially if you regularly work on multiple projects or for various clients. Maybe it’ll help.

hourible.com

Posted on Programming

Heaps/Hashlink on Android

I wrote Hexlock using Haxe and Heaps, which went quite well – until I decided to compile it for Android. But it’s there now and we can all relax. Aside from getting Heaps setup to compile on Android, I ran into a few other issues.

  1. Touch event coordinates were wildly inflated

This made it seem like my interaction events weren’t going through at all (because in a sense they were not), but I tracked this back to touch events being multiplied by 10,000. They come in as a value between 0 and 1, and I discovered that they’re multiplied by windowWidth (which makes sense), but then divided by something called TOUCH_SCALE. It turns out I needed to add -D hl_ver=1.12.0 to my compile.hxml file to get it to use the appropriate TOUCH_SCALE.

2. I wanted to trigger vibration on Android

So I did. Click here to if you want to know how: triggering vibration on Android in Heapse/Hashlink.

3. Handling the back button

If you’re playing, the back button should take you back to the level select screen. If you’re at the level select screen, it should do what the back button does on Androids and minimize the app. And now it does! If you want to know how, check out dealing with the Android back button for Heaps/Hashlink.

4. hxd.Save didn’t work out-of-the-box

At this point I was too tired (lazy) to dig into why, so I just implemented my own method of saving using Android shared preferences. It seems to work! If you’d like to learn how, click here: saving data for Android with Heaps/Hashlink.

Again, I am not saying these are the best, correct, or even good ways to handle those issues. They’re just what I did, and they worked for me.

Posted on Programming

Heaps/Hashlink saving data for Android

When I was making Hexlock, I was disappointed but not overly surprised to find that hxd.Save didn’t work out-of-the-box on Android. It’s quite possible that it would have after some simple changes, but at that point I was too tired to dig through source files to try to discover why and what those changes may be.

So I just implemented my own method of saving using Android shared preferences.

The first step was to load and store data differently when compiling for Android, which wasn’t too bad because I already had my own wrapper to handle that. A lot of this will contain code specific to Hexlock, which you can ignore or not at your discretion. Basically I keep a map of levels (JSON strings) to score data, load it once when the game starts, and save the entire map out on a new high score.

public static function loadSaves():Void {
	var defValue = new Map<String, Highscore>(); // The default value
	#if android
		var bytes = Main.load(getInstance().name); // These are hl.Bytes
		var data = @:privateAccess String.fromUTF8(bytes); // Convert it to a String

		// This is 100% overkill but a wise man once said:
		// It is better to just throw in random possible values
		// than to wait for Android Studio to finish compiling and check 
		// what they actually are.
		if (data.length == 0 || data == "null" || data == null) { 
			// Welp, at least we have the default value
			getInstance().data = defValue;
			return;
		}

		// This code is basically taken from hxd.Save
		var obj : Dynamic = haxe.Unserializer.run(data);

		// set all fields that were not set to default value (auto upgrade)
		if( defValue != null && Reflect.isObject(obj) && Reflect.isObject(defValue) ) {
			for( f in Reflect.fields(defValue) ) {
				if( Reflect.hasField(obj, f) ) continue;
				Reflect.setField(obj, f, Reflect.field(defValue,f));
			}
		}
            
		getInstance().data = obj; // That's our data now            
	#else
		// If it's not for Android, just use hxd.Save
		getInstance().data = Save.load(defValue, getInstance().name);
	#end
}
public static function save(level:Level, highschore:Highscore) {
	getInstance().data.set(Json.stringify(level), highscore); // Add the score
	#if android
		var data = haxe.Serializer.run(getInstance().data);
		Main.save(getInstance().name, data);
	#else
		Save.save(getInstance().data, getInstance().name);
	#end
}

For both save() and loadSaves() hxd.Save is used, except when compiling to Android, because I added -D android to my compile-to-c.hxml. There may already be a hlNative one #shrug. I didn’t see one, and adding -D android was easy enough.

You can see from above that the code requires the function Main.save(String, String) and Main.load(String); getInstance().name is just a constant string identifier like ‘my_cool_highscore_file’.

Since those functions are only referenced when compiling for Android, they only need to be defined when compiling for Android, which is convenient because hl.Bytes isn’t available on non-hashlink targets.

And in Main.hx:

#if android
@:hlNative("Java_io_heaps_android_HeapsActivity")
public static function save(name:String, data:String) {}
#end

#if android
@:hlNative("Java_io_heaps_android_HeapsActivity")
public static function load(name:String):hl.Bytes { var i = 4; return hl.Bytes.fromValue("null", i); }
#end

Who knows if I needed a body for load(String) – not me – but it worked, so I didn’t look into it further.

When compiling to C code for Hashlink, calls to the function Main.save() get replaced with Java_io_heaps_android_HeapsActivity_save(), which I defined in my jni.c1 file, and of course an equivalent transformation happens to Main.load().

JNIEXPORT vbyte* JNICALL Java_io_heaps_android_HeapsActivity_load(vstring *name) {
	(*jvm)->AttachCurrentThread(jvm, &thisEnv, 0); //this is important to avoid threading errors
	jclass cls = (*thisEnv)->FindClass(thisEnv, "io/heaps/android/HeapsActivity");
	jmethodID method = (*thisEnv)->GetStaticMethodID(thisEnv, cls, "loadData", "(Ljava/lang/String;)Ljava/lang/String;");

	char *cname = hl_to_utf8(name->bytes);
	__android_log_print(ANDROID_LOG_DEBUG, "JNI.c", "loading... %s", cname);

	// Is this necessary? Who knows! Better safe than seg fault though.
	char *buf = strdup(cname);
	jstring jstrBuf = (*thisEnv)->NewStringUTF(thisEnv, buf);

	jstring result = (*thisEnv)->NewStringUTF(thisEnv, "null");
	if (method > 0)
		result = (jstring) (*thisEnv)->CallStaticObjectMethod(thisEnv, cls, method,jstrBuf);
    
	char *cresult =   (*thisEnv)->GetStringUTFChars(thisEnv, result, 0);
	__android_log_print(ANDROID_LOG_DEBUG, "JNI.c", "returned... %s",cresult);


	free(buf); // Free the string
	return cresult;
}

JNIEXPORT jstring JNICALL Java_io_heaps_android_HeapsActivity_save(vstring *name, vstring *data) {
	(*jvm)->AttachCurrentThread(jvm, &thisEnv, 0); //this is important to avoid threading errors
	jclass cls = (*thisEnv)->FindClass(thisEnv, "io/heaps/android/HeapsActivity");
	jmethodID method = (*thisEnv)->GetStaticMethodID(thisEnv, cls, "saveData", "(Ljava/lang/String;Ljava/lang/String;)V");

	char *cname = hl_to_utf8(name->bytes);
	char *cdata = hl_to_utf8(data->bytes);

	char *bname = strdup(cname);
	char *bdata = strdup(cdata);

	__android_log_print(ANDROID_LOG_DEBUG, "JNI.c", "saving %s: %s", bname, bdata);

	jstring jname = (*thisEnv)->NewStringUTF(thisEnv, bname);
	jstring jdata = (*thisEnv)->NewStringUTF(thisEnv, bdata);

	if (method > 0)
		(*thisEnv)->CallStaticVoidMethod(thisEnv, cls, method, jname, jdata);

	free(bname);
	free(bdata);
}

To find out more about where thisEnv and jvm come from, check out triggering vibration on Android from Heaps/Hashlink.

Haxe strings come through Hashlink as vstrings, and the JNI uses jstrings, so you can see me converting between the two, above. Onload I just send back the char* instead of dealing with transforming it back into a vstring, and handle that in Haxe (which you can see even further above). Is this a memory leak? Who knows! Not me. But if it is, oh well – the load function is only called once during the apps lifetime.

As you can see, the C code assumes there will be saveData() and loadData() functions in io.heaps.android.HeapsActivity, and there will be, because we’re about to add them!

static public void saveData(String name, String data) {
	SharedPreferences sharedPref = getContext().getSharedPreferences(name, Context.MODE_PRIVATE);
	SharedPreferences.Editor editor = sharedPref.edit();
	editor.putString(name, data);
	editor.apply();
}

static public String loadData(String name) {
	SharedPreferences sharedPref = getContext().getSharedPreferences(name, Context.MODE_PRIVATE);
	return sharedPref.getString(name, "null");
}

Above you can see that I just use name for both the SharedPreferences file and for the key in it. Whoops, that’s embarrassing. I could have used another string defined in the java, or just omitted specifying a name completely but it works and that’s good enough for me.

If you want to find out more about the files I’m referencing (including a Git repo hosting them), you can do so here: Hello World; Heaps on Android.