2022 in review: IoT, ray tracing and web

Another year has come to pass, and it’s time for some more reflection! Despite its many flaws, the (only? :s) one way this year has been a successful one is through all the code I’ve managed to write. So without delving more into all the melancholy, let’s jump right in on what really matters in life and think of all the great projects I’ve had:

I started the year with embedded development in C#, then did two intensive months of game development - starting by learning Vulkan and ray tracing from scratch, learned Kubernetes and Terraform, a bit of cloud, worked on my homepage, and then did a bit more of embedded stuff by writing some programs to monitor and control factories in Minecraft.

Embedded .NET with Meadow

For the first half of the year I got back to embedded development and managed to drastically improve the UX and reliability of setting short timers. This is all thanks to the Meadow platform having reached a much better maturity compared to how it was the year or two before, when I first started working on it. I even wrote a post about the adventures I had when designing and implementing that system. There’s also a short Youtube clip illustrating how it’s all used in my daily life. At that point I was feeling really good having “shipped” something useful, and moved on to the next topic. Before this (for many many years) I had used the same launcher, but the countdowns were opened to a browser window.

Plus, over time I’ve gotten really used to the system, and thought that I couldn’t ever live without it. This assumption was however challenged a month or two ago, when I upgraded the firmware on the device running the display hardware, instead of the other device I was supposed to develop serial port things on. But what the hell. Let’s just upgrade it to the new OS and get all the promised stability and performance upgrades while at it. Maybe those would even eliminate the very rare restarts when a watchdog timer set by me times out.

One of the big new features of that new release was the inclusion of a new linker, which is able to trim out unused code, greatly reducing both the code size and the startup speed. Unfortunately, this specific linker isn’t implemented the same way as the one by Microsoft in the full .NET SDK, and there seems to be no working developer-facing options to alter its behaviour yet - it's all beta. The way it’s currently implemented strips out code needed by reflection-based serialization, leaving me stranded with my API calls.

One could think that I could always go back to the old OS, but the matter has two obstacles. The first being the tooling, as it is still under active development, and there’s no apparent way to install a previous version. Though I highly doubt it’s been blocked yet, and there’s a responsive Slack channel to ask about these things. But the second one is quite embarrassing. The last time I worked on the codebase, it was when I moved the server side stuff to my Kubernetes server (more on that later), and it turned out to be a more involved operation than originally foreseen.

When I finally got it to work, I was rather frustrated, and seemed to have forgotten to make the final commit - but there’s really no excuse for this kind of irresponsible behaviour. So after I had done the OS update and the code updates to go with it, I no longer had a working version to go back to. Rather than taking the time to fix things, I’ll just have to live with my mistakes until the new OS is at a serviceable state. At least I’ve had plenty of things to keep me busy and not thinking about it. Quite the theme overall…

Vulkan and ray tracing

The timing for my next expedition couldn’t have been better. After I had neatly finished the work on the timer (but before the OS update), there was a bit of time to relax. How unheard of. When it was time for my summer holiday, I was already anxious to try something new. For a long time I’d been meaning to learn Vulkan, and now I also had the spirit and the time for it. And not just Vulkan, if things would progress well.

I think it took me about a week to go through most of the tutorial. I had originally anticipated it to take the whole month, so my spirit was further elevated by that triumph! As detailed in the relevant post, one of my long-lived dreams has been to build a game with realtime soft shadows. I’ve always thought the math for it to be beyond my abilities - except now. Armed with Vulkan and powerful hardware, perhaps I could just brute force my way to it.

And it all started rather promisingly. I learned more than I could have hoped for, and even got good results on the screen, at interactive frame rates.But the quality fell short of what I’d hoped for. And with that, so fell my motivation to work on it. Despite this, I still managed to achieve some rather great things after the initial blog post, namely multiple importance sampling (MIS) and the integration of Nvidia Realtime Denoisers (NRD). Both improving the quality and performance by at least an order of magnitude or two. Almost enough. Almost.

It sucks that I haven’t documented my journey better than I did, as this is the most impressive thing I’ve achieved in a long time. There’s some nice screenshots on my Twitter page, and some short compression-butchered video segments on my Youtube page, but that’s it.

But the reality of it is that the intensive ~two months I worked on the project really burned me out for a long while, despite all the impressive things I achieved. I really enjoyed what I did, but everything has its limits. Though truth be told, RT is perhaps something that I don’t really wish to pursue more. It was very rewarding, but as a concept it’s not really something that interests me that much - I was really only after the results. I thought that simple 2D shadows would be easy to implement with the new technology, but the truth is that they are actually even more complex than 3D ones, at least when implemented the way I approached them. Sunken costs kept me engaged.

But when it comes to using Vulkan as a graphics API, I really did enjoy it a lot. It’s so much nicer than OpenGL, as it’s without all the technical debt of immediate mode rendering and old hardware pipelines, while explicitly supporting all the new things that make development easier and the code more performant, such as pre-compiled shaders and cached pipelines, and multithreaded API design. Though it’s still not easy to use it optimally by any means - especially when it comes to supporting heterogeneous hardware. I already liked OpenGL, and I like Vulkan even more.

A few years back I made yet another version of my unicorn game project - codenamed Unnamed Spacefaring Game: Energized (with modern OpenGL and C#) - and I thought that the code I wrote for it was diamond-grade. Not quite. Or maybe it was, compared to how the previous iterations were implemented on top of a decade+ old Python codebase. I also had big plans for writing the engine side in such a way that it would be able to run efficiently on many-core systems. This wasn’t easy, and it was further complicated by the APIs offered by OpenGL. The solution I arrived on was ambitious, and I feel really bad I never got far enough to really test it in battle.

Vulkan on the other hand has native support for many frames-in-flight and multiple descriptor sets. These features could be used to drastically simplify the resulting architecture, perhaps even completely eliminating the need for all those ambitious inventions I had. I’m very conflicted on how to proceed. Everything sounds almost too simple. Though I guess there’s still some demand for building infrastructure for better multithreading, but the big picture has warped so much that the old plans don’t apply anymore, and I’d have to spend more time planning against that new future to be able to say anything conclusive.

But what I do know is that the code I wrote for that RT Unnamed Building Game was a lot better than that of USGE. I’m really hoping to get back to UBG someday, as its scope is also a lot smaller than USGE’s; if I’d be able to keep the idea alive, yet yeet the RT things, it would be something that could realistically be shipped, and perhaps sold. Though, without RT most of the novelty and graphical appeal wears off…

I guess it’s all about managing scope; it would also be possible to ship a version of USGE with only a few months of development. It would be a technology demo, but a fully playable game nonetheless. But there’s been enough gamedev for now.

Homelab and custom auth

As told above, I had used up (almost every waking moment, I might add) of my summer holiday for learning Vulkan and RT, and there was still a lot to do. Meanwhile at work the then-current project had evaporated over the holiday, and for a moment I was left without one. This gave me the energy to continue learning, but still I faced that eventual burnout. This “opportunity” meant that I could pivot my focus at both work and at my own time to learning about Kubernetes, and forget about all the unconqueable things in ray tracing.

Plus, yet another of my long-time dreams had been to have a solid infrastructure for running self-hosted software - both of my own making, and by other people. And doing it securely. So I didn’t really mind the literal double-timing of learning new things so soon again.

And a lot of new things I learned. It took a bit of iteration, but now I have a somewhat satisfactory hierarchy of Kubernetes manifests, and a small-yet-equally-satisfactory collection of services described by those manifests. All hosted on an ebay’d tinyminimicro-sized server I bought just for this purpose. The server's awesome: it's tiny, x64, and consumes about 6 watts at idle. Pity it has only 8 GB of RAM, and doesn't have the PCI-E connector for a 10+ Gbit network card soldered in. Also, I’d still like to do the manifests over one more time - this time using Terraform for avoiding some repetition on specific parts of the infrastructure. A bit more on that in the next section.

I had hoped to create a VLOG episode about all things homelab, but that didn’t happen this year. But what I instead have is a two-part special about one facet of it - namely authentication and authorization. I’ve materialized a lot of my dreams this year, and the theme continues :)

I set up a monitoring stack with Postgres, Grafana and Prometheus, and migrated some of my own server applications to the new hardware (countdowns, hometab, …). I also set up an instance of changedetection.io. All of these things benefit from unified access control in the form of single sign-on (SSO). As illustrated in the first part of the VLOG, I had initially set up Authelia for this task, but it was not to my satisfaction.

Moonforge

So I did what I do best, and made my own. While this is a bold move even by my standards, I kept things realistic by having Keycloak do the actual user authentication, and having my service continue from that. It’s called Moonforge. All it does is minting and validating JWTs based on configurable policy. There’s more information about this in the second part of the VLOG.

But in short, Authelia felt really nasty because it had a single login cookie for everything. With Moonforge, the user logs in to Keycloak, and then to Moonforge via OIDC, and afterwards to the actual applications with OIDC-like semantics, even if the applications don’t natively support it, thanks to special reverse proxying routes spanning some paths even in each application’s own domain. The JWT cookies for these logins are application-specific, and can’t be used to access other applications unless the user specifically allows it on a case-by-case basis.

I really think that having my own service for auth is for the best - as long as I haven't made a mistake in something elementary. As further detailed in the VLOG, the alternatives are just not that appealing. Authelia requires me to fully trust the applications it protects, and standalone Keycloak is too complex to configure: I’m bound to make a mistake. If I build my own, I know for sure how to configure it. And since I’ve been doing backend stuff for a long time, I hope for everyone’s sake that I’ve learned to build a simple one that is reasonably secure. After all, the harder parts are still done by Keycloak - for now. And I’ve already spent enough time looking for ready-made solutions.

Besides, having an auth(n|z) solution of my own has been yet another long-time aspiration. I think I did some initial UI planning as far back as 2015, and it took until now to implement a system. It looks nothing like originally envisioned, but that is partly due to how the focus so far has been on authz instead of authn. Plus, I haven’t really decided whether I want to let it have its own visual identity, or whether to have a more standardized appearance around my core css.

While the visual identity is a definite nice-to-have, the actual technology behind the scenes is what it’s really all about. In its current form it replaces Authelia with something more secure, without offering much more than that. The foundation I’ve built everything on feels really solid and well grounded. The cookie paths and domain are sensible, and there’s been effort in building everything with a focus on security. There’s extra care in JWT validation, and things like embedded keys and alternative signature algorithms have been explicitly disabled - strange how some JWT libraries don’t even have the option to disable these features. And the small scope has definitely helped.

As I’ve mentioned many times now, for a project of this scope, having Keycloak be part of it is sensible. But I’m hoping to find the time to replace it with my own implementation. When I was still working on Tracker, I implemented a login flow similar to how for example WhatsApp Web works now. That type of passwordless flow was really nice, and I’ve wanted to replicate it in a more generic setting, while also adding some other security features. Hence the dream. In the VLOG I talk about all the things I’d like to add and how they’d strengthen the security of all the services relying on the system - pushing more and more of the auth functionality on to better trusted companion devices, outside the device and the server the login is being performed on. There's also plans to harden the surface which mints the JWTs by requiring input from a trusted device. I’d also like to get hands-on experience interfacing with security keys such as Yubikeys.

Moonforge is far from being feature-complete, but it’s perfectly serviceable for its current task. I feel good, and with that 'final part of the 1st stage' of my Kubernetes infrastructure implemented, I now have what I wanted: a stable foundation for hosting and building self-hosted services and applications. Next up is Terraform, better uptime monitoring, perhaps distributed logging and most importantly improved machine-to-machine authentication. And maybe even making more VLOGs on the topic.

Terraform, cloud and static HTML generation

With the things at home looking so good now, why stop there? The next thing I could be improving is my (still perfectly functioning) decade+ old homepage. The current site is hosted on a single server, and the PHP in it dynamically serves WikiCreole-based content. I’ve been meaning to replace it for the past two years with static generation and geo-replicated hosting, but the time hasn’t been ripe. Except now*.

There’s been demand at work to learn more about the cloud, and I was able to use this to my advantage. I was required to set up a static website on AWS S3 and CloudFront, so why not use what I learned for the new site? And while at it, why not do it properly. This is again a topic that would have made a great VLOG episode or a post of its own, but I’ll try to be brief.

At work I was given Terraform to accomplish the task, and I quickly fell in love with it. It allows for writing all the infrastructure as “code”, a bit similiar to how Kubernetes manifests work; precise and to some degree self-documenting. And making changes is easy, as the user doesn’t have to know the exact commands, they only need to tell how they want the end result to look like. It’s a really nice way to work.

The best advanced feature of Terraform are the “modules”, which allow for defining complex reusable components, which in turn expand to a set of simpler pieces of infrastructure. While that’s an overkill for just a single static website, I’m looking forward to migrating all(?) my Kubernetes manifests to Terraform. While raw manifests are mostly fine, the biggest pain point for me has been in defining firewall rules and HTTP(S) ingresses. These are very verbose, and contain a lot of dupliction between each other. With Terraform I should be able to write a single custom utu_http_ingress module, which defines all those at once. There’s also some special additions that need to be done to each Traefik (the reverse proxy) HTTP ingress route entry in order to support the special login paths needed by Moonforge - having the module transparently do this in a centrally defined location is awesome.

With the basic infra up, the more fun part was uploading the content. I could have used some type of shell script and aws-cli to upload the site content, but it turned out to be really easy to use the SDK and build my own program. With it I could easily incorporate some extra steps, such as only uploading the files that have been changed / renamed between each run, and more importantly transforming the uploaded filenames and paths. The static site generator includes the .html file extension on all content, and I was trivially able to remove that. There was also some special logic on how “index” pages and paths worked with the PHP site, and I was able to take that into account, too.

And lastly, I learned that CloudFront is able to compress the content it serves, but this feature depends on the file types and the available CPU time on each edge server. The only way I can ensure that the content is always served compressed is to save it pre-compressed to S3, and then return Content-Encoding header with the correct value for each request. But now that I control the upload process, this was no problem :)

And speaking of content, that’s a topic for another post, again. A few years back I stumbled on statiq, and found it to be the least-sucking tool in the .NET world, so that’s what I chose and started to slowly port over the site itself, and the content. Instead of WikiCreole and PHP I had Markdown and C# HTML compiled down to plain HTML.

With this new technology stack I could also accomplish some of my long-time desires for ever so slightly improving content-discoverability on the site. Along with the old stuff, the site also serves as the companion for my VLOG, containing the scripts, some technical specs and other commentary on each episode. I wanted to make it easier to move between different episodes and projects, so I implemented a sidebar navigator for this.

Site generation in statiq is a multi-stage process, with one of the stages being content discovery and “front-matter” parsing. This way each page knows what other pages exist, and I’m able to pull similar pages from the metadata to the sidebar - either by the folder structure, or by a special grouping key. This metadata is also used to build navigation breadcrumbs.

*The tool still sucks, though. It tries so hard, but fails. There’s a live-reload server for avoiding full rebuilds when working on individual pages and templates, and a caching system so that those full rebuilds can be avoided also when publishing. Unfortunately, both have been broken for several years. The live reload server needs to be restarted whenever a page containing an error is opened (so don’t save a live-reload page until the code compiles!), and the caching system doesn’t purge the metadata cache, so pages appear multiple times in it, breaking the sidebar.

I was going to sidestep all that by making a hot-reload system of my own, by building on the “hot” reload technology I invented for USGE and UBG and UBG's asset pipeline. Newer dotnet supports unloadable assemblies once again, so it’s possible to keep the compiler and dependencies loaded in memory, but still dynamically reload all the user code - a sub-second operation. That way I could delete the cache between rounds, and avoid the server dying whenever an error occurs, by simply very quickly restarting it after each run instead. Unfortunately statiq always starts a text-based debug console meant for live-reload even when not using it, and never cleans it when exiting, thereby failing to run again in the same process. The relatively complex code which does some high-level orchestration of the generator is rather tightly coupled with the code controlling the debug console, so it’s not an easy task to separate these two. Even cleaning up afterwards is not trivial. FML.

Despite this, the preview for the new site is live, and the DX is a bit better than before. And it’ll get better, won’t it? And if not, I’ll just have to make my own generator. There’s still work to be done in upgrading imagepaste, perhaps the contact form, and transferring over some other static content. I’d also like to improve the page load latency, so I’ll have to find a way to cache the CSS file, yet reliably invalidate it without rebuilding the whole static site. That might be impossible to do without scripting. But for now, at least the primary content has been migrated succesfully.

Minecraft and Lua

It’s been a tight year, and I wanted to take my mind off other things, so I found myself gravitating towards modded Minecraft once again. Plus, it’s just so damn addictive it’s hard to keep away for too long.

This time I’ve been playing Create: Above and Beyond, a non-expert modpack with strong progression. The focus is on Create, a “low-tech” automation mod, which has large moving machinery which in my opinion fits the theme of Mincraft really well. As the technology level keeps increasing over the playthrough, some products benefit from automating them with programmable in-game computers. It’s also fun to monitor some production chains, such as the amount of charcoal fuel.

Few years ago I already built some libraries for saving metrics to Postgres (with plans to migrate to TimescaleDB), so this time I was able to build an in-game Lua program, a metrics gateway server and a Grafana dashboard in two short hours :) This was a good primer for the more complex factory-controlling programs.

While the modpack is heavily focused on doing things with just Create, some things would get needlessly complicated when done with just in-world redstone circuits. Just like I really enjoyed describing infrastructure with Terraform, I also enjoyed avoiding complex circuits with Lua code. It’s a lot more self-documenting, and the code itself is creeper-proof :p Some might argue that the beauty of the game is using those simpler primitives for achieving the end goal, but at some point it just becomes plain frustrating. Though in this case I’m not very good with Lua, and I’m not even trying to be: I know the bare minimum, and use that to make the programs. I guess that’s my way of playing the same game.

And as it was beautifully noted to me, I guess I’m in the right industry when even relaxing and playing video games leads me back to writing code. Or alternatively, I’m just a sick and troubled mind, which can’t ever truly let go and just relax (:

Closing words

Whoa. It seems I’ve been really active this year. Maybe too much so. But, for now my only real regret is that I haven’t documented my doings even better. This post ended up a lot longer than I first anticipated trying to cover even the majority of what I’ve done. Especially the gamedev things. But at least I have something to show for it, and I did at least try with Twitter.

Thanks for reading! If you are - for some strange reason - after for even more to read, can I recommend you the 15 year anniversary post of the blog. It’s a lot shorter, I promise!

PS. I’m also working on a little something to highlight all the wonderful private code commits of mine, and also my other online doings in a central place. Stay tuned.

VLOG season 2, sad progress report

As it turns out, I really did do it all over again. Including the part where I don’t really publish anything even though I really wanted to.

So far, I’ve scripted, filmed, edited and uploaded three whole episodes for the second season of the VLOG. The first episode is about the whys of the season, second more about general background about me, with the third episode finally talking more about the game project this season is to be about.

I’ve been quite satisfied with where the videos have been heading, but that’s also the problem. They are not there, yet. They are mostly just fluff, without proper content that would be useful in a broader sense.

What I would have liked to show is me building cool things, while making insightful commentary. To make matters worse, I have built cool things and thought a lot of insightful thoughts. But I’ve done it off-camera.

First: I don’t know how to efficiently present things afterwards. All those previous episodes required writing a script – a task which takes a considerable amount of time to reach the quality I’m satisfied with, and then some extra time to edit it together. I am rather awkward when I try to present things unprepared, plus it makes the editing process much more tedious.

Second: I haven’t been able to find the right mind for doing things live. Most of the time I struggle to find the energy to focus and do things. I’d rather just code in short bursts and then recharge. Or at least have the ability to do so. Sure, I may have pulled “all-dayers” all through last week coding IoT things. But if I set up the camera, I feel pressured to produce value, and be energetic in commenting the stuff I do, all the while also looking presentable. That’s a lot of extra to ask, and I just haven’t been able to do it. But I’d really like to. I even tried to make the live coding editing easier by introducing a recording pedal (tested with my recent Minecraft videos), but it’s of no use when I can’t even start due to the reasons above. (Plus, the project is so very overwhelmingly ambitious…)

So now instead of coding and providing quality content, I don’t produce anything, and feel really bad when coding things off-camera. It was supposed to be the polar opposite!

I’m sad and depressed, and I don’t know how to proceed.

VLOG season 2

OMG. I did it! …again?

I’m not sure if I’ve mentioned it here before, but I have a VLOG in finnish. A few years ago, I shot and edited eight episodes of me talking about what I’ve been doing, or what I’ve been cooking. I also made one special episode about some low-level serverless technology alternatives with C# and dynamic code compilation and execution, including syntax tree editing. I didn’t dare to publish any of these, but they exist.

Now I’ve began the second season, with more focus on technology. Perhaps gamedev. And this time it might just be good enough for public release! The format itself is still subject to evolution, but the two episodes I’ve so far completed serve as an introduction to the series and the reasons behind its existence.

Though, truth be told the first episode isn’t that good, and made me hesitate on the whole thing. Ultimately, I decided that I’d shoot an episode or two more, and if they are good enough, they could perhaps redeem the farce that is the first episode. I think it would be quite bad if the only episode available was the first one, and it was unbearable to watch. But if there’d also be some better episodes immediately available, the viewer could perhaps skip the first one and, and decide to like the series based on those later episodes :3

I’m now at the point where I have one better episode ready. Although even that starts a bit weary. But it gets better! Content-wise I’m still debating. So far, the series has been only about me, and isn’t really useful for anyone ­– unless they just want to get to know me better. That would be perfectly fine if I had a fanbase, but the case is completely the opposite, so I’m not sure why I’m bothering with this. But, as said, at least the episodes still have the purpose of laying the foundations for the episodes to follow, should someone want to invest (more of) their time in all this right now, or at a later date.

I’m also still not sure of the best way to present the auxiliary information about each episode. Not that anyone would really care. First of all, I have a short description of each video in YouTube’s video description field. That is fine. But I also have some ‘technical’ notes about the video there just in case; perhaps to deter some obvious commenters. Much of these notes are also duplicated on my own website, but not all. And vice-versa the site contains some notes not on the video description. I’d like to unify these somehow. I’d like to have as much information on my own site as possible, yet I feel like there should also be some on the video description for those obvious cases. But maintaining these two in sync is a pain, and they have each have their own purposes :( So what do?

But, anyway. Here’s the page for the new season. It goes a bit more in depth into the production of individual episodes. There's also the near-complete script available for each video if you just want a quick overview of the stuff. Then there's also those video links. Videos themselves are still unlisted, but the links are there D:

About pride and accomplishment in optional multiplayer games

(This is effectively a rant about how I am incompatible with MMORPGs)

As Destiny 2 has been feeling very stale for a long time, I’ve shifted my gaze to other games. There was a rather long burst of Borderlands 3, and then a bit of Roboquest, and a longer phase on Gunfire Reborn. And all the time I’ve had a tiny longing towards Guild Wars 2. A longing that has been growing in such a way that now I can’t wait to play it. I’m also very happy that they just announced a lot of details about an upcoming expansion, including the release date. What a coincidence. Although the release is about six months away still; plenty of time to get bored, and I kinda already am. Allow me to explain:

Destiny 2, Guild Wars 2 and Borderlands 3 all have a mountain on content in them. And they are great games, with great gameplay. Sounds great, right? That a lot of content I’ve been really enjoying, taken time to get good at, and/or maxed out on. I’m on the very peak of (almost) everything. But it is not as simple as this. Things are (almost) too easy, and there is little challenge left or rewards to earn which I can do on my own. Which brings us to the following:

My time is limited.

Outside of expansions(!), a lot of content in D2 and in GW2 is just replaying old content. In D2 it is the age-old formula of bounties and the season pass, and in GW2 the latest one is the quest for a legendary amulet. These offer nothing new to the game, and just direct playing the old content again and again for some reward. I’m all for replayability, but these literally offer nothing new, or change the experience in any way.

And actually, D2 makes things even worse. It’s a loot-shooter. But the bounties require to use your less good loot. And that’s basically the content. Or well, some bounties just tell you to do X three times. And then repeat that YYY times. And if you don’t complete those other bounties while at it, you are basically throwing away almost all “progress” and ability to better “enjoy” further content.

And in GW2’s case, the new questline requires to replay both the story-content and some open-world aspects of the past several years. While this is a good opportunity for the player to spot if there’s any foreshadowing in the story, that’s about all the value there is. No skips for lengthy dialogues, and nothing to change the experience. Just a mountain of playing it all again. And the fact that I’ve already played it once doesn’t net me anything.

Then why play? Like I already mentioned with D2, if that work would be completed, it would (even greatly) enhance the ability to enjoy the new expansions, and the other repeating content. But in D2’s case the bounties are so ingrained in the game nowadays that even the expansions are filled with bounties that punish using the weapons and subclasses you enjoy.

And with GW2 (especially after the very recent legendary armory feature), a legendary piece of equipment is the literal best-in-slot that replaces everything that would ever go in that slot. It has the same stats as the otherwise best stats containing Ascended-rarity items, but it allows for free and unlimited stat swapping. After that you don't need anything else on that slot ever again. In a game like Build Wars 2, that’s the hot shit, and highly desireable. You'd be mad not to pursue that.

It’s all about the economy and playtime — and psychology

In a boring game, wouldn’t it be nice to be able to switch playstyle at will, and for free? Or in case of looter-shooters, wouldn’t it be nice to be able to sometimes enjoy our hard-earned loot, and get new loot?

I’d enjoy those things, but things just aren’t meant to be. In D2 that means being broke and longing for new fun and interesting ways to play, just brand-new content out of reach. And as it just happens, in GW2 that also means being broke and longing for new fun and interesting ways to play, with brand-new content just out of reach. Even when the games and the reward structures are completely different. The essence of all this seems to revolve around accessibility, skill, balance, long-term investment and perceived value, and efficiency. It’s quite complicated, but I’ll try to render out my own experience in relation to this:

In “short”, a lot of the content in these two games is balanced for good equipment and depending on content, almost no skill. Some content on the other hand might require near-literal godlike skill and/or a lot of time — or just a larger amount of less able players.

In D2 the open-world sandbox enemies are frail, and they die from about anything. But it’s also fun to mow down large amounts of red bars, even though there could be even more of them. But to get new ways of destruction, or any kind of real challenge, the content to play changes. There’s the adjustable-difficulty 3-player nightfall strikes or 6-player raids, and also the 3-player dungeons to explore. Strikes are the only piece of content that has matchmaking, and even that stops right as the actually challenging difficulties start. All non-matchmaking content is balanced in such a way that a lone solo player has little chance to really even begin playing them, let alone finish them (dungeons and lost sectors being the exception).

And the game makes this exceptionally hard for so-called hardcore-casuals (which I like to call myself). Every few months a new season begins, and rises an arbitrary “power cap” on equipment. It also raises the power level required on all content to match. Effectively undoing any investment towards difficult content. Soloing content like dungeons or master-tier lost sectors is something the game’s creators reserve for the sweatiest players – those with time to grind the game and increase that arbitrary power level to a sufficient value in order to match the level of the enemies. But I don’t have that kind of time. So even if I was as skilled as them, I just can’t play the same content as them, as I haven’t done the ever-elusive the numbers game beforehand.

With GW2 this changes slightly. Lot of the solo content open-world content does have challenge, but sooner or later it starts to essentially feel like the infinite variety of oatmeal. Different, but the same. The game tries perhaps combat this by being a theme-park MMO. Every playable area is vastly different than others, and as such the world feels disconnected. But then there’s some things that can’t be soloed. And everything gets very easy with more players.

In all these cases, the rewards stay the same. More players, more easy, a lot more rewards in the same time span. But at least with these rewards it would be possible to change the way the game is played in order to keep the experience fresh. Is there really no good way in the middle?

In D2 I could play with the equipment I already own and like, but would eventually grow tired. Or I could try the challenging content, and not really get anywhere. With the most fun weapons gated in that content. In GW2 I can either keep soloing challenging content and miss out on a lot of rewards. I could still purchase a limited number of new ascended-tier gear with new stats, but would eventually go broke. Or I could purchase less-able and a lot cheaper exotic-tier equipment, but I’d only be making the game intentionally a lot harder, while also missing out on even more rewards, further limiting my ability to change things up and stay in a nice position in the game.

In GW2 the most long-term cost-conscious choice would be to craft a full set of legendary weapons, armor and trinkets. Then I could just enjoy playing with what I want. But the amount of work is legendary. Just to get the gated materials for one armor weight class (out of 3), it would take an estimated 500-1000 hours of constant gameplay via WvW over 24 weeks. More if there are gaps on some weeks. Alternatively, via PvP the gated materials for the in about 280-330 hours over 6-24 months (but still a good number of hours every two months, or else things take a lot longer). Then there’s also the weapons and trinkets, and the normal materials for all these. And that is not cheap. But then again, legendaries are the be-all end-all of equipment. Equipment-wise there's nothing left after acquiring them.

WvW is just grind when solo, but PvP can be really engaging. But then it, too, eventually turns to rewards and tryharding, and starts to feel like a chore. Just like everything else. And if only I had better, more predictable teammates.

Then there’s the (5 out of 6, already have one) legendary trinket and their quests. I have no estimate on how long they take; PvE ring and accessory have similarly lengthy quests as the amulet I spoke of earlier. Second ring and accessory are PvP and WvW only, and take time comparable to multiple armor pieces. And then the weapons, which are thankfully mostly just about money, but still have a lot of gated stuff. But the weapons are perhaps the most irrelevant of these, and I already have few of them.

Let’s finally talk about multiplayer

Nearly all these problems are solvable. There’s so much more content gated in and behind raids (in either game), or fractals, or dungeons, or even WvW. Simply play them with a group for the intended experience. A lot of perfectly balanced challenge, and great rewards. Just like all things should be.

But that is the problem. It all requires a group. Not only is my time limited, but my social energy is exceptionally limited. Luckily things are easier with people I know; and I really used to enjoy doing guild content in Guild Wars 2. Unluckily the schedules and expectations eventually just took a toll on me. I just couldn’t find the social, mental or even physical energy (due to sleep problems) to always be there for the group, and fell out. People missed me, for a while. Then life went on, and getting back became hard. Then even later many people stopped playing, or found new groups, and there was nothing left.

Now I’d have to find a whole new group, and find the constant energy for it. Or alternatively I could look and fight really hard outside of the game, and eventually land in less-organized pick-up groups for a single instance of some content. But to make that happen, I’d already be expected to be master of that very content. And be expected to talk, fluently. If I can’t do that, I can’t ever even begin enjoy any of that gated story content, challenge or rewards. I really like the games, but would like them even more if I could play them they way I want, and all the content. This is not just the fear of missing out. This is missing out.

In the end I’m like Sisyphus. Forever doomed to meagre repeating work with pride and accomplishment in sight, but always just out of reach.

Top things to pursue

My long-time readers might know or guess that I struggle with anxiety about wanting to do too many things, and that I always try stay productive even when I should relax. I was recently prompted to make a ranked list of 20 things I’d like to pursue, and forget everything except the top 3. I shall now combine these concepts: I’ll make the list, but won’t forget a thing. And as everything doesn’t always have to be perfect, it is not ranked. At least to the absolute final degree. Kek. Also, true to myself the list is a mixed combination of ‘work’ and ‘leisure’. Because leisure is still a serious business, and can’t be taken lightly.

So anyway, in a surprisingly small amount of time I came up with this list, which I’ll just leave here. I feel that something important might still be missing, but this is what I came up with. And as nothing is ever truly complete, I might augment this one later. I'll try to leave a note.

• Game development
• Articulation and verbal skills via/and VLOGs
• Gaming
• Expanding social life
• Embedded programming
• Home automation
• Television and movies
• ‘Home’-server, high-availability computing, serverless and modern web infra
• a) in the cloud
• b) self-hosted
• Getting really good at cooking
• Transition in fashion
• DAW-centric music
• Skill-based sports
• Travel
• Long- and short-range radio communication, both data and voice
• Photography and videography
• Demoscene music and synchronized visuals, also on a stage; performance coding
• Writing
• Playing tabletop RPGs
• Designing my dream home together with professionals

Dreaming about Ampere

Hello again,

quite a while since the last update. Again. But at this point you should know me well enough to expect it. Or maybe you just happened to read the previous post. Go figure.

Anyway.. One lesser thing I've been pondering is how me upgrading to an ultrawide display might have played a part in me not finding as much joy from gaming as I used to. Sure, this a minor thing considering all the other factors, but a factor nonetheless. Ar at least the lack of the best possible GPU is a good excuse to not play games. Irregardless of that, I've been in the process of upgrading my GPU for a while now, and I was extremely happy with how good of a product Nvidia managed to launch with Ampere!

"A great generational leap in performance", good pricing, and even the Founder's Edition models seem very good. Typically FEs themselves have been a bit lacking, but now it seems that they might actually be the better product! I'll still of course have to wait for the reviews, but I don't remember being this exicted about hardware in quite a while. It's also one of the very few things I've really been excited about the whole year.

As that's not all. I recently got myself the Valve Index VR kit. The waiting list for it was so long that I'd almost forgotten about it. Now, I've mostly been using it so get some exercise in form of Beat Saber, but I also did play through Arizona Sunshine. AS has some flaws, but the gunplay itself was rather nice. I also started Half-Life Alyx a while ago elsewhere, but haven't really gotten to play it more. This is unfortunate. Instead, I've been busy playing Minecraft, and just simply keeping busy.. This is hopefully maybe changing, but we'll see. Anyway. What saddens me a bit is how increasing the render resolution scale in AS made the game a lot more clearer, but completely killed the FPS. Maybe the new GPU will chance that! And most certainly I'll get that sweet 144 Hz on Destiny 2 again, too!

Hmh. It seems that "whiles" are the staple of my timeframes :p And while this post feels a bit lackluster, it feels good to produce some content once in a while. Maybe I'll even get motivated to write some development-themed post at some point, too. We can all hope, at least :) We'll see, we'll see...

And I know what you are thinking. About everything. We'll see.

Plans

Everything I planned for. Crumbling before my very eyes. And I’m not even talking about the pandemic.

I kinda thought that now that I got my studies finished, I’d get a chance to focus on what is important and make time to do things. But that didn’t quite plan out, as there surfaced some things that’ll be requiring my attention. And not just for a little while, but for several years :( And some things that directly contradict everything.

Though it (mostly, not completely) depends on whether I think about those things or not (: It would be wise to give some thought to one of them, but thinking about it doesn’t really help it at the moment… Was this vague enough? :s

But, as they say: it is important to try to keep sense of normality in a time of crisis. Remains to be seen how well that'll pan out.

I graduated

1.5 months ago. I guess it should feel like something? Clearly not, as I've been in no hurry to blog about it in more depth. And it only took 8.5 years :d

That is also probably the reason. I've been working almost full-time for about 4.5 years now, and my studies were already somewhat complete even when starting to work. During that time they just progressed slowly and sluggishly, but relatively surely. The two larger challenges were passing swedish and then of course the thesis. Everything has just been one large marathon, with the goal of just reaching the finish line.

And now I did. Sure, it's an acomplishment, but is it really anything that special? The goal was always there and reaching it didn't really come as a surprise; there was no great and overwhelming feeling reaching it. The race just ended, and there was little more to it than that.

* * *

But actually, there is a bit more. In order to get to that let's talk a bit about the thesis itself. Like my Bachellor's, Master's was about something that I was researching at the time. For many years I've been interested in backend development. Now everything's finally giving so much return of interest functionality-wise that it's worth to invest in deployment. What good is it to be able to write backends with only little effort when making them available and easily accessible is uncharted territory.

So for my thesis I wanted to take a look into the process that went to creating backends in a way that made them easier to deploy. I didn't want to chomp on a piece too big, so I purposefully didn't seek an automated deployment pipeline. I knew better than that. But the real friends are the ones we made along the way, so buckle up ;)

I had previously written a simple restaurant menu parser for my Nokia e51 in PHP. When the page was requested, the server fetched the menus from preset restaurants and rendered a slimmed down HTML-only page with just the menu contents. This was very advantageous compared to opening the site(s) of the restaurants manually and waiting for needless data transfer over a slow mobile connection, and then the slow rendering of complex content and scripts on the low-power mobile device.

Anyway. When the parser was in need of an upgrade I ported it to this new thing called .NET Core and hosted it on a now-retired iteration of usva. Then at one point I also dockerized it and made it run on Azure. This was done because that was the only application running on usva, and now I could do other things with the server (for example turn it off :p). But anyway. That was the first step on the journey of improved hosting.

Several years passed till the thesis. I needed an application to use as the base for it, and after making a list of possibilities the menu parser was kinda the only reasonable choice. Other alternatives would have been far too work-intensive in their implemenation itself. I needed a tried concept so that I could focus on just researching deployment and hosting (and common design things). But of course I couldn't not upgrade the application, so for the thesis I took the menu parser logic of the old system, and wrote a whole new site, including a history database for the foods and even some user-specific stuff as an example. I also added an admin API and some other features such as centralized structured logging.

While the academic goal was to present some design guidelines for containerized applications, the pratical goal was to create the basic building blocks that could be later used for more involved DevOps concepts, namely build and deployment automation. At the start I was not sure how these would end up looking, or even if I'd ever get to see them.

But ultimately I ended up with something I can be quite satisfied with! I could use the Dockerfile to automate building the application, and then further use Docker compose to define the runtime: namely the exposed ports, configuration and the whole set of services (application, updater-parser, database and log storage). What still remains to be improved is how the database is used, as there remained some manual steps for database migrations. But for everything else the process was automated quite far.

The goal was to simplify hosting and deployment, and it really did get a lot simpler. Just one command to build the app, another to copy it over to the target server, and one final command to update the running instances. But what is even more imporant is that it is the same three commands no matter the app. This opens the way to efficiently build automation in the future by replacing just those commands. But it also makes the manual work easier.

* * *

The work done on the thesis is just a part of a larger continuum about backend development. While the thesis (and graduation in general) is a great step, the work never ends. And I'm already busy with the "next" steps - as indicated by the surprisingly many blog posts as of late. There is always so much more work to be done to get better. This same train of thought extends to my other hobbies, too. In the shadow of the continuum, there just doesn't seem to be any time to feel the pride and acomplishment of the past.

My own private Nuget feed (featuring packages for simpler web APIs): the brave new world might really be a thing?

Wow, what an unexpectedly productive end-of-the-year this is becoming! A blog post, again! Let’s just hope that this is the new normal, and not just something preceding a less-than-stellar episode.

* * *

Like I’ve begun to outline in the earlier posts, I’ve been rethinking my “digital strategy”. It is a pretty big thing as a whole, and I’m not sure if I even can sufficiently outline all the aspects both leading to it and executing it. But I shall continue on trying to do it. This time I’m focusing on sharing code. Jump to the final section if you want to skip the history lesson and get to the matter at hand.

As some of you might know, I started programming at a very early time, relatively speaking. I was self-taught, and that time PHP was starting to be the hot stuff, whereas JavaScript was just barely supported by widely-used browsers. No-one even imagined DevOps because they were too busy implementing SQL-injection holes to their server-side applications which they manually FTP’d over, or developed directly in production, also known as “the server”. And I guess I was one of those guys.

Then I found Python and game programming. And as everyone should know, there really isn’t that much code sharing in game programming. At least if you are trying to explore things and produce things as fast as possible without too much planning (not that this doesn't to apply to everything else, too). Python was also good for writing some smaller one-off utilities, snippets and scripts for god-knows-what. It could also be adopted to replacing some PHP-scripts. But then I found that the concept of application servers exists, opening up the way for a completely new breed of web applications (with some persistance without a database, also EventStream and WebSockets). There was so much to explore, again.

I was satisfied with Python for a very long time. Then I bumped to Unity3D, where the preferred language is C#. A language much faster than Python, at parts due to strong typing. I wasn’t immediately a fan: it took a few years to affect my other programming projects. After that I finally acknowledged all these benefits, and for about 3-4 years now, I’ve been trying to (and eventually succeeding to) shift from writing all my programs, scripts and snippets in Python, to writing them in C#. This was also catalyzed by that fact that I worked on a multi-year C# project at work. And now I can’t live without strong typing and all the safety it brings. And auto-completion! The largest, and almost only, obstacle was related to the ease of writing smaller programs and scripts. With Python I could just create a new text file, write a few lines, and execute it. Whereas with C# this meant starting up Visual Studio, creating a whole new project, writing the code, compiling it, finding the executable, and then finally executing it. The effort just wasn’t justified for snippets and small programs. But for something larger, like reinventing Tracker once again, or for extract-transform-load tools, it began to make sense. (And games of course, but at this time they didn’t really happen anymore, as I was too focused on server-side.)

Then this all changed almost overnight when I found LinqPad. It provided a platform to write and execute fragments of C# code, without the need to even write them to disk (a bit like with Python’s REPL). This obviously obliterated the obstacle (khihi) I was having, and opened a way to start writing even those tiniest of snippets in C#, while also allowing me to use the “My Extensions” feature of LinqPad together with the default-imported packages to share some code between all these programs. And of course I could just individually reference a DLL in each script.

* * *

Parallel to this is also maybe the fact that I was now starting to get so experienced, that I had already “tried everything” and what was left was to enjoy all this knowledge, and apply it to solving problems :o

With Python it never really got to a point where I “had” to have larger amounts of shared code or needed to package it some way. There were few lesser instances of it, but I was able to solve it in other ways. But now with C# things have continued to evolve, and the need for sharing code between projects, programs and snippets is becoming a reality.

For why this is only happening now, there were multiple aspects at play. As I said, almost everything was always bleeding-edge, and there was no need to version things. In part this was due to the fact that there really wasn’t any actively maintained code actively running. Everything was constant rediscovery, and as such there really wasn’t any code to share. Rather, everything was constantly forked and improved upon without thinking about reuse too much. Good for velocity. Also, as programming really happened on only one computer, code could be shared by just linking other files. Code running elsewhere was either relatively rare, or in maintenance-only mode.

With age and experience this is now changing. Or, well… That, but also the fact that I’ve gotten more interested in smaller and more approachable online-services - services which have the ability to create immediate value. A bit like in the old times! In order to make those services dependable, a more disciplined approach is needed. While Tracker will always have a special place in my heart, there seems to be better things to spend my time on. At the same time another strong reason is that I find myself creating and executing code on multiple computers and platforms, making filesystem-based approaches unsustainable.

* * *

All this has finally pushed things to a point where I should really be looking into building some kind of infrastructure to share code between all these instances. For a time, I leveraged on the ability of LinqPad to reference custom DLLs and was quite happy building per-project utilities with it. But now my development efforts seem to be focused all over one smaller slice of the spectrum, with multiple opportunities to unify code. This is especially important considering the fact that I’m also looking to make it more viable to both create and execute code on multiple environments, be it my laptop, work computer or a server. In my master’s thesis researched on using Docker Compose to simplify running code on servers. As indicated in the earlier blog post, I’m now trying to continue on that path and make that task even simpler by utilizing serverless computing. With both Docker and serverless it becomes somewhat impossible to easily and reliably consume private dependencies from just the filesystem.

Instead, these dependencies should be in some kind of centralized semi-public location. And a private Nuget-repository is just the thing! As outlined in these new blog posts, my idea is to build some kind of small, easily consumable private ecosystem of core features and libraries so that I can easily reference these shared features and not copy-paste code and files all over the place. I started the work of building this library a tiny tiny while ago, but quickly decided that I should go in all the way. I’m not saying it will be easy, but it should certainly be enlightening.

The first set of projects to make it to the repository are those related to building web APIs. While I’ve on many occasions planned on using tools like gRPC, sometimes something simpler just is the way to go. I’m not sure how thoroughly I’ve shared my hatred for REST, but I like something even simpler if going that direction. This set of projects is just that. A common way to call HTTP APIs the way I want. Ultimately this comes down to one single thing: wrapping application exceptions behind a single HTTP status code and handling them from there. How else am I supposed to know if I’m calling the wrong fcking endpoint, or if the specific entity doesn’t happen to exist??! Although, as we all know, writing HTTP libraries is a path to suffering. But maybe one with a narrow focus can succeed?

Anyway, maybe my impostor syndrome gets something out of this. Or not. It seems that everyone else is releasing public packages and open source code. But I am not. But maybe, just maybe, this will eventually change that thing.

* * *

The task of actually setting up a Nuget repository ended up being easier than I’d like to think. As the first reaction I thought about using some kind of self-hosted server for it, as I had had much success with setting up servers for other pieces of infrastructure lately. Unfortunately, the official Nuget.Server-package one was a bit strange, and I wasn’t sure if it even ran without IIS. I also didn’t want to spend too much time searching for alternative open-source implementations. So, I decided to try the cloud this once! Microsoft has a free 2 GB version of Azure Artifacts for hosting not only Nuget-packages, but also packages for Node.js and Python. I decided to test it, and then migrate to a self-hosted one should I need to save on hosting costs in the future.

As I wanted a private Nuget feed, I first created a private project (or team?), and then set up Personal Access Tokens so that I could automatically fetch and push packages with my credentials. To actually use the feed with the Nuget command-line tool, I ended up adjusting my local per-user NuGet.Config file. I think that I also had to first install the Azure Artifacts credential prodiver using a PowerShell script. But anyway. This way I should now be able to use the packages in any new project, without having to explicitly reference the repository and access tokens: nuget sources add -Name DeaAzure -Source "https://pkgs.dev.azure.com/_snip_/index.json" -Username anything -Password PAT_HERE. I had to manually add to feed to LinqPad, but that was just as easy: F4 -> Add NuGet… -> Settings.

For actually creating the packages I just used Google and followed the first (or so) result. This means I wrote a nuspec-files for the target projects, built them manually and then used nuget pack and nuget push. I also wrote a script automating this. But, after I later tried doing the same with a newer version of the nuget CLI I got some warnings. It seems that it is now possible to specify the package details directly via csproj-files. Apparently this should also simplify things a bit: now I manually add the generated DLL to the package, the newer way might do it automatically. It did feel a bit strange to manually how to know how to include the DLLs to the correct directory it the nupkg. I’ll try to blog about the updated version of this process in more detail later (edit: here). That is, after I get to implementing it.

While I was relatively hopeful about this whole thing above (though the last paragraph kinda changed that trend), I’m actually not sure what kind of things I’ll have to do when I want to use the feed with dockerized builds. I might have to bundle the config to each repository. Remains to be seen. It will also be interesting to see how my workflow adapts to both using, but also for releasing these new Nuget-packages. How about debugging and debug symbols? Will there be build automation and tests? How is versioning and version compatibility handled? Automatic updates and update notifications? Telemetry? For a moment everything seemed like it was too easy, I’m glad it didn’t actually end that way! /s

Edit: it seems that at least the debug symbols of my own packages work directly with LINQPad when using Azure Artifacts. For some reason I did have to manually remove the older versions of some of my packages from LINQPad's cache from the disk in order to use the symbols I included in the newer packages of mine (the older ones didn't include symbols, yet the newer one had a different version number). But after that they worked :)

An update: a brave new world is coming?

Hmm. Starting to write this post is surprisingly hard. Despite the fact that I’ve already written one blog post earlier this year. Well, anyway… Now that I’m finally graduating (more about that on a later post!11), I’ve had some extra time to spend on productive things and re-evaluate my focus. And what an exciting turn of events has it been!

I’ve already spent one weekend on briefly investigating the state of serverless computing. This netted me an instance of Node-Red, but also MQTT, NATS, MongoDB and Redis (even though I already had one) to toy around a bit. All these are running on top of Docker Compose on a new, previously under-utilized incarnation of Usva. Docker Compose is actually a somewhat new tool for me. I’ve meant to look into it (or Kubernetes) for quite a while, but never got around to it. That is, until I made it a part of my thesis. Now this research already paid back – setting up all those services using Compose was a breeze!

Testing out Node-Red actually kicked off a larger process. It demonstrated that utilizing a range of ready-made tools isn’t actually all that bad and that there actually seems to be a lot of tools now that suit my way of doing things. And well, now that I’ve somewhat reinvented all of the software involved at least once, I can finally get to actually using existing ones properly :p I'll have to postpone Tracker once again while I discover this new brave world.

Node-Red offers most of all a nice platform to host short pieces of code. There is a web UI for coding and testing, and after that the code remains on the instance and keeps on executing. No need to worry about setting up routing or cronjobs or any of that! There are also some plugins for using stuff like databases and message queues with minimal work. While Node-Red doesn’t seem very scalable on anything remotely complicated (see a price tracker below), it did manage to verify the concept of serverless for me. And tinkering with it was actually quite fun and easy!

* * *

While a bit unrelated, as a part of this I also began experimenting on using events and message queues for communication on a larger scale than before. This shouldn’t come as a surprise though; in one iteration of Tracker I already began heading this way. And this kind of indirect messaging actually also plays a role with IoT devices – directly addressing them might not be possible due to how they have been networked. But why is this important? For two somewhat equally important reasons. The first is that for the longest time I’ve been in great pain due to how hard service discovery can be. But if services subscribe to queues instead of needing direct addressing, service discovery gets almost completely eliminated.

And second, it is because IoT is relevant now! I backed Meadow F7 on Kickstarter during the spring; it is a full .NET Standard 2.0 compatible embedded platform in Adafruit Feather form-factor, with a possible battery life of about up to two years. C# and low power in one small package!!! It even has integrated Wi-Fi and Bluetooth. I should be getting it in just a few weeks. Finally! But I guess more on that on a later post.

So, what does this all mean? With service discovery being taken care of by a message broker, and hosting solved by serverless and docker-compose (at least in theory), there comes an unexpected ability to realistically develop microservices and other lighter pieces of code. And possessing some easily programmable embedded devices opens up a way for a whole new level of interactions. Though first I have to replace Node-Red with a proper serverless solution… Well, not have to, but...

I already have plenty of development ideas, but I guess they too will have to wait for another post!

Trying something new

Let's try something new. Or actually, something old.

What I mean by this, is that it is once again time to do some self-reflection. But this time I'd like to do it in finnish, as that is my native language and the language of most of my thought process. It should thereby be a lot easier to unload my thoughts with great precision as there is no translation overhead nor thinking in two languages simultaneously. Although, as I might have previously mentioned, another language helps to look at the thing more objectively. Anyway, it is low(er) effort, just what I need now. And there is the potential for feeling good of the end result.

This type of deeper reflection poses a great question, though. Is this something I dare to publish? Things are eternal in the internet; no backsies. And yet: once published, don't I just hope this gets read?

This was written at 2 am. Try to enjoy.

* * *

Minä olen ihan helvetin kyllästynyt siihen, että en nykyään jaksa oikein tehdä mitään. Öisin ei uni tule ja päivällä väsyttä. Mieli olisi kuitenkin täynnä kaikkea siistiä ja hienoa mitä haluttaisi tehdä, mutta kaikki jää kuitenkin toteuttamatta. Joko mielenkiinto vain menee saman tien kun yrittää toteuttaa, tai sitten ei vaan fyysisesti jaksa.

Oli kyse sitten TV:n katsomisesta, tarinallisten pelien pelaamisesta, liikunnasta tai luovasta toiminnasta; lopputulos on sama. Ei mitään. Parempiakin päiviä toki on, mutta ne tuntuvat olevan aina vain harvinaisempia ja harvinaisempia. Olen kärsinyt tästä ongelmasta nyt varmaankin kymmenkunta vuotta, mutta vasta viime vuosina se on edennyt sen verran pahaksi, että havahduin sen oikeasti huomaamaan. Ja hakemaan apua. Ikävä kyllä tämä on sellaisia asioita, joissa mitään pikaratkaisua ei ole - ainakaan pidemmällä aikavälillä.

Tai sit olen vaan ihan helvetin laiska.

Kirjoitan tästä asiasta, koska kyllä näistäkin asioista pitäisi voida puhua. Edes jollekin. Toisaalta yritän tässä hieman taas vaihtaa lähestymistapaa ja saada edes jotain aikaan. Minun on nimittäin pitänyt jo pitkään saada käyntiin - kaiken muun lisäksi - oma varsin vapaamuotoinen vlogi eli videopäiväkirja. Mutta enpä ole siihenkään pystynyt, ja se harmittaa. Taustoja vlogin perustamishalulle on monia. Erinomainen aihe vaikkapa aloitusjaksoksi, siis. Mutta ei jaksa. Melkoinen muna-kana-ongelma. Tämä teksti yrittää osaltaan olla eräänlainen kiertotie asialle.

* * *

Vlogi, miksi? Syitä en osaa asettaa tärkeysjärjestykseenkään, mutta tässä tälläinen puoliksi auki pureskeltu pohdinta:

Innostuin pari vuotta sitten "travel film" tyyppisestä Youtube-sisällöstä. Verrattain lyhyitä kiiltokuvamaisia tarinoita matkailusta. Kantavana teemana ennen kaikkea matkan hauskat kokemukset ja se mitä tuntemuksia matka herätti videon tekijässä siinä hetkessä. Näille videoille ehkä hieman omainen carpe diem -asenne oli minulle jollain tavalla pysäyttävä. Kuten aimmin Life is Strange -taustaisesti avauduin, hetkestä nauttiminen kun on ollut minulle aina varsin haastavaa. Ehkä voisin jotenkin kyetä emuloimaan tuota tunnetta jos tekisin itsekin vastaavia videoita? Lopulta ehkä oppiakin nauttimaan?

Toisaalta videoiden tekeminen olisi varsin erinomainen tilaisuus harjoitella esiintymistä, tai vielä tärkeämmin ennen kaikkea itsensä sanallista ja eleellistä ilmaisua. Ne, jotka minut tuntevat ovat luultavasti huomanneet sen, kuinka sekään ei ole minulle kovin helppoa. (Kommentoikaa toki, vaikka anonyymisti. Nettisivuilta löytyy yhteydenottolomake, joka ainakin luultavasti toimii. Tai jos on muutakin.) Kun kameralle koittaisi selittää fiiliksiään, niin niitä olisi pakko miettiä ja muodostaa kaikesta tapahtunesta mielipide, parhaassa tapauksessa kesken tapahtuman. Tämän jälkeen tapahtumaa voisi ihan eri tavalla havainnoidakin ja oikeasti yrittää uppotua siihen siinä hetkessä. Kokonaisvaltaisempi kokeminen rikastuttaa elämää ja tuottaa iloa ihan eri tavalla. Tai näin on ainakin hypoteesi.

Videoiden tekeminen on toki myös aivan oma maailmansa. Se on jotain erilaista kuin se ohjelmoinnin ja videopelien maailma missä yleensä kuljen. Mutta kuitenkin jollain tavalla tuttua. Tekeminen on toki laaja termi. Siihen kuuluu ainakin editointia, kuvaamista ja jopa suunnittelua. Kaikki näistä asioita, joissa voisi myös kehittyä ja oppia paljon uutta. Editointi on luovaa tekemistä, joskin ikävän työlästä. Kuvaaminen puolestaan - ainakin kuten itse näen asian - on erityisen hieno asia. Sillä kuten selostaminen, niin myös kuvaaminen pakottaa tarkastelemaan senhetkistä tilannetta eri kulmista (sanan kaikissa merkityksissä :p) ja ehkä arvostamaan tapahtuvaa. Jopa verrattain yksinkertaisetkin asiat voivat tuottaa suurta iloa kun niihin keskittyy. Viimeiseksi mainitsin suunnitelmallisuuden. Siinä minulla olisi kehitettävää, joskin hieman eri tavalla kuin ensiksi arvaa. En ole millään tavalla spontaani ja olen mukavuusalueellani kun saan miettiä tulevaa rauhassa. Mietin kaikkea kuitenkin varsin käytännönläheisesti, enkä tunteella. Kuinka ottaa se mukaan suunnitteluun? Ajatella, millaisia tuntemuksia voisin asioista kokea. Kun asetelmaa vie vielä pidemmälle se kääntyy, ja ollaan yllättävän syvien kysymysten äärellä. Mitä tehdä, jotta tuntisin haluamiani tunteita. Mistä oikeasti pidän?  Mikä minut tekee onnelliseksi?

Vlogaaminen on siis ehkä paitsi uusi laaja harrastusmahdollisuus, niin myös suurta potentiaalia omaava työkalu. Mutta se on myös muuta. Uskallan sanoa, että eräs ihmisen perustarve on tulla huomatuksi. Tämä blogi - ja jatkossa ehkä myös vlogi - ovat minun tapani huutaa tyhjyyteen, että minä olen olemassa. Minulla on myös vahva usko siitä, että voisin oikeasti tehdä laadukasta sisältöä, josta on ihmisille hyötyä. Joko viihteenä, tai asiasisältönä. Mutta olenko valmis tekemään työtä sen eteen, että pääsen sinne asti? Ja mikä riittää? Ehkä on vain parempi olla miettimättä asiaa.

* * *

Korostettakoon tässä suuren avautumisen vanavedessä nimittäin vielä tuota aiempaa lyhyttä tekstiä täydellisyyden tavoittelusta. Se on aivan helvetin vaikeaa, mutta jostain syystä siihen vain silti pyrkii. Ja kun sitä ei vaan voi saavuttaa, ja on niin kovin helppo menettää motivaatio. Oli kyse sitten ohjelmoinnista tai videoiden tekemisestä, tai jostain muusta. Miksi yrittää jotain, kun tietää, että sen voisi tehdä aina vielä vähän paremmin. Miksi julkaista mitään työn alle saatua, kun tiedossa olisi vielä loputon kasa parannuksia. Miten tyytyä hyvään? Tai jopa keskinkertaiseen? Miten uskotella itselleen, että keskinkertainenkin voi kelvata? Miten hyväksyä se, että aina on kriitikoita?

Kuinka heittäytyä ja antaa ei-täydellisille, mutta realistisille ja aidoille tuotoksilleen mahdollisuus onnistua sellaisina kuin ne ovat? Kuinka tuntea ylpeyttä niistä ja niiden tekemisestä? Mikä on lopulta arvokasta?

About Tracker, and being perfect

How hard can it be to just make a simple service to track a phone? Not that hard. Unless you want to be a perfectionist and overengineer the whole thing, several times over. And that is why I never seem to get anywhere close to finishing Tracker.

In the last five or so years I've learned a lot about web services and would of course like to apply ALL that knowledge somewhere. A hobby project seems like the perfect place!

• Code generation and custom tooling: avoiding repeative tasks by writing ten times as much code to maintain. And then there is this one thing that was not in the original specification and you'd basicly have to rewrite the whole thing.
• High availability and fault tolerance: considerable additional design and implementation complexity. Good luck getting anything ready when working on the project just some occasional weekends.
• Metrics: a whole universe in itself to set up, integrate and maintain.
• Devops, service orchestration and distributed configuration management: yet another whole universe, with yet more design and implementation complexity. As if you even get to this point. Sure you can devops your stub services, but that if anything is wasted time.
• Distributed tracing: better reinvent the wheel again! That way it works just the way I want.
• Extremely high performance: each small part is tuned for the best achiveable performance before integrating it to the larger system, and never finishing anything.

And of course everything needs to be P E R F E C T!

What good is a if it doesn't function as well as theoretically possible? This intra-process message queue can only pass 300 000 messages per second?? It should be able to do at least 500 000!! Better spend a week trying to optimize it before moving on to the next thing! A decryption routine that can only decrypt 20 000 packets per second? Needs to be able to do at least twenty times as much! Better to just give up or spend ages looking for a faster implementation.

How about this config file? Why it is on the file system? How to make it seamlessly work with multiple environments with different configs. What if those configs change dynamically? Better get to work already, there is much to do!

Hey, see that single point of failure? You'd better engineer it away! What failure point? I need to first map the whole system and get realtime metrics from every single thing it does, or doesn't.

And hey, see that middleware solution for solving that one problem you have? Better not use it and write my own. That way I learn so much more about the problem domain and can leave out unnecessary parts for blazing fast performance.

I'd like to have public test server for this? I wonder how I can set that up automatically. And better make sure that it does database migrations and all that automatically.

And did you know that you could run your code inside a custom operating system to save on context switching time with networking?

Oh, and there is this web application, too. What if I used modern JavaScript this time? And then the mobile app needs to be rewritten, too.

And what about...!

/dies

Mission accomplished?

Oh god. I’ve been more busy than ever. I guess I really got a life like I was wondering the last time. Or I just suck with time management; have I really done anything? So… Let’s review the year despite there still being plenty of it still left and see if we can answer the question in the title.

First of all, as the previous post might have made you guess, I kinda lost the motivation to develop the Tracker project further, at least for now. Again. There is just so much work to be done, and I’m just one person. And with all the things I’m going to list below I just didn’t have the time, either. I really hope that I can get back to it at a later time and better focus. This was also a conscious decision in part. The project did, and would have taken so much time, and I just had to prioritize. I also mentioned how it is a potential topic for the Master’s thesis. But before I can dive into that, I must finish my Bachelor’s. And that means passing a course in that disgusting language that is not to be named. But I’ve really tried to learn it! Been doing Duolingo almost every day since April. In addition I’ve almost completed one other online course on it as suggested by a teacher. I really really really hope that I can pass the course by the end of this year. I should have all the motivation I need, but somehow that still isn’t quite enough :(

I’ve also tried to be a bit more social, the camera I bought serving as a catalyst. Let’s just say that it is a work in progress :S A work in progress with results! I have started a vlog, and so far I’ve shot, edited and uploaded eight (8) full episodes! Although none are public, at least at the moment. And the latest three aren’t even on Youtube because I haven’t dared to yet purchase a music licensing subscription. I’ve also been getting back to photography and started an Instagram account and been trying to post my “best” pictures there. It is a rather random collection of pictures without any great theme, so getting any real followers will probably be rather impossible. Not that they appear without “marketing”... But that is not important. Right? Right??

Oh, and I also bought a house. Wow. So much space. “Oispa tilaa” (“[I wish I had] more space”) was already becoming quite a catchphrase for me. So much space!! This, of course, wasn’t a decision done lightly and took considerable time investment, too. And furnishing the whole house is an ongoing process. More time sinks! I’m unable to describe this in more detail currently. But I’ve talked about the house in the vlog a bit, and will talk more later. Maybe even a blog post, but don’t try to push it.

And last but certainly not the least: my very hard work really paid off, and I finally found a relationship in January! This was and is a huge time sink, but I couldn’t be happier! But this is all you need to know about that.

For the next year I hope to get more free time which I can spend on gaming, code, tv / movies and some other ambitions, like more camera and social stuff. Maybe music. And other™ things! Remains to be seen how it will turn out.

And the answer for the topic? You decide.

PS. I’m currently dying trying to figure out a replacement for my five year old Nexus 5 which has been slowly dying over the past few years. The time for replacing is really close, but I just can’t find single satisfactory device on the market!

Getting a life?

I've just been so very busy with everything.

So, first off: work. I've continued working my day-job almost full-time, and that has reflected on the amount of free time I have. On the weekdays I never have the time nor the energy for anything productive, and weekends must be shared with everything else.

Despite all that I got some great work done on Tracker v3 earlier this year. I should write a separate blog post about it, as it is a big topic. But in short I ended up writing my own schema and code generation instead of FlatBuffers. Now initially the messages are serialized as JSON, but I have plans to move to a more effective format when I get everything else done. I also spent time figuring out an actor system called Akka.Net, which I also used for networking initially. Akka can be thought as a message processing framework. Unfortunately the networked performance of Akka isn't very good, and the message-based communication seemed to get too complex when combined with (potentially branching) asynchronous communication. I had a good prototype, but moved on to a "simpler" custom service and networking implementation focusing on request-response model. I'm basing the networking code on Tracker v2 and made some good progress, but it is still in progress. My biggest challenge currently is that I would have to decide on what kind of internal architecture I want to support, and have A LOT of open questions and pending work (oh god, will it ever be done???):

• Support service instance sharing?
• Other ways of sharing data?
• Service instance pooling, for example for database workers?
• Task-based or not Task-based?
• How to best support asynchronity, while keeping good performance?
• Message queueing and buffering? Optimized data structures and piping code.
• Back-pressure?
• Am I really going to write my Master's Thesis on this??

And as if this isn't enough, I've been FINALLY trying to up my social life. And I can say without a doubt that this has taken a considerable amount of my energy. Without giving too much back. Depressing. I've made some new friendish connections, but nothing more than that.

And - again - as if I already didn't have enough things to do, I also tried making some music, as I couldn't resist getting a Novation Launchpad Pro. Another reason was that I was going to make all kinds of non-music stuff with it, but so far that has limited to making a rather simple Guild Wars 2 key mapping for it, for playing a music-themed minigame. Haven't done anything new with it lately.

Then there is also another time sink. I've wanted a good video camera for a long long time for various reasons, so I finally acted on that. Got a GoPro Hero 5 and the Karma Grip. Was going to take them to a holiday trip, but the delivery dragged on and on. So no holiday video. I shot some test material with, watched a lot more in Youtube, and in the end got bit by a camera-bug. This infestation caused me to upgrade the GoPro to a Panasonic Lumix GH5 with a 20mm f1.7 lens (H-H020, the only lens that was available to ship immediately). I've had the camera for about a month now, and am in the process of getting a 12-35mm f2.8 lens (H-HSA12035), because the camera has amazing video quality, and I'd like to use it more than the 20mm allows. The only weakest point is the autofocus. But despite this I have yet again a new hobby: vlogging. I already shot and edited the pilot-episode (cooked some food) on one Saturday (took almost the whole day!).

The episode turned out quite ok, but I'm still debating whether I dare to publish it, even after some tweaks. The thing is, that I've continued the self-discovery catalyzed by Life is Strange, and that video shows more of the unfiltered and buried personality of mine. I feel I'm not yet ready to expose that to the world. But even if I never publish that - or any other episode - shooting the video was a fun and therapeutic experience. The editing not that much .P

So, a lot has happened, and will continue to happen, but at a lot slower pace. At least when it comes to programming projects. I feel great sadness about this, but I guess this is called getting a life?