Tag: Dive Deep

Adventures With APIs

Adventures With APIs

By | | Comments 0 Comment

I’ve written before about the advantages of knowing how to dig around in DevTools to reverse engineer website interfaces. This week I’ve had three further instances of doing this work to good effect.

The Friendly Skies

Firstly, I travel a lot, enough that I now have Executive Platinum status on American Airlines. This means I’m first in line for complimentary upgrades, but only if there are seats available. So I wanted an easy way to go straight to a complete seat map to look at availability without needing to go through a full search on the website. Turns out there’s a magic URL that does just that, and all you need to do is pass it some parameters. So I present to you, a quick and dirty seat map lookup form. Give it a try!

While that’s cool, I wondered if I could make it simpler by leveraging a flight data API like the one from FlightLabs. That was pretty straightforward as well. Though I couldn’t embed it in WordPress, I did script it up in Python for your enjoyment. Just pass it a flight number, and it’ll do the rest. Neat!

Shake It Off

This week Taylor Swift tour tickets went on sale, and needless to say it broke Ticketmaster, despite their best attempts to add friction via pre-registration to enter a lottery to win a code to join a queue to enter a room to maybe get lucky enough to click fast enough to buy tickets. Sadly I was unsuccessful at securing seats despite dozens attempts across several days. But I did learn something about the API Ticketmaster used to check queue status, so all was not lost.

When I first joined the queue, the following was displayed:

Of course I was curious: how many more than 2000 people were there… 5000? 25000? A million? So I opened up DevTools, and took a look at the calls coming back from the server to check status. Lo and behold, there was a wealth of info in an easily digested JSON block:

Wouldn’t it have been helpful to display that information to the user? At least the exact users in line, and the expected service time value. No idea why it wasn’t shown, other than Ticketmaster not being known as a terribly customer obsessed company.

Automating ******** Across ****

The final example can’t speak of publicly other to say my penchant for automation will save my employer a sizable amount of money. In these uncertain economic times, that’s always a good thing.

That Lovin’ Feeling

That Lovin’ Feeling

By | | Comments 0 Comment

I’ve done a fair share of production debugging in my career. There’s a heroic Dopamine rush that comes with it, that feeling of diving deep on a problem in a critical situation, finding the solution, and then implementing it to the delight of your customers and teammates (well, either delight or they graciously allow you to live another day).

A similar feeling is experienced any time you build something with your own two hands and see it come to life; that joy is why this blog is named what it is, because I love to build. But as a manager of a technical team, I don’t get the chance to directly build solutions as often as I might like, and when I do, it often represents a failure of some kind. To be successful as a leader, one needs to learn to let the joy of building go, at least the hands-on kind.

Instead, what a manager must cultivate is the joy of watching others succeed, especially those to whom they are charged to mentor as direct reports. Being thrilled when a person shares a difficult problem they solved; rejoicing when a struggling individual responds to your coaching with positive growth. Giving others the spotlight and serving their needs while hidden from view. Learn to get a Dopamine hit from such experiences, and you’ll do well.

And here’s the real kicker: when you achieve it, you’ll have scaled your ability to enjoy your job beyond what was possible before, because you now have a whole team of people whose successes are a source of happiness, versus just your own. Ultimately you’re still building, but now it’s through others.

Expect The Unexpected

Expect The Unexpected

By | | Comments 0 Comment

Like many other languages, Python has the notion of default values for function parameters, for example:

def compute_final_price(cost, sales_tax_rate=0.08):
    return cost * (1.0 + sales_tax_rate)

This code behaves as you would expect:

compute_final_price(10.0, 0.07)  # returns 10.7
compute_final_price(10.0)  # returns 10.8

However, interesting things happen in more complex cases, because the default value is evaluated once, at the time the function is defined, and not when it’s executed.

my_rate = 0.05

def compute_final_price(cost, sales_tax_rate=my_rate):
    return cost * (1.0 + sales_tax_rate)

compute_final_price(10.0)  # returns 10.5
my_rate = 0.06
compute_final_price(10.0)  # still returns 10.5

You can prove to yourself the value is evaluated at definition time by running help(compute_final_price), and you’ll notice the default value has already been computed:

Help on function compute_final_price in module __main__:

compute_final_price(cost, sales_tax_rate=0.05)

The above can be pernicious if the default value is an object that’s modified inside the function:

def add_to_list(item, my_list=[]):
    my_list.append(item)
    return my_list

add_to_list(4, [1, 2, 3])  # returns [1, 2, 3, 4]
add_to_list(1)  # returns [1]
add_to_list(1)  # returns [1, 1]

A common workaround for the above is to do the following:

def add_to_list(item, my_list=None):
    if my_list is None:
        my_list = []
    my_list.append(item)
    return my_list

add_to_list(1)  # returns [1]
add_to_list(1)  # returns [1]

I’ve encountered folks that insist one should never use default values except for None and the above pattern, but I’ve found most of the time I’m not mutating such defaults, and the behavior described here is fine. But still, the more you know.

Crossing The Rubicon

Crossing The Rubicon

By | | Comments 0 Comment

There are a plethora of resources for those just getting started with software development, and best I can tell, most of them will do the job adequately well. But there isn’t nearly as much on what is needed next: guidance on how to get from “early intermediate coder” to “seasoned software engineer”. In brief, here are some suggestions:

  1. Study existing high-quality code; if you don’t know what good looks like, it’s nearly impossible to produce it.
  2. Write a lot of code. There’s no substitute for practice and putting in your time.
  3. Find someone who can give you feedback on the code you write, and humbly iterate per the guidance.

In my experience, an efficient way to do all three is to find a couple open source projects that need support, dig into their code, and submit contributions. You start from a base of quality code, get to write more, and you’ll get feedback through pull requests. And the icing on the cake is that it’s all done in public.

Spend a few years of doing the above and you’ll be well on your way to next-level programming expertise. It’ll also teach you how to code with a distributed and decentralized community of stakeholders, and that’s no small thing.

Throwing In The Towel

Throwing In The Towel

By | | Comments 0 Comment

I’ve said before that I enjoy thinking about organizations, continuously optimizing them to give the highest probability of the desired outcomes. Organizational ergonomics, if you will. I’m also a mathematician by training and a nerd, which is what made Applying the Universal Scalability Law to organizations such a fascinating read, because it puts statistical weight behind its arguments.

In the same vein, I thoroughly enjoyed the following two articles that take a mathematical approach to understanding variability in project estimation (a perniciously difficult problem in all technical work):

The latter’s provocative title hopefully piques your interest enough to read further. You won’t regret it!

My Kingdom For A Streak

My Kingdom For A Streak

By | | Comments 1 comment

Because I’m a LeBron fan, when it comes to Wordle, I care more about longevity of sustained greatness (i.e. my perfect streak of winning games) than I do about having epic individual performances (i.e. having a lot games where I’ve won in 2 or 3 guesses). That means I need a way to keep my statistics even if I move between devices or need to replace one.

Almost to quadruple 100!

It turns out (at least as of this writing, who knows if the NYT will change it) that statistics are stored in a simple JSON object in browser local storage. On a laptop this storage is easily manipulated using DevTools, and on mobile it’s a bit tougher but still not too bad, at least on Android.

Both of the above, however, rely on GUIs. It got me thinking if there was a scriptable approach. First, I found a Python library (natch) that can speak the Chrome DevTools protocol. Then I had to figure out how to use it to read and write to local storage. That wasn’t too hard (even though the example here is in Node, the technique was portable). Finally I needed to make the library connect to both a laptop-based browser (easy), and my Android phone via adb (not so easy). Luckily I stumbled on the correct magical incantation in this post. Put it all together, and boom, I can now backup my Wordle streak, and easily transfer it between devices, using a script:

import chrome_local_storage

laptop_storage = chrome_local_storage.ChromeLocalStorage(port=9222)
phone_storage = chrome_local_storage.ChromeLocalStorage(port=9223)

wordle_stats = laptop_storage.get('games/wordle', 'nyt-wordle-statistics')
phone_storage.set('games/wordle', 'nyt-wordle-statistics', wordle_stats)

Want to see how I did it? Well, you’re in luck, because I packaged up my implementation and published it on PyPI. The source code is also on Github. Enjoy!

Hooray For Heuristics

Hooray For Heuristics

By | | Comments 3 comments

Besides the two resolutions I made for 2022, I’ve decided to try out a meta-resolution: every year from here on out, I will resolve to read the same number of books as years I am old (inspired by the coincidence that I finished 42 books last year, which happened to match my age). I track all my reading on Goodreads, where you can follow along if you’d like.

Given the above challenge, I wanted to determine how much of a time investment was going to be involved, and especially wanted an easy way to break it into daily reading targets that would keep me on pace. To do so, I needed two pieces of data: an average book size in pages, and an average time spent per page. My gut feel for these values was 300 pages and 1 minute, which led to a nifty conclusion: if I let A be my age, and aim to read A pages per day, which takes roughly A minutes, I should be able to easily complete my goal of A books over the course of the year (365 > 300, but I expect I’ll miss days here and there). Plugging in my current age of 43, that means a modest investment of 43 minutes per day is all it takes to achieve what otherwise sounds like a difficult goal. Isn’t that neat?

Neat enough that I wanted to validate my assumptions. For average book size, I downloaded the last 10 years of my reading records from Goodreads (I’ve been doing this a while): 84218 pages divided by 288 books gives an average size of 292 pages. My guess was pretty dang close, cool!

To measure my reading speed, I timed how long it took me to read 10 pages of three representative books: Multipliers (business/engineering non-fiction), Lifting the Veil (religious non-fiction), and The End of Eternity (science fiction). Resultant times were 6.5, 10.5, and 10.5 minutes for 10 pages, respectively, which averages out to 0.9 minutes per page. Once again, my intuition was reasonable.

One final statistic worth pondering: if I can hold to this meta-resolution, how many more books can I expect to read before I shuffle off this mortal coil. Thanks to Google, I know average life expectancy for a male in the United States is 75, so we’ll say I’ve got 32 years left. Thanks to Gauss, I can easily compute a sum from 1 to N with the formula N * (N+1) / 2. The sum of 1 to 75 is thus 75 * 76 / 2 = 2850, and now we need to subtract off years 1 through 42, which sum to 42*43 = 903, for a final result of 2850-903 = 1947 books. My Goodreads backlog is only 99 books long, so I guess I better start adding to it. Any suggestions?

Off To The Races

Off To The Races

By | | Comments 0 Comment

In my previous post I mentioned an issue I had when building a CDK construct. As promised, today I’ll go through the problem I found: a dreaded race condition, which anyone who’s spent much time debugging software knows is a pernicious type of situation where behavior varies depending on the order in which various parts of the system execute, causing intermittent failures.

For background, part of the power of CDK is that it provides a framework for executing raw AWS API calls as part of a larger deployment. This is useful in numerous circumstances. For my construct, it enabled me to output several Managed Blockchain parameters that are available via API call but not from CloudFormation.

Under the hood these API calls are executed in a Lambda function that is created just for this purpose. This function has an IAM role, to which various permission policies are applied. For efficiency, it is only created once during a deployment, and then shared across all the API calls in the stack.

In order to keep my code well-organized, I’ve broken out the API calls in several places: one to gather data on the network member, and another to gather data for each peer node. And as a security best practice I want the permissions to be scoped as narrowly as possible. That means at each point in my construct where I call the function, I attach a policy that allows access only to the specific member or node being queried, via an explicit identifier.

Here’s the problem: IAM is an eventually consistent service, and thus policy updates are not immediately effective. Typical propagation time is only a few seconds, but it can take longer in certain circumstances. For the first custom API call in a CDK stack this is not an issue. The policy and role is created, and then the Lambda is created, the latter taking over a minute to be fully instantiated because it upgrades its dependencies at launch. However, on subsequent calls, because the Lambda is already warmed up, it runs immediately after the preceding policy update, and about half the time said policy is not yet effective, and the function fails due to a permission error.

It’s the “sometimes it works, sometimes it doesn’t” nature of race conditions that make them so difficult to track down. Thankfully I was able to identify and document my experience and pass it along to the CDK team. Anyone want to take a crack at a solution? I described several possible approaches in my write-up, with the “simple retry logic” approach likely being the best.

A Number Of Numbers

A Number Of Numbers

By | | Comments 0 Comment

Back in my math major days in college, I was introduced to the Online Journal of Integer Sequences. It’s exactly what it says on the tin. As part of a class we were encouraged to contribute, which I did.

A few weeks ago I had another idea for a submission, and to my surprise no one else had added it, so once again I had opportunity to contribute a little piece of Internet history.

Here’s a complete list of the sequences I’ve authored over the years:

If the above isn’t enough online notoriety, check out my only published mathematical work, A Probabilistic View of Certain Weighted Fibonacci Sums. I was only a mild contributor, but still got an authorial credit, which is pretty cool.