I was at a (pretty boring) course the a couple days this week, and also madly trying to get some stuff done for my moonlight job, so that disrupted my attempt at keeping a regular pace of blogging up.

Something I think would behoove most programmers is to learn at least one or two other programming language families. Java and C# are too close. Smalltalk or Lisp on the other hand. It comes down to one general idea…

Experience with new ideas is part of it. Each language has its idiosyncracies that are sometimes useful elsewhere. But the most important value is when one tries hard to understand why the language or framework designers did as they did. That’s the real eye opener. Too many people see something unfamiliar or that doesn’t make immediate sense and proclaim the designers and implementers brain dead, kludge it, and move on.

This is usually a mistake. Not only does that person make inefficient use of time by trying to fight the environment that’s supposed to improve productivity, it’s missing a meaningful learning opportunity. Realistically, even if the designers are less intelligent in every way (unlikely), they had much, much more time to consider the problem space. So it’s totally worth taking time to pause and think about it differently. Instead of thinking “this is stupid, I can’t get it to do what I want it to do”, wonder “under what conditions could this solution possibly make sense?”

It may very well be that the designer missed something. It may be that there is a serious flaw. Showing that helps the designer learn something. But looking for the designer’s error first, and stopping there, misses all the opportunities to learn something in those cases where in fact the designer was right after all. And those opportunities are a delight, because it’s almost as good as having a dedicated mentor, older and wiser, right there explaining and teaching you tricks of the trade.

I do my own stock investment and purchases. One great read for ideas is TD Waterhouse’s Newcrest research, which it makes available to those who have brokerage acounts with them. One has to log in to their website, navigate a few pages, and then look at the reports one at a time.

Two annoyances for me: 1) I don’t remember to log in everyday and check, and 2) the session timeout for the website is 15 minutes so that I have to log back in after reading a longish paper.

Being me, I can’t stand anything to do with computers and not being able to do something I believe should be possible. I don’t want to navigate the website and read the PDFs one at a time, I’d much rather have it automatically be picked up and sent to me via email, which I already have a habit of reading.

So after a few hours in the dead of night, and several more after some death-like sleep, I finally hit upon a setup that works for me. Here’s what I learned:

On my side

• wget doesn’t support saving session cookies between calls
• curl is a more powerful and flexible tool than wget
• LiveHttpHeaders is a Mozilla plugin that lets one see exactly what the browser is sending and receiving - better than a logging proxy, for HTTPS purposes.

On TD’s side

Their process to get to the market research goes something like this:

1. Copy the user values in the login form to a hidden form on the same page, and submit that instead. The hidden form has an authorization token that is generated everytime you read the front page, so you can’t just submit the form as the first step. Get a session cookie for your troubles.
2. Get a pointless page where TD says they’re loading your profile. All it seems to do is reload to the user’s start page.
3. From the start page, load the MarketsAndResearch page. It looks like we get another kind of cookie for our troubles: skipping this step yields “session timed out” messages. This yields a page composed of a few frames: header, side bar, and the main content page.
4. Here’s where it gets a little weird. On loading the main content page, you get back an unusual hidden form. It’s prepopulated with a magic number (the form value is called “magicno”), other form values called Blob1, Blob2, and Blob3. Blob3 is empty, but Blob1 and Blob2 have a really long string of characters in each (300+?). It posts this to some other server, www.tdcanada.wallst.com, and an ASP page. Said ASP (ASP.NET according to the HTTP headers) takes the magic values and gives back a redirect page. The redirect page has a few query parameters that identify you to the redirect target. Get a session cookie (presumably for this wallst.com server) for your troubles.
5. After that, it’s pretty straightforward. All the session cookies have been obtained, it’s a matter of parsing some HTML and javascript function calls to figure out which URL and what query parameters to use to get the report for the day you want. It’s a bit obtuse; the first call to get a report creates a window that calls a different URL (with similar arguments), and that window calls some “cgi-bin/upload.dll?” URL to finally get the PDF.

Applying a little reasoning, it looks like TD decided on a Java general portal, but their research people are running ASP.NET. So the Java portal goes to the trouble of contacting the ASP.NET server behind the scenes, agreeing on some magic values, and then giving the browser these values use on the ASP.NET server. The ASP.NET server then knows that the user has been given the thumbs up already by the main portal. The ASP.NET server then tells the browser to go to a certain place, with a certain number in the query parameter, to get the session cookie.

Put that way, sounds like some kind of back-alley deal.

In the end, I’m just happy I was ultimately successful. Other than the learning experience, spending this much time to solve such a small amount of manual labour seems pointless; I’m happy to have done it for the knowledge though.

I must admit, the TD website is a little more “kludgy” then I might have imagined. I was particuarly surprised that their site causes IE to prompt the user about a potential security breach, because TD is reusing an SSL certificate that’s for a different TD server.

I attended a couple technical seminars today. One was pitching what’s basically a rapid application development platform - basically a canned, integrated set of solutions for common enterprise application problems. Like VB is to languages, so this was to applications. The other was on service oriented architecture: pros, cons, directions.

On the surface, I have to admit I’m technically snobbish enough to get a little irate at one can see cosmetically. Blah blah, our product can do anything, blah blah, service oriented architecture will butter your bread.

However, that attitude stems from arriving at the seminar with an attitude of “make me have to believe you”. Another attitude I find much more useful is to ask “why does he think that?”. “Because he’s stupid” is not an acceptable answer. At worst, “he fell into a trap that I know is a trap because I’ve seen that trap up close myself before”. Even if you don’t agree at the superficial level, you can turn up some interesting things.

For instance, in the product seminar, I think the product is being pitched as more than it cna really do. However, the way the product is being pitched, and hammering home the claim that Java and .NET projects would never be as productive as with this tool made me think: it’s true in a way. As our application needs get more advanced, our tools aren’t advancing as fast. Sure, the main programming languages advance… but since when was the human response to increasing adversity to concentrate on improving our general capabilities? We specialize, and develop tools that are really good for a narrow problem space. And this, I think, is the brilliant vision of the product I got from the product seminar. Sure you could do it in Java or .NET. But it’s like building a house by learning carpenty first to make the boards, then smithing to make the hammers and nails, then architecture to design it… That company realized there was a gap: on the one end, we have very complex goals; on the other end, we have tools, but they address such a basic complexity level (hammers and nails) leaving a big gap that still has to be addressed (architecture).

Wow, that was longer than I thought it would take to say. At any rate, I’m glad I went (and not only because the breakfast was good!).

YouSendIt looks legit, but I haven’t tried it myself. It professes a 1GB maximum, and it’ll host your file securely for you, emailing the recipient that he/she can pick it up. Not sure what their business model is, but if it’s as advertised, that’s a useful site to remember.

I’m really starting to miss being a techie. At least there, there’s some level of minimum objectivity (does it work?) that one can count on.

I thought I asked for something pretty nicely, and it really was for work that I’m not best suited for. I suspect, however, that the recipient was deliberately looking for a negative way to interpret it and saw it as an attempt to offload work to someone else.

That part doesn’t bother me. It’s the not so subtle rejection. In fact, I’d suspect that I was being told off, but with words that still yielded plausible deniability. However, thinking of it that way won’t help me keep an open mind.

I guess this is good for me - it takes all kinds to make the world go ’round. But does it ever feel nasty in the moment.