For fun, I've been looking into Java's memory management tools, jhat and jmap.

Jmap works fine. I'm running Linux, so I find the Java process that I want to analyze by using ps -ef |grep java. Then I run jmap -dump:format=b,file=java_pid.hprof pid where pid is the Java process id. This generates a heap dump. You can repeatedly do this to get snapshots of the heap over time. Neat! Some of the heap dumps are quite large, over 1G in size.

Once you've got the heap, what can you do with it? You can use jhat - at least in theory. According to the jhat documentation, "jhat enables you to browse heap dumps using your favorite webbrowser." Sounds cool! But when I attempt to run it, I always get an OutOfMemoryError:

me@me:~/heaps$ ~/jdk1.6.0_10/bin/jhat java_pid403M.hprof
Reading from java_pid403M.hprof...
Dump file created Sun Oct 18 10:17:15 PST 2009
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Hashtable.rehash(Hashtable.java:356)
at java.util.Hashtable.put(Hashtable.java:412)
at com.sun.tools.hat.internal.model.Snapshot.addHeapObject(Snapshot.java:153)
at com.sun.tools.hat.internal.parser.HprofReader.readInstance(HprofReader.java:728)
at com.sun.tools.hat.internal.parser.HprofReader.readHeapDump(HprofReader.java:474)
at com.sun.tools.hat.internal.parser.HprofReader.read(HprofReader.java:226)
at com.sun.tools.hat.internal.parser.Reader.readFile(Reader.java:79)
at com.sun.tools.hat.Main.main(Main.java:143)

The smallest heap dump that I have is 400M, and it produced this error. Not so cool, really. What to do - run jmap on the jhat process?

I tried doing what I've done in the past when running into an OutOfMemoryError - reset the JVM's max heap size. The documentation says you can pass flags to the JVM running jhat by using "-J". Almost as if they knew you'd need it, they give the example of passing in the max heap size as 512MB with "-J-Xmx512m". So I tried using "-J-Xmx1024m", and the process continued further, but failed in the end:

me@me:~/heaps$ ~/jdk1.6.0_10/bin/jhat -J-Xmx1024m java_pid403M.hprof
Reading from java_pid403M.hprof...
Dump file created Sun Oct 18 10:17:15 PST 2009
Snapshot read, resolving...
Resolving 8502679 objects...
Chasing references, expect 1700 dots....................
[ a whole lot more dots, but not 1700, and then ]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Hashtable.rehash(Hashtable.java:356)
at java.util.Hashtable.put(Hashtable.java:412)
at com.sun.tools.hat.internal.model.Snapshot.addHeapObject(Snapshot.java:153)
at com.sun.tools.hat.internal.parser.HprofReader.readInstance(HprofReader.java:728)
at com.sun.tools.hat.internal.parser.HprofReader.readHeapDump(HprofReader.java:474)
at com.sun.tools.hat.internal.parser.HprofReader.read(HprofReader.java:226)
at com.sun.tools.hat.internal.parser.Reader.readFile(Reader.java:79)
at com.sun.tools.hat.Main.main(Main.java:143)

Disappointing! I wonder if anyone uses jhat at all, or if it's just a nice toy for very small heap dumps.

In fact, if you search the internet for "jhat OutOfMemoryError" you will find some very interesting links to follow up on... I'd like to get jhat working, just to see what it does. But meantime, I've also stumbled across the Java performance blog, and I'll soon be trying out the Eclipse Memory Analyzer.

is this in my job description?

OK, my job doesn't actually have a description. I work at a company which doesn't believe in that kind of thing. My job title is "software engineer", and I imagined I'd be doing software-engineer-y things when I signed on. And usually, that's the case.

Last week, though, I had to spend a couple of days doing things that would normally be done by a business analyst. It was made particularly onerous by having to wade through enormous amounts of legalese, the kind of thing that makes my head vibrate (in a bad way). In addition, I was forced into more-than-usual contact with one of the testers that I don't get along with so well.

The fact that I was assigned to do this work isn't a signal of any kind. I'm pretty sure it was just expedient. There just aren't any BA's at my company. Given that, I think the work that I wound up doing should have fallen into the product manager's hands, but he's already overloaded with other work, and tries to avoid involvement in my product when possible. Of course, I am also overloaded with work, but the buck stops with me, it seems.

two databases, one column

I feel fortunate that I don't experience a "WTF?!" moment on a daily basis. Although sometimes, it does feel like I'm in Groundhog Day, perpetually experiencing the same WTF?!s over and over again. Today was such a day.

The application that I'm working on talks to two databases. In one database, there's a table called user with columns like id, firstname, lastname, ssn, and so on. The information in some of the rows in this table has also been stored in the second database. This is done for reasons of efficiency, so that when querying the second database, we don't always have to dip into the first database to get user information.

This seems perfectly reasonable to me, except for the way that it was done. Way back in the stone ages of our application, someone decided to store all the user information in a single column in the second database. It is stored as a set of key-value pairs in a CSV string, in a column called user, like this: "user=3378,first=John,last=Smith,ssn=123456789,..."

Since I first encountered this, I've been cursing scratching my head about why someone would decide to store all of this information in a single column. It makes queries more difficult to read and write, because it requires regular expressions to extract single items like the social security number from the user column. I can imagine this is also a performance hit.

In addition, whoever decided on this format did not replicate the user table column names for the keys in the CSV string. So the user.firstname column in database 1 maps to first in the CSV, user.lastname maps to last, and so on. This makes life more difficult when searching the code base for all queries related to a particular piece of user information. I recently fixed a bug caused by this mapping difference.

I'd love to have a conversation with the person who decided to do things this way. It would either be enlightening, or just plain crazy.

networking

I just laid down some serious cash out of my own pocket to go to a professional conference. I should do this more often, I guess. It's difficult to justify it based on results. Does the conference pay for itself, in the long run? I have my doubts. But I expect it will considerably improve my mood, at least for a little while, and I can really use that.

Before going, I should to do some reading about how to network effectively. Without feeling (or acting) like a leech.

please talk louder, I can't hear you

(200/365) Shhhhh CC BY 2.0
by dm-set

The personnel manager of my company is seated in an office near my cubicle. This morning, when I arrived, she was engaged in what turned out to be a lengthy conversation with her assistant. The conversation was loud, and large pieces of it drifted into my consciousness. Quite a lot of it was of a personal nature. At times the conversation would become muted, at which point I surmised they were talking about something (possibly work related) that they didn't want others to hear about. This was an indication to me that they were aware of the fact that people sitting nearby could hear every word of what they were saying.

Later, around 10 am, the guy in the cubicle adjacent to me began having a loud and detailed telephone conversation about his car loan. This went on for about five or ten minutes.

Then at around 2 pm, a guy further away began having a rancorous phone conversation, apparently with a car mechanic, about the unsatisfactory service he'd received. Despite the fact that he is two rows away from me, I could hear every word.

In between all this was the incessant chatter from two nearby cubicle dwellers who can't seem to stop free associating about non-work related topics for a large part of the day.

I think my number one complaint about my current job is the incessant noise due to inconsiderate coworkers who have different standards about what constitutes appropriate workplace discussion and the volume thereof.

I am a programmer, working at a software company. That means I spend long hours thinking in front of a computer. Ideally, I'd be sitting in an almost noise-free environment, with infrequent interruptions. For example, right now I am at home typing at my computer. There's a bit of noise from the apartment across the hall - I sometimes hear a child making, well, child noises. Sometimes there's car noise from the parking lot. Overall, it's pretty quiet, and there's no sound of people talking. At home, this is true all day long.

Normally, I am a very focused person, and I find I can frequently block out the conversation noise at work. After about a year at my current place, that no longer seems to work. I've noticed that several coworkers use headphones. I don't like headphones; they eventually begin to hurt my ears. Further, I do not focus well with background music drilling into my brain non-stop. I work much better with quiet, and it's frustrating being forced to take steps to block out noise which should not be there in the first place. Because these people are not even having work-related discussions. They are just blabbing about personal stuff during work hours.

The company culture overall encourages this. I'm within earshot of a couple of more senior people who have loud phone conversations (work related or personal) from within their offices. These are not cubicle dwellers, but they still find it necessary to leave their doors wide open when talking on the phone. What's amazing to me is that these people do actually sometimes close their doors when on the phone. Presumably this means something particularly private is being discussed. Don't they realize that no one wants to hear their non-private discussions, either? Just be considerate and shut the door when having a conversation, for crying out loud!

In my list of grievances at my current job, this ranks pretty high. The noise is bothersome, but just as aggravating is the apparent lack of consideration by numerous employees. Sometimes it seems like these people are deliberately talking loudly as a way of showing how important they are. Or maybe they're just thoughtless blowhards. Either way, I wish they'd STFU. But I guess that's not going to happen, so I need to put my ass in gear and start looking for a new job.

how much time will it take?

Clock Top / CC BY 2.0
by Chris Metcalf

I'm one of just a few developers working on a poorly supported software product. The product was barely maintained for the last several years, since it was first released. Many moons ago, it was decided to give the product a complete overhaul, front end and back. Meantime, the main caretaker of the product wisely decided to leave for greener pastures. He was tasked to add as many new features as he possibly could before leaving, and he did so.

He pretty much had free rein over the product while adding enhancements and rewriting code. So far as I can tell, there was no extra pair of eyes checking to see what he did. Management trusted this developer to do everything in the "correct" way. The problem with that lack of oversight is that you are trusting a developer to do everything the way you expect without much (or any) feedback. In many cases, even the product owner doesn't know what to expect, so this imputes impossible, magical abilities on a developer. Naturally, some things are not working correctly, and additions have to be made.

Next thing you know, I've been assigned to finish up the work that was started by the departing developer. There are areas of the application that are incomplete. The code base is quite large, and the application is complicated. I don't know what things work or don't work until I happen to run across them while debugging some other problem.

I've been asked to supply an estimate for how much more time will be required to take this project to completion. So far, I have equivocated, saying that it is very difficult to make that estimate at this point. I am still unfamiliar with a lot of the code base. No one knows how many bugs are in the current iteration of the product. I don't have a specification for what needs to be done, or a checklist of things that have been changed or are still needed.

Truth is, it looks to me like there is still a lot of work ahead, but I can't tell how quickly that can be done. It's even harder to make an estimate because one developer keeps getting pulled away to put out fires in another development effort, so progress is even more unreliable than it usually is in software development.

How do you make an estimate in a case like this? I have a gut feeling for how much more time is needed, but I can't justify it. I'd like to take a few days to run through every feature in the product and see just what is and isn't working. At least then I'd have an idea of how much stuff is still broken. But management would not approve that. I'm supposed to spend every minute developing or bug fixing.

So I can't come up with any justifiable estimate. Meantime, management decides to supply their own deadline. I believe this date was pulled out of thin air; it certainly is not realistic. I don't understand why you'd pull a deadline out of a hat and not expect to be disappointed. I've seen this happen with another project. They kept rescheduling the deadline for months. It makes no sense.

fragile (code) - handle with care

shards CC BY 2.0
by zappowbang

I'm working on a "legacy" software system. Legacy means different things; in this case I mean it has been around for a few years. The original developers are long gone, and there are numerous sections which are a mystery. Even more annoying, the code produces mystery entries in the database. There are columns with fields that have dual, hidden meanings. There are extra rows for undocumented special cases.

One of my colleagues, let's call him "Mr Eager-Beaver," recently decided he'd had enough, and removed some lines of code which had been adding apparently extraneous information into the database. He's not an idiot, so he tried doing this on a test system first, and found that nothing went wrong. So he argued for committing the change, and so it was.

Unfortunately, the application is extremely large and complex. Being finite, I'm sure you could design a suite of tests which check every possible feature in the application. However, the company motto is "we will test no code before its time." The end result is that we actually have a very limited suite of tests for this application. Further, from a QA perspective, you can't just go in and hit a few pages, click a few buttons, and declare a code change has done no damage.

So ever since this change was committed, we've been uncovering bugs related to it. In a few dusty corners of source code, the application expects to find the missing information in the database, but doesn't, and barfs.

It wouldn't be so bad if these bugs were easy to find, or if they could all be dumped on the original hero who did the refactoring. No; what happens is that you have no idea what is causing the bug until, after several hours of research, the realization dawns on you: "Oh, this is due to Eager-Beaver's code change!" At that point, you can fix the thing pretty easily, and there's no point in reassigning the bug to him.

So far I have had two such bugs assigned to me. Eager is actually a pretty decent guy, so while I do get an urge to come over and give him a good metaphysical ass-kicking when I finally discover the source of the bug, I don't. I can only hope he's been assigned his own fair share of these bugs. And that he'll be as nice to me, whenever he uncovers some code crime of my own.