Blog of Barry

New Features in JacobSix

We are working hard to keep making JacobSix a simple, yet powerful Project Collaboration application.  The last sprint has been a very productive one.  Many late nights and lost weekends have added up to some very good progress on JacobSix.

Here are some of the major features additions:

• Ability to follow a Project - There is a little pin to the right of the project name in the Overview section of the project page.  If this is pinned (pin facing down), you will be following the Project.  What does this mean?  Well, you will get emails when anything changes in the Project, such as new messages, new Milestones or Groups, added Links or Notes.  Pretty much anything at all.  This allows you to keep up to date on any happenings in a Project so you can react quickly.  Eventually, following a Project will lead into IM messages in the future.
• Ability to follow an Action - The same pin you can put on a Project you can put on an individual Action.  If you don't want to following the entire project, you can just concern yourself with one Action.  When an action is pinned, you wil receive emails if anything about that action changes.
• Condensed View of Action Lists - You can click the magnifying glass icon above the action list to collapse the action details.  This allows you to fit more on the screen.  You can always expand an individual action to see the details.  The condensed view mode carries across all action lists and your workbook.
• Displaying Avatar next to Messages - In the Recent Messages list, we now show the avatar of the person who posted the message.
• Invitations by User Name - You can now search for users that are collaborating with you on other projects to invite them to a new project.
• Burndown Chart - For those of you who are into Scrum Agile Project Management, then a burndown chart is available.  This chart is based on the Estimated and Actual hours, using the Project Start and Due or Milestone Start or Due dates to build the range.  This chart is in the More Charts page.
• Commenting on Actions - You can now comment on an Action.  The action assigned and any followers of the action or project will be emailed of any new comments.  This is a good way to communicate and ask questions about an Action.

We have many more enhancements coming soon.  Here is a rundown of what we will be adding soon:

• Project Feed - The project feed will give you a running stream of all activity in a project.  So, if someone closes an action, it will be in the feed.  If someone adds an action, in the feed.  If a new message is posted, in the feed.  The feed is a temporal view of all things happening in your project.  We will provide an RSS link to this feed so it will be easy to follow day to day the projects progress.
• Contacts - The ability to store users in a contact list so you can easily invite them to future projects and also communicate with them.
• User Page - A page dedicated to a user, showing any actions assigned to that user for common projects.

I hope you are finding JacobSix useful and it is helping you with your projects.  If you have any suggestions, feedback, or praise, please let us know at http://getsatisfaction.com/tahelpya/products/tapingya_jacobsix.  We appreciate all suggestions and consider them all seriously.  We want to make JacobSix as useful as we can.

More features in JacobSix. Rolling through the sprint.

I have spent a large amount of time optimizing the site as much as possible given the GAE limitations and recent slowdown.  What I mean by recent slowdown is the exceedingly long response times that happen.  This is not all the time, but enough to become annoying.  A request that typically takes 300ms to complete is taking 20 seconds in some cases.  Using Google App Engine is both a blessing and a curse.  You get many benefits but also are at the mercy of the Google Infrastructure.  Could be anything between point A and B.  So, I cannot pin the blame entirely on Google.  Maybe something in the network.  Who knows.

Besides performance improvements, I have also added Action Filtering to the "All Actions" page and "All Closed Actions" page.  I'll be creating screencasts soon to demo some of these features and guide people through the site.  It is not a complex application (that was the point), but some of the features are not completely obvious (let me know if this is not the case, I'll be happy to hear this).  From the All Actions page, there is a little arrow button next to the refresh button at the top right of the listing.  Clicking this will expand the filter options.  You can filter by assigned, group, and priority.  This is a combined filter.  So, choosing assigned and priority will filter on both, not either.

Have fun with the feature and let me know other filters you want to see.  I will add due date filtering (between this date and that date).  That one is in the list.

Where is the slowdown in my GAE App?

Click here to download:

profile_helper.py (1 KB)

Recently I noticed JacobSix not performing very well.  Requests would take a long time sometimes, not always.  Well, I think that's the nature of GAE and you cannot expect consistent response times even from the same exact request doing the same amount of work.  Sometimes it takes 300ms, sometimes 3000ms for the same request.  Well, I wanted to make sure it was not something I injected into the code to cause the slowdown so I wanted to time each section of of the request to find out how long it is taking to complete that section.

I created a Profiler class in Python to do just this.  You simply create an instance of the Profiler class.

self.profiler = Profiler(name="Project Controller", enabled=True)

To start profiling a section of code, you call the start() method:

self.profiler.start()

This marks the start time at UTC Now.

To mark a section, you call the mark() method.

self.profiler.mark('BEFORE FETCH')
... some code here
self.profiler.mark('AFTER FETCH')

When you are done timing, you can call the stop() method:

 
self.profiler.stop()
self.profiler.report()

The report() method outputs the results of the timing to logging.info() so you can check it in your logs.  Here is an example output:

======PROFILE REPORT Project Controller======
09-03 05:00PM 12.837 Mark - Name - Between - Accumulatve
09-03 05:00PM 12.837 0 - BEFORE FETCH - 0ms - 0ms
09-03 05:00PM 12.837 1 - AFTER FETCH - 10ms - 10ms
 09-03 05:00PM 12.837 2 - BEFORE JSON - 0ms - 10ms
09-03 05:00PM 12.838 3 - AFTER JSON - 0ms - 10ms
09-03 05:00PM 12.838 4 - BEFORE RENDER - 0ms - 10ms
09-03 05:00PM 12.838 5 - AFTER RENDER - 0ms - 10ms
09-03 05:00PM 12.838 ==============================
 09-03 05:00PM 12.838 TOTAL TIME: 0:00:00.010875 (10ms)
09-03 05:00PM 12.838 ======END PROFILE REPORT======

The report shows you each mark with the description given when you called the mark() method.  Next it shows you the time between each mark section.  The first mark is showing you how long (in ms) it took between the starting time and the mark time.  The next mark shows you how long between mark 0 and 1 and so on.  The accumulative time is just that.  Finally, the total time shows you how long it took between the start() and stop() calls.

There may be better ways out there, but this is what I came up with real fast.  I've attached the source if you want it.  Maybe messy, but I had to put it together quick to test some sections of my code out.  I have some improvements in mind.

GAE definitely forces you to be efficient.  In my case, it turns out that most of the processing was in generating the JSON of the results and very little time was in the FETCH.  However, this is not consistent.  Sometimes, it seems to take GAE much longer to both FETCH and access the memcache.

Another deployment of JacobSix

I'm up again until midnight working on JacobSix.  This will catch up to me soon and I'll definitely crash and burn for a few days.  Going to sleep at midnight and waking at 5:00 AM with the boys is taking its toll.  However, the positive side is that JacobSix is getting better every day.  I worked all night working out bugs with the Invitation workflow.  A major mess, but I think I have it cleaned up.  There were quite a few other issues that I cleaned up tonight.  For example, the "More Charts" was broken if you are filtering on "With Milestones."  Turns out, GAE cannot do "__KEY__" only queries with an inequality filter ( != ).

Another bit of work is to try to increase performance in many of the Datastore queries.  I just don't think it's my issue after looking at the logs and seeing so many inconsistent numbers coming from GAE.  On one request, the processing time is 300ms.  Immediately after, the same exact query comes back as 2000ms.  I don't get it.  I did clean up the indexes, which may help a bit since half of them were not in use.

There's a ton more work to be done to handle errors a bit better.  I'll probably take a step back and clean things up before moving into the next half of this sprint.

If you tried JacobSix and like it, leave me a comment.

Google App Engine Getting Better

I just read this post at the GAE blog:

http://googleappengine.blogspot.com/2009/02/roadmap-update.html

Basically, GAE is getting XMPP support and inbound email processing.  Wow!  The possibilities for this framework keep on growing.  I can only imagine what I can do with XMPP.  Complete IM integration into my GAE applications.  The inbound email processing is even better.  Now, I can imagine with JacobSix and email being sent with Action status updates or new Actions.  Same with the XMPP.  Simply use the JacobSix bot to check on your top actions.

I'm looking forward to all that is coming.

More new features at JacobSix

Burned the midnight oil the last night and added the final two features of the "More" tab at JacobSix.  Notes and Links.  

Notes provides a simple WYSIWYG interface allowing HTML markup text entries.  You can pretty much put any HTML in the note and they are listed in order of update time.  The Notes keep a history of previous versions, which will eventually be exposed so you can rollback changes or view an older version.  For now, it's just a great way to keep information about a project.  Because it accepts HTML, you can embed iframes, which means video or any other embed that's out there.  I haven't tested the dangers of this, which I'm sure are many.  So for now, we'll give it a try and restrict the HTML later if necessary.

Links provides a place to keep any URL you want to save.  Bookmarks or Favorites basically.  However, like TaPingYa (the messaging application), the Links will be collected automatically from around the project.  So, if you post a message to the Project that has a URL, it will automatically be saved in your Links.  You can delete these later if you want.  This just makes it easy to see all those URL's that get posted in various places.  The auto collection is not complete yet, but soon.

I also added some logic to check the state of Google App Engine.  The service goes down for maintenance now and then, so JacobSix needs to handle this better.  I put some code in place to deal with this, but messed up the logic so it didn't go down so well.  Next time!

Google App Engine Maintenance

JacobSix is down once more because of Google App Engine maintenance for about an hour.  This is normal, but what makes me sad is I put logic in to check for this and report a nice message to users.  My logic was bad and now they will only get a nasty error message on the screen.  Sad, so sad.  I need to get some QA going on this application.  Any volunteers?