Working on large code bases for large web sites

I watched a presentation today from a Facebook engineer that ended with valuable information. Generally an interesting presentation about the ups and downs they faced with their code. Surprisingly blunt about technology choices as well.

Around 46:40 he talks about Practical Advice. This is the part that caught me.

Evolution of Code Design at Facebook

Don’t Branch

This I wholeheartedly agree with. In my opinion when you decide to branch code to do any form of refactoring you immediately create new problems for yourself. The first problem is “when” to merge back into trunk. Do you merge after everything in your branch is perfectly tested? Or do you wait till both your branch and trunk are perfectly tested? If there was bug fixing going on in trunk while you’re branched testing has to occur for both – but which version do you accept – the trunk or the branch?

 

The second problem is what to do with conflicts of the merge that exist on the edges of your changes. Do you refactor the trunk just enough to fit the edges? Or do you change your branch to accomodate those edges in trunk? Do you allow these edges to change your design in the branch? And how much time do you spend doing this – just enough to mash it together or enough to have long lasting durability?

The easy answer in my opinion is stay in trunk. Get over your fear of “what happens when my code change breaks the rest of it”. That’s what revisions are for and if you’re smart with your unit testing and judicious with your commits you should never run into broken builds. Most often in my experience developers who immediately reach for a branch are developers who don’t have sufficient tests for their code (me included).

Confession: I am often saying “I’ve never been mad at myself for branching”. And the reason I say this is because I work(ed) in environments where trunk is abused and brittle. Or often too many developers are making too many changes at once. I say it with the sentiment that I want to branch and not have to worry about what the flaky trunk might do. And when I merge, my branch is the winner in all cases. But always (and I mean always) this causes me to have to make code changes in trunk after I merge. Always. And usually afterwards I say “should’ve stayed in trunk”.

Take advantage of product churn

Ah. I love this one. It’s so obvious but yet so hard to really get. In most large websites there is a product group of folks who decide the features. They change their minds a lot. And that is exactly their job. So when they do, why would you stick with a design and/or architecture that solved the previous problem. Or why would you try to design something so long lasting? Take advantage of a clean slate.

 

There are plenty of arguments to be made for architecting solid systems that can support change. My point is don’t get bound up in architecture decisions so tightly that when change does occur you become adverse. Be flexible and open to the idea that product teams may come to you and say feature X is the next best thing. And that feature may create a ripple in your architecture. If it does, roll with it and re-architect.

It’s my experience that the reason why you get this adversity to re-architect is generally due to the size of the development staff. When you’re dealing with a large staff it is nearly impossible to make quick changes to the architecture. Every developer has to be at a minimum aware of the change, and most of them often implement the change. And when the number of developers is high, the slower the change can occur.

Focus on usability

This is targeted at developer APIs and frameworks, and I do agree but I’m less adamant about it. Or should I say that my version of adoption isn’t so much about winning hearts and minds as much as working with people willing to adopt. And there is a big difference. Top down mandates are a result of working with a development staff that is just not very good with adoption. Organic adoption happens with a certain chemistry and culture. Develop those things first then adoption comes with it. This is a tricky bit to do. And in my experience top down mandates have always been easier (especially in larger development groups). I wish it were different, and I realize my personal style is much different than others.

 

It all depends at which level you are. For instance I hate being told “which way to do it”. But if I’ve come up with a better way I wish I could give the top down mandate. Organic adoption is always better, but it just doesn’t always work.

Generally I agree with all of these kinds of advice. But admittedly I’m more liberal in my approach to design/architecture/coding. I find myself more open than others to change and especially using yet unfamiliar technology. I dislike rigor and unnecessary structure or process, but I’m not opposed to smart and reasonable standards. But the advice given was sound in my opinion.

HIG is dead, long live HIG

At work on numerous occasions I’ve had debates with co-workers about what a good JavaScript library provides. Specifically we debate about “widgets”. Or rather I always chide about them :)

Widgets are those things that the library creates for you like text box, button, drop-down and the infamous Calendar. And in doing so the library relieves you of some effort, and/or provide you some uniformity. That second bit is what my co-workers always bring to the table. Their point is that if you have 100 developers all building software for your website – you really want them building these widgets in the exact same way to help with maintenance/updating and general look and feel. And that’s a huge point. It’s critical. And on occasion they bring up companies like Apple to point to the fact that when you build software for the Mac you get these “widget” libraries built in, so that your app looks like a “Mac app”. If the core library gets updated to change the menu bars for example, your app receives the same treatment for free. No work on your part.

I always retort with “Ok but let’s not build every single widget out there for our library” or more often “we don’t need a library that has all that built in already.” And what I mean by that is, in my opinion, I’ve never seen websites in particular use the exact same widgets across multiple pages. Sure there are the few, staple widgets that in fact are the same across whole sites. But that list is small. No more than 10. Everything else is some variation of a staple. And if that’s the case it would be better to have a library that provides the absolute core set of widgets, but has huge extensibility built into it. Which is to say, a developer could start with the core widget, but extend it to add features for his/her own needs.

The bottom line is that there has to be uniformity across all the widgets (colors, button positions etc). Most often this is referred to as HIG (Human Interface Guidelines). However, it’s my opinion that HIG should not always be followed. Developers always start with HIG, but usually it falls away after a special new feature is requested and it doesn’t quite fit into whatever HIG they’ve adopted. What do you do? Do you change the intent of the new feature to match the HIG? Or do you side-step the HIG to get the new feature just right? And why would a JavaScript library be involved in that decision?

That last question is always where I end. Why would I use a library that caused me to choose one way or the other? If for example the library had “all the widgets I needed” with little extensibility, I’d be forced to look at reducing the feature possibly to fit into the capability of the library or because of the HIG. And is the HIG always the final answer? But if the library had all the extensibility I needed, is it more possible I could meet the HIG by developing a 1-off widget?

This is a tricky situation. There are a lot of ramifications in both directions. Generally speaking you’ll stay out of trouble with a library that “does it all”. And more often than most people would like to admit, you’ll encounter a circumstance where that library doesn’t do “this one thing”. I don’t claim to be right, but I stand by my style and philosophy. I usually lean on the side of “give the developers the room” instead of the other side “give the developers all they’ll need”.

Position yourself, then say “yes”

This morning my reading led me to 2 different blog posts about successful people. The first mentioned it was about becoming successful by actually doing something. The statement was when you decide to do something, opportunity comes.

The second post was about positioning yourself for success instead of relying on skill, luck or some other form of intangible. The statement was not everyone has equal skill or equal luck, however everyone can show up and position themselves.

I can relate to all that. I’ve done that a lot in my life. My baseball career in college was strictly positioning and doing. I didn’t get a scholarship right away. In fact I didn’t get recruited. Nor did I even get invited. Rather I just showed up to a coaches office and said “I’m Clint from Parker – I called earlier.”

Yes. I called coaches in Kansas (with the help of my high school coach: Thanks Coach Flax). I called and said I want to play. I was an average ball player from a 3A school in a small town in Arizona. Most phone calls ended with coaches saying sorry no thanks. Most had full rosters. Except for two teams: Dodge City and Garden City. They both nonchalantly said “sure come on out.”

I packed everything I owned into the back of the S-10 I had and drove to Kansas. I had $120 for gas and food. I had never been further east than Phoenix. I didn’t even really have an appointment with either of these coaches. I just knew they would be trying out other guys that they actually recruited in the next week. They said I could show up and try out.

The try outs were pretty lopsided. I wasn’t ready. I was scared. But I wanted it bad. I threw myself all over the field. Literally. I was the only player that had dirt on my jersey. I didn’t get a single hit during batting. I fouled off everything. I couldn’t see the ball. During strength and endurance I decided that I was going to be noticed. They noticed me because when it was all over I was puking all over the dugout.

Dodge City offered me a spot. Not a real spot. A walk-on spot. That meant they would let me continue to try out. But it came with the best circumstances. Coach called a financial aid office, then called the housing office and before I knew it I had a Pell grant, a loan and all my dorm room reservations set. I had school paid for and a place to stay. Over the course of 1 night coach had made all that happen. I didn’t hit the ball once in batting practice. But because I swung at everything and puked all over the place he felt I deserved another chance to try out.

I eventually walked on and got a full ride scholarship. I still have my NJCAA Letter of Intent. It’s the same letter that all of the “recruited” players sign. Same piece of paper. But I had to earn it over the course of a semester. Trying out every day.

That part of my life determined a lot of who I am today. It showed me that if I wanted something I could position myself to do anything. If I say “yes” that it means most of the hard work is over. Even in my work I do this. I get asked “is this doable?” and I usually say yes. Some people would say this is a detriment. I get in over my head quite a bit. In fact I fail quite a bit too. Most of the time the work is bigger than I originally felt it would be. But I work through it. I finish. Always.

And because I work through it, and because I position myself to be there, and because I say yes: I always win. I win because I get the experience, the knowledge and the opportunity. Others may see failure because I don’t hit the ball on the first at bat. I see success because I’m in the game.

Observations on expectations and praise

There are a few things I’ve observed recently that have made me understand work relationships a little better.

Regarding expectations, I’ve noticed that those who are expected of and feel they deliver on those expectations often get disappointed. This is usually due to the fact that what they delivered was expected and therefore not praised as some work of heroics. Or maybe they weren’t praised at all. Generally: do as you’re asked and nothing happens.

Put another way, those who continue to deliver as expected are generally asked to do more of the same. And with each delivery less and less praise is received (if at all). This is strictly because it is expected to be done. However this disappoints those who deliver on expectations because everyone could use a “pat on the back”.

Now consider those who deliver more than expected. This often receives some form of “praise”. Naturally praise is given when “above and beyond the call of duty” work is performed. For those who did the work ,this feels great. And sometimes it’s motivational. People will want to do it again.

Let’s then consider when “above and beyond the call of duty” work is performed again and again. That creates a new normal. A new expectation level. Above and beyond is now expected. Praise then wanes. Disappointment brews in those who do the work. This demotivates those doing the work. Sometimes indignation creeps into those doing the work.

In all cases bad blood is created between those asking for the work and those doing the work.

In my opinion there are things that each side can change that may prevent disappointment and bad blood.

For those doing the work: do it for yourself – not for praise from others. Remember that work is what you make it. If you are lucky enough to really enjoy your work, leave it at that. If you don’t enjoy your work, well that’s an entirely different problem. Expecting praise will lead to disappointment because it’s never enough. Face it, people are busy and what you do in the big picture is yours. No one else is going to really appreciate your work. Learn to praise yourself and learn to need only yourself to feel good about what you do. If you are happy doing exactly as you’re expected, good for you. The world needs more of you. If you want to do more and go beyond the call of duty, good for you. You should be proud of yourself for being a person who wants to do more. But in either case don’t get ruined when others don’t provide you some form of recognition. Recognize yourself.

For those asking for the work: remember what people are sacrificing. At work it’s easy to say that within the window of 8-5, people are willingly giving up life’s pleasures such as free time and leisure activities. But don’t forget that there are emotional and psychological sacrifices made all the time. Pride and ego are big things, and you shouldn’t inadvertently stomp all over them by simply forgetting to throw a little praise around. If people are delivering on your expectations, remind them that you know. Let them know that you’re paying attention. Rub their pride a little with an easy “thanks” or “good job”.

“No good deed goes unnoticed”. Bullshit. They do all the time. So keep doing those deeds for yourself and learn to notice others doing the same.

I’m not Clint J Hill

I have the fortune of having a historical figure’s name. However I’m not Clint J Hill.

I’ll never be #1 in a Google search – and I’m Ok with that. But for those that have left comments here at my blog believing I’m the other Clint J Hill, I apologize. You’ve found the wrong Clint J Hill.

The 1 big tip – I don’t go by “Clinton”.