Programming

Performance testing persistent maps

2011-03-16T19:18:00.000-07:00

Persistent maps have the excellent property that they share structure, which enables them to produce copies efficiently. I recently implemented a very simple RecordedMap which keeps a history of all changes that occur so that you can rewind time. Naturally I wanted to know what the overhead I would pay for this feature was, so put together some tests to see:

Map Implementation	Time taken
RecordedMapModel	00:00:00.9722224
ClojureRecordedMapModel	00:00:04.9654442
Dictionary	00:00:00.1727407
Copy Dictionary	00:00:17.3513137
FSharpMap	00:00:00.7329950
PersistentTreeMap	00:00:04.4958646

Comparing a mutable dictionary directly to an immutable persistent FSharpMap we can see a performance cost of about 5x time, which is a little painful. More than I was hoping for, but well within an order of magnitude. So there is definitely a trade-off here.

Keeping a full history of all previous versions of the map adds very little overhead, less than 50% (RecordedMapModel). Using a copy dictionary approach was about 20x as slow.

The Clojure persistent map did not perform as well as the FSharpMap, but still was much better than the mutable copy approach, and within an order of magnitude from having no history. I am confident that the performance hit here is because I am doing some funky casting and/or reflection because I'm not providing type hints. A true test would use an implementation in Clojure and avoid reflection. I didn't go there as I wanted to keep the tests as similar as possible.

Source code used to test is available here:
https://github.com/timothypratley/timetraveltests
I commented out the Clojure parts as you need to get the dependency to make that work. If you want to, simply add Clojure as a reference and copy the clojure binaries into the output folder. I suggest you build from source and match .NET framework targets.

For a 5x time cost you can have an immutable dictionary with a fully recorded history. Something that would take 20x as long if you had to copy the data around. An important difference here is that the persistent map version uses shared structure and so will take up far less memory. Too bad I don't know a good way to measure this.

Yay for clever data-structures.

2010-02-15T16:44:00.000-08:00

I was reminiscing about my youth and loaded up an old game I wrote in high school... Pizza Delivery! Totally a blast to play you drive around to flashing locations on the minimap to deliver pizzas and have to refuel occasionally. Going off the road makes your car blow up :)

download pizza.zip

Examples are great value - 2 for the price of 1

2010-01-04T15:12:00.000-08:00

Tests and documentation. Everyone is quick to label these as essential. But trying to include time for them in a project plan is always a battle. Tests and documentation are hard. But because they are seemingly easy to understand activities most people don't realize the effort and skill required to produce them. Even though these activities pay for themselves in the long run, they can end up the first thing on the chopping block when looking for things to cut from a project. This article provides a motivation as to why an example framework should be an integral part of development.

Tests are great for making sure stuff works. But tests and documentation are often the last thing on a programmers mind. Even if you have a testing regime in place, implementing useful tests is actually quite an art that requires experience and thought.

Examples are a subset of tests.
Actually examples are a really important subset of tests because they inevitably are concerned with the API you are exposing, not the internals of your implementation. This is a very high payoff place to have tests. So much so you can think of examples as 'essential tests'. Not only does this mean they are important to provide, but it also means if you are providing examples then you are generating high value tests without additional deep thinking or experience required.

Examples make excellent documentation.
Examples are a really great way to learn anything new. This is especially true in programming. Including examples in documentation of code communicates clearly what the code does and how it should be used. They also get the reader thinking about scenarios where they could use this stuff in ways that reading a description often does not convey.

One thing people fear about documenting examples is 'what if they get out of date?'. Indeed this is a concern. But if our examples are really just important tests, then this concern disappears. Of course our examples will be kept up to date because we are using them as tests. Important tests that we need to pass. If we can achieve both testing and documentation from one activity the payoff to cost is much more appealing.

Another concern is if the examples are too far removed from the code then they wont be created as part of the coding process. We want our development cycle to be closely knit with producing examples so these don't become a separate activate which could fall by the wayside. Performing documentation as a separate task rarely works well unless you have someone dedicated on the case. Documentation generated from source code improves the chance of survival.

So here are my requirements for a useful example framework:

Define examples in one place once
Automatically part of test suite
Automatically included in documentation
Examples live in code

How might we achieve this in Clojure?
First I'd like to note that all the parts are already there, which is great! Clojure has a built in test framework and an autodoc project which supports usage documentation.

Capture examples in workflow
When developing a new function I always end up using it in some way and experimenting with it a little in a REPL. The first thing I do is write a function stub, then immediately after a very basic call to the function. From there I refine back and forward filling out the function, adding calls to it and fixing problems. Those experiments to just after the function definition then get turned into tests when I'm finished. Being able to do this is a really useful feature of Clojure. Clojure has an unfair advantage here being a functional language, it naturally lends itself to small discreet examples matching directly the function definitions.

What is missing?

Unit tests do not automatically appear as usage in generated doc
Unit tests are user named and not related to symbols or available for lookup
Usage documentation is not tested, and is separate from the tests or code
Usage documentation is not available at the REPL via (doc) or (doc-usage)
Usage documentation coverage does not exist yet
There is no support for multiple usage or tests per symbol

These issues can all be overcome and I'm volunteering my help to address them. I've made a start on how I think this should be tackled and am now looking for feedback on whether these are useful improvements to make and the right approach to take.

The code is on github.

The README describes what my goals are, what is remaining, and gives build instructions.

The central part is an example macro which can used very similarly to deftest. The difference being it is associated to a symbol for lookup later from autodoc or (doc-usage) and has a slightly more restrictive form to more suit output formatting.

I've implemented all the wikibook examples. Now they are in s-expressions I am confident I can easily adapt them to a new system very easily.

Lastly I want to re-iterate that Clojure already supports excellent test workflow and documentation. I've found it useful to myself to make some minor tweaks to achieve my goals. Now I'm at the (doc) lookup and autodoc generation part I want to more closely align with what Tom and Stuart are doing with the aim to help provide usage examples for Clojure itself.

Clojure's Quiet Achiever

2009-11-07T05:11:00.000-08:00

The sequence abstraction is at the heart of much of Clojure's power, but receives little attention. I want to tell you what's so great about it!

Iteration and recursion are considered fundamental programming concepts and show up early in any 101 course no matter what the paradigm. The sequence abstraction almost completely throws these tools out the window. You don't need them. That's the big point I want to make here; this huge fundamental part of programming is unnecessary once you have a good sequence abstraction. I can't oversell how much complexity that removes. There must be a catch...

By my count there are 80 functions in Clojure core for manipulating sequences in some way. When you first are exposed to them the natural reaction is a feeling that the cure is worse than the disease, that something simple like a loop is now a mammoth set of functions you need to learn and understand. Loops are just one thing, surely sequences are more complex. However consider for a moment that loops do lots of different things. You have to understand every loop you come across from start to end. Sequence functions abstract this away.

A well worn example is (reduce * (range 2 10)) computes the factorial of 10. We have a sequence of number that need to be multiplied together, so we can express it like that instead of an iterative expression. Lots of languages support the heavy lifters map and reduce, so what's so special about Clojure? My premise is that Clojure has a fuller set of supporting functions which mean you almost never have to drop down to loop/recur. It also has lazy sequences and persistent data structures which mean it will be fast, and you don't need to worry about the implementation.

Another simple example... Say we have a vector of numbers
(def v [1 2 3 4 5])
And we wanted to perform some operation on the sum of each item and the next item. We might implement this imperatively like so:
for i=0; i<v.length-1; i++
operate( v[i] + v[i+1] );

So how would this look like in sequence operations?
(map operate (map + v (rest v)))

Lets break that down,
(rest v) the sequence of v without it's head: (2 3 4 5)
we are mapping the + operator to two sequences: (1 2 3 4 5) and (2 3 4 5)
which will result in a new sequence (3 5 7 9)
which we map the operation over.

You can read the statement as
"create a sequence of results of calling operate on a sequence made from adding items from a sequence of v to a sequence of v without it's head."
user=> (map inc (map + v (rest v)))
(4 6 8 10)

The parts can be individually understood. They can be combined to perform pretty much anything you need to do. As individual pieces they are simple enough to understand and make building blocks to describe all sorts of transformations. This is true of all means of abstraction and means of combination, and especially true of functional abstractions. The sequence abstraction can be extended also. Check out this interesting library which allows you to do SQL like queries on sequences.

Clojure really exploits the sequence abstraction well to remove a major source of mutation and state: iteration/recursion. Sequences are built into the language and well supported with many useful utility functions.

Live long Java

2009-11-07T04:37:00.000-08:00

One thing I really love about Java is the quality of its libraries. The famous Java Ecosystem. For just about any problem domain you'll be able to pickup something that works great and is well supported. Most of the time you wont even need to leave the massive standard library. They really do have something for everyone. And usually there is a pretty clear winner so you don't have to spend your time trialling solutions or wondering how to integrate them.

Lets look at client/server communications for example. Sockets socket sockets, so much fun to be had! So many choices to be made. Will you use threads? Will you support TSL? What wire format to choose? Are you after performance, or simplicity? Yes you can tackle all those problems with due vigor, but someone already has! Netty gives you a completely configurable framework so you can treat all those decisions as options and just focus on your client/server protocol. This is no small thing, decoupling your messaging from the transport layer. For example UDP and TCP are totally different abstractions and it is a non-trivial task to come up with a solution which allows you to plug in either and still work.

Java really has reached a very strong level of 'this is how you solve that problem'. And for that, I hope it stays around for a long time.

Believe the computer

2009-11-04T05:40:00.000-08:00

If you support software you are probably familiar with this scenario:

A person responsible for system X seeks your help because "System Y is broken because it is reporting problem Z with X". The general premise is that it must be System Y's fault, because X is not reporting any error. You respond "X has problem Z". But they can't find any problem Z, it must be Y. Well it is true, Y is not infallible, it could be reporting incorrectly or not doing the right thing. You check the logs, follow the interactions, check any values/times/events... and come to the conclusion "Y indicates X has problem Z". After explaining your findings, they fix Z in X. I've been on both sides of the equation.

Why do we find it so hard to imagine an error occurring on 'our' end? Well if we could imagine the error, we would already be on the path to fix it. Also sometimes checking the 'other end' quickly reveals something outside your control. The second reason is however a big red warning flag. If 'this end' does not provide enough information to tell you whether it is successfully interacting with some other system, it is time to add that feature. This is the first step to being able to diagnose this problem and avoid getting caught in the same cycle for the next one. Once you have that information, you are one step closer to imagining the real cause.

This topic is closely related to the phenomenon whereby when stuck with a seemingly unsolvable problem, simply telling someone else about it can point out an answer without any input from the other person. Some workplaces find this so compelling they set up a doll or teddy bear that developers have to speak to before they ask any one else. The great thing about computer systems is they can be vastly more informative than teddy bears. If you believe them!

Prefix niggle: Comparators

2009-09-16T17:50:00.000-07:00

Infix comparators are very intuitive:

x < y

Which reading literally sounds like "x is less-than y". Visually the arrow indicates we are testing for x being smaller than y.

Prefix notation for comparators has always been something hard for me to think about:

(< x y)

Reading literally sounds like "less-than x y", which takes a bit of mental untangling. Visually it requires a bit of cognition to distinguish < is less-than and > is greater-than because there is no context.

However if instead of calling < "less-than?", we call it "ascending?" then grammatically the sentence become much easier to understand: "ascending? x y". Also the arrow is more easily interpreted visually as we don't require any context to consider it to be a ramp ascending < or a ramp descending >.

Why learn Clojure?

2009-08-19T16:42:00.000-07:00

Clojure is expressive and powerful. It provides the concise expression capabilities of Lisp in a practical environment. It also leverages the JVM, anything you can do in Java you can do in Clojure with far less ceremony. There are loads of other reasons but lets dive into some syntax to get a feel for the language.

Clojure programs are written in forms. Forms enclosed in parenthesis indicate function calls:


(+ 1 2 3)

calls the '+' function with arguments 1 2 3 and returns the value 6, the sum of arguments.

New functions can be defined using defn:


(defn average [x y] (/ (+ x y) 2))

Here x and y are symbols representing the input arguments.
Function '/' is called to divide the sum of x and y by 2.
Note that forms are always in prefix notation, with function followed by subsequent arguments.
Now average can be invoked as


(average 3 5)

and will return 4.
In this example, 'average' is a symbol, whose value is a function.

Clojure provides easy access to the JVM:


(.show (javax.swing.JFrame.))

This calls the show method on the result of (javax.swing.JFrame.) which constructs a new Jframe. Note the full stop before the method call and the full stop after the construction. We can combine them without the usual boilerplate code:


(doto (javax.swing.JFrame.)
  (.setLayout (java.awt.GridLayout. 2 2 3 3))
  (.add (javax.swing.JTextField.))
  (.add (javax.swing.JLabel. "Enter some text"))
  (.setSize 300 80)
  (.setVisible true))

Functions can be passed to other functions:


(map + [1 2 3] [4 5 6])

returns (5 7 9)
map is a function which takes another function and calls it with arguments taken from following collections. In our case we have provided the function '+' and two vectors of integers. The result is a list of the results of calling '+' with arguments taken from the vectors. Using functions as arguments to other functions is very powerful. We can use our previously defined average function with map like so:


(map average [1 2 3] [4 5 6])

returns (5/2 7/2 9/2)

Functions can also return other functions:


(defn addx [x] (fn [y] (+ x y)))

Here addx will return a new function that takes 1 argument and adds x to it.


(addx 5)

returns a function that can be called with 1 argument and will add 5 to it.


(map (addx 5) [1 2 3 4 5])

returns (6 7 8 9 10)
We called map with a the result of addx, which was a function that takes an argument and adds 5. That function was called on the list of numbers we supplied.


(reduce + (range 1 100 2))

(range 1 100 2) creates a lazy sequence of numbers 1 3 5 7 ... 99. 1 is the starting point, 100 is the end point, 2 is the step. reduce calls the + function. First it calls + with two arguments, the first two numbers supplied by range. Then it calls + again with the previous result and the next number, until all the numbers are exhausted. Clojure has lots of support for sequences, collections and high level operations. As you learn them, you will find very expressive ways to write tasks such as this.


(defn factorial [n]
  (reduce * (range 2 (inc n))))

Clojure provides hash-maps which are very useful in many programming tasks. Maps are written between {} just like vectors are written between [] and lists are written between ().


{:name "Tim", :occupation "Programmer"}

Is a map which associates the keyword :name with "Tim" and :occupation with "Programmer". Note that the comma is treated as whitespace by Clojure but may be optionally supplied to help visually group things that go together. Keywords are preceeded by : and provide a convenient way to name fields, however keys don't have to be keywords.

assoc returns a new map with a value associated to a key:


user=> (assoc {:a 1, :b 2} :b 3)
{:a 1, :b 3}


(defn map-count [map key]
(assoc map key (inc (get map key 0))))

This function takes a map as input, looks up a key and increments how many times that key has been counted. (get map key 0) just looks up key in map and returns its value if found, else 0. inc adds one to that value. (assoc map key value) will return a new map with key associated to value. So as you can see the map is not modified in any way. A new map is returned with a new value associated to the key supplied.


(reduce map-count {} ["hi" "mum" "hi" "dad" "hi"])

Results in {"dad" 1, "mum" 1, "hi" 3} because reduce starts with an empty map which is used as the input to the first function call, then the resulting map is used for the next and so forth. The strings are used as keys. Now we can write a more useful function which takes a string and counts the words in it:


(defn count-words [s]
  (reduce map-count {}
          (re-seq #"\w+" (.toLowerCase s))))
(count-words "hi mum hi dad hi")

Gives the same result. Note that re-seq here splits the input string into words based upon the regular expression provided.

This should have given you a taste of the syntax and the power of
Clojure.
Try it out!

Rational choice

2009-05-26T06:49:00.000-07:00

Science has answered many questions and illuminated many truths. As our understanding of nature expands it becomes clearer that our existence is explainable by the physical properties of our universe. Taking this further, our existence is actually defined by these properties. Being fully defined, how can a rational mind believe in choice?

At my will, I can think to bend my arm, and then I bend my arm, and it bends. Surely this is proof of choice? Did I not just choose to do and execute that choice? Well actually my brain state interacted with my muscles by firing electrical currents. This current was triggered by a chemical process, driven by physical molecules. Those interactions all colluded to result in my arm moving. What part did I play in it? The event happened, I precipitated the event. Or did I just observe it? All these questions are turning away from choice and focusing on understanding what the "I" label means.

As we chase the cause and effect trying to determine what originated the choice we soon run out of sources within our own body. The event was influenced by our environment, which was a result of our upbringing, and before that our genes. Everything leading to the event of me bending my arm was a beautiful, logarithmically complex chain of events from the big bang, expansion of the cosmos, formation of RNA, formation of life, evolution, on a scale of time that is incomprehensible. It was all inevitable. There is only one outcome. What appears to be random is just our lack of understanding. An understanding that can never be reached because its only manifestation is its entirely.

When you watch a river flow, you can spot eddies and patterns in the water. You are looking at completely different water but the shape of the bank causes these patterns to persist. Complex shapes and dances form. We ourselves are similar. We appear as an entity in full self control, but actually we are entirely a result of a universal history. Our bodies are just cells which have developed in a symbiosis which allowed them to grow. Our thoughts occurring in the neural network of our chemical brain. Most of the molecules are transients from eating, drinking and breathing which get absorbed transformed and ejected continuously. Even our digestion is governed by bacterial processes which come from entirely external sources. Everything about us is governed by causes and effects that we cannot fully measure, but which are bound by the laws of physics like some giant game of dominoes.

So we are observers it seems. Conscious enough to watch the movie but even our own actions are the result of processes that cannot be controlled. The script is unknown to us, but is already fully determined and unchangeable.

Pick a religion and it comes up empty on the question of choice also. It is possible to conceptualize ourselves as some kind of supernatural dice machine with a mystical link to the physical world. Even so, that would remove all possibility of the rationality required to make a choice and reduce it to a random event.

So how can a rational mind believe in choice?

Simple stuff for Linux

2008-07-10T19:09:00.000-07:00

Man(ual) pages

To get help on a particular command:
$ man
eg. man printf
Man stands for manual. It is split in to sections.

Section # Topic
1 Commands available to users
2 Unix and C system calls
3 C library routines for C programs
4 Special file names
5 File formats and conventions for files used by Unix
6 Games
7 Word processing packages
8 System administration commands and procedures

In the example above it pulls the printf man page from the shell which is most likely not the one wanted.
So run:
$ man 3 printf
to get the C library printf routine
To quit from a man page, press "q"
To search a man page, type "/"

To search the man pages
$ apropos

Shell shortcuts

^D - logout
cd - - return to the previous directory you were in
!$ - use the last arguement from the previous command
(ctrl)c - cancel a command

Unix utilities

To search through all the .h files in the projects directory and directories below for the definition of "Cont_"
find ~/projects -name "*.h" -exec grep -H "typedef struct Cont_" {} \; -print

Use awk to print one column from a text file.
eg: Given a file, ctr.log, with lines like
ctr key: 1983224, iso: 2210, ctr no: TTNU1519344 , etd: 1215752400
ctr key: 1983225, iso: 22G1, ctr no: FSCU3621588 , etd: 1215752400

$ grep "iso: 22G1" ctr.log | awk ' { print $8 } ' | sort | uniq
This will find all of the containers with the iso 22G1 in the ctr.log file; print the 8th column, the container number; sort them and remove any duplicates.

$ grep "iso: 22G1" ctr.log | wc -l
This will count how many 22G1 containers there are by counting the number of lines.

CVS

2008-07-03T03:24:00.000-07:00

Usage

cvs -d (repository) login
cvs -d (repository) checkout (module)
cvs update
cvs diff
cvs commit

NB: you can also shorten usage, eg: cvs up works just like cvs update.

I recommend always doing a cvs diff before any commits. This shows you exactly what you have changed and will be committing, and can avoid accidentally introduced typos.

Avoid mixing formatting changes with logic changes. Doing so makes it very difficult to figure out exactly what changed. Formatting changes typically effect large portions of the file but have no impact on the meaning of the file. Logic changes typically effect only small portions of the file but greatly effect behavior of the program you are editing. If you mix them, cvs tools wont be able to help you discover what really changed to cause some new behaviour. So avoid making formatting changes, and batch them up into one big commit which changes nothing but the format. This works well with indentation tools such as astyle.

Checking changes

cvs annotate (file) > tmp
Will create you a replica of the file, marked with the person who last changed each line, and what date they made the change. This is useful for finding out who is responsible for some part of the code.

cvs log (file)
Gives you the list of revisions, who made them and when.

cvs diff -r (revision) (file)
Shows you the differences between the current file, and what it was as of the specified revision. This is useful when trying to figure out what has been introduced into a new version if you are looking for an obscure bug.

Debugging

2008-07-03T03:08:00.000-07:00

GDB can really be a life saver, even if used for only the most basic functions.

When programs crash they create a core file (sometimes you need to do some configuration to ensure this happens and what the file is called).
gdb myprog core
(This loads the information dumped when the program crashed)

bt (or backtrace)
provides the call stack when the error occured. It shows you exactly on which line of which file the problem occured (if you have debug symbols).

info locals
prints out information about local variables, usually at this point you will go oh damn that should not be NULL, and realize what needs to be fixed.

print myvariable
if you want to check a particular variable.

up
down
move up or down the stack to look at variables in a different scope.

If you are writting a service, you can create a script that loads a core and does a bt and info locals and emails you. For example:
# See if there was a crash
corefile=/var/crash/corefiles/core.${PROCESS}.$UID.$pid
if [ -f $corefile ]; then
gdb $bindir/${PROCESS} $corefile < $basedir/sh/gdb_script > mail.out
mail -s "${PROCESS} crashed" me@email.com <>
rm mail.out
fi

Where gdb_script just contains:
backtrace
info locals

Sometimes (especially with C++) the stack gets corrupted and provides no information. Now you have to either resort to printfs in the code, interactive GDB or use electric-fence/valgrind.

Logging

If you are writting a non-trivial program, chances are you have a log file that you write to. The more useful information you have in the log, the less hit and miss you have to do to narrow down the problem. A big part of this usefulness is simply choosing unique and searchable strings for your output, and formatting the information in ways that make it relatively easy to grep for. Some people like to put file/function/line information in the log. This can be useful, but my preference is to stick with a timestamp and a concise, unique message which includes not only a description of the event, but any id information associated linked to the event. I never have the same log string twice in a program if I can help it.

Two things that will save you hours of pain are:
1) Always expect the worst, always check for bad inputs or undefined cases and at the very least write an error message to your log file. Always check that pointers passed into a funciton are not NULL before dereferencing, always check for indices outside the array bounds, elses in if trees and defaults in case statements. In large projects the unexpected always happens and the sooner you can localise where, the better. If your program doesn't handle it, at least you KNOW about it and can fix it.
2) Write to the log file regularly. A good rule of thumb for me is whenever processing something external, changing state, or responding to some event. The more information you have about what happened leading up the the problem the better. What you don't want is repeatitive or overly verbose spam, as it can bury the real information. However so long as you use unique strings, you can usually use text processing techniques to quickly dig up what you want.

Less is better than VIM for log files

A final note on using log files: Use less to read them.
less is a command just like more, but is ironically much more powerful.
It has all the searching functionality of VIM, but is a better choice than VIM because if your log file is large (which it will be) VIM uses vast system resources (memory, tempory files) and is slow to load.

Some common shortcuts you should know [work in less and VIM]:
(shift)G - goto end of file
?(string) - search backward for (string)
/(string) - search forward for (string)
n - repeat last search (in last direction)
If you press shift+F, it will start acting like "tail -f". You can stop auto refreshing by Ctrl+C

Keep a seperate editor open that you can copy and paste important lines into. This allows you to build up a timeline of important events which will allow you to understand why the impossible has happened.

Building with Makefiles

2008-07-03T02:43:00.000-07:00

Always compile with warnings on!
gcc -Wall foo.c
The -Wall means Warn all. Without it you will have subtle flaws in your program that will be impossible to debug. Behavior that just does not make sense because of some trivial syntax or cast precision loss. It does not take a lot of effort to produce warning free code, but it does take a lot of effort to find a bug that the compile could have told you about. You will also learn quite a lot about the subtleties C/C++ in a controlled environment, instead of the lost in the wilderness with wild animals way.

Build debug symbols (-g), you wont regret it when you get a phone call at 2am in the morning.

Use compiler optimisation (-O1 -O2 or -O3), it really does improve performance dramatically.

When in doubt, make clean.
Often you are working with other people's make files. Maybe edited by multiple people afterward. They or you might not have set the dependencies up in a fool proof manner, resulting in a broken build. If you have library dependencies, make clean;make on those too! Spend some time reading yosefk if you are depressed about this and need cheering up. But quite often if you are having some weird problem it is a broken build. Make clean will either fix it or at least give a compile error as to what is wrong. (eg: a function might have been redefined with different arguments, but if the .o is not rebuilt you will link to it incorrectly and crash!)

Makefiles can be complicated things, but really we want them to do something very simple: If anything changes, recompile it! Dependencies can be generated for you to do this, so you don't need to sit there figuring them out.
gcc -MMD
creates a .d file for all your
-include $(OBJS:.o=.d) will include these dependency files (if they exist, and the .o can't exist if they don't) so this will always keep you up to date and avoid make clean paralysis.

You can make rules to automatically compile your stuff for you. Check this out, it even puts the .o files and .d files into an obj directory so it wont clutter up your filesystem:
$(OBJDIR)/%.o: %.cpp Makefile
@$(RM) -f $@
@mkdir -p $(DIRS)
$(CPP) -MMD $(CFLAGS) $(INCLUDE) -c $< -o $@ 2>&1 | more

If you are lazy it can even find all your source files to compile:
SRCS = $(shell find -name 'tests' -prune -o -name '*.cpp' -printf "%P ")
DIRS = $(shell find $ -name 'CVS' -o -name '$(OBJDIR)' -o -name 'tests' $ -prune -o -type d -printf "$(OBJDIR)/%P ")
OBJS = $(patsubst %.cpp,$(OBJDIR)/%.o,$(SRCS))

However sometimes it is preferable to be able to specify the source manually if you have multiple executables to build.

VIM

2008-07-03T02:04:00.000-07:00

VI(M) is a very powerful editor which requires you to survive a learning period as it is different from almost any interface you have used before. If you survive, you will love it, if not you will hate it.

So lets us vi a file right now!
vi test.txt

The first thing that whacks you in the face is that you cannot type text into your document! Why? VI has two modes:
1) command mode
2) edit mode
And you initially start in command mode. You can enter edit mode by pressing 'i'. Now you can type into the document. You can always get back to command mode by pressing escape. Back in command mode you can type :x to exit and save. That is the major hurdle that most people don't jump. There are plenty of good VIM documents on the web explaining basic usage to build on from there.

Here are some development specific tips once you are comfortable with VIM as an editor which make it far more productive than an IDE:

Use ctags

If you were recently editing but navigated away and want to go back to where you were, press u for undo then (ctr)r for redo.

Use .
This is 'repeat last command'
Suppose you called a variable 'notvalid' and then later decided you wanted it to be 'isvalid':
/notvalid
(searches forward for occurance of notvalid)
cw !isvalid
(change word - replaces notvalid with !isvalid)
(esc)
/
(search again)
.
(repeat last command - this will replace the notvalid we just found with !isvalid)
/
.
(and again)

Of course you could also use a global search and replace:
:%s/notvalid/!isvalid/g
Which is less typing, but I prefer to check each replace as I go, so the search and repeat is good for that. You can do very complex editing very quickly using this technique, just by putting a little thought into how to modify text in a repeatable way.

:make
will build your project from within VIM. The advantage of this is that it will take you instantly to the line and file of any error that might occur in the build.
:cn and :cp can be used to move forward and backward through the error messages.

Customise your ~/.vimrc
This is what will transform VIM into a customised IDE.

syntax on
Will do syntax highlighting

set viminfo='10,\"100,:20,%,n~/.viminfo
au BufReadPost * if line("'\"") > 0|if line("'\"") <= line("$")|exe("norm '\"")|else|exe "norm $"|endif|endif
Will remember the last line you were in when you lasted edited a file.

set expandtab
Is usually the easiest way to ensure tab size is irrelevant.

set smartcase
Will allow you to search for strings case-insensitive unless you use capitals. There are many more handy ones, look up the documentation and go nuts!

set autoindent
set smartindent
set softtabstop=4
set shiftwidth=4
set showmatch
set ruler
set incsearch
set ignorecase
set number
set cindent
set backspace=2
set laststatus=2

%
If you press this while over a brace, it will take you to the complimentary brace. This is a life saver if the code isn't formatted how you like it and you can't be sure what nest you are in.

(shift)G
Go to end of file

?
search backward

/(up) or %(up) or ?(up)
each time you press (up) it goes back to the previous string so you can modify it. (down) goes forward through the list.

I
insert at start of line

A
append at end of line,
just press (esc) after if you dont want to put any text in

8yy
(navigate)
p
Will copy 8 lines and paste it where you want it

8dd
Will delete 8 lines of code

8(down)
Will move down 8 lines
I use this to measure how much I want to cut copy or delete.
Alternatively you can use visual select mode
(ctrl)v

ctags

2008-07-03T01:19:00.000-07:00

ctags allow you to quickly navigate source code by jumping to the definition of a tag (eg: a function or variable declaration). This is vital on large projects, as without it you are stuck with grep or find.

ctags -R
will recursively build a tags file which is recognised by VIM (and other editors). Now if you edit a source file, you will be able to press (ctrl)] to jump to the definition of a tag, and (ctr)t to return back to where you were.

You need to keep the tags up to date, so in your Makefile add the following rule:
tags: $(SRCS) Makefile
-ctags -R
This will rebuild the tags if any of your source files change.

Large projects typically have nested directories. By default VIM does not search up for a central tags file, but you can make it do this by adding to your ~.vimrc
set tags=tags;/

What doesn't work too well is if your project depends on an external library. If the library is part of your source tree you can have a top level tags file, however how to keep this up to date is unknown to me. Also you will run into duplicate tags which your editor may or may not deal with nicely.