To simulate a coast line, we start with a simple closed loop of segments, like a square! Then we go through an iterative process of randomly breaking each segment into two smaller ones: A point is chosen at random near by the center of the segment, and the segment is replaced with the two segments that connect the original end points, through the new middle point. Each of the new segments goes through the same process, until the segments' lengths are small enough for our taste. Here, it would be when the length is of just a couple of pixels, and the iterative process becomes meaningless.
On the left we see the segments of each iteration in gray, and the final result in black. On the right we see the filled in area, which looks very 'natural'.
See the code here
